TWiki> VOTech Web>ResourceDiscovery>OntologyOfObjectTypes>InferenceEngineTests (2007-11-20, AlexandreRichard)EditAttach

Inference Engines tests

We tested the capabilities and performances of various description logics reasoners.

Conditions under which the tests were made :

  • Test files : the different versions of the CloneCompliant ontology
  • Interface: Protégé 3.2 (beta build 318) / DIG interface
  • Test = "classify taxonomy" (consistency check then classification)
  • Sets of 10 tests for each version of the file and each reasoner
  • Tests durations format ( [minimum_time]-[maximum_time] [mean_time] ), all times in seconds
  • RacerProWinXP = RacerPro 1.9 - WindowsXP
  • RacerWinXP = Racer 1.7.24 - WindowsXP
  • FaCT++WinXP = FaCT++ 1.1.3 - WindowsXP
  • PelletWinXP = Pellet 1.3 - WindowsXP
  • RacerProLinux = RacerPro 1.9 - Debian Linux kernel 2.6
  • RacerLinux = Racer 1.7.24 - Debian Linux kernel 2.6
  • PelletLinux = Pellet 1.3 - Debian Linux kernel 2.6
  • Pellet could not run all the tests, independently of performance (currently does not support qualified cardinality restrictions)

version RacerProWinXP RacerWinXP FaCT++WinXP PelletWinXP RacerProLinux RacerLinux PelletLinux Concepts Defined Restrictions Roles Logic Description
01 0.3-0.3 0.3 0.3-0.4 0.4 0.3-0.4 0.3 0.2-0.3 0.3 0.3-0.4 0.4 0.5-0.6 .05 0.3-0.7 0.5 17 0% 0 0 ALC TOP sub-level & EMSpectrumRange branch
02 0.3-0.3 0.3 0.3-0.4 0.4 0.3-0.4 0.4 0.3-0.3 0.3 0.4-0.4 0.4 0.5-0.6 0.5 0.3-1.5 0.5 26 0% 0 1 ALC AstrObject sub-level
03 0.3-0.6 0.4 0.4-0.7 0.5 0.3-0.5 0.4 0.3-0.3 0.3 0.4-0.7 0.5 0.5-0.6 0.6 0.5-0.6 0.6 26 4% 1 1 ALC isObservedIn & X-raySource definition
04 0.3-0.9 0.4 0.4-0.5 0.5 0.3-0.5 0.4 0.3-0.3 0.3 0.4-1.1 0.5 0.6-0.6 0.6 0.6-1.4 0.8 26 19% 5 1 ALC Objects "_____Source" defined
04alt 0.2-0.9 0.5 0.4-0.5 0.5 0.4-0.5 0.4 QCR non supported 0.4-0.6 0.5 0.6-0.6 0.6 QCR non supported 26 19% 5 1 ALCQ Same as 04 but definitions using QCR (discontinued)
05 0.3-0.8 0.4 0.5-0.6 0.6 0.4-0.4 0.4 0.4-0.4 0.4 0.3-0.7 0.5 0.6-0.6 0.6 0.5-1-2 0.6 32 34% 11 1 ALC InfraredSource & RadioSource hierarchies, a few rdfs:comment
06 0.4-0.8 0.4 0.6-0.6 0.6 0.4-0.4 0.4 0.3-1.1 0.4 0.3-1.1 0.5 0.6-1.8 0.8 0.4-0.8 0.7 44 25% 11 1 ALC part of CompositeObject's hierarchy
07 0.5-0.9 0.8 1.2-1.2 1.2 0.5-1.2 0.7 0.5-2.3 0.6 0.7-2.2 1.0 1.1-1.3 1.3 0.5-1.0 0.9 101 12% 13 5 SIN CompositeObject's properties & definition added
07test 0.5-0.9 0.8 1.3-1.3 1.3 0.5-0.7 0.6 0.6-0.6 0.6 0.8-0.8 0.8 1.4-1.4 1.4 0.9-1.0 1.0 101 18% 26 5 SIN a few more definitions & NC with a Transistive Universal on CompositeObject (tests)
08 0.5-0.9 0.9 1.3-1.4 1.4 0.5-0.9 0.6 0.6-0.7 0.7 0.8-1.5 1.1 1.4-1.8 1.5 0.9-1.5 1.1 101 22% 33 5 SIN some definitions under CompositeObject
08test2 1.4-2.3 1.7 1.9-2.2 2.1 16.5-18.9 17.1 0.7-0.8 0.7 1.5-1.5 1.5 1.8-2.2 2.0 1.0-1.2 1.1 101 23% 37 5 SIN Changes on PlanetaryNebula definition (tests)
09test2 1.5-1.6 1.6 2.4-2.5 2.5 15.4-16.2 15.9 QCR non supported 1.2-1.7 1.6 2.2-2.9 2.4 QCR non supported 101 23% 37 5 SIQ same as 08test2 but using QCR
10 1.7-2.2 1.8 2.7-3.5 3.1 > 2h 0.7-0.9 0.8 1.4-2.0 1.7 2.8-3.7 3.0 1.0-1.3 1.2 101 24% 40 5 SIN DoubleStar definition & hierarchy
10testSHIN 1.2-2.7 1.4 1.9-2.2 2.0 >2h 2.9-3.9 3.5 1.5-2.6 1.7 2.0-3.0 2.2 3.6-5.9 3.8 101 24% 36 5 SHIN like 10 but with role hierarchy
11 2.4-3.1 2.7 4.1-5.0 4.6   Java : out of memory 3.1-7.4 3.9 4.5-7.5 5.2 Java : out of memory 101 27% 50 5 SIN GalaxyPair, Multiplestar, SuperClusterOfGalaxies definitions
12 2.7-3.5 2.9 4.2-4.3 4.3     2.0-6.7 4.3 4.6-4.7 4.6   102 26% 48 5 SIN definitions under Galaxy, Star, etc.
13new 2.8-5.2 3.2 4.2-4.9 4.6     4.1-6.4 5.1 4.3-4.6 4.5   102 28% 53 5 SIN more definitions (tests)
14 2.4-3.2 2.6 4.5-4.7 4.6     3.2-5.2 4.7 4.7-4.9 4.8   102 26% 48 5 SIN more definitions (corrected)
15 2.0-2.4 2.1 5.5-5.7 5.6     2.9-5.1 4.4 6.0-6.7 6.4   102 26% 48 5 SIN(D) Equivalent to the 'Simbad5.owl' with the '_____Component' hierarchies and Vizier/Simbad namespaces
15testPN 2.5-6.0 3.3 11.0-13.4 11.6     2.6-5.6 4.8 11.7-12.7 11.9   104 26% 53 5 SIN 'if_PlanetaryNebula' classification test
16 1.9-3.2 2.6 5.5-5.7 5.6     2.3-3.6 3.2 5.0-7.5 5.8   102 26% 49 5 SIN(D) Vizier keywords added
17 2.4-5.9 3.2 5.4-5.8 5.6     2.1-4.3 3.2 5.7-6.0 5.8   102 26% 49 5 SIN(D) Part of the Simbad short codes and names added
18 2.4-2.7 2.6 5.4-6.0 5.6     2.1-4.4 3.1 5.4-6.0 5.6   102 26% 48 5 SIN(D) Simbad short codes and names added & a few corrections on definitions
19 2.5-2.7 2.7 5.4-5.9 5.7     2.1-4.6 3.1 4.3-5.2 4.7   102 26% 48 5 SIN(D) Additionals corrections and disjoints
19testSHIN 1.6-4.8 1.9 2.5-3.0 2.6     1.5-2.9 1.8 2.8-2.9 2.8   102 27% 42 5 SHIN(D) hierarchie de propiétés et modifications impliquées (test)
20testSIQ 5.3-10.6 5.9 20.8-21.2 20.9     6.9-12.0 7.4 21.5-21.6 21.6   103 27% 51 5 SIQ(D) modification de définitions avec QCR (tests)
20testSHIQ 3.6-6.2 4.9 3.4-3.9 3.6     4.8-6.9 5.1 3.5-4.1 3.7   103 27% 36 5 SHIQ(D) ver.20 avec hierarchie de propiétés et modifications impliquées (test)

Notes:

  • CloneCompliant is a thouroughly checked version of the ontology of Simbad object types currently in progress, it's written in OWL 1.1 (evolution of OWL-DL) and its versions make a good test set since they grow both in size and in complexity
  • Protégé takes between 0.2 and 0.6s to synchronize itself with the reasoner
  • Concept=class, role=property (synonyms)
  • QCR = Qualified Cardinality Restriction, i.e: (\geq n R C) or (\leq n R C)
  • Logics: S = ALC+transitive roles (e.g. SIN = ALCIN+transitive roles)

-- AlexandreRichard - 07 Jun 2006

Edit | Attach | Print version | History: r17 < r16 < r15 < r14 < r13 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r17 - 2007-11-20 - 14:29:45 - AlexandreRichard
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback