0.981555sss.com

ResultsoftheOntologyAlignmentEvaluationInitiative2016ManelAchichi1,MichelleCheatham2,ZlatanDragisic3,JeromeEuzenat4,DanielFaria5,AloFerrara6,GiorgosFlouris7,IriniFundulaki7,IanHarrow8,ValentinaIvanova3,ErnestoJimenez-Ruiz9,10,ElenaKuss11,PatrickLambrix3,HenrikLeopold12,HuanyuLi3,ChristianMeilicke11,StefanoMontanelli6,CatiaPesquita13,TzaninaSaveta7,PavelShvaiko14,AndreaSplendiani15,HeinerStuckenschmidt11,KonstantinTodorov1,CassiaTrojahn16,andOndˇrejZamazal171LIRMM/UniversityofMontpellier,Francelastname@lirmm.
fr2DataSemantics(DaSe)Laboratory,WrightStateUniversity,USAmichelle.
cheatham@wright.
edu3Link¨opingUniversity&Swedishe-ScienceResearchCenter,Link¨oping,Sweden{zlatan.
dragisic,valentina.
ivanova,patrick.
lambrix}@liu.
se4INRIA&Univ.
GrenobleAlpes,Grenoble,FranceJerome.
Euzenat@inria.
fr5InstitutoGulbenkiandeCiencia,Lisbon,Portugaldfaria@igc.
gulbenkian.
pt6Universit`adeglistudidiMilano,Italy{alfio.
ferrara,stefano.
montanelli}@unimi.
it7InstituteofComputerScience-FORTH,Heraklion,Greece{jsaveta,fgeo,fundul}@ics.
forth.
gr8PistoiaAllianceInc.
,USAian.
harrow@pistoiaalliance.
org9DepartmentofInformatics,UniversityofOslo,Norwayernestoj@ifi.
uio.
no10DepartmentofComputerScience,UniversityofOxford,UK11UniversityofMannheim,Germany{christian,elena,heiner}@informatik.
uni-mannheim.
de12VrijeUniversiteitAmsterdam,TheNetherlandsh.
leopold@vu.
nl13LASIGE,FaculdadedeCiencias,UniversidadedeLisboa,Portugalcpesquita@di.
fc.
ul.
pt14TasLab,InformaticaTrentina,Trento,Italypavel.
shvaiko@infotn.
it15NovartisInstitutesforBiomedicalResearch,Basel,Switzerlandandrea.
splendiani@novartis.
com16IRIT&UniversiteToulouseII,Toulouse,France{cassia.
trojahn}@irit.
fr17UniversityofEconomics,Prague,CzechRepublicondrej.
zamazal@vse.
czAbstract.
Ontologymatchingconsistsofndingcorrespondencesbetweense-manticallyrelatedentitiesoftwoontologies.
OAEIcampaignsaimatcomparingTheofcialresultsofthecampaignareontheOAEIwebsite.
ontologymatchingsystemsonpreciselydenedtestcases.
Thesetestcasescanuseontologiesofdifferentnature(fromsimplethesauritoexpressiveOWLon-tologies)andusedifferentmodalities,e.
g.
,blindevaluation,openevaluation,orconsensus.
OAEI2016offered9trackswith22testcases,andwasattendedby21participants.
ThispaperisanoverallpresentationoftheOAEI2016campaign.
1IntroductionTheOntologyAlignmentEvaluationInitiative1(OAEI)isacoordinatedinternationalinitiative,whichorganisestheevaluationofanincreasingnumberofontologymatchingsystems[18,21].
Itsmaingoalistocomparesystemsandalgorithmsopenlyandonthesamebasis,inordertoallowanyonetodrawconclusionsaboutthebestmatchingstrategies.
Furthermore,ourambitionisthat,fromsuchevaluations,tooldeveloperscanimprovetheirsystems.
Tworsteventswereorganisedin2004:(i)theInformationInterpretationandIntegrationConference(I3CON)heldattheNISTPerformanceMetricsforIntelli-gentSystems(PerMIS)workshopand(ii)theOntologyAlignmentContestheldattheEvaluationofOntology-basedTools(EON)workshopoftheannualInternationalSemanticWebConference(ISWC)[41].
Then,auniqueOAEIcampaignoccurredin2005attheworkshoponIntegratingOntologiesheldinconjunctionwiththeInter-nationalConferenceonKnowledgeCapture(K-Cap)[4].
From2006untilnow,theOAEIcampaignswereheldattheOntologyMatchingworkshop,collocatedwithISWC[19,17,6,14,15,16,2,9,12,8],whichthisyeartookplaceinKobe,JP2.
Since2011,wehavebeenusinganenvironmentforautomaticallyprocessingeval-uations(§2.
2),whichhasbeendevelopedwithintheSEALS(SemanticEvaluationAtLargeScale)project3.
SEALSprovidedasoftwareinfrastructure,forautomaticallyexe-cutingevaluations,andevaluationcampaignsfortypicalsemanticwebtools,includingontologymatching.
IntheOAEI2016,allsystemswereexecutedundertheSEALSclientinalltracks,andevaluatedwiththeSEALSclientinalltracks.
Thisyearwewelcomedtwonewtracks:theDiseaseandPhenotypetrack,sponsoredbythePistoiaAllianceOntologiesMappingproject,andtheProcessModelMatchingtrack.
Addi-tionally,theInstanceMatchingtrackfeaturedatotalof7matchingtasksbasedonallnewdatasets.
Ontheotherhand,theOA4QAtrackwasdiscontinuedthisyear.
Thispapersynthesisesthe2016evaluationcampaign.
Theremainderofthepaperisorganisedasfollows:inSection2,wepresenttheoverallevaluationmethodologythathasbeenused;Sections3-11discussthesettingsandtheresultsofeachofthetestcases;Section12overviewslessonslearnedfromthecampaign;andnally,Section13concludesthepaper.
2GeneralmethodologyWerstpresentthetestcasesproposedthisyeartotheOAEIparticipants(§2.
1).
Then,wediscusstheresourcesusedbyparticipantstotesttheirsystemsandtheexecution1http://oaei.
ontologymatching.
org2http://om2016.
ontologymatching.
org3http://www.
development.
seals-project.
euenvironmentusedforrunningthetools(§2.
2).
Finally,wedescribethestepsoftheOAEIcampaign(§2.
3-2.
5)andreportonthegeneralexecutionofthecampaign(§2.
6).
2.
1TracksandtestcasesThisyear'sOAEIcampaignconsistedof9tracksgathering22testcases,anddifferentevaluationmodalities:Thebenchmarktrack(§3):Likeinpreviouscampaigns,asystematicbenchmarkse-rieshasbeenproposed.
Thegoalofthisbenchmarkseriesistoidentifytheareasinwhicheachmatchingalgorithmisstrongorweakbysystematicallyalteringanontology.
Thisyear,wegeneratedanewbenchmarkbasedontheoriginalbiblio-graphicontologyandanotherbenchmarkusingalmontology.
Theexpressiveontologytrackoffersalignmentsbetweenrealworldontologiesex-pressedinOWL:Anatomy(§4):TheanatomytestcaseisaboutmatchingtheAdultMouseAnatomy(2744classes)andasmallfragmentoftheNCIThesaurus(3304classes)describingthehumananatomy.
Conference(§5):Thegoaloftheconferencetestcaseistondallcorrectcor-respondenceswithinacollectionofontologiesdescribingthedomainofor-ganisingconferences.
Resultswereevaluatedautomaticallyagainstreferencealignmentsandbyusinglogicalreasoningtechniques.
Largebiomedicalontologies(§6):Thelargebiotestcaseaimsatndingalign-mentsbetweenlargeandsemanticallyrichbiomedicalontologiessuchasFMA,SNOMED-CT,andNCI.
TheUMLSMetathesaurushasbeenusedasthebasisforreferencealignments.
Disease&Phenotype(§7):Thedisease&phenotypetestcaseaimsatndingalignmentsbetweentwodiseaseontologies(DOIDandORDO)aswellasbe-tweenhuman(HPO)andmammalian(MP)phenotypeontologies.
Theevalua-tionwassemi-automatic:consensusalignmentsweregeneratedbasedonthoseproducedbytheparticipatingsystems,andtheuniquemappingsfoundbyeachsystemwereevaluatedmanually.
MultilingualMultifarm(§8):ThistestcaseisbasedonasubsetoftheConferencedataset,translatedintotendifferentlanguages(Arabic,Chinese,Czech,Dutch,French,German,Italian,Portuguese,Russian,andSpanish)andthecorrespondingalignmentsbetweentheseontologies.
Resultsareevaluatedagainstthesealign-ments.
InteractivematchingInteractive(§9):Thistestcaseoffersthepossibilitytocomparedifferentmatch-ingtoolswhichcanbenetfromuserinteraction.
Itsgoalistoshowifuserinteractioncanimprovematchingresults,whichmethodsaremostpromisingandhowmanyinteractionsarenecessary.
Participatingsystemsareevaluatedontheconferencedatasetusinganoraclebasedonthereferencealignment,whichcangenerateerroneousresponsestosimulateusererrors.
Instancematching(§10).
Thetrackaimsatevaluatingtheperformanceofmatch-ingtoolswhenthegoalistodetectthedegreeofsimilaritybetweenpairsoftestformalismrelationscondencemodalitieslanguageSEALSbenchmarkOWL=[01]blindEN√anatomyOWL=[01]openEN√conferenceOWL=,90%andRiMOM85%).
Wearguethattheseresultsonthedatalinkingsub-taskareduetotheproblemofselectingthemostappropriatemappingwhenanumberofpossiblealternativesareavailable.
BothAMLandRiMOMaresuccessfulinprovidingasetofcandidateDBpediaentitiesastargetmappingwithagivenOWLinstance(i.
e.
,highrecallvalue).
Ontheopposite,thecapabilitytochoose/selectthemostappropriatemappingamongthesetofavailableoptionsisstillchallengingandonlyAMLsucceedsinprovidinghigh-qualityresultsonthistask(i.
e.
,highprecisionvalue).
10.
2ResultsoftheSYNTHETICtaskUOBMandSPIMBENCHtasksaretwooftheevaluationtasksofinstancematchingtoolswherethegoalistodeterminewhentwoOWLinstancesdescribethesamerealworldobject.
Forthersttask,thedatasetshavebeenproducedbyalteringasetofsourcedataandgeneratedbySPIMBENCH[37]withtheaimtogeneratedescriptionsofthesameentitywherevalue-based,structure-basedandsemantics-awaretransforma-tionsareemployedinordertocreatethetargetdata.
While,forthelattertaskthedatasetshavebeengeneratedwiththeUniversityOntologyBenchmark(UOBM)[30]andtransformedwiththeLANCEbenchmarkgenerator[36].
Forbothtasks,thetransformationsappliedwereacombinationofvalue-based,structure-based,andsemantics-awaretestcases.
Thevalue-basedtransformationscon-sidermainlytypographicalerrorsanddifferentdataformats,thestructure-basedtrans-formationsconsidertransformationsappliedonthestructureofobjectanddatatypepropertiesandthesemantics-awaretransformationsaretransformationsattheinstancelevelconsideringtheTBoxinformation.
ThelatterareusedtoexamineifthematchingsystemstakeintoaccountRDFSandOWLsemanticsinordertodiscovercorrespon-dencesbetweeninstancesthatcanbefoundonlybyconsideringinformationfoundintheTBox.
Westressthataninstanceinthesourcedatasetcanhavenoneoronematchingcounterpartinthetargetdataset.
AdatasetiscomposedofaTBoxandacorrespondingABox.
SourceandtargetdatasetssharealmostthesameTBox(differencesintheprop-erties,duetothestructure-basedtransformations).
ForSPIMBENCH,thesandboxscaleis10Ktriples≈380instanceswhilethemainboxscaleis50Ktriples≈1800instances.
WeaskedtheparticipantstomatchtheCreativeWorksinstances(NewsItem,BlogPostandProgramme)inthesourcedatasetagainsttheinstancesofthecorrespondingclassinthetargetdataset.
ForUOBM,thesandboxscaleis14Ktriples≈2.
5Kinstanceswhilethemainboxscaleis60Ktriples≈10Kinstances.
Weaskedtheparticipantstomatchalltheinstancesthatarenotcommontothetwodatasets.
Forbothtasks,weex-pectedtoreceiveasetoflinksdenotingthepairsofmatchinginstancesthattheyfoundtorefertothesameentity.
TheparticipantstothesetasksareLogMap,AMLandRiMOM.
Forevaluation,webuiltagroundtruthcontainingthesetofexpectedlinkswhereaninstancei1inthesourcedatasetisassociatedwithaninstanceinthetargetdatasetthathasbeengener-atedasanaltereddescriptionofi1.
Thewaythatthetransformationsweredone,wastoapplyvalue-based,structure-basedandsemantics-awaretransformations,ondifferenttriplespertainingtooneclassinstance.
Thesystemswerejudgedonthebasisofprecision,recallandF-measureresultsthatareshowninTables44and45.
SandboxtaskMainboxtaskPrecisionF-measureRecallPrecisionF-measureRecallLogMap0.
9580.
8510.
7660.
9810.
8140.
695AML0.
9070.
820.
7490.
90.
8160.
747RiMOM0.
9840.
99210.
9910.
9951Table44.
ResultsoftheSPIMBENCHtask.
LogMaprespondswellregardingtheSPIMBENCHtask,whiletheperformancedropswhenmatchingthedatasetsoftheUOBMtask.
LogMapisautomaticanddoesnotrequirethedenitionofacongurationleincontrasttoAMLandRiMOM.
SandboxtaskMainboxtaskPrecisionF-measureRecallPrecisionF-measureRecallLogMap0.
7010.
320.
2070.
6250.
0440.
023AML0.
7850.
6650.
5770.
5090.
5120.
515RiMOM0.
7710.
8210.
8770.
4430.
4770.
516Table45.
ResultsoftheUOBMtask.
AMLrespondswellregardingtheSPIMBENCHtask,whiletheperformancedropswhenmatchingthedatasetsoftheUOBMtask.
AMLhadtoturnoffthereasonerinordertohandlemissinginformationaboutthedomainandrangeofTBoxproperties.
LogMapandAMLproducelinksthatarequiteoftencorrect(resultinginagoodprecision)butfailincapturingalargenumberoftheexpectedlinks(resultinginalowerrecall).
RiMOMperformsbetterthananyothersystemformostofthetasks;itperformsexcellentinthecaseofSPIMBENCHbut,althoughitexhibitsthebestresultsfortheSandboxtrackofUOBM,itsperformancedropsfortheMainboxtrack.
ForRiMOM,theprobabilityofcapturingacorrectlinkishigh,buttheprobabilityofaretrievedlinktobecorrectislower,resultinginahighrecallbutnotahighprecision.
ThemaincommentsfortheSPIMBENCHandUOBMtasksare:–LogMapandAMLhaveconsistentbehaviourregardingSandboxandMainbox.
–RiMOMhasaconsistentbehaviourfortheSPIMBENCHtaskandaninconsistentbehaviourfortheUOBMtask.
–AllsystemsperformedwellfortheSPIMBENCHtask.
–TheUOBMdatasetsseemtobemore"difcult"forbothIMsystems,andthisdif-cultystemsfromthedatasetitself,ratherthanfromthetransformationsimposedbyLANCE.
–TheUOBMdatasetsseemtobemoredifcultforbothIMsystems,andthisdif-cultystemsfromthedatasetitself,ratherthanfromthetransformationsimposedbyLANCE.
Inparticular,animportantsourceofdifcultyforthesystemsisthattheURIsoftheinstancesinthedatasetlookverysimilartoeachother,soeventhechangeofaURIcanleadtofalsepositivesorfalsenegatives.
10.
3ResultsoftheDOREMUStaskTheDOREMUStask,havingitspremieratOAEI,containsrealworlddatasetscomingfromtwomajorFrenchculturalinstitutions—TheBnF(FrenchNationalLibrary)andthePP(PhilharmoniedeParis).
ThedataareaboutclassicalmusicworksandfollowtheDOREMUSmodel(onesinglevocabularyforbothdatasets)issuedfromtheDORE-MUSproject15.
Eachdataentry,orinstance,isabibliographicalrecordaboutamusicalpiece,containingpropertiessuchasthecomposer,thetitle(s)ofthework,theyearofcreation,thekey,thegenre,theinstruments,tonameafew.
Thesedatahavebeencon-vertedtoRDFfromtheiroriginalUNI-andINTER-MARCformatandanchoredtotheDOREMUSontologyandasetofdomaincontrolledvocabulariesbythehelpofthemarc2rdfconverter16,developedforthispurposewithintheDOREMUSProject(for15http://www.
doremus.
org16https://github.
com/DOREMUS-ANR/marc2rdfmoredetailsontheconversionmethodandontheontologywereferto[1]and[29]).
Notethatthesedataarehighlyheterogeneous.
WehaveselectedworksdescribedbothattheBnFandatthePPwithdifferentdegreesofheterogeneityintheirdescriptions.
Thedatasetshavebeenselectedinthreesub-tasks.
Nineheterogeneities.
Thistaskconsistsinaligningtwosmalldatasets,BnF-1andPP-1,containingabout40instanceseach,bydiscovering1:1equivalencerelationsbetweentheirinstances.
Thereare9typesofheterogeneitiesthatthesedatamanifest,thathavebeenidentiedbythemusiclibraryexperts,suchasmultilingualism,differencesincat-alogues,differencesinspelling,differentdegreesofdescription(numberofproperties).
Fourheterogeneities.
Thistaskconsistsinaligningtwolargerdatasets,BnF-2andPP-2,containingabout200instanceseach,bydiscovering1:1equivalencerelationsbetweentheinstancesthattheycontain.
Thereare4typesofheterogeneitiesthatthesedatamanifest,thatwehaveselectedfromthenineinTask1andthatappeartobethemostproblematic:1)Orthographicaldifferences,2)Multilingualtitles,3)Missingproperties,4)Missingtitles.
TheFalsePositivesTrap.
Thistaskconsistsincorrectlydisambiguatingtheinstancescontainedintwodatasets,BnF-3andPP-3,bydiscovering1:1equivalencerelationsbetweentheinstancesthattheycontain.
Wehaveselectedseveralgroupsofpairsofworkswithhighlysimilardescriptionswherethereexistsonlyonecorrectmatchineachgroup.
Thegoalistochallengethelinkingtoolscapacitytoavoidthegenerationoffalsepositivesandmatchcorrectlyinstancesinthepresenceofhighlysimilarbutstilldistinctcandidates.
9heterogeneities4heterogeneitiesFalsepositivetrapPrec.
F-m.
Rec.
Prec.
F-m.
Rec.
Prec.
F-m.
Rec.
AML(th=0.
2)0.
9660.
9180.
8750.
9340.
8480.
7760.
9210.
8860.
854AML(th=0.
6)0.
9620.
8620.
7810.
9430.
830.
7410.
8530.
7730.
707RiMOM0.
8130.
8130.
8130.
7460.
7460.
7460.
7070.
7070.
707Table46.
ResultsoftheDOREMUStaskResultsOnlytwosystemsreturnedresultsonthetrack:AMLandRiMOM.
NotethatAMLhasbeenconguredwithtwodifferentthresholds.
Theresultsoftheirperfor-mances,evaluatedbyusingprecision,recallandF-measure,oneachofthethreetaskscanbeseeninTable46.
ThebestperformanceintermsofF-measureisprovidedbytheAMLtoolwithathresholdof0.
2onalltasks.
11ProcessModelMatchingIn2013andin2015thecommunityinterestedinbusinessprocessmodellingconductedanevaluationcampaignsimilartoOAEI[3].
Insteadofmatchingontologies,thetaskwastomatchprocessmodelsdescribedindifferentformalismslikeBPMNandPetriNets.
WithinthistrackweofferasubsetofthetasksfromtheProcessModelMatchingContestasOAEItrackbyconvertingtheprocessmodelstoanontologicalrepresen-tation.
Byofferingthistrack,wehopetogaininsightsinhowfarontologymatchingsystemsarecapableofsolvingthemorespecicproblemofmatchingprocessmod-els.
Thistrackisalsomotivatedbythediscussionsattheendofthe2015OntologyMatchingworkshop,wheremanyparticipantsshowedtheirinterestinsuchatrack.
11.
1ExperimentalSettingsWewereusingtherstdatasetfromthe2015ProcessMatchingContest.
Thisdatasetdealswithprocessingapplicationstoauniversity.
ItconsistsofninedifferentprocessmodelswhereeachdescribestheconcreteprocessofaspecicGermanuniversity.
ThemodelsareencodedasBPMNprocessmodels.
WeconvertedtheBPMNrepresenta-tionoftheprocessmodelstoasetofassertions(ABox)usingthevocabularydenedintheBPMN2.
0ontology(TBox).
ForthatreasontheresultingmatchingtaskisaninstancematchingtaskwhereeachABoxisdescribedbythesameTBox.
Foreachpairofprocessesmanuallygeneratedreferencealignmentsareavailable.
Typicalactiv-itieswithinthatdomainareSendingacceptance,Invitestudentforinterview,orWaitforresponse.
Theseexamplesillustrateoneofthemaindifferencesfromtheontologymatchingtask.
Thelabelsareusuallyverb-objectphrasesthataresometimesextendedwithmorewords.
Anotherimportantdifferenceisrelatedtotheexistenceofanexecu-tionorder,i.
e.
,themodelisacomplexsequenceofactivities,whichcanbeunderstoodasthecounterparttoatypehierarchy.
OnlyfewsystemshavebeenmarkedascapableofgeneratingalignmentsfortheProcessModelMatchingtrack.
Wehavetriedtoexecuteallthesesystems,however,someofthemgeneratedonlytrivialTBoxmappingsinsteadofmappingsbetweenac-tivities.
Aftercontactingthedeveloperofthesystems,wereceivedthefeedbackthatthesystemshavebeenmarkedmistakenlyandaredesignedforterminologicalmatch-ingonly.
Wehaveexcludedthemfromtheevaluation.
Moreover,wetriedtorunallsystemsthatweremarkedasinstancematchingtools,whichhavebeensubmittedasexecutableSEALSbundles.
Oneofthesetools(LogMap),generatedmeaningfulresultsandwasalsoaddedtothesetofsystemsthatweevaluated.
Finallyweevaluatedthreesystems(AML,LogMap,andDKP),oneofthesesystemswasconguredintwodifferentsettingsrelatedtothetreatmentofevents-to-activitymappings.
ThiswasthetoolDKP.
ThuswedistinguishbetweenDKPandDKP*.
Inourevaluation,wecomputedstandardprecisionandrecall,aswellasthehar-monicmeanknownasF-measure.
Thedatasetweusedconsistsofseveraltestcases.
Weaggregatedtheresultsandpresentthemicroaverageresults.
ThegoldstandardweusedforourrstsetofevaluationexperimentsisbasedonthegoldstandardthathasalsobeenusedattheProcessModelMatchingContestin2015[3].
Wemodiedonlysomeminormistakes(resultinginchangeslessthan0.
5percentagepoints).
Inordertocomparetheresultstotheresultsobtainedbytheprocessmodelmatchingcommunity,wepresentalsotherecomputedvaluesofthesubmissionstothe2015contest.
Moreover,weextendedourevaluation("Standard"inTable47)byanewevalua-tionmeasurethatmakesuseofaprobabilisticreferencealignment("Probabilistic"inTable47).
Thisprobabilisticmeasureisbasedonagoldstandardwhichismanuallyandindependentlygeneratedbyseveraldomainexperts.
Thenumberofvotesoftheseannotatorsareappliedassupportvaluesintheprobabilisticevaluation.
Foradetaileddiscussion,pleasereferto[28].
11.
2ResultsTable47summarisestheresultsofourevaluation.
"P"abbreviatesprecision,"R"isrecall,"FM"standsforF-measureand"Rk"meansrank.
Theprex"Pro"indicatestheprobabilisticversionsoftheprecision,recall,F-measureandtheassociatedrank.
Thesemetricsareexplainedbelow.
ParticipantsoftheProcessModelMatchingContestin2015(PMMC2015)aredepictedingreyfont,whileOAEI2016participantsareshowninblackfont.
TheOAEIparticipantsarerankedonposition1,8,9and11withanoverallnumberof16systemslistedinthetable(whenusingthestandardmetrics).
NotethatAML-PMatthePMMC2015wasamatchingsystemthatwasbasedonapredecessorofAMLparticipatingatOAEI2016.
ThegoodresultsofAMLaresurprising,sinceweexpectedthatmatchingsystemsspecicallydevelopedforthepurposeofprocessmodelmatchingwouldoutperformontologymatchingsystemsappliedtothespecialcaseofprocessmodelmatching.
WhileAMLcontainsalsocomponentsthatarespecicallydesignedfortheprocessmatchingtask(aooding-likestructuralmatchingalgorithm),itsrelevantmaincomponentsarecomponentsdevelopedforontologymatchingandthesub-problemofinstancematching.
ParticipantsStandardProbabilisticMatcherContestSizePRFMRkProPProRProFMRkAMLOAEI-162210,7190,6850,70210,7420,2830,4102AML-PMPMMC-155790,2690,6720,385140,3770,3980,3874BPLangMatchPMMC-152770,3680,4400,401120,5320,2720,3608DKPOAEI-161770,6210,4740,53880,6860,2190,3339DKP*OAEI-161500,6800,4400,53490,7720,2110,33110KnoMa-ProcPMMC-153260,3370,4740,394130,5060,3020,3785KMatch-SSSPMMC-152610,5130,5780,54460,5630,2740,3687LogMapOAEI-162670,4490,5170,481110,5940,2910,3903Match-SSSPMMC-151400,8070,4870,60840,7610,1920,30712OPBOTPMMC-152340,6030,6080,60550,6480,2580,3696pPalm-DSPMMC-158280,1620,5780,253160,2100,3350,25816RMM-NHCMPMMC-152200,6910,6550,67320,7830,2970,4311RMM-NLMPMMC-151640,7680,5430,63630,6810,1970,30613RMM-SMSLPMMC-152620,5110,5780,54370,5160,2420,32911RMM-VM2PMMC-155050,2160,4700,296150,3090,2940,30114TripleSPMMC-152300,4870,4830,485100,4860,2100,29315Table47.
ResultsoftheprocessmodelmatchingtrackIntheprobabilisticevaluation,however,theOAEIparticipantsgainposition2,3,9and10,respectively.
LogMaprisesfromposition11to3.
The(probabilistic)precisionimprovesover-proportionallyforthismatcher,becauseLogMapgeneratesmanycorre-spondenceswhicharenotincludedinthebinarygoldstandardbutareincludedintheprobabilisticone.
TherankingofLogMapdemonstratesthatastrengthoftheprobabilis-ticmetricliesinthebroadeneddenitionofthegoldstandardwhereweakmappingsareincludedbutsoftened(viathesupportvalues).
Figures11(a)-(b)showtheprobabilisticprecision(ProP)andtheprobabilisticre-call(ProR)withrisingthresholdτonthereferencealignment(0,000;0,375;0,500;0,750).
ThematcherLogMapmainlyidentiescorrespondenceswithhighsupport(ofwhichmanyarenotincludedinthebinarygoldstandard).
ThiscanbeobservedbytheminorchangeintheProPandthesignicantincreaseintheProRwithhigherτ.
ForthematcherAML,theoppositeeffectcanbeobserved.
TheProPdecreasesdramaticallywithrisingτ(accompaniedbyaweakincreaseoftheProR).
Thisindicatesthatthematchercomputesahighfractionofcorrespondenceswithlowsupportvalue(whicharepartlyincludedinthebinarygoldstandard).
ForthematchersDKPandDKP*,withincreasingτ,aminordecreaseinProPandincreaseinProRcanbeobserved.
TheProPdecreases,sincethenumberofcorrespondencesinthenon-binarygoldstandarddecreases(withrisingτ).
Atthesametime,theProRincreaseswithalowernumberofcorrespondences(withrisingτ).
Figure11(c)displaystheprobabilisticF-measure(ProFM)withrisingthresholdτonthereferencealignment.
AMLachievesbestresultswithτ=0,375sincethismatcheridentiesahighfractionofcorrespondenceswithlowsupportvalue(whichcanalsobetrivialcorrespondences).
Fordetailsabouttheprobabilisticmetric,pleasereferto[28].
TheresultsdepictedinTable47andFigure11indicatethattheprogressmadeinontologymatchinghasalsoapositiveimpactonotherrelatedmatchingproblems,likeitisthecaseforprocessmodelmatching.
Whileitmightrequiretorecongure,adapt,andextendsomepartsoftheontologymatchingsystems,suchasystemseemstoofferagoodstartingpointwhichcanbeturnedwithareasonableamountofworkintoagoodprocessmatchingtool.
Wehavetoemphasisethatourobservationsaresofarbasedononlyonedataset.
Moreover,onlythreeparticipantsdecidedtoapplytheirsystemstothenewtrackofprocessmodelmatching.
Thus,wehavetobecautioustogeneralisetheresultsweobservedsofar.
Inthefuturewemightbeabletoattractmoreparticipantsintegratingmoredatasetsintheevaluation.
12LessonlearnedandsuggestionsThelessonslearnedfromrunningOAEI2016werethefollowing:A)Thisyear,assuggestedinpreviouscampaigns,werequestedtoolregistrationinJuneandpreliminarysubmissionofwrappedsystemsbytheendofJuly.
Thismea-surewassuccessfulinreducingthenumberofsystemswitherrorsandincompati-bilitieswiththeSEALSclientduringtheevaluationphaseashadhappenedinthepast.
However,notallsystemscompliedwiththedeadlines,andsomedidhaveproblems,whichstilldelayedtheevaluation.
Infutureeditions,wemustbemorestrictinenforcingtheparticipationprotocol.
B)Thanksinparttothenewsubmissionschedule,thismarkedtherstOAEIeditionwhereallparticipantsandalltrackswereevaluatedusingtheSEALSclient.
Nev-ertheless,somesystemdevelopersstillstruggledtogettheirsystemsworkingwiththeclient,mostlyduetoincompatibleversionsoflibraries.
Thisrecurringproblem,plustheeffortrequiredtoupdatetheSEALSclient'slibraries,leadtotheconsid-erationofwhetheritwouldnotbebettertodevelopasimpler,morestreamlinedevaluationsolution.
(a)Probabilisticprecision(b)Probabilisticrecall(c)ProbabilisticF-measureFig.
11.
Changeinmetricvalueswithrisingthresholdτ.
C)ThecontinuedabsenceoftheSEALSwebportaldidnotseemtoaffectparticipa-tion,astheGoogledrivesolutionforsubmissionwaswellreceivedbythepartici-pants.
OAEImaymovetowardsacloud-basedsolution.
D)Whilethenumberofparticipantsthisyearwassimilartothatofrecentyears,theirdistributionthroughthetrackswasuneven.
Long-standingtrackshadnoshortageofparticipants,butalasthesamewasnottruefortheInteractive,ProcessModel(new)orInstance(newdatasets)tracks.
OnereasonforthisisthattheOAEIdatasetshavebeenreleasedtooclosetothesubmissiondeadlinetoallowsystemdevel-operstodeveloptheirsystemstotacklethemall—thetimingisbarelysufcienttoallowseriousdevelopmentfocusingononenewdataset.
Thus,withprizemoneyonofferononeofthenewtracks,itisnosurprisethatsystemdeveloperswerepolarisedtowardsthattrackandeschewedtheothernewones.
WeshouldconsideranticipatingthedeadlineforinitialreleaseofOAEIdatasets,particularforthosethatarenew,inordertogivesystemdevelopersmoretimetotacklethem,therebyincreasingparticipation.
E)TheincreasingvarietyofOAEItracksalsoposesdifcultiestosystemdevelopersinconguringtheirsystemstohandledifferenttypesoftasks.
Itisnoteworthythatonlytwosystems,bothofwhicharelong-termOAEIparticipants,havetackledalltracks—andoneofthemdidsousingexternalcongurationlesspecifyingthetypeoftask.
Onesolutiontofacilitateparticipationinmultipletrackswouldbetohavetheevaluationclienttransmittothesystemthespecicationsofthetask,e.
g.
,whetherclasses,properties,and/orindividualsaretobematched,andwhetheronlyaspecicsubsetofthemaretobematched.
Thiswouldalsomakethetasksmorerealistic,inthesensethatinnormaluse,auserwouldprovidetotheontologymatchingsystemthistypeofinformation.
F)WithregardtothelowparticipationintheProcessModelandInstancetracks,itmeritsconsideringwhetherenforcingadherencetotheSEALSclientandontology-baseddatasetswerenotdeterrentfactors.
ItshouldbenotedthattheProcessModelMatchingContest(PMMC)receivedamuchlargernumberofparticipantsin2015thandidtheProcessModeltrack,andthatthereisaconsiderablenumberofpub-licationsondatainterlinkingsystems,butonlyoneoftheseparticipatedintheInstancetrack.
G)Inpreviousyearsweidentiedtheneedforconsideringnon-binaryformsofeval-uation,namelyincaseswherethereisuncertaintyaboutsomeofthereferencemappings.
Arstnon-binaryevaluationtypewasimplementedinlastyear'sCon-ferencetrack,butthisyeartwonewtracksfollowedsuit:DiseaseandPhenotypewheretheevaluationwassemantic,andProcessModel,whereitwasprobabilistic.
Thesenewstrategiesshouldprovideafairerevaluationofthesystemsincomplextestcases.
ThelessonslearnedinthevariousOAEI2016trackwerethefollowing:largebio:Whilethecurrentreferencealignments,withincoherence-causingmappingsaggedasuncertain,maketheevaluationfairtoallsystems,theyareonlyacom-promisesolution,notanidealone.
Thus,weshouldaimformanuallyrepairingandvalidatingthereferencealignmentsforfutureeditions.
phenotype:Theprizeofferedinthistrack,thankstothekindsponsorshipofthePistoiaAllianceOntologiesMappingproject,waspositivelyacceptedbythecommunityandhelpedattractnewparticipants.
However,italsohadapolarisingeffect,withsomesystemsfocusingexclusivelyinthistrack.
Infutureeditions,wewillconsiderincludingaprizeacrossOAEItracksinordertomotivatedeveloperstosuccessfullyparticipateinmorethanonetrack.
interactive:ThenewfunctionalityoftheOracleallowingsystemstosubmitasetofuptothreeconictingmappings,ratherthanamappingatatime,wassuccessfullyex-ploitedbyonenewparticipatingsystem.
Nevertheless,thistrack'sparticipationhasremainedlow,asmostsystemsparticipatinginOAEIfocussedexclusivelyonfullyautomaticmatching.
Wehopetodrawmoreparticipantstothistrackinthefutureandwillcontinuetoexpanditsoastobetterapproximaterealuserinteractions.
processmodel:TheresultsofthenewProcessModeltrackhaveshownthatthepartic-ipatingontologymatchingsystemsarecapableofgeneratingverygoodresultsforthespecicproblemofprocessmodelmatching.
Thisshowsthatthebasiccom-ponentsofanontologymatchingsystemcanalsobesuccessfullyappliedtootherkindofmatchingproblems.
instance:Inordertoattractmoreinstancematchingsystemstoparticipateinvaluesemantics(val-sem),valuestructure(val-struct),andvaluestructuresemantics(val-struct-sem)tasks,weneedtoproducebenchmarksthathavefewerinstances(intheorderof10000),ofthesametype(inourbenchmarkweaskedsystemstocompareinstancesofdifferenttypes).
Tobalancethoseaspects,wemustthenproducedatasetswithmorecomplextransformations.
13ConclusionsOAEI2016sawthesamenumber(21)ofparticipantsasinrecentyears,withahealthymixofnewandreturningsystems.
WhilesomenewparticipantsweremainlydrawnbytheallureofprizemoneyinthenewDiseaseandPhenotypetrack,theveryfactthattherewasprizemoneyonoffershowsthatinterestinontologymatchingisnotwaning,whichbodeswellforthefutureofOAEI.
AllthetestcaseswereperformedontheSEALSclient,includingthoseintheinstancematchingtrack,whichisgoodnewsregardingtheinteroperabilityofmatchingsystems.
Furthermore,thefactthattheSEALSclientcanbeusedforsuchavarietyoftasksisagoodsignofitsrelevance.
Unlikepreviousyears,thisyeartherewasnonoticeableimprovementwithregardtosystemruntimes—forinstance,thedistributionofruntimesinAnatomyandLargeBiomedicalOntologieswasapproximatelythesameaslastyear.
Therewasalsonoprogresswithregardtotheabilitytohandlelargeontologiesanddatasets,asthenumberofsystemsabletocopewiththeLargeBiomedicalOntologiesdatasetinfullwasthesameaslastyear,andallsystemsabletocopewiththeInstanceSyntheticdatasetwereestablishedsystemsalreadyknownfortheirabilitytohandlelargedatasets.
Finally,therewasnoprogresswithregardtoalignmentrepairsystems,withonlyafewreturningsystemsemployingthem.
Asaconsequence,incoherentalignmentsarecommon.
WithregardtoF-measure,somereturningsystemsshowedsubstantialimprove-ments,butoverall,theimprovementsinF-measureweresubtleinAnatomyandLargeBiomedicalOntologies,andnon-existentinConference.
Ashasbeenthetrend,mostsystemsfavourprecisionoverrecall.
Mostoftheparticipantshaveprovidedadescriptionoftheirsystemsandtheirex-perienceintheevaluation.
TheseOAEIpapers,likethepresentone,havenotbeenpeerreviewed.
However,theyarefullcontributionstothisevaluationexerciseandreectthehardworkandcleverinsightpeopleputintothedevelopmentofparticipatingsys-tems.
Readingthepapersoftheparticipantsshouldhelppeopleinvolvedinontologymatchingndoutwhatmakesthesealgorithmsworkandwhatcouldbeimproved.
TheOntologyAlignmentEvaluationInitiativewillstrivetocontinuetobearef-erencetotheontologymatchingcommunitybyimprovingboththetestcasesandthetestingmethodologytobetterreecttheactualneedsofthecommunity.
Evaluatingon-tologymatchingsystemsremainsachallengingbutcriticaltopic,whichisessentialtoenabletheprogressofthiseld[38].
Moreinformationcanbefoundat:http://oaei.
ontologymatching.
org.
AcknowledgementsWewarmlythanktheparticipantsofthiscampaign.
Weknowthattheyhaveworkedhardtohavetheirmatchingtoolsexecutableintimeandtheyprovidedusefulreportsontheirexperience.
Thebestwaytolearnabouttheresultsremainstoreadthepapersthatfollow.
WewouldalsoliketothankthePistoiaAlliance9whichsponsoredtheDiseaseandPhenotypetrackandfundedtheprizeforthewinners.
WeareverygratefultotheUniversidadPolitecnicadeMadrid(UPM),especiallytoNan-danaMihindukulasooriyaandAsuncionGomezPerez,formoving,settingupandprovidingthenecessaryinfrastructuretoruntheSEALSrepositories.
WearealsogratefultoMartinRingwaldandTerryHayamizuforprovidingthereferencealignmentfortheanatomyontologiesandthankElenaBeisswangerforherthoroughsupportonimprovingthequalityofthedataset.
WethankKhiatAbderrahmaneforhissupportintheArabicdatasetandCatherineComparotforherfeedbackandsupportintheMultiFarmtestcase.
WealsothankfortheirsupporttheothermembersoftheOntologyAlignmentEvaluationIni-tiativesteeringcommittee:YannisKalfoglou(Ricohlaboratories,UK),MiklosNagy(TheOpenUniversity(UK),NatashaNoy(StanfordUniversity,USA),YuzhongQu(SoutheastUniversity,CN),YorkSure(LeibnizGemeinschaft,DE),JieTang(TsinghuaUniversity,CN),GeorgeVouros(UniversityoftheAegean,GR).
MichelleCheathamhasbeensupportedbytheNationalScienceFoundationawardICER-1440202"EarthCubeBuildingBlocks:CollaborativeProposal:GeoLink".
JeromeEuzenat,ErnestoJimenez-Ruiz,ChristianMeilicke,HeinerStuckenschmidtandCassiaTrojahndosSantoshavebeenpartiallysupportedbytheSEALS(IST-2009-238975)Eu-ropeanprojectinpreviousyears.
DanielFariawassupportedbytheELIXIR-EXCELERATEproject(INFRADEV-3-2015).
ErnestoJimenez-RuizhasalsobeenpartiallysupportedbytheSeventhFrameworkProgram(FP7)oftheEuropeanCommissionunderGrantAgreement318338,"Optique",theEPSRCprojectsDBOntoandED3,theResearchCouncilofNorwayprojectBigMed,andtheCentreforScalableDataAccess(SIRIUS).
CatiaPesquitawassupportedbytheFCTthroughtheLASIGEStrategicProject(UID/CEC/00408/2013)andtheresearchgrantPTDC/EEI-ESS/4633/2014.
OndˇrejZamazalhasbeensupportedbytheCSFgrantno.
14-14076P.
References1.
ManelAchichi,RodolpheBailly,CecileCecconi,MarieDestandau,KonstantinTodorov,andRapha¨elTroncy.
Doremus:Doingreusablemusicaldata.
InISWCPD:InternationalSemanticWebConferencePostersandDemos,2015.
2.
JoseLuisAguirre,BernardoCuencaGrau,KaiEckert,JeromeEuzenat,AloFerrara,RobertWillemvanHague,LauraHollink,ErnestoJimenez-Ruiz,ChristianMeilicke,An-driyNikolov,DominiqueRitze,FrancoisScharffe,PavelShvaiko,OndrejSvab-Zamazal,CassiaTrojahn,andBenjaminZapilko.
Resultsoftheontologyalignmentevaluationinitia-tive2012.
InProc.
7thISWContologymatchingworkshop(OM),Boston(MAUS),pages73–115,2012.
3.
GoncaloAntunes,MarziehBakhshandeh,JoseBorbinha,JoaoCardoso,SharamDadash-nia,ChiaraDiFrancescomarino,MauroDragoni,PeterFettke,AvigdorGal,ChiaraGhi-dini,PhilipHake,AbderrahmaneKhiat,ChristopherKlinkm¨uller,ElenaKuss,HenrikLeopold,PeterLoos,ChristianMeilicke,TimNiesen,CatiaPesquita,TimoPeus,AndreasSchoknecht,EitamSheetrit,AndreasSonntag,HeinerStuckenschmidt,TomThaler,IngoWeber,andMatthiasWeidlich.
Theprocessmodelmatchingcontest2015.
In6thInterna-tionalWorkshoponEnterpriseModellingandInformationSystemsArchitectures,September3-4,2015Innsbruck,Austria,pages127–155,2015.
4.
BenjaminAshpole,MarcEhrig,JeromeEuzenat,andHeinerStuckenschmidt,editors.
Proc.
K-CapWorkshoponIntegratingOntologies,Banff(Canada),2005.
5.
OlivierBodenreider.
Theuniedmedicallanguagesystem(UMLS):integratingbiomedicalterminology.
NucleicAcidsResearch,32:267–270,2004.
6.
CaterinaCaracciolo,JeromeEuzenat,LauraHollink,RyutaroIchise,AntoineIsaac,VeroniqueMalaise,ChristianMeilicke,JuanPane,PavelShvaiko,HeinerStuckenschmidt,OndrejSvab-Zamazal,andVojtechSvatek.
Resultsoftheontologyalignmentevaluationini-tiative2008.
InProc.
3rdISWContologymatchingworkshop(OM),Karlsruhe(DE),pages73–120,2008.
7.
SilvanaCastano,AloFerrara,LorenzoGenta,andStefanoMontanelli.
CombiningCrowdConsensusandUserTrustworthinessforManagingCollectiveTasks.
FutureGenerationComputerSystems,54,2016.
8.
MichelleCheatham,ZlatanDragisic,JeromeEuzenat,DanielFaria,AloFerrara,GiorgosFlouris,IriniFundulaki,RogerGranada,ValentinaIvanova,ErnestoJimenez-Ruiz,etal.
Resultsoftheontologyalignmentevaluationinitiative2015.
In10thISWCworkshoponontologymatching(OM),pages60–115,2015.
9.
BernardoCuencaGrau,ZlatanDragisic,KaiEckert,JeromeEuzenat,AloFerrara,RogerGranada,ValentinaIvanova,ErnestoJimenez-Ruiz,AndreasOskarKempf,PatrickLam-brix,AndriyNikolov,HeikoPaulheim,DominiqueRitze,FrancoisScharffe,PavelShvaiko,CassiaTrojahndosSantos,andOndrejZamazal.
Resultsoftheontologyalignmentevalu-ationinitiative2013.
InPavelShvaiko,JeromeEuzenat,KavithaSrinivas,MingMao,andErnestoJimenez-Ruiz,editors,Proc.
8thISWCworkshoponontologymatching(OM),Syd-ney(NSWAU),pages61–100,2013.
10.
JimDabrowskiandEthanV.
Munson.
40yearsofsearchingforthebestcomputersystemresponsetime.
InteractingwithComputers,23(5):555–564,2011.
11.
JeromeDavid,JeromeEuzenat,FrancoisScharffe,andCassiaTrojahndosSantos.
ThealignmentAPI4.
0.
Semanticwebjournal,2(1):3–10,2011.
12.
ZlatanDragisic,KaiEckert,JeromeEuzenat,DanielFaria,AloFerrara,RogerGranada,ValentinaIvanova,ErnestoJimenez-Ruiz,AndreasOskarKempf,PatrickLambrix,Ste-fanoMontanelli,HeikoPaulheim,DominiqueRitze,PavelShvaiko,AlessandroSolimando,CassiaTrojahndosSantos,OndrejZamazal,andBernardoCuencaGrau.
Resultsoftheontologyalignmentevaluationinitiative2014.
InProceedingsofthe9thInternationalWork-shoponOntologyMatchingcollocatedwiththe13thInternationalSemanticWebConference(ISWC),RivadelGarda,Trentino,Italy,pages61–104,2014.
13.
ZlatanDragisic,ValentinaIvanova,PatrickLambrix,DanielFaria,ErnestoJimenez-Ruiz,andCatiaPesquita.
Uservalidationinontologyalignment.
InTheSemanticWeb-ISWC2016-15thInternationalSemanticWebConference,Kobe,Japan,October17-21,2016,Proceedings,PartI,pages200–217,2016.
14.
JeromeEuzenat,AloFerrara,LauraHollink,AntoineIsaac,CliffJoslyn,VeroniqueMalaise,ChristianMeilicke,AndriyNikolov,JuanPane,MartaSabou,FrancoisScharffe,PavelShvaiko,VassilisSpiliopoulos,HeinerStuckenschmidt,OndrejSvab-Zamazal,Vo-jtechSvatek,CassiaTrojahndosSantos,GeorgeVouros,andShenghuiWang.
Resultsoftheontologyalignmentevaluationinitiative2009.
InProc.
4thISWContologymatchingworkshop(OM),Chantilly(VAUS),pages73–126,2009.
15.
JeromeEuzenat,AloFerrara,ChristianMeilicke,AndriyNikolov,JuanPane,FrancoisScharffe,PavelShvaiko,HeinerStuckenschmidt,OndrejSvab-Zamazal,VojtechSvatek,andCassiaTrojahndosSantos.
Resultsoftheontologyalignmentevaluationinitiative2010.
InProc.
5thISWContologymatchingworkshop(OM),Shanghai(CN),pages85–117,2010.
16.
JeromeEuzenat,AloFerrara,RobertWillemvanHague,LauraHollink,ChristianMeil-icke,AndriyNikolov,FrancoisScharffe,PavelShvaiko,HeinerStuckenschmidt,OndrejSvab-Zamazal,andCassiaTrojahndosSantos.
Resultsoftheontologyalignmentevalu-ationinitiative2011.
InProc.
6thISWContologymatchingworkshop(OM),Bonn(DE),pages85–110,2011.
17.
JeromeEuzenat,AntoineIsaac,ChristianMeilicke,PavelShvaiko,HeinerStuckenschmidt,OndrejSvab,VojtechSvatek,WillemRobertvanHage,andMikalaiYatskevich.
Resultsoftheontologyalignmentevaluationinitiative2007.
InProc.
2ndISWContologymatchingworkshop(OM),Busan(KR),pages96–132,2007.
18.
JeromeEuzenat,ChristianMeilicke,PavelShvaiko,HeinerStuckenschmidt,andCassiaTro-jahndosSantos.
Ontologyalignmentevaluationinitiative:sixyearsofexperience.
JournalonDataSemantics,XV:158–192,2011.
19.
JeromeEuzenat,MalgorzataMochol,PavelShvaiko,HeinerStuckenschmidt,OndrejSvab,VojtechSvatek,WillemRobertvanHage,andMikalaiYatskevich.
Resultsoftheontologyalignmentevaluationinitiative2006.
InProc.
1stISWContologymatchingworkshop(OM),Athens(GAUS),pages73–95,2006.
20.
JeromeEuzenat,MariaRosoiu,andCassiaTrojahndosSantos.
Ontologymatchingbench-marks:generation,stability,anddiscriminability.
Journalofwebsemantics,21:30–48,2013.
21.
JeromeEuzenatandPavelShvaiko.
Ontologymatching.
Springer-Verlag,Heidelberg(DE),2ndedition,2013.
22.
DanielFaria,ErnestoJimenez-Ruiz,CatiaPesquita,EmanuelSantos,andFranciscoM.
Couto.
TowardsAnnotatingPotentialIncoherencesinBioPortalMappings.
In13thIn-ternationalSemanticWebConference,volume8797ofLectureNotesinComputerScience,pages17–32.
Springer,2014.
23.
ValentinaIvanova,PatrickLambrix,andJohanAberg.
Requirementsforandevaluationofusersupportforlarge-scaleontologyalignment.
InTheSemanticWeb.
LatestAdvancesandNewDomains12thEuropeanSemanticWebConference,ESWC2015,Portoroz,Slovenia,May31–June4,2015.
Proceedings,pages3–20,2015.
24.
ErnestoJimenez-RuizandBernardoCuencaGrau.
LogMap:Logic-basedandscalableon-tologymatching.
InProc.
10thInternationalSemanticWebConference(ISWC),Bonn(DE),pages273–288,2011.
25.
ErnestoJimenez-Ruiz,BernardoCuencaGrau,IanHorrocks,andRafaelBerlanga.
Logic-basedassessmentofthecompatibilityofUMLSontologysources.
J.
Biomed.
Sem.
,2,2011.
26.
ErnestoJimenez-Ruiz,ChristianMeilicke,BernardoCuencaGrau,andIanHorrocks.
Eval-uatingmappingrepairsystemswithlargebiomedicalontologies.
InProc.
26thDescriptionLogicsWorkshop,2013.
27.
YevgenyKazakov,MarkusKr¨otzsch,andFrantisekSimancik.
ConcurrentclassicationofELontologies.
InProc.
10thInternationalSemanticWebConference(ISWC),Bonn(DE),pages305–320,2011.
28.
ElenaKuss,HenrikLeopold,HanVanderAa,HeinerStuckenschmidt,andHajoA.
Reijers.
Probabilisticevaluationofprocessmodelmatchingtechniques.
InLecturenotesincom-puterscience.
Conceptualmodeling:35thinternationalconference,ER2016,Gifu,Japan,November14-17,2016,pages279–292,2016.
29.
PasqualeLisena,ManelAchichi,EvaFernandez,KonstantinTodorov,andRapha¨elTroncy.
Exploringlinkedclassicalmusiccatalogswithoverture.
InISWCPD:InternationalSeman-ticWebConferencePostersandDemos,2016.
30.
L.
Ma,Y.
Yang,Z.
Qiu,G,Xie,Y.
Pan,andS.
Liu.
TowardsaCompleteOWLOntologyBenchmark.
InESWC,2006.
31.
ChristianMeilicke.
AlignmentIncoherenceinOntologyMatching.
PhDthesis,UniversityMannheim,2011.
32.
ChristianMeilicke,RaulGarcaCastro,FredericoFreitas,WillemRobertvanHage,ElenaMontiel-Ponsoda,RyanRibeirodeAzevedo,HeinerStuckenschmidt,OndrejSvab-Zamazal,VojtechSvatek,AndreiTamilin,CassiaTrojahn,andShenghuiWang.
MultiFarm:Abench-markformultilingualontologymatching.
Journalofwebsemantics,15(3):62–68,2012.
33.
BorisMotik,RobShearer,andIanHorrocks.
Hypertableaureasoningfordescriptionlogics.
JournalofArticialIntelligenceResearch,36:165–228,2009.
34.
HeikoPaulheim,SvenHertling,andDominiqueRitze.
Towardsevaluatinginteractiveontol-ogymatchingtools.
InProc.
10thExtendedSemanticWebConference(ESWC),Montpellier(FR),pages31–45,2013.
35.
CatiaPesquita,DanielFaria,EmanuelSantos,andFranciscoCouto.
Torepairornottorepair:reconcilingcorrectnessandcoherenceinontologyreferencealignments.
InProc.
8thISWContologymatchingworkshop(OM),Sydney(AU),pages13–24,2013.
36.
TzaninaSaveta,EvangeliaDaskalaki,GiorgosFlouris,IriniFundulaki,MelanieHerschel,andAxel-CyrilleNgongaNgomo.
Lance:Piercingtotheheartofinstancematchingtools.
InInternationalSemanticWebConference,pages375–391.
Springer,2015.
37.
TzaninaSaveta,EvangeliaDaskalaki,GiorgosFlouris,IriniFundulaki,MelanieHerschel,andAxel-CyrilleNgongaNgomo.
Pushingthelimitsofinstancematchingsystems:Asemantics-awarebenchmarkforlinkeddata.
InWWW,CompanionVolume,2015.
38.
PavelShvaikoandJeromeEuzenat.
Ontologymatching:stateoftheartandfuturechallenges.
IEEETransactionsonKnowledgeandDataEngineering,25(1):158–176,2013.
39.
AlessandroSolimando,ErnestoJimenez-Ruiz,andGiovannaGuerrini.
Detectingandcor-rectingconservativityprincipleviolationsinontology-to-ontologymappings.
InTheSeman-ticWeb–ISWC2014,pages1–16.
Springer,2014.
40.
AlessandroSolimando,ErnestoJimenez-Ruiz,andGiovannaGuerrini.
Minimizingcon-servativityviolationsinontologyalignments:Algorithmsandevaluation.
KnowledgeandInformationSystems,2016.
41.
YorkSure,OscarCorcho,JeromeEuzenat,andToddHughes,editors.
Proc.
ISWCWorkshoponEvaluationofOntology-basedTools(EON),Hiroshima(JP),2004.
Montpellier,Dayton,Link¨oping,Grenoble,Lisboa,Milano,Heraklion,Kent,Oslo,Oxford,Mannheim,Amsterdam,Trento,Basel,Toulouse,PragueDecember2016