The ECOC framework is a powerful tool to deal with multiclass ca tegorization problems This li brary contains both stateoftheart coding oneversus one oneversusall dense random sparse random DECOC forestECOC and ECOCONE and decoding des igns hamming ID: 23356
Download Pdf The PPT/PDF document "Journal of Machine Learning Research ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
JournalofMachineLearningResearch11(2010)661-664Submitted8/09;Revised1/10;Published2/10Error-CorrectingOuputCodesLibrarySergioEscaleraSERGIO@MAIA.UB.ESOriolPujolORIOL@MAIA.UB.ESPetiaRadevaPETIA.IVANOVA@UB.EDUComputerVisionCenterEdiciO,CampusUAB,08193,Bellaterra,Barcelona,SpainEditor:ChengSoonOngAbstractInthispaper,wepresentanopensourceError-CorrectingOutputCodes(ECOC)library.TheECOCframeworkisapowerfultooltodealwithmulti-classcategorizationproblems.Thisli-brarycontainsbothstate-of-the-artcoding(one-versus-one,one-versus-all,denserandom,sparserandom,DECOC,forest-ECOC,andECOC-ONE)anddecodingdesigns(hamming,euclidean,inversehamming,laplacian,b-density,attenuated,loss-based,probabilistickernel-based,andloss-weighted)withtheparametersdenedbytheauthors,aswellastheoptiontoincludeyourowncoding,decoding,andbaseclassier.Keywords:error-correctingoutputcodes,multi-classclassication,coding,decoding,opensource,matlab,octave1.Error-CorrectingOutputCodesTheError-CorrectingOutputCodes(ECOC)framework(DietterichandBakiri,1995)isasimplebutpowerfulframeworktodealwiththemulti-classcategorizationproblembasedontheembeddingofbinaryclassiers.GivenasetofNcclasses,thebasisoftheECOCframeworkconsistsofdesigningacodewordforeachoftheclasses.Thesecodewordsencodethemembershipinformationofeachclassforagivenbinaryproblem.Arrangingthecodewordsasrowsofamatrix,weobtainacodingmatrixMc,whereMc2f 1;0;1gNcn,beingnthelengthofthecodewordscodifyingeachclass.Fromthepointofviewoflearning,Mcisconstructedbyconsideringnbinaryproblems,eachonecorrespondingtoacolumnofthematrixMc.Eachofthesebinaryproblems(ordichotomizers)splitsthesetofclassesintwopartitions(codedby+1or-1inMcaccordingtotheirclasssetmembership,or0iftheclassisnotconsideredbythecurrentbinaryproblem).Then,atthedecodingstep,applyingthentrainedbinaryclassiers,acodeisobtainedforeachdatapointinthetestset.ThiscodeiscomparedtothebasecodewordsofeachclassdenedinthematrixMc,andthedatapointisassignedtotheclasswiththeclosestcodeword.Severaldecodingstrategieshavebeenproposedinliterature.ThereaderisreferredtoEscaleraetal.(2008)foramoredetailedreview.AnexampleofanECOCdesignisdescribedinFig.1.TheECOCdesignsareindependentofthebaseclassierapplied.Theyinvolveerror-correctingproperties(DietterichandBakiri,1995)andhaveshowntobeabletoreducethebiasandvarianceproducedbythelearningalgorithm(KongandDietterich,1995).Becauseofthesereasons,ECOCshavebeenwidelyusedtodealwithmulti-classcategorizationproblems.c\r2010SergioEscalera,OriolPujolandPetiaRadeva. ESCALERA,PUJOLANDRADEVA ECOCcodingdesignfora4-classproblem.White,black,andgreypositionscorrespondstothesymbols+1,-1,and0,respec-tively.Oncethefourbinaryproblemsarelearnt,atthedecodingstepanewtestsampleXistestedbythenclassiers.Then,thenewcodewordx=fx1;::;xngiscomparedwiththeclasscode-wordsfC1;::C4g,classifyingthenewsamplebytheclassciwhichcodewordminimizesthedecodingmeasure. Figure1:ECOCdesignexample.2.LibraryAlgorithmsTheECOCslibraryisaMatlab/OctavecodeundertheopensourceGPLlicense(gpl)withtheimplementationofthestate-of-the-artcodinganddecodingECOCdesigns.Amainfunctiondenesthemulti-classdata,coding,decoding,andbaseclassier.Alistofparametersarealsoincludedinordertotunethedifferentstrategies.Inadditiontotheimplementedcodinganddecodingdesigns,whicharedescribedinthefollowingsection,theusercanincludehisowncoding,decoding,andbaseclassierasdenedintheuserguide.2.1ImplementedCodingDesignsTheECOCdesignsoftheECOClibrarycoverthestate-of-the-artofcodingstrategies,mainlydi-videdintwomaingroups:problem-independentapproaches,whichdonottakeintoaccountthedistributionofthedatatodenethecodingmatrix,andtheproblem-dependentdesigns,wherein-formationoftheparticulardomainisusedtoguidethecodingdesign.2.1.1PROBLEM-INDEPENDENTECOCDESIGNSOne-versus-all(RifkinandKlautau,2004):NcdichotomizersarelearntforNcclasses,whereeachonesplitsoneclassfromtherestofclasses.One-versus-one(Nilsson,1965):n=Nc(Nc 1)=2dichotomizersarelearntforNcclasses,splittingeachpossiblepairofclasses.DenseRandom(Allweinetal.,2002):n=10logNcdichotomizersaresuggestedtobelearntforNcclasses,whereP( 1)=1 P(+1),beingP( 1)andP(+1)theprobabilityofthesymbols-1and+1toappear,respectively.Then,fromasetofdenedrandommatrices,theonewhichmaximizesadecodingmeasureamongallpossiblerowsofMcisselected.SparseRandom(Escaleraetal.,2009):n=15logNcdichotomizersaresuggestedtobelearntforNcclasses,whereP(0)=1 P( 1) P(+1),deningasetofrandommatricesMcandselectingtheonewhichmaximizesadecodingmeasureamongallpossiblerowsofMc662 ERROR-CORRECTINGOUPUTCODESLIBRARY2.1.2PROBLEM-DEPENDENTECOCDESIGNSDECOC(Pujoletal.,2006):problem-dependentdesignthatusesn=Nc 1dichotomizers.ThepartitionsoftheproblemarelearntbymeansofabinarytreestructureusingexhaustivesearchoraSFFScriterion.Finally,eachinternalnodeofthetreeisembeddedasacolumninMcForest-ECOC(Escaleraetal.,2007):problem-dependentdesignthatusesn=(Nc 1)Tdi-chotomizers,whereTstandsforthenumberofbinarytreestructurestobeembedded.ThisapproachextendsthevariabilityoftheclassiersoftheDECOCdesignbyincludingextradichotomizers.ECOC-ONE(Pujoletal.,2008):problem-dependentdesignthatusesn=2Ncsuggesteddichotomizers.Avalidationsub-setisusedtoextendanyinitialmatrixMcandtoincreaseitsgeneralizationbyincludingnewdichotomizersthatfocusondifculttosplitclasses.2.2ImplementedDecodingDesignsThesoftwarecomeswithacompletesetofECOCdecodingstrategies.Thenotationusedreferstothatusedin(Escaleraetal.,2008):Hammingdecoding:HD(x;yi)=ånj=1(1 sign(xjyji))=2,beingxatestcodewordandyiacodewordfromMccorrespondingtoclassCiInverseHammingdecoding:IHD(x;yi)=max(D 1DT),whereD(1;2)=HD(yi1;yi2),andDisthevectorofHammingdecodingvaluesofthetestcodewordxforeachofthebasecodewordsyiEuclideandecoding:ED(x;yi)=q ånj=1(xj yji)2AttenuatedEuclideandecoding:AED(x;yi)=q ånj=1jyjijjxjj(xj yji)2Loss-baseddecoding:LB(r;yi)=ånj=1L(yjij(r)),whererisatestsample,Lisaloss-function,andisareal-valuedfunctionRn!RProbabilistic-baseddecoding:PD(yi;x)= log Õj221n:Mc(ij)=0P(xj=Mc(;)jj)+K,whereKisaconstantfactorthatcol-lectstheprobabilitymassdispersedontheinvalidcodes,andtheprobabilityP(xj=Mc(;)jj)isestimatedbymeansofP(xj=yjijj)=1 1+eyji(ujfj+wj),wherevectorsuandwareobtainedbysolvinganoptimizationproblem(Passerinietal.,2004).Laplaciandecoding:LAP(x;yi)=ai+1 ai+bi+K,whereaiisthenumberofmatchedpositionsbe-tweenxandyibiisthenumberofmiss-matcheswithoutconsideringthepositionscodedby0,andKisanintegervaluethatcodiesthenumberofclassesconsideredbytheclassier.Pessimisticb-DensityDistributiondecoding:accuracysiRnini siyi(n;ai;bi)dn=1 3,whereyi(n;ai;bi)=1 Knai(1 n)biyiistheb-DensityDistributionbetweenacodewordxandaclasscodewordyiforclassci,andn2R[0;1]Loss-Weighteddecoding:LW(r;)=ånj=1MW(;)L(yji(r;)),whereMW(;)=H(ij) ånj=1H(ij)H(;)=1 miåmik=1j(hj(rik);;);j(xj;;)=1;ifxj=yji0;otherwise.miisthenumberoftrainingsamplesfromclassCi,andrikisthekthsamplefromclassCi663 ESCALERA,PUJOLANDRADEVA3.ImplementationDetailsTheECOCsLibrarycomeswithdetaileddocumentation.Auserguidedescribestheusageofthesoftware.Allthestrategiesandparametersusedinthefunctionsandlesaredescribedindetail.Theuserguidealsopresentsexamplesofvariablesettingandexecution,includingademole.Aboutthecomputationalcomplexity,thetrainingandtestingtimedependsonthedatasize,codinganddecodingalgorithms,aswellasthebaseclassierusedintheECOCdesign.AcknowledgmentsThisworkhasbeensupportedinpartbyprojectsTIN2009-14404-C02andCONSOLIDER-INGENIOCSD2007-00018.ReferencesURLhttp://www.gnu.org/licences/E.Allwein,R.Schapire,andY.Singer.Reducingmulticlasstobinary:Aunifyingapproachformarginclassiers.JournalofMachineLearningResearch,1:113141,2002.T.DietterichandG.Bakiri.Solvingmulticlasslearningproblemsviaerror-correctingoutputcodes.JournalofArticialIntelligenceResearch,2:263282,1995.S.Escalera,OriolPujol,andPetiaRadeva.BoostedlandmarksofcontextualdescriptorsandForest-ECOC:Anovelframeworktodetectandclassifyobjectsinclutterscenes.PatternRecognitionLetters,28(13):17591768,2007.S.Escalera,O.Pujol,andP.Radeva.Onthedecodingprocessinternaryerror-correctingoutputcodes.IEEETransactionsinPatternAnalysisandMachineIntelligence,99,2008.S.Escalera,O.Pujol,andP.Radeva.Separabilityofternarycodesforsparsedesignsoferror-correctingoutputcodes.PatternRecognitionLetters,30:285297,2009.E.B.KongandT.G.Dietterich.Error-correctingoutputcodingcorrectsbiasandvariance.Inter-nationalConferenceofMachineLearning,pages313321,1995.N.J.Nilsson.LearningMachines.McGraw-Hill,1965.A.Passerini,M.Pontil,andP.Frasconi.Newresultsonerrorcorrectingoutputcodesofkernelmachines.IEEETransactionsonNeuralNetworks,15(1):4554,2004.O.Pujol,P.Radeva,,andJ.Vitria.DiscriminantECOC:Aheuristicmethodforapplicationdepen-dentdesignoferrorcorrectingoutputcodes.IEEETransactionsinPatternAnalysisandMachineIntelligence,28:10011007,2006.O.Pujol,S.Escalera,andP.Radeva.Anincrementalnodeembeddingtechniqueforerror-correctingoutputcodes.PatternRecognition,4:713725,2008.R.RifkinandA.Klautau.Indefenseofone-vs-allclassication.TheJournalofMachineLearningResearch,5:101141,2004.664