UnachievableRegion Precisionrecallcurvescannotgothroughtheunachievableregion BoydetalICML2012 UnachievableRegioninPRSpace June2820122 Introduction Unachievableregionvarieswiththedataset OurGoal ID: 450154
Download Pdf The PPT/PDF document "Introduction" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Introduction UnachievableRegion Precision-recallcurvescannotgothroughtheunachievableregion. Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,20122 Introduction Unachievableregionvarieswiththedataset. OurGoal Understandandcharacterizetheunachievableregion. Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,20122 IllustrationofanUnachievablePoint 100positiveexamples 200negativeexamples Holdprecisionconstantat0.2 Actual PN P 2080N 80120 1002000.2recall Actual PN P 40160N 6040 1002000.4recall Actual PN P 60240N 40-40 1002000.6recall Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,20126 UnachievableRegion ExamplePRcurveswith=0:33. RandomGuessing:randomlyassignedlabelsaccordingtoclassskew MinimumPRCurve:worstpossibleranking Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,20129 AUCPRComparisons Unachievableregionalwaysincluded\forfree"inAUCPR RangeofAUCPRscoresis[AUCPRMIN;1]andthusdependsonskew AUCPRnotcomparablefromdi erentskews Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201212 AUCPRComparisons Unachievableregionalwaysincluded\forfree"inAUCPR RangeofAUCPRscoresis[AUCPRMIN;1]andthusdependsonskew AUCPRnotcomparablefromdi erentskews Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201212 AUCPRComparisons Unachievableregionalwaysincluded\forfree"inAUCPR RangeofAUCPRscoresis[AUCPRMIN;1]andthusdependsonskew AUCPRnotcomparablefromdi erentskews Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201212 AUCNPRNormalizedareaunderprecision-recallcurveAUCNPR=AUCPR�AUCPRMIN 1�AUCPRMIN Pros RangeofAUCNPRis[0;1]regardlessofskew Withsameskew,preservesorderingofAUCPR Cons NogoodinterpretationasanareainPRspace AUCNPRofrandomguessingisnotsimple Stillsensitivetoskew Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201213 Downsampling Oftendownsamplepositivesandnegativesdi erently ChangesunachievableregionforPRanalysis CohortStudy Shouldpreservethetrueskew Case-controlStudy Articiallymakestheratioofnegativestopositives1:1,2:1,etc. Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201214 AggregatingResultsSometimeswanttocombineresultsfromproblemswithdi erentskews Cross-validationfolds Multipletasks AUCNPRisastepintherightdirection AUCNPRrangeis[0,1]foreachfold/task MeanAUCNPRgivesmeanfractionofachievableareaobtained Butmoreworkisneeded! Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201217 F1ScoreandUnachievableRegion F1=2pr p+r Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201218 F1ScoreandUnachievableRegion F1=2pr p+r Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201218 Conclusions Awarenessoftheunachievableregioniscriticalforprecision-recallanalysis Shownhowtocomputethisregionfromjusttheclassskew Recommendation:Alwaysshowunachievableregioningures AUCNPR:arststeptowardsscoresthataccountfortheunachievableregion Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201220 Conclusions Awarenessoftheunachievableregioniscriticalforprecision-recallanalysis Shownhowtocomputethisregionfromjusttheclassskew Recommendation:Alwaysshowunachievableregioningures AUCNPR:arststeptowardsscoresthataccountfortheunachievableregion Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201220 Conclusions Awarenessoftheunachievableregioniscriticalforprecision-recallanalysis Shownhowtocomputethisregionfromjusttheclassskew Recommendation:Alwaysshowunachievableregioningures AUCNPR:arststeptowardsscoresthataccountfortheunachievableregion Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201220 Conclusions Awarenessoftheunachievableregioniscriticalforprecision-recallanalysis Shownhowtocomputethisregionfromjusttheclassskew Recommendation:Alwaysshowunachievableregioningures AUCNPR:arststeptowardsscoresthataccountfortheunachievableregion Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201220 Acknowledgments ElizabethS.Burnside JudeShavlik CIBMTrainingProgram(NIH5T15LM007359) NIGMSgrantR01GM097618-01 NLMgrantR01LM011028-01 NIEHSgrant4R01ES017400-03 ERDFthroughtheProgrammeCOMPETE PortugueseGovernmentthroughFCT ProjectHORUSref.PTDC/EIA-EIA/100897/2008 EuropeanUnionSeventhFrameworkProgrammeFP7/2007-2013 ResearchFundK.U.Leuven(CREA/11/015andOT/11/051) EUFP7MarieCurieFarerrIntegrationGrant(#294068) FWO-Vlaanderen(G.0356.12) UWCarboneCancerCenter Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201221 ThankyouQuestions? Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201222 ModifyingF1ScoreAfewdesirablepropertiesforamodiedF1score,f0,f0(r;p)=0ifp=r 1�+rf0(r1;p)f0(r2;p)i r1r2f0(r;p1)f0(r;p2)i p1p2 Impossible! 0=f0(0;0)f0(0;)f0(1;)=0 Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201223 ModifyingF1ScoreAfewdesirablepropertiesforamodiedF1score,f0,f0(r;p)=0ifp=r 1�+rf0(r1;p)f0(r2;p)i r1r2f0(r;p1)f0(r;p2)i p1p2 Impossible! 0=f0(0;0)f0(0;)f0(1;)=0 Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201223 ModifyingF1ScoreRelaxedpropertiesforamodiedF1score,f0,f0(r;p)=0ifp=r 1�+rf0(r1;p)f0(r2;p)i r1r2f0(r;p1)f0(r;p2)i p1p2 Onepossiblef0 f0(r;p)=(0ifp2(p�)r p�+(1�)rifp� Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201224 ModifyingF1ScoreRelaxedpropertiesforamodiedF1score,f0,f0(r;p)=0ifp=r 1�+rf0(r1;p)f0(r2;p)i r1r2f0(r;p1)f0(r;p2)i p1p2 Onepossiblef0 f0(r;p)=(0ifp2(p�)r p�+(1�)rifp� Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201224 ProofofUnachievablePointsTheorem Theorem Precision(p)andrecall(r)mustsatisfypr 1�+rwhere=pos pos+negistheproportionofpositiveexamples. p=TP TP+FPTP TP+(1�)n=rn rn+(1�)n=r r+(1�) Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201225 ProofofMinimumAUCPRTheorem Theorem TheareaoftheunachievableregioninPRspaceandtheminimumAUCPR,forproportionofpositives,isAUCPRMIN=1+(1�)ln(1�) AUCPRMIN=Z10r 1�+rdr=r+(�1)ln((r�1)+1) r=1r=0=1 (+(�1)(ln(1)�ln(1�)))=1+(1�)ln(1�) Boydetal.(ICML2012) UnachievableRegioninPRSpace June28,201226