internationaljournalofmedicalinformatics78(2009)1 - PDF document

internationaljournalofmedicalinformatics78(2009)1…9 journalhomepage:www.intl.elsevierhealth.com/journals/ijmi STARE-HI„StatementonreportingofevaluationstudiesinHealthInformaticsJanTalmon,ElskeAmmenwerth Articlehistory:Received4July2008Accepted3September2008 Keywords:MedicalInformaticsPublishingstandardsResearchdesign Objective:DevelopmentofguidelinesforpublicationofevaluationstudiesofHealthInfor-maticsapplications. Correspondingauthor.Tel.:+31433872253.E-mailaddress:(J.Talmon).1386-5056/$…seefrontmatter©2008ElsevierIrelandLtd.Allrightsreserved. internationaljournalofmedicalinformatics78(2009)1…9 Thisstudyisbasedonexperiencesfromeditors,reviewers,authorsofsystematicreviewsandreadersofthescienti“cliterature.Theapplicabilityoftheprincipleshasnotbeenevaluatedinrealpractice.Onlywhenauthorsstarttousetheseprinciplesforreporting,shortcomingsintheprincipleswillemerge.©2008ElsevierIrelandLtd.Allrightsreserved. 1.Introductionapplicationoftechnologies,methods,andinterventionsisrecognisedasgoodpracticeandethicallyimportantinallaspectsofhealthcaree;thetechnologies,methods,andinterventionshavetobeproventobesafe,effec-tive,andthemostappropriatecomparedwithothermethodsandalternativesolutions.Giventheessentialroleofinformationtechnology(IT)systemsonthedeliveryofmodernhealthcare,andthedepen-denceofhealthprofessionalsandorganizationsonthem,itisimperativethattheyarethoroughlyassessedthroughrobustevaluationsaswithanyotherformofhealthprocessortechnology.ThisprincipleisadvocatedandelaboratedintheDeclarationofInnsbruckk.InthepastdecadesithasbeendemonstratedthatITsys-temscannotonlybebene“cial,butalsocanhaveunintended,potentiallydetrimentaleffectsasdocumentedbyy,justlikeotherinterventionsin(social)environmentsasexten-sivelydescribedinin.ItisimperativetomonitorITsystemimplementationsandtheireffectsduringthewholelifecycleofthesystem.Unintendedandintendedconsequencesaswellasthesocio-technicalcircumstancesunderwhichtheseoccurareimportanttoreportsincetheyprovideinsightthatcaninformsystemdevelopersandimplementationsofsimilarsystemselsewhere.Althoughtheremightbepressuretosup-pressbadnews,thereareseveraloverridingimperativesforreportingunintendedconsequencesconsequences.AlsoassessmentofthedevelopmentandimplementationprocessitselfprovidesindicationsonwhatisgoodpracticeandhencecontributestosuccessfulHealthInformaticsapplicationsapplications.Whenitcomestodecisionshowbest(ifatall)touseITsystemsforaparticulartaskinthehealthcaredeliverypro-cess,objectiveappraisalofopportunitiesandoptionsrequiresaccesstoavailableevidence.Partofthisevidencecanbefoundinthescienti“cliterature.BesidesthefactthatthevolumeandcoverageofevaluationstudiesintheliteratureissmallgiventheimportanceandimpactofHealthInformaticsonmod-ernhealthcarepracticeanddeliveryy,theevidencethatisavailableistosomeextentdif“culttoappreciateduetopoorreportingpracticeactice.ThereisgrowingpublishedevidenceoftheimpactofHealthInformaticsonhealthcaree,and,increasingly, Weusethetermevidenceinabroadsense.Itisnotnecessar-ilyrestrictedtosomeoutcomemeasurelikemorbidity,mortalityorcompliance,butincludesalsouserexperiencesascollectedwithquestionnairesorqualitativestudiesandresultsfromtheanalysisofimplementationprocessesforexample.Consequently,evidence-basedhealthinformaticsisbasedonbothquantitativeandqualitativeresearchresults.WeusethetermHealthInformaticsratherthanMedicalInformatics.Thelatterhastheannotationthatitconcernsthepro-reviewsappearthatsummarizetheavailableevidenceintheformofanarrativereview,asystematicreview,oraformalmeta-analysis(e.g..g.).However,thewidevarietyinthekindsofITsystemsandtheirapplicationdomainsaswellasthevariouskindsofoutcomemeasureshavelimitedthegeneral-isabilityof“ndings,andthushavehamperedmeta-analysis,asreportedforexamplebyy.OtherstudiestoohaveshownthatpublicationsontheevaluationofITinterventionsinhealthcarehaveseveralshortcomingsthatseverelyhampertheproperappraisalofthesepublications,seethereviewin[12,pp.243…323]DuringaspeciallyconvenedexpertworkshoponHealthInformaticsevaluation…HISEVAL…sponsoredbytheEuropeanScienceFoundationandheldinInnsbruckk,theconcernwasraisedthatwithoutproperguidelinesforthedesign,plan-ning,execution,andreportingofevaluationstudiesinHealthInformatics,itwouldbedif“culttobuiltupaproperevidencebasethatcanbeusedtomakeinformeddecisionsregardingITinterventionsinhealthcare.Inotherdomainsofmedicine,theseproblemshavealsobeenidenti“ed.Workdoneintheearly1990sleadtothepublicationoftheCONSORTstatementin19961996.TheCONSORTstatementprovidesguidelinesforthepublicationofrandomizedcontrolledclinicaltrials(RCTs).Thisstate-menthasbeenadoptedbymanymedicaljournals.Laterthisstatementwasrevisedvisedandhasbeenextendedtocoverspeci“ckindsofRCTdesignssuchasclusterRCTsRCTsorRCTsassessingnonpharmacologictreatmentseatmentsandhasbeenshortenedforreportingRCTsinjournalandconferenceabstractsacts.SeveralotherguidelineshavebeendevelopedfollowingtheapproachoftheCONSORTstatement;forexam-pletheQUOROMstatementforreportingofmeta-analysesyses,STARDforreportingofdiagnosticstudiesstudiesandSTROBEforobservationalstudiesstudies.Anoverviewofthevariousguidelineshasbeenpublishedinin.SuchanoverviewisalsoavailableattheEQUATOR-networkwebsitebsite.CONSORThasprovenitsvalueovertime.StudieshavedemonstratedthatthereismorequalityinthereportingofcontrolledstudiesaftertheintroductionoftheCONSORTT.Guidelinesforgoodreportingofstudiesarelikelytohaveanin”uenceonthequalityofthestudiesthemselvesaswell,becauseoftherequirementforacleardemonstrationofsoundscienti“cmethodology.HealthInformaticsisasigni“cantareaofhealthsystemsinvestment,andpotentiallyaffectseveryprofessionalandpatient.ItisthereforeevidentthatHealthInformaticsshouldadoptsimilarrobustguidelinesastobuildamoresolidevi-dencebase.HealthInformaticsapplicationspotentiallyhave cessingofinformationrelatedtothemedicalconditionofpatients,butthereareapplicationsthatcoverabroaderscope,includingthehealthofabroaderpartofthepopulation. internationaljournalofmedicalinformatics78(2009)1…9 effectsonhealthcareorganizations,healthcaredeliveryandoutcomes,thereforeaHealthInformaticsapplicationmaynotdirectlyaffectthemedicalconditionofthepatient…asdrugsdo…butitwillgenerallyhaveanindirecteffectbyassistingthecaregiversintheirdecisionsandtheirpatientmanagement.ThestudydesignsthatarecoveredbyCONSORTandotherreportingguidelinesarenotalwaysthemostappropriateinHealthInformaticsevaluationresearchh.Otherstudydesigns,bothqualitativeandquantitativeones,arefrequentlyTheseobservationshaveledustothedevelopmentofguidelinesforreportingofevaluationstudiesinHealthInfor-matics,whichbuilduponworkofothers,yetalsotakeintoaccountsomespeci“cissuesthatareoftencentraltoHealthInformaticsevaluationstudies.Theyareintendedtobeappli-cableforaspectrumofquantitativeandqualitativestudydesignsasfoundinHealthInformaticsresearch. 2.ObjectivesTheobjectiveofSTARE-HI(STAtementontheReportingofEvaluationstudiesinHealthInformatics)istoprovideguide-linesforwritingevaluationreportsinHealthInformaticswhichcanbereliablyinterpretedbysubsequentreaders;andbydoingthistoimprovethequalityofpublishedevaluationstudiesinHealthInformatics;andthustoimprovetheevi-dencebaseofHealthInformatics.Theseobjectivesareachievedbypresentingguidelinesforreporting,whichareformattedasachecklistwithenoughdetailtoguideauthorswithoutmakingreportingtoocomplex.Thispaperdescribestheprocedurefollowedtocometotheseguidelines.ItgivesintabularformtheissuesthatshouldbeincludedinareportofanevaluationofanITinterven-tioninhealthcare.Next,thevariousitemsarediscussedinmoredetail.Adescriptionoftherationaleofeachitemandexampleswillbepublishedinasubsequentseparatepaper. 3.MethodAninitialsetofitemswasdraftedbytheeditorialteam(rep-resentedbytheauthorsofthispaper)basedondiscussionsattheHISEVALworkshopandontheirexperiencewithassessingthequalityofpapersforeitherarevieworameta-analysisoraspartoftheeditorialprocessasreviewersandeditors.TheCONSORTstatementstatement,criteriaforreviewersofbiomedi-calinformaticsmanuscriptsuscripts,theQUOROMstatementstatement,theSTARDstatementstatementandothermoregeneralrecommen-dationsonpublicationqualitysuchasswereusedasfurtherreferencematerial.Foreachitem,abriefdescriptionwasmade.Preliminaryversionsofthelistofitemsandtheirdescriptionswerepresentedtoanddiscussedwithcolleaguesworld-wideatworkshopsatMedinfo2004,MIE2005,MIE2006,AMIA2006,andThelistofitemsandthedescriptionswasre“nedbytheeditorialteambasedonthecommentsandsuggestionsreceived.TheresultingdraftwasthenpublishedonthewebsiteoftheEFMI-workinggrouponevaluation(EFMI-WGEVAL),andallmembersofthatworkinggroupwereinformedofthecallforcommentsattheendof2006.Thesecom-mentswereassessedbytheeditorialteamandincorporatedinthedraftdocument.Inspring2007,aseconddraftofSTARE-HIwasputonthewebsiteofEFMI-WGEVAL,andpotentialauthorswereinvitedtouseandvalidateSTARE-HI.ThisversionofSTARE-HIwasalsousedinapilotstudytoassessthequalityofreportingofevaluationstudiesinaselectedgroupofMedicalInformaticsjournalsinonepartic-ularyearear.BoththeusageofthedraftofSTARE-HIandthisassessmentstudyprovidedfeedbackonvariousaspectsofSTARE-HI.Allreceivedcommentshavebeentakenintoaccountlead-ingtoa“naldraftofSTARE-HI.This“naldraftwasonceagainmadeavailablethroughtheEFMI-WGEVALwebsiteanddistributedamongmembersoftheAMIAWorkingGrouponEvaluationfora“nalroundofcomments,whichhavebeenincludedintheguidelinesaspresentedhere. 4.STARE-HIrecommendationsThescopeofSTARE-HIistoprovideguidelinesforthereport-ingofevaluationsinHealthInformatics,independentofevaluationmethodused.Therefore,theseguidelineshaveageneralcharacter,withamainfocusonthedescriptionofthecontextinwhichthestudytookplace,thedescriptionofthemethodology,generalrecommendationsforthereportingofresults,andthestructuringofthediscussion.Incaseswhereastudydesignhasbeenselectedforwhichalreadyamorespe-ci“cguidelineexists(e.g.arandomizedtrial),STARE-HIshouldbeusedinawaywhichiscomplementarytothemorespeci“cguideline.WeconsidertheHealthInformaticsapplicationastheobjectofevaluation.Thisassessmentcancoverthetechni-calartefact(orsystem)assuchincludingitsfunctionality,butalsotheeffecttheartefacthasonthesurroundingorganiza-tionalenvironment,theworkingproceduresandsocialandpsychologicalissues,aswellasitseffectonpatientshealthandoutcomes.STARE-HIisnotprimarilydevelopedforassess-inginformationresourcesthataremadeavailablethroughIT,forexamplepatientguidesontheInternet,orforstudiesongeneralattitudesofhealthcareprovidersandpatientstowardtheuseofITinhealthcare.Table1,welistthepreferredstructureofapublicationonanevaluationstudy.Inthenextsection,wethendescribetheitemsinmoredetail.Fortherationaleandforexamplesofgoodpractice,werefertoasubsequentpaperthatisbeingdeveloped.ThereadershouldbeawarethatSTARE-HIisintendedasaguideline,notarigidstructuralstandard.Wedorecog-nisethattheorderofitemsandthedetailtowhichthevariousitemsmaybedescribedinareportorpapermaydependontheaudience,thetypeofpaper,theavailablespace,andthetypeofstudy.Thetargetreadermustalsobeawareofthemoredetailedinstructionoftherespectivesci-enti“cjournalthatmayaffecttheorderofthesectionsandimposerestrictionsontheirlength.Inaddition,notallissues internationaljournalofmedicalinformatics78(2009)1…9 Table1…TheSTARE-HIprinciples:itemsrecommendedtobeincludedinHealthInformaticsevaluationreports. Item#Item 1Title2Abstract3Keywords4Introduction4.1Scienti“cbackground4.2Rationaleforthestudy4.3Objectivesofstudy5Studycontext5.1Organizationalsetting5.2Systemdetailsandsysteminuse6Methods6.1Studydesign6.2Theoreticalbackground6.3Participants6.4Study”ow6.5Outcomemeasuresorevaluationcriteria6.6Methodsfordataacquisitionandmeasurement6.7Methodsfordataanalysis7Results7.1Demographicandotherstudycoveragedata7.2Unexpectedeventsduringthestudy7.3Study“ndingsandoutcomedata7.4Unexpectedobservations8Discussion8.1Answerstostudyquestions8.2Strengthsandweaknessesofthestudy8.3Resultsinrelationtootherstudies8.4Meaningandgeneralisabilityofthestudy8.5Unansweredandnewquestions9Conclusion10Authorscontribution11Competinginterests12Acknowledgement13References14Appendices presentedmaybeofrelevanceforareportforaparticularstudy. 5.Descriptionofitems1.TitleThetitleshouldgiveaclearindicationofthetypeofsystemevaluated,thestudyquestionandthestudydesign.TheuseofthetermevaluationŽ(orassessmentŽorstudyŽ)pre-cededbyaspeci“cationofthetypeofstudyinthetitlehelpstodetectevaluationstudies(e.g.EvaluationoftheeffectofaCPOEsystemonmedicationerrors:aretrospectiverecordanalysisŽ).2.AbstractTheabstractshouldpreferablybestructuredandmustclearlydescribetheobjective,setting,participants,mea-sures,studydesign,majorresults,andconclusions.Incasetherearemajorlimitationsinthestudy,theseshouldbementionedaswellell.StructuredabstractsarealreadymandatoryfortheInternationalJournalofMedicalInfor-matics,MethodsofInformationinMedicine,JAMIAfortheirresearchpapers,andmostmedicaljournals.Agivenscien-ti“cjournalmayhavedetailedguidelinesfororganizationoftheabstract.3.KeywordsTosecureproperindexingin,forexample,Medlineitisessentialtoprovidegoodkeywords.AmongthekeywordsshouldbeevaluationŽandkeywordsdescribingthetypeofsystemevaluated(e.g.EHR,LIS,telemedicine),thesetting(e.g.primarycare,secondarycare),theoutcomemeasures,thestudydesign(e.g.RCT,before-and-after,“eldstudy).ItisadvisedtouseMESHtermsasprovidedbyywheneverpossiblesincethatwillenhanceretrievabilityofthepaperinsearches.4.IntroductionTheintroductionshouldprovidethereaderwiththeback-groundfortherestofthepaper.Hence,itshouldaddressthefollowingissues:4.1.Scienti“cbackgroundThescienti“cbackgroundisadescriptionofwhatisalreadyknownaboutthe(typeof)systemthatisobjectofstudy.Weusethetermsysteminabroadsense.Itdoesnotnec-essarilyrefertoacompletesystem,butcanberestrictedtoacertainfunctionalityofalargersystem,ortheusageofamoregeneralapplicationinaspeci“cdomainorforaspeci“cpurpose,etc.Thetermcoversbothhardwareandsoftwaresystems,functionalitiesandalgorithms,andtheorganizationalandsocialenvironmentwhererelevant.4.2.RationaleforthestudyDescribesbrie”ythemotivationforthestudy:whatarethespeci“creasonstoperformthestudy(scienti“cinter-est,justi“cationforexpenditure,insightintoproblems,addressingopenresearchquestions)?Isthestudypartofalargerresearch,developmentorimplementationproject?Fromwhichstakeholderviewpoint(ifany)isthestudyperformed?Ifpossible,itshouldalsobementionedwhatin”uencethe“ndingsofthestudymayhave.4.3ObjectivesofthestudyThespeci“cstudyquestionsandhypothesesmustbedescribedasconciselyaspossible.Itshouldthenbestatedwhereappropriatewhetheranyformalpermissionwasobtainedforexamplefrominstitutionalreviewboards(IRBs),ethicscommittees,staffcommittees,andthelike.5.StudycontextInformationonstudycontextisimportantforthelaterassessmentofgeneralisabilityofresults.Clearly,givingall Somejournalsmayrequirethatthisinformationisprovidedattheendofthemanuscript.RequirementswithrespecttoIRBapprovalmayvaryfromcountry-to-country.Wheneverparticipantsrunacertainriskbyparticipatinginastudy,IRBapprovalshouldbesought.Currently,itisoftenconsideredappropriatetoseekIRBapprovalforstudiesthataskhealthcareprofessionalsfortheiropinionsaboutITapplicationsinaspeci“csituation(e.g.howtheyperceivetheCPOEsystemthatwascurrentlyinstalledintheirorganization).Weusethetermgeneralisabilityinabroadsense.Itnotonlycoversthenotionthatiscommonforquantitativestudies,butalsotheassessmentofthe“ndingsbyareaderontheapplicabilityoftheresultsinhisorherownenvironment.Aqualitativestudyontheperceivedusefulnessofaparticularsysteminprimarycaremaynotbegeneralizableto/applicableinatertiarysetting. internationaljournalofmedicalinformatics78(2009)1…9 detailsonthecontextisnotfeasible„theauthorsofaneval-uationpaperhavetodecidetowhichextentinformationisneededtosecurethevalidityandgeneralisabilityofthepaper.5.1OrganizationalsettingThisshoulddescribethehealthorganization(s)wherethesystemisbeingevaluated,includingitsgeographicalloca-tionandpreferablyitsname.Itshouldindicatewhatkindofhealthcarefacilityitis(primary,secondary,tertiarycare,homecare,etc.).Incasethewholeorganizationisnotinvolved,itshouldbemadeclearwhichdepartmentshavebeeninvolvedintheevaluation(e.g.a12-bedintensivecareunitofa320-bedurbanreferralhospital.).5.2.SystemdetailsandsysteminuseThesystemdescriptionmustpermitthereadertounder-standhowthesystemworks(orisintendedtowork).Theauthorsmayrefertoatechnicaldescriptionprovidedelse-where,andthismayreducethetechnicaldescription,butsuf“cientdetailisstillneededfortheevaluationstudyreporttobeself-contained.Systemsdetailsshouldcom-prisetheaimoftheITsystem(e.g.laboratorysystem,administrativesystem,nursingdocumentationsystem,CPOEsystem),typeofsystem(home-grown,opensourceorcommercialsystem),thetypeofinformationthatismanagedbythesystem(e.g.drugorders,nursingcareplans),andtheclinicalorothertaskssupportedbythesystem(e.g.orderingprocesses,nursingdocumentationprocesses).Thedescriptionshouldalsoincludeinformationon(1)howwide-spreadthesystemisusedinthefacilityinwhichthesystemisevaluated,forhowlongandforwhatpurposeand(2)numberandprofessionsoftheusersofthesysteminthatfacility.Anyadditionalinformationtodetailrelevantaspectsofthecontextinwhichthestudywasconductedshouldbementioned(e.g.customizationofsoftware,usertraining,additionalattentiontothestudy,systemonlyimplementedshortlybeforetheevaluation).6.MethodsThissectionofapaperdescribesinsuf“cientdetailthestudydesignandthemethodsusedinthestudy.STARE-HIcontainsitemswhichhavebeendealtwithinmoredetailinguidelinesformedicalstudies;whereappropriatewewillrefertothese.6.1.StudydesignThisdescribestheoverallstudydesignandthemotivationforchoosingit.Thedescriptionofstudydesigncomprisesthetypeofstudy,forinstanceobservationalstudy(casestudy),quasi-experimentalstudy(e.g.before…after,withorwithoutcontrol;interruptedtime-serieswithorwithouton…off-design),orexperimentalstudy(RCT).ForanRCT,theauthorsshouldfollowtheCONSORTguidelinesforallpartsofSection6.Incaseofnon-RCT,thetaxonomypresentedinincanbeusedtodescribethetypeofstudy.Itshouldalsobespeci“edwhetheritisalaboratorystudy,simulation,oractual“eldstudy.Ifapplicable,statereasonsforchoiceoflevelofobservation/participation(patients,professionals,clinics,hospitals,etc.).Throughoutthedescriptionofthestudydesign,theauthorsmustshowtheirawarenessofspeci“candpotentialstudydesignbiases(forthispurposeseethereviewofbiasesin[12,pp.257…289,34,pp.209…215]bymakingastatementontheirpotentialimpactandhowthatishandled.6.2.TheoreticalbackgroundofthestudyWhereappropriate,statethetheories…withsuf“cientref-erences…onwhichthestudyisbased,whichguidedtheselectionofthemeasurementinstrumentsusedandwhichformthebasisforinterpretationoftheresults(e.g.theuseracceptancemodelthatguidedaquantitativesurveyortheorganizationaltheoriesthatguidedaqualitativestudy).6.3.ParticipantsDescribethemethodsofselectionofparticipatingusers,patients,units,hospitals,etc.,including…ifapplicable…inclusionandexclusioncriteriaforeachtypeofparticipantinastudy.Incaseofacontrolledtrial,itshouldbespeci“edhowparticipantswereallocatedtointerventionandcontrolgroups(randomizationorotherapproaches„refertoCON-SORTformoredetailsdetails).Statethebasisofsamplesizecalculationsifapplicable(powercalculations).6.4.Study”owGivesuf“cientdetailsondateofbeginningandendoftheoverallstudyandanystudyperiods;givecleardescriptionanddateofintervention(inexperimentalstudies).Incaseofastudyinwhichseveralmethodshavebeenused,spec-ifywheneachmethodwasusedforwhichgroup.A”owdiagramshouldbeusedtosummarizetheexperimentalstudydesigns(liketheRCT”owchartasrequiredinCON-SORTT).Forobservationalstudies,useadiagramshowingstudyactivitiesovertime.Ineachcase,indicatetimelineandmarkanyimportantdatessuchasbeginningofstudy,intervention,endofstudy,whereappropriatecomparedtodevelopmentmilestonesofthesystem(place/phaseinlifecycle).6.5.OutcomemeasuresClearystateoutcomemeasuresorotherevaluationvari-ablesofinterestthatwereusedinthestudy.De“netoasuf“cientdetailthekeyconceptsinthestudysuchasmed-icationerrororusersatisfaction.Inopenqualitativestudies,nopre-de“nedoutcomemeasurecanbede“ned;however,whencertainaspectsaremoreinthefocusoftheresearcherthanothers,thesecanbestatedhere(seealsoSection6.2).6.6.MethodsfordataacquisitionandmeasurementThissectionshouldprovidesuf“cientdetailsuchthatoth-ersareabletoduplicatethestudyortousesomeofthemethodsforotherstudies.Allrelevantaspectsofappliedmethods(e.g.questionnaire,interview,observation,log“leanalysis,chartreview)shouldbedescribed.Examplesofaspectsarelocationandset-tingofdatacollection,numberandtypeofinterviews,typeanddurationofobservations,whetherdatacollectionwasretrospectiveorprospective,professionalbackgroundoftheinterviewers,blindingofobserverand/orpartici-pantsand/oranalysts,etc.Itshouldalsobeidenti“edwhichoutcomemeasuresarecoveredbyeachoftheselectedForeverymeasurementorobservation,informationontheirvalidationhastobegiven,withreferencestoear-lierworkwherenecessary(e.g.wasthereapre-testwithassessmentofinter-raterreliability?Wasthequestionnairepreviouslyvalidated?).Newlydesignedmeasurementtools internationaljournalofmedicalinformatics78(2009)1…9 shouldbedescribedinmoredetail,fulldisclosureofsuchtoolsshouldbegivenintheappendicesorassupplementary6.7.MethodsfordataanalysisThissectiondescribesthemethodsusedfordataanalysis.Theselectionofthosemethodsdependsondataacquisitionmethodsandstudyquestions.Whenseveralmethodsareused,combinethedescriptionofdataacquisitionanddataanalysisforeachmethod.Forquantitativedata,statethestatisticaltechniquesthathavebeenusedforanalysis.Fortheanalysisofquali-tativedata,indicatetheanalysismethodsindetail.Foralldataanalysismethods,indicateanysoftwareproductTriangulationmaybeusedtocombinedatafromvarioussourcesces.Whentriangulationisused,itshouldbespeci“edwhatkindoftriangulationwasapplied(methods,measures,data,investigatorortheorytriangulation),andhowthedatawerecombined.Throughoutthedescriptionofthedataacquisitionanddataanalysis,theauthorsmustshowtheirawarenessofspeci“candpotentialdataanalysisbiases(formoreinformation,[12,pp.290…313][34,pp.209…215]7.ResultsTheresultsectionpresentsmainlythedataobtainedfromapplyingthemethodsasdescribedinSection6.Depend-ingonthetypeofstudy,theorganizationandnamingofsectionsmaybedoneinadifferentway.TheinterpretationanddiscussionoftheresultsshouldbelefttothediscussionSections7.2and7.4areofspecialimportanceforbutnotlimitedtoqualitativestudies,asoneoftheirobjectivesistoobtainnewinsightforexamplesocialandorganizationalaspectsofHealthInformaticsapplications.7.1.DemographicandotherstudycoveragedataGivebasicnumbersonthesizeofthestudy,forexam-plenumberofusersobserved/interviewed,documentsormedicalrecordsanalysed,distributionandreturnrateofquestionnaires,numberofobservationdays,pagesoftran-scriptsanalysed,etc.Whenthestudymeasuresarerelatedtopersons,baselinedemographicdataand/or(clinical)characteristicsofstudyparticipants(users,patients,andunits)shouldbegiven,suchasagegroups,professions,usagepatterns,patientsdiagnosticscores,etc.Inparticularinqualitativestudiesthecharacteristicsandqualitiesoftheparticipantsmaybeofmoreimportancethansheernumbers.Informationonnumberandtypeofdrop-outsshouldbeaddedaswellwithidenti“cationofreasons.Whereappropriate,baselinedatamustbegivenforrelevantgroupsseparately(e.g.forcontrolandinterventiongroupsintrials,orfordifferentprofessionsoragegroupswhenthatisrelevant).7.2.UnexpectedeventsduringthestudyAnyeventthatmayhavein”uencedstudydesignand/orresultshastobedescribed(e.g.deviationsfromtimeline,systemupdatesduringthestudy,staffchanges,educa-tionalinterventions,systemfailure,highdrop-outrateinonegroupandchangesinmanagementororganizationalstrategyduringthestudyperiod).Ifpossible,theseeventsshouldberelatedtothetimelineofthestudy.Theauthorsshouldindicatetowhatextenttheseunexpectedeventsmightin”uence(bias)thestudy“ndings.7.3.Study“ndingsandoutcomedataThisisthemajorsection,presentingtheresultsofthestudy.Foreachstudyquestion,outcomevariableandevaluationcriterion,suf“cientdatashouldbepresented.Qualitativedatamaybepresentedastext.Quotationsfromparticipantsshouldbeusedtoillustratemajorpoints.Thesequotesshouldbeanonymous,buthaveanindicationofthetypeofpersonbeingquoted(e.g.Nurse3).Quantitativedatacanbepresentedintablesand“gures.Typically,eachtable,“gure,etc.,shouldbereferencedinthetext.Themostimportantorrelevantresultsshouldbeemphasisedhere,andspecialnotionshouldbegiventounexpectedorstrikingresultssuchasdifferencesbetweengroups.Absolutenumbersshouldalwaysbeprovided;notonlyrelativenumbers.7.4.UnexpectedobservationsAnyunintended(positiveornegative)side-effectsofthesystemthatwerenotinthefocusofthestudybutthatseemremarkableshouldbereportedhere.Thiscouldbe,forexample,theobservationofbottlenecksintheclinicalwork-”owaftersystemimplementation,severeorganizationalproblemsthatseemrelatedtothenewsystem,orpersistentunsolicitedresponsesonaspeci“ceffectofasystemdur-ingaqualitativestudythatfocussedonotheraspects.HeretheauthorsmayreportaboutthedifferencebetweentheintendeduseasdescribedinSection5.2andtheobservedactualuse.ThedifferencewithSection7.2isthatunexpectedeventsmayin”uencethe“ndings(andmayhavecausedchangesofthestudyprotocol),whileunexpectedobservationsrelatetoissuesthataroseduringthestudythatcouldleadtoadditionalinsights,furtherrecommendations,potentialexplanationsforthe“ndingsorfutureresearchtopics.8.DiscussionThediscussionshouldbeacriticalinterpretationandassessmentofthestudyresultsandthestudyitselfinviewofthestudyquestions.Wesuggestauthorsshouldmakethediscussionstructuredwiththefollowingclearsubheadingssubheadings.€8.1.AnswertostudyquestionsInterpretthedataandansweryourstudyquestion(s).WhereasinSection7theresultsarepresentedindetail,inthispartofthereporttheanswerstothestudyquestionsareinfocus.Makeexplicitreferencetothespeci“cstudyques-tionseitherbyrestatingthemorbyothercross-referencemechanisms.8.2.StrengthsandweaknessesofthestudyThissectioncontainsacriticaldiscussionofthemeth-odsused.Describethestrongandweakpointsofthestudy,forexampleconcerningthestudydesign,compa-rabilityofinterventionandcontrolgroupwithrespecttobaselinedata,studyexecution,confounders,internal DetailedadviceonhowtopresentdataintablesandgraphsisoutofscopeofSTARE-HI.Guidanceonthistopiccanbefoundinseveralpublications,forexampleinE.R.Tufte,TheVisualDisplayofQuantitativeInformation,GraphicsPress. internationaljournalofmedicalinformatics78(2009)1…9 andexternalvalidityof“ndings,completenessofacquireddata,drop-outofparticipants,representativenessoftheparticipants,low-orhigh-responserates,etc.ReferherealsototheinformationpresentedinSection7.2.Discussanybiasesthatcouldbepresentandthatwouldin”u-encethe“ndingsofthestudyortheinterpretationofthe8.3.ResultsinrelationtootherstudiesMakeclearwhatexactlyisnovelaboutyourresults.Describetowhatextenttheresultsareinagreementwith“ndingsofothersandinthislightprovideinformationaboutthecomparabilitywiththestudysetting.Whenthereisdisagreementwith“ndingsofothers,discusspossiblereasons.8.4.MeaningandgeneralisabilityofthestudyDescribethemeaningofthestudy“ndings,bothforthevar-iousstakeholdersinthestudy,forotherinstitutionsandforHealthInformaticsingeneral.Inthiscontext,discussthegeneralisability/applicabilityofthestudyforotherorgani-zations.ReferherealsototheinformationgiveninSections5.1,5.2and8.2.8.5.UnansweredandnewquestionsDiscusswhetherthestudyhasshednewlightonanissueand/orhasraisednewquestions.Describewhatresearchshould/couldbeperformedinthefuturetofurtherimproveourknowledgeaboutthesystemanditseffects.ReferherealsototheinformationpresentedinSection7.4.9.ConclusionTheconclusionsummarizesthemain“ndings,discussestheimpactofthe“ndingsandhowtheyrelatebacktothebigpicturedescribedintheSection1,givesrecommenda-tionsbytheauthorsandprovidesashortoutlookforfutureresearch.10.AuthorscontributionAnincreasingnumberofjournalsrequiremakingthecon-tributionsoftheauthorstoapaperexplicitoratleasttomakeclearthateachauthorquali“esforauthorship[29].Itisrecommendabletomakethatinformationpartofthepaperandnotonlyinthecoveringletterwhilesubmittingthemanuscript.11.CompetinginterestsAstatementoftheinterests,“nancialorotherwise,theauthorsmayhavewithrespecttosubjectofstudyandwhichmay…butnotnecessarilyhave…in”uencedthedesignofthestudyand/ortheinterpretationoftheresults[29].Thisinformationshouldalsomakecleartherelationofthestudyteamtowardsthesystembeingevaluated(developers,users,operators,internalqualityassurance,externalteam,12.AcknowledgementsAcknowledgeany“nancialorothersupportyougotwhenconductingthestudyorwritingthepaperpaper.€13.ReferencesReferencesshouldadheretothejournalguidelines.Whennostrictguidelinesareprovidedbythejournal,thestylerec-ommendedbytheNLMispreferreded.Theauthorsshouldincludeonlyreferencesthatarereallyneededfortheirargu-ments.Itisimpropertoprovidealonglistofreferencesonlytodemonstratethatmanypapershavedealtwithaspeci“cissue.Onlythekeypapersshouldbeincluded.Prefersuchlistsshouldberestrictedtoreferencesthecontentsofwhicharediscussedinthepaper.14.AppendicesAnysupportingmaterialshouldbeincludedinappendices.Thisincludesnecessary,detaileddescriptionsofspeci“cmeasurementmethods/tools(e.g.aquestionnaire),spe-ci“cdataanalysistechniquesanddetailedstudyresults.Currently,moreandmorejournalsallowsuchadditionalmaterialbeingstoredonthepublisherswebsite.Thisalsoopensuptheabilitytoinclude(partsof)audio-visualandmulti-dimensionalmaterialcollectedinthestudy. 6.DiscussionSTARE-HIwasdevelopedtoprovideguidelinesforwritingandinterpretingevaluationreportsinHealthInformatics,bydoingthistoimprovethequalityofpublishedITevaluationstudiesinHealthInformaticsandthustoimprovetheevidencebaseofHealthInformatics.ItencouragestransparencyinreportingofITevaluationstudies.STARE-HIwasdevelopedinaniterativeprocessinvolvingvolunteerexpertsfromvariousHealthInformaticsdomains.Noformalprocedures(e.g.Delphitechnique)orvotingpro-cedurewasusedtodevelopthisversion,butallcolleaguesinterestedcouldsubmitcommentsthatwereassessedbytheeditorialgroup.SeveralpresentationsofdraftsofSTARE-HIatvariousworkshopsandconferenceswereusedtomakesurenoimportantaspectwasmissed.STARE-HIborrowsfromearlierworkinotherdomainssuchasCONSORTforreportingofRCTs;however,toourknowledge,STARE-HIisthe“rstapproachtodevelopspeci“cguidelinesforthereportingofHealthInformaticsevaluationstudies.Althoughtheguidelinesarewrittenforthepublicationofevaluationstudiesinthescienti“cliterature,theycanalsobeusedforotherreportingpurposes.Althoughthestructuremightrequiresomechanges,forexampletheAbstractcouldbecomeanexecutivesummaryandtheconclusionscouldbecomeasetofrecommendations,theissuesarelargelyvalidforanykindofreportingofstudies. 7.ConclusionWepresentSTARE-HIasaguidelinetoreportITevaluationstudiesinhealthcarewithdetailedrecommendationsforeachaspectthatisparticularlyrelevantforanevaluationstudy.WhetherSTARE-HIisfeasibleforthebroadrangeof(quan-titativeandqualitative)HealthInformaticsevaluationpaperscanonlybeshownwhenitisusedbyauthorsandeditors.Weinviteanybodytoreporttheirexperiencethatmaybeincor-poratedinsubsequentupdatesofSTARE-HI.Subsequentmorerigidevaluationstudiesonthisquestionareplanned.Weperceivehigherqualitypublicationsofevaluationstud-iestobeanimportantsteptowardsthevisionofscienti“callyvalidevidence-basedHealthInformatics.Suchanevidencebasewillhelp:todemonstratethevalueofHealthInformaticsapplica- internationaljournalofmedicalinformatics78(2009)1…9 SummarypointsWhatwasalreadyknownbeforethisstudy:ThequalityofreportingofHealthInformaticsevalua-tionstudiesisvariableandisopenforimprovement.Guidelinesforreportingdohelpauthorsinaddressingthecriticalissuesinapublication.Agreedguidelinesforreportinghaveanimpactonthequalityofpublishedpapers.Whatthisstudyhasadded:HealthInformaticsevaluationstudiesshouldaddressissuesthatareparticularforour“eldandnotcoveredbyotherguidelinesforreporting.DespitethevarietyinstudydesignsfoundinHealthInformaticsevaluationstudies,asetofgeneralprinci-plesforreportingofsuchevaluationstudiescouldbeassembledintoacomprehensiveguideline.toassistdecisionmakersin“ndingevidenceonthemostappropriatesystemsandapproaches,toguidethe“eldtowardsdevelopmentsthathavemaximalbene“cialimpactonhealthcaredelivery,andtodemonstratethatHealthInformaticshasasolidscienti“cbasisandhenceisadisciplinethatcancontributetofurtherimprovementofthequalityofcare. 8.AuthorscontributionsTheideaforSTARE-HIwasraisedduringtheHISEVALwork-shopinInnsbruckk.J.T.tooktheinitiativetodevelopSTARE-HI,heistheguarantorofthestudy.J.T.andE.A.drafteda“rstlistofissuesanddraftedthe“rstversionofthemanuscript.J.B.,N.d.K.,P.N.,andM.R.allcontributedbycriticallyassessingtheitemsandtheirdescriptionsinsev-eraliterations.Theyhavemadesuggestionsforexpansionandprovidedvariouspartsofthetext.J.T.andE.A.integratedthevariouscontributionsandeditedthe“nalversionofthemanuscript.Allauthorshaveapprovedthis“nalversion. CompetingInterestsJTiseditoroftheInternationalJournalofMedicalInformatics.HewaschairoftheworkinggrouponTechnologyAssessmentandQualityDevelopmentoftheInternationalMedicalInfor-maticsAssociation(IMIA)withJBasco-chair.CurrentlyNdKchairsthisworkinggroup.EAischairoftheWorkingGrouponAssessmentofHealthInformationSystemsoftheEuropeanFederationofMedicalInformatics(EFMI).JBandPNareco-chairsofthisworkinggroup. AcknowledgmentsThisworkhasbeenvalidatedandenrichedbycommentsfrommanyduringitspreparationanddevelopment.Inpar-ticularweacknowledgethecontributionsof:JosAarts,EmilyCampbell,PetraKnaup,ChristofMachan,ZahraNiazkhani,ChristianNøhr,HabibPirnejad,JoshuaRichardson,RainerRöhrig,DeanF.Sittig,MuratSincan,ChristaWessel,JohannaWestbrookandJeremyWyatt.Endorsements:EFMIhasformallyendorsedSTARE-HIintheirboardmeetingof30May2007inBrijuni,Croatia.InNovember2007,theworkinggrouponEvaluationoftheAmericanMedicalInformaticsAssociation(AMIA)endorsedSTARE-HIduringtheAMIA2007AnnualMeeting.TheGeneralAssemblyofIMIAhasadoptedtheSTARE-HIpaperasaformalIMIAdocumenton25May2008. [1]D.Sackett,W.Rosenberg,J.Gray,R.B.Haynes,S.Richardson,Evidencebasedmedicine:whatitisandwhatitisnt,BMJ312(7023)(1996)71…72.[2]E.Ammenwerth,J.Brender,P.Nykanen,H.U.Prokosch,M.Rigby,J.Talmon,Visionsandstrategiestoimproveevaluationofhealthinformationsystems.Re”ectionsandlessonsbasedontheHIS-EVALworkshopinInnsbruck,Int.J.Med.Inform.73(June(6))(2004)479…491.[3]E.Ammenwerth,N.T.Shaw,Badhealthinformaticscankill„isevaluationtheanswer?MethodsInf.Med.44(1)(2005)1…3.[4]J.S.Ash,M.Berg,E.Coiera,Someunintendedconsequencesofinformationtechnologyinhealthcare:thenatureofpatientcareinformationsystem-relatederrors,J.Am.Med.Inform.Assoc.11(March…April(2))(2004)[5]E.Tenner,WhyThingsBiteBack:TechnologyandtheRevengeofUnintendedConsequences,VintageBooks,NewYork,NY,1997.[6]M.Rigby,Evaluation:16powerfulreasonswhynottodoit„and6over-ridingimperatives,in:V.Patel,R.Rogers,R.Haux(Eds.),MEDINFO,IOSPress,2001,1198…1202.[7]J.Talmon,Evaluationandimplementation:acallforaction,Methods.Inf.Med.45(Suppl1)(2006)S11…S15.[8]M.Rigby,Evaluation„theCinderellascienceofICTinhealth,MethodsInf.Med.45(Suppl1)(2006)S114…S120.[9]P.S.Whitten,F.S.Mair,A.Haycox,C.R.May,T.L.Williams,S.Hellmich,Systematicreviewofcosteffectivenessstudiesoftelemedicineinterventions,BMJ324(June(7351))(2002)[10]E.Ammenwerth,N.deKeizer,Aninventoryofevaluationstudiesofinformationtechnologyinhealthcaretrendsinevaluationresearch1982…2002,MethodsInf.Med.44(1)(2005)44…56.[11]B.Chaudhry,J.Wang,S.Wu,M.Maglione,W.Mojica,E.Roth,etal.,Systematicreview:impactofhealthinformationtechnologyonquality,ef“ciency,andcostsofmedicalcare,Ann.Intern.Med.144(May(10))(2006)742…752.[12]J.Brender,HandbookofEvaluationMethodsforHealthInformatics,AcademicPress,NewYork,2006.[13]C.Begg,M.Cho,S.Eastwood,R.Horton,D.Moher,I.Olkin,etal.,Improvingthequalityofreportingofrandomizedcontrolledtrials.TheCONSORTstatement,JAMA276(August(8))(1996)637…639. internationaljournalofmedicalinformatics78(2009)1…9 [14]D.G.Altman,K.F.Schulz,D.Moher,M.Egger,F.Davidoff,D.Elbourne,etal.,TherevisedCONSORTstatementforreportingrandomizedtrials:explanationandelaboration,Ann.InternMed.134(April(8))(2001)663…694.[15]M.K.Campbell,D.R.Elbourne,D.G.Altman,CONSORTstatement:extensiontoclusterrandomisedtrials,BMJ328(March(7441))(2004)702…708.[16]I.Boutron,D.Moher,D.Altman,K.F.Schulz,P.Ravaud,MethodsandprocessesoftheCONSORTgroup:exampleofanextensionfortrialsassessingnonpharmacologictreatments,Ann.Intern.Med.148(February(4))(2008)[17]S.Hopewell,M.Clarke,D.Moher,E.Wager,P.Middleton,D.Altman,etal.,Consortforreportingrandomizedtrialsinjournalandconferenceabstracts,Lancet371(January)(2008)[18]D.Moher,D.J.Cook,S.Eastwood,I.Olkin,D.Rennie,D.F.Stroup,Improvingthequalityofreportsofmeta-analysesofrandomisedcontrolledtrials:theQUOROMstatement.Qualityofreportingofmeta-analyses,Lancet354(November(9193))(1999)1896…1900.[19]P.M.Bossuyt,J.B.Reitsma,D.E.Bruns,C.A.Gatsonis,P.P.Glasziou,L.M.Irwig,etal.,Towardscompleteandaccuratereportingofstudiesofdiagnosticaccuracy:theSTARDinitiative.Standardsforreportingofdiagnosticaccuracy,Clin.Chem.49(January(1))(2003)1…6.[20]E.A.vonElm,D.Altman,M.Egger,S.J.Pockock,P.C.Gotzsche,J.P.Vandenbroucke,etal.,Thestrengtheningthereportingofobservationalstudiesinepidemiology(STROBE)statement,Lancet(2007).[21]M.E.Falagas,E.I.Pitsouni,Guidelinesandconsensusstatementsregardingtheconductionandreportingofclinicalresearchstudies,Arch.Intern.Med.167(May(9))(2007)877…878.[22]TheEquator-NetworkWebsite,2008,Availablefrom:http://www.equator-network.org/(citedJune42008).[23]D.Moher,A.Jones,L.Lepage,UseoftheCONSORTstatementandqualityofreportsofrandomizedtrials:acomparativebefore-and-afterevaluation,JAMA285(April(15))(2001)1992…1995.[24]M.Egger,P.Juni,C.Bartlett,Valueof”owdiagramsinreportsofrandomizedcontrolledtrials,JAMA285(April(15))(2001)[25]H.A.Heath“eld,I.E.Buchan,Currentevaluationsofinformationtechnologyinhealthcareareofteninadequate,BMJ313(October(7063))(1996)1008.[26]B.Kaplan,Evaluatinginformaticsapplications„somealternativeapproaches:theory,socialinteractionism,andcallformethodologicalpluralism,Int.J.Med.Inform.64(November(1))(2001)39…56.[27]D.G.Altman,Betterreportingofrandomisedcontrolledtrials:theCONSORTstatement,BMJ313(September(7057))(1996)570…571.[28]E.Ammenwerth,A.C.Wolff,P.Knaup,H.Ulmer,S.Skonetzki,J.H.vanBemmel,etal.,Developingandevaluatingcriteriatohelpreviewersofbiomedicalinformaticsmanuscripts,J.Am.Med.Inform.Assoc.10(September…October(5))(2003)512…514.[29]InternationalCommiteeofMedicalJournalEditors.Uniformrequirementsformanuscriptssubmittedtobiomedicaljournals:writingandeditingforbiomedicalpublication,2007,Availablefrom:www.ICMJE.org(citedJune42008).[30]J.Talmon,E.Ammenwerth,T.Geven,Thequalityofreportingofhealthinformaticsevaluationstudies:apilotstudy,in:MEDINFO,Brisbane,Australia,2007.[31]R.B.Haynes,C.D.Mulrow,E.J.Huth,D.G.Altman,M.J.Gardner,Moreinformativeabstractsrevisited,Ann.Intern.Med.113(July(1))(1990)69…76.[32]NationalLibraryofMedicine,MedicalSubjectHeadings,2008,Availablefrom:http://www.nlm.nih.gov/mesh/meshhome.html(citedJune232008).[33]A.D.Harris,J.C.McGregor,E.N.Perencevich,J.P.Furuno,J.Zhu,D.E.Peterson,etal.,Theuseandinterpretationofquasi-experimentalstudiesinmedicalinformatics,J.Am.Med.Inform.Assoc.13(January…February(1))(2006)16…23.[34]C.Friedman,J.Wyatt,EvaluationMethodsinBiomedicalInformatics,2ed.,Springer-Publishing,NewYork,NY,2006.[35]J.Greene,C.McClintock,Triangulationinevaluation:designandanalysisissues,Eval.Rev.9(5)(1985)523…545.[36]B.Kaplan,D.Duchon,Combiningqualitativeandquantitativeapproachesininformationsystemsresearch:acasestudy,MISQuart.12(4)(1988)571…586.[37]M.Docherty,R.Smith,Thecaseforstructuringthediscussionofscienti“cpapers,BMJ318(May(7193))(1999)[38]K.Patrias,Citingmedicine:theNLMstyleguideforauthors,editorsandpublishers[Internet],2nded.,2007,Availablefrom:http://www.ncbi.nlm.nih.gov/books/bookres.fcgi/citmed/frontpage.html(citedJune92008).