optimism,agreeability,mood,propensitytouseextremecategories,andotherch - PDF document

optimism,agreeability,mood,propensitytouseextremecategories,andothercharacter-istics,andsodoingsomethingaboutthisÔÔresponse-categorydifferentialitemfunction-ingÕÕ(orDIF)shouldbeahighpriorityforresearchers.ThemethodologyofanchoringvignettesattackDIFwithnewtypesofsupplementalsurveyquestionsthatmakeitpossibletoconstructacommonscaleofmeasurementacrossrespondents,alongwithspeciallydesignedstatisticalmethodsforanalyzingtheresultingdata.Anchoringvignetteshavenowbeenusedtomeasurenumerousconceptsandhavebeenimplementedinsurveysinover80countriesbyagrowinglistofsurveyorganizationsandresearchersinavarietyofacademicÞelds.Inthispaper,wedescribethisapproachanddevelopimprovedstatisticalmethodsforanalyzing,evaluating,andselectinganchor-ingvignettesthatrequirefewerassumptions,canextractmoreinformationfromthesamesurveyquestions,andshouldsaveinresearchcosts.Previousapplicationsofanchoringvignetteshaveusedasmanyas12vignettesperself-assessmentquestion.However,addingthismanyadditionalquestionsforeachself-assessment,oreventheÞveusedbyKingetal.(2004),maybeprohibitivelyexpensiveinsomesurveysand,weshow,areoftenunnecessarytocorrectDIF.Insomecases,thenecessarycorrectionmaybeachievedwithasinglevignette.Inothercases,morevignettesmaybeinformative.Inallcases,themethodsweintroducetoevaluatetheefÞcacyofeachvignettetoimproveinterpersonalincomparabilityshouldbeofdirectpracticalusetoresearchers.WebegininSection2bysummarizingtheanchoringvignettesapproach.Section3thenprovidesamoregeneraldeÞnitionofthisapproach,anewformalization,amoregenerallyapplicableanalyticalmethod,andanillustrationofthemethodsoffered.Section4thendevelopsnewwaysofevaluatingtheinformationavailableinasetofanchoringvignettesanddetectingvignettesthatmaybeempiricallyunnecessaryorlessusefulandthosethatmayviolatekeyassumptionsofthetechnique.Section5givesseveralexamplesofeval-uatingvignettesinpractice,andSection6concludes.ThenewinformationourmethodsrevealgreatlyincreasestheefÞcacyofthenonparametricestimator,makingitapowerfulalternativetotheparametricapproachandonethatrequiresconsiderablylessstringentassumptions.TheTechniqueofAnchoringVignettesAvariantofaquestionaskedinnumeroussurveysseekstomeasurewhatpoliticalscien-tistscallpoliticalefcacyHowmuchsaydoyouhaveingettingthegovernmenttoaddressissuesthatinterestyou?(1)Nosay,(2)Littlesay,(3)Somesay,(4)Alotofsay,(5)Unlimitedsay.Forthisquestion,asmostothers,politicalscientiststypicallytheorizethateachrespondenthasanlevelofefÞcacythatmaydifferfromthereportedlevelduetomeasurement,respondentÔÔconsiderations,ÕÕthesurveyinterviewsetting,positivitybias,orotheraspectsofDIF.PoliticalscientiststypicallyviewtheactuallevelofefÞcacyasarelativelyobjec-tivepsychologicalstate:RespondentswhogenuinelyfeelpoliticallyefÞcaciousaremorelikelytoparticipateinpolitics,writeletterstopublicofÞcials,contributetopoliticalcampaigns,debatepolicywiththeirfriends,andfeelmoregenerallypartofthepoliticalsystem.ThedifferencebetweentrueunderlyingperceivedpoliticalefÞcacyandthere-portedlevelmaydifferduetoavarietyofmeasurementfactorsincludingidiosyncratic Alibraryofanchoringvignetteexamplesusedintheseandothersurveys,andothermaterials,canbefoundathttp://gking.harvard.edu/vign/.Oursoftwareforanalyzinganchoringvignettesisathttp://wand.stanford.edu;seeWand,King,andLau(forthcoming).GaryKingandJonathanWand Tobemoreexplicit,twoassumptionsenableustoregardthistrichotomousrecodedanswerasDIFfreeandfreelycomparedacrossdifferentgroupsofpeople.TheÞrstisresponseconsistency,whichistheassumptionthateachrespondentusesthesurveyresponsecategoriesinthesamewaytoanswertheanchoringvignetteandself-assessmentquestions.DifferentpeoplemayhavedifferenttypesofDIF,butanyonepersonmustapplythesameDIFinapproximatelythesamewayacrossthetwotypesofquestions.Secondis,whichistheassumptionthatthelevelofthevariablerepresentedinthevignetteisunderstoodbyallrespondentsinthesamewayapartfromrandommeasurementerror.Ofcourse,evenwhenrespondentsunderstandvignettesinthesamewayonaverage,differentrespondentsmayapplytheirownuniqueDIFsinchoosingresponsecategories.Thus,unlikealmostallexistingsurveyresearch,anchoringvignettesallowandulti-matelycorrectfortheDIFthatmayexistwhensurveyrespondentschooseamongtheresponsecategories,buttheyassumelikemostpreviousresearchtheabsenceofDIFintheÔÔstemquestion.ÕÕItseemsreasonabletofocusonresponse-categoryDIFasthemainsourceoftheproblembecausethevignettesdescribebehaviorsorpsychologicalstatesintendedtobemoreobjectiveandforwhichtraditionalsurveydesignadvicetoavoidDIF(suchaspretesting,cognitivedebrieÞng,andwritingquestionsconcretely)islikelytoworkbetter(althoughthemethodsdescribedheremayalsohelptoidentifyproblematicvignettes).Incontrast,responsecategoriesdescribemuchmoresubjectivefeelingsandattitudesattachedtowordsorphrasesthatarebytheirnatureimprecise;theresponsesshouldthereforebehardertolayoutinconcretewaysandavoidDIFwithoutanchoringvignettes.ThispointalsoseemsconsistentwiththeÞndingintheliteraturethattraditionalsurveydesignadvicecanworkespeciallybadlyforLikert-typeresponsescales(SchwarzOneissuewiththeDIFcorrectionintheexamplethusfaristhattheÞve-categorypoliticalefÞcacyresponseisreducedtoonlyathree-categoryDIF-correctedvariable,andsoinformationmayhavebeenlost.Asitturnsout,however,wecanrecoveradditionalinformationbyaddingmorevignettes.Forexample,Kingetal.(2004)alsousethis[Imelda]lackscleandrinkingwater.Sheandherneighborsaredrawingattentiontotheissuebycollectingsignaturesonapetition.Theyplantopresentthepetitiontoeachofthepoliticalpartiesbeforetheupcomingelection.withthesamesurveyquestionandresponsecategories.Ifsurveyrespondentsrankthetwovignettesinthesameorder,wecancreateaDIF-freevariablebyrecodingtheself-assessmentintoÞvecategories:(1)lessthanMoses,(2)equaltoMoses,(3)betweenMosesandImelda,(4)equaltoImelda,and(5)greaterthanImelda.Ofcourse,wecanobtainconsiderablymorediscriminatorypowerbyaddingmorevignettes.InthepoliticalefÞcacyexample,Kingetal.(2004)useÞvevignettespresentedtotherespondentinrandomorder.Morevignettesofcoursealsocomewithanadditionalassumption:thatalltherespond-entsunderstandthevignettesasfallingalongthesameunidimensionalscale,eveniftheydonotusethescaleandresponsecategoriesinthesameway.Inthevignettesabove,forexample,ImeldapresumablyhasmorepoliticalefÞcacythanMoses,butifmanyrespond-entsindicatedotherwise,thismightbeanindicationofmultipledimensionsbeingtappedbythevignettes.Evenifunidimensionalityholdsasassumed,usingmultiplevignettesgivesrisetootherpotentialchallenges.Forexample,randomerrorinperceptionsorresponsesmayproduceinconsistenciesinvignetterankings.Otherrespondentsmaynotperceivethedifferencebetweensomevignettesandmaygivethemtiedresponses.Wedealwiththeseissuesinthisstudy.GaryKingandJonathanWand whichoftheconditionsontherightsideofequation(1)aretrueandthensummarizewiththevectorofresponsesthatrangefromtheminimumtomaximumvaluesamongalltheconditionsthatholdtrue.Valuesofthatareintervals(orvectorvalued),ratherthanscalar,representthesetofinequalitiesoverwhichtheanalystcannotdistinguishwithoutfurtherassumption;wereferinformallytocasesthathaveintervalvaluesasbeingcen-soredobservations.Table1givesall13examplesthatcanresultfromtwovignetteresponsesandaself-assessment.Examples1Ð5havebothvignettescorrectlyorderedandnottied,withtheresultforbeingascalar.Thevignetteresponsesaretiedinexamples6Ð8,whichproducesacensoredvalueforonlyiftheself-assessmentisequaltothem.Examples9Ð13areforsurveyresponsesthatincorrectlyorderthevignettes.Withineachset,theexamplesareorderedbymovingfromlefttoright.ThisgeneralizeddeÞnitionforclariÞestheimpactofties(asinexamples6and8)andinconsistencies(asinexamples9and13)amongthevignettesthatoccurinagroupstrictlygreaterthanorlessthan.Notethatallfouroftheseexamplesinthetablehavescalar(uncensored)valuesfor.Thisisappropriatesincewemightreasonablyexpectrespond-entstobemorelikelytogivesometiedorinconsistentanswersamongvignettesthatarefarfromtheirownself-assessmentevenwhentheycorrectlyrankthevignettesthatmatterneartheirownvalue.Forexample,ifwearemeasuringheightandarespondentknewhisorherheighttowithinaninch,heorshestillmighthavedifÞcultycorrectlyrankingtheheightsoftwotrees200and206feettall,swayinginthebreeze.Yet,thesamerespondentwouldpresumablyhavenodifÞcultyunderstandingthatbothtreesaretallerthanhimselforherself.WethusregardtherespondentÕsmisorderingofvignettesinthiswaytonothaveanycensoringeffectontheestimate.AParametricSupplementThenonparametricestimatordiscussedinSection3.1recodesthevignettesandself-assessmentquestionsintoasingleDIF-freevariable.Sincethisunusualdependent Table1Allexampleswithtwovignettes:thistablegivescalculationsforthenonparametricforallpossibleexamples(sansnonresponse)withtwovignetteresponses,(intendedtobeorderedas),andaself-assessment,Surveyresponses12345Exampley100000100000100000100000110000010102,3,40000110000100101,2,3,4100011,2,3,4,5010012,3,4,500001GaryKingandJonathanWand foragivenvectorofvaluesoftheexplanatoryvariables,.Thisofcourseisexactlytheorderedprobitmodel.WenowgeneralizethismodelintheÞrstoftwowaysbyaddingnotationforvectorvaluesof,whichwedobyalteringtheobservationmechanisminequation(2)toThiscensoredorderedprobitmodelcanalsobeusedasameansofestimatingcausaleffectsunderthemaintainedassumptions.Onewouldmerelyestimatethemodelandexactlyasunderorderedprobit(suchasbyusingtheproceduresinKing,Tomz,andWittenberg2000).Researcherscouldestimatetheprobabilityofarespondentansweringinanyindividualcategorybyusingequation(3).Tocomputehistograms,andforoccasionalotherpurposes,weareabletoaddconsider-ablerobustnessbyintroducingasecondgeneralizationoftheclassicorderedprobitmodelthatconditionsthecalculationoftheprobabilityofbeinginaspeciÞc(single)categoryontheobservedvectororscalar(usingatechniqueanalogoustothatinKing[1997]andKingetal.[2004],AppendixB).Thiscalculationissimply wheretheusualunconditionalprobabilityPr()isdeÞnedinequation(3).Thenewexpressioninequation(5)conditionsonbynormalizingtheprobabilitytosumtoonewithinthesetandzerooutsidethatset.Forscalarvaluesof,thisexpressionsimplyreturnstheobservedcategory:Pr(1forcategoryand0otherwise.Forvector-valued,equation(5)putsaprobabilitydensityoverthecategorieswithinwhichintotalsumtoone.Avirtueofthismethodisthatpredictionsofscalar-valuedobservationsareÞxedattheirobservedvalues,independentofthemodel,andthusarealwayscorrectnomatterhowmisspeciÞedthemodel.Inaddition,predictionsofvector-valuedobservationsarere-strictedtowithintheirobservedrange,alsowithcertaintyandindependentofmodelingchoices.Themethodusesallavailableinformationintheself-assessment,vignettes,andexplanatoryvariablestoestimatethedistributionoffrequenciesforthevector-valuedobservationsratherthanmerelyassuminganarbitrarydistributionexante.Therobustnessofthisconditionalapproachisdirectlyrelatedtosizeoftherange.Analogoustotheinßuenceoftheproportionofmissingdatainmultipleimputation,asmallerrangeofvectorvaluesoffersrelativelygreaterconÞdenceintheresultinghistogramindependentofthemodelingassumptions,sincetheresultisboundedbytheknowninformationinthedata(HeitjanandRubin1990;Kingetal.2001).EmpiricalIllustrationKingetal.(2004)analyzepoliticalefÞcacydatainChinaandMexicoandshowthat,withananchoringvignettesDIFcorrection,therankingofthetwocountriesswitchescompared Iftworesponsecategoriesareobservedaspartofthevector-valuedforthesamesetofobservationsandfornoothers,someofthethresholdparametersarenotidentiÞed.ThenonidentiÞcationinthisunusualspecialcaseispartialinthesensethatthelikelihoodstillindicatesthemassinthesumofthetwocategories.TheproblemcanbeeasilyaddressedbycombiningthetwocategoriesorbyusinginformationtoconstructapriorfortheÕs.GaryKingandJonathanWand totherawsurveyresponses,withtheresultbeingmuchmoreinlinewithreality.However,theiranalysisdealswithtiesandinconsistenciesbyspreadingthemuniformlyovervector-valuedobservations,whichisnotaplausibleapproachingeneral.ThetopleftgraphinFig.1reproducestheÞgurefromtheirarticle,andthetendencytowardauniformhisto-gramisobvious,althoughofcoursewewillneedtogobeyondthenearlyuniformobservedhistogramstoshowevidenceofbias.ThebottomtwographsinFig.1presentreanalysesofthesesamedatausingtheconditionalcalculationsfromthecensoredorderedprobitmodel.Becauseofsomeveryslightlypopulatedcategories,wecombinedcategoriesintherightbottomgraphfromthesamemodel(andtherighttopgraphfromtheuniformmodelforcomparison).Thatis, 1234567891011 0.00.10.20.30.4 123,45,67,89,10111234567891011123,45,67,89,1011 0.00.10.20.5 Censored Ordered Probit 0.00.10.20.30.40.5 Censored Ordered Probit, Collapsed 0.00.10.20.30.40.5 Fig.1TwomethodsofanalyzingaDIF-freemeasureofpoliticalefÞcacy.Thehorizontalaxisoneachgraphrangesfromalowlevelofsayingovernmentonthelefttoahighlevelontheright.Thetoptwographsallocatevectorvaluesofuniformlyovertherange,whereasinthebottomtwo,theyareallocatedaccordingtothecensoredorderedprobitmodel.Fortherighttwographs,categorieshavebeencombinedforvisualclarity;thefractionineachofthecombinedcategoriesisdenotedbyaverticallinedividingtherelevantbaronthegraphinproportiontothesizesofthetwocategories.TheshadedbarsrepresentChinesedata,theclearbarsMexican. TheplottedproportionsaretheconditionalÞttedprobabilitiesfromacensoredprobitmodelwithcovariatesage,yearsofeducation,maledummy,andaChinadummy.Thecutpointsforthecensoredorderedprobitareassumedconstantforallobservations.ComparingIncomparableSurveyResponses amongrespondentsontheunderlyingcontinuousscalewhenvignettesarechosentodeÞnecategoriesofsuchthatrespondentsaresortedequallyacrossallthecategories: Ourimmediategoal,then,istodeÞneameasureoftheinformativenessordiscrimi-natorypowerofasetofvignettesforagivenself-assessmentquestion.Inotherwords,weneedtodeÞneacontinuousfunction)thattakesthesetoffrequencies,argumentsandreturnsarealnumberindicatingtheinformationinthefrequenciesandhencetheinformationinthevignettesthatdeÞne.Thisrealnumbershouldbeataminimum,whichwewilldenoteaszero,whenonecategorycontainsallrespondentsandatamaximumwhenrespondentsarespreadacrossthecategoriesequally.Toavoidhard-to-justifyapplication-speciÞcassumptionsabouttherelativeimportanceofeachofthecategoriesof,werequiretheargumentsoftobesymmetricinthatifwereorderedÕs,wouldreturnthesamenumber.Thesymmetryrequirementcouldeasilybedroppedifsuchinformationwereavailableforaparticularapplication.Ofcourse,manyfunctionssatisfythesesimplecriteria,butwecannarrowdownthesearchtoauniquechoicebyaddingtwomorerequirements,bothneededtocopewiththepossibilityofconsideringandcomparingthismeasurewithdifferentnumbersofvignettes.First,sincemorecategoriesof(whichresultfromusingmorevignettes)shouldneveryieldlessdiscriminatorypower,werequireinthecaseofnotiesorinconsistenciesthatbeamonotonicallyincreasingfunctionofthenumberofvignettesandhencethenum-berofcategories,21.Second,whenaddinganewvignetteanddecomposingacategoryintosmallerbins,theamountofexpectedinformationintheunionofthesesmallerbinsshouldremainthesameastheoriginalundecomposedbinandtheexpectedinfor-mationintheotherunaffectedbinsshouldremainunchangedbytheadditionofthenewForexample,supposewebeginwithasinglevignette(1),sothathas2categories,withproportionslabeled,and,respectively,andwewishtocompute).Thethreeproportionsrefer,respectively,tooutcomeswheretheself-assessmentislessthan,equalto,andgreaterthan.Nowconsideraddinganadditionalvignettewithavaluehigherthantheexistingone:.Thiseffectivelybreaksupthegroupofrespondentswithself-assessmentsthatfalltotherightof(previouslythethirdevent,withfrequency)intotheself-assessmentfallingbetween,equalto,andgreaterthan,andofcourse,thedecompositionislogicallyconsistent:.SincetheÞrsttwoevents,beinglessthanandequalto,havenotchangedwiththeintroductionof,theÞrsttwoproportions,,areunchanged,andsowerequireourmeasureoftheinformationthattheyconveytoalsobeunchanged.Thesecondcriterion,then,isthatweoughttobeabletocomputeidenticalvaluesoftheinformationinthevignettesbyapplyingthefunctiontothesetofallÞveprobabilitiesbycomputingaweightedaverageoftheinformationingroupedcategories)andtheinformationinthecomponentsofthethirdcategory,).Moreformally,thisamountstoaddingaconsistencyrequirementsothatinthisexamplewherethesecondtermquantiÞesseparatelyhowmuchinformationthesecondvignetteadds.Thus,weshallrequireageneralizationofthis,whichstatesthatwhenmultiplewaysofhierarchicallydecomposing)exist,theyshallallyieldidenticalvalues.ComparingIncomparableSurveyResponses thefrequencydistribution,fromthevector-valuedobservations,arepartiallyunknown,andsoweshallestimatethem.However,asweshallsee,estimationinthiscontextdoesnotinvolveanyaddeduncertaintyorassumptions.Instead,theÔÔestimationÕÕprocessinthiscontextinvolvescalculatingtheinformationwearecertainthevignettesandself-assessmentprovide.ToÞxideas,Table2showswhathappenswithÞveselectedorderingsofthreevignettes.Foreach,itlistsallpossiblevaluesof(dependingontheself-assessmentresponse).Thus,theÞrstrowisthecanonicalcasewiththevignettesuniquelyrankedinthecorrectorder.(Vignettevaluesareintendedtobeorderedbythenumberintheirsubscript,anditemsinbracesareeithertiedorinconsistent.)Thisproducesonlyscalarvaluesfor.Thesecondorderingcangeneratefourpossiblescalarvaluesofandonevector-valuedresponse,andsoon.Tocomputeafullfrequencydistribution,wefollowÞvesteps.First,sortallthescalarvaluesofintotheirappropriatebins.Second,parameterizefrequencydistributionsofresponsesforeachuniquecombinationofthevectorvalues.Forexample,forallthevignetteresponsesthatfollowthepatterninthesecondrowinthetableandwheretheself-assessmentleadstoavector-valuedset,set,z1,z2],wehave2,3,4.Inthissituation,weassignunknownfrequencies,andtothethreecategories,respec-tively.Thevaluesofthesefrequenciesareunknown,butweknowthattheysumtoone:1(meaningthattheprobabilityoftakingonthevalues1,5,6,or7is0),andsothenumberoffreeparametersforthisexampleisonlytwo.Wealsomakesimilarparameterizationsforthevector-valuedresponsesinthethirdandfourthrowsofTable2,yieldingeightfreeparametersfortheentireproblem.Third,weestimatethevalues,byaprocedurewedescribeshortly,andaddtheestimatedÕs(weightedbythenumberofrespondentsforwhichtakesonthesamevalue)intotheappropriatebinsandintowhichwehavealreadyputthescalarvalues.Atthispoint,wehaveanestimateofthefullfrequencydistribution,andsoasaÞnalstep,wecomputetheinformationinthevignettesbyapplyingtheentropyformulainequation(8).Theonlyremainingquestion,then,istoestimatetheunknownparameters.Wedosobyminimizingequation(8).Ifweonlyusetheinformationinthevignettesandself-assessmentresponses,thentheminimumentropyisexactlytheinformationinthatweexistsinourdata.Anyotherinformationthatmayexistwouldbeestimatedandhencewouldrequirepotentiallyincorrectmodelingorotherstatisticalassumptions.Asit Table2SelectedvignetteorderingsandpossiblevaluesofObservedrankingofvignettesforrespondentiPossiblevaluesofC(dependingony1,2,3,4,5,6,72,3,4,5,6,72,3,4,5,62,3,4,5,61,2,3,4,51,2,3,41,2,3,4,52,3,4,51,2,3,4,5,6,7Note.Bracesintherightcolumndenotevector-valuedresponsesfor AlthoughtheoptimizationprocedureproducesestimatesoftheÕs,theyareancillaryparametersandareofnoparticularinterestinandofthemselves.Becauseourcriteriaindicatethatweareindifferentamongallhisto-gramswiththesameentropy,theonlyrelevantquantityproducedbythisprocedureisthevalueoftheminimumentropy.Wewouldbeinterestedindifferencesbetweentwodensitieswiththesamelevelofentropyif,forexample,wehadpreferencesformeasuresthatprovidedmoreprecisionatacertainportionofthescaleorifweonlywishedtoidentifysomespeciÞcpercentileorfractionofrespondents.Inthesesituations,alternativeformalcriteriawouldprobablyleadtoameasureofweightedorrelativeentropy,suchasthatcomputedfromtheKullbackandLeibler(1951)distance,KLComparingIncomparableSurveyResponses Bothoftheseinequalitiesareinconsistentandyetclearlyequation(11a),wheretheself-assessmentandthetwovignettesaretied,seemsagooddeallessproblematicthanequation(11b),wherethevignettesareoutoforder.TheproblemisthatthereexistmanypossibleorderingsoftiesandinconsistenciesthatleadtothesamerangeforOurimmediategoal,then,isametricwithwhichwecanorderthetwoexamplesandanyothersthatarise.WedothisbyadaptingideasfromtheÞeldofstatisticalgenetics.ResearchersinthatÞeldoftenneedtocomparetwosequencesofDNA,whereeachiteminthesequenceiscomposedofoneoffourpossibleletters(orbasepairs).Sinceanaturalorderingbetweenanysequencesdoesnotexist,developingametricbasedonthesimilarityoforÔÔdistanceÕÕbetweentwosequencesisneeded.Researchersthusoftenusethedistance,whichisthenumberofswitchesinindividuallettersthatittakestoturnonesequenceintoanother(Linetal.2002).Byaroughlyanalogouslogic,wecomputethedistancebetweentheexpressionsinequations(11a)and(11b)bytheminimumnumberofsingle-unitmovesinvignettere-sponsesuntilisscalarvalued.Tosimplifythiscalculation,weÞrstdeÞneasthesetofvignettesandthatexcludesallvignettesstrictlygreaterthanandanothersetthatisstrictlylessthan.Then,weaddorsubtract1fromachosenvignetteresponseoneachround(sothatthenewÔÔresponseÕÕisstillbetween1and)andcomputetheminimumnumberofsuchstepsrequireduntilhasascalarvalue;welabelthisdistance.ThespeciÞcstepsinasequenceoflengtharenotnecessarilyunique,butthekeyisthatweareindifferentbetweenallsequencesthatgivethesamevalueofTable3computesforeachexpression,onestepatatime.Itshowsthatalthough2forboth,1forequation(11a)but2forequation(11b).With,wehaveameasureofhowfaranyindividualsurveyresponseisfromhavingascalar-valued.ItindicateshowmucherrorÑrandomornonrandomÑexistsinthedata.Otherthingsequal(suchasentropy),choosingvignettesthatreducetheaverageamongobservationsinthedatawouldseemtobeagoodidea.Inaddition,wesuggestseveralstrategiestousetoprovidehintsaboutthepossibleexistenceofnonrandomerror.First,wesuggestexaminingahistogramof,sincerandomoccurrencesoftiesorinconsistenciesthatimplicateshouldnormallyleadtoaunimodalhistogram,withtheÞrstmodenear0,butnonrandomviolationsmayyieldadditionalmodes.ThispointshouldbecheckedforÔÔedgeeffects,ÕÕsincedifferentpatternsofmayresultwhenisnear0or.Todothis,wemerelyexamineaseparatehistogramofforrespondentsgivingeachvalueofFinally,werecommendexploringstatisticallybytryingtopredictitwithavailableexplanatoryvariables.Itmaybethatwecanidentifyastratumofrespondentswhoconceptualizetheunderlyingscaledifferentlythanothers.Ifso,wemightbeableto Table3Countingswitchdistanceuntilisscalarvalued:twoexamplesStep12345GArrowsidentifyitemsmovedandthedirectiontheymoved,fromtheprevioussteponthelineabove.ComparingIncomparableSurveyResponses vignette4notworthwhile,unlessonewerewillingtomakestatisticalmodelingassump-tionsandthusfocusonestimatedentropyandtheverticaldimensionoftheÞgure.Ifwecouldaffordtoincludeonlyfourvignettes,wewouldnotuse2345,1345,or1245sincetheyaredominatedby1235,1234,orboth.Outofthe10combinationsofthreevignettes,123or125wouldappeartobelikelychoices.Whethertoaddvignettes3and4tothesubset125,then,dependsononeÕstrustintheorderedprobitmodelingassumptions.ThesecanbejudgedonlyinpartbycheckingtheÞtoftheorderedprobitmodeltotheuntiedobservations.Inaddition,oneshouldkeepinmindthattheamountgainedbygoingfromsubset125to12345(verticallyonthegraph)isonlyabouthalftheentropygainedbygoingfromonevignettetothree.Soifonecanaffordthreevignettes,125wouldappeartobeagoodchoiceaccordingtothesecriteria.Whethertoincludemoredependsonhowcomfortableoneiswiththeassumptions.Finally,wenotethattheapplicationwehavebeenreferringtohereconsidersonlythehistogramoftheentiresample.IfthegoaloftheresearchistocomputethedensityofpoliticalefÞcacyineachcountryseparatelyorofsomecausaleffect,oneshouldevaluatethesequantitiesdirectly,inafashiondirectlyanalogoustotheanalysisweperformedhere.HealthInordertoillustratedifferentfeaturesofentropyresults,wenowconductthesameanalysisfortwodifferentvariables.Botharecomponentsofhealthwithself-assessmentsandvignettesaskedaspartofthe2002WorldHealthSurvey(conductedbytheWorldHealthOrganization)inadifferentsurveyinChina.Webeginwithananalysisofvignettesaboutthequalityofsleepandrestfulnessduringtheday.Forexample,oneoftheÞvevignettesaskedis[Noemi]fallsasleepeasilyatnight,buttwonightsaweekshewakesupinthemiddleofthenightandcannotgobacktosleepfortherestofthenight.(theotherscanbefoundattheanchoringvignettesWebsite),withsurveyquestionInthelast30days,howmuchofaproblemdidyouhave[doesÔÔnameÕÕhave]duetonotfeelingrestedandrefreshedduringtheday?(1)None,(2)Mild,(3)Moderate,(4)Severe,(5)Extreme/CannotDo.Figure3analyzesthesedatawithanentropygraphinaformdirectlyparalleltothatinFig.2.Aparticularlyinterestingfeatureofthisgraphisthatitgivesseveralexamplesofasinglestrategicallyplacedvignetteprovidingmoreinformationthanasetofthreeapparentlyless-well-placedvignettes.Inparticular,eithervignette1or2aloneprovidesmorediscriminatorypowerthanvignettes3,4,and5dotakentogether.Indeed,thisistruewhetherwemeasurediscriminatorypowerbyeitherestimatedorknownentropy.Simi-larly,acombinationoftwovignettes(1and2)provideasmuchormoreinformationthantwosetswithfourvignettes(1345and2345)andmanywiththreevignettes.Finally,weofferananalysisoftheself-care,witharepresentativevignette:[Victor]usuallyrequiresnoassistancewithcleanliness,dressingandeating.Heoccasionallysuffersfrombackpainandwhenthishappensheneedshelpwithbathinganddressing. TheestimatedentropyforthesleepquestionsarecalculatedfromtheconditionalÞttedprobabilitiesofacensoredprobitmodelwithcovariatessex,age,weight,yearsofschooling,andmaritalstatus.Thesameholdsforself-careability,subsequently,exceptthatheightisexcludedfromthesetofcovariatessinceitÕsinclusionleadstoanunidentiÞedcutpoint.NocountrydummyisincludedbecauseÞguresareforChinaonly.GaryKingandJonathanWand andthesurveyquestion:Overallinthelast30days,howmuchdifÞcultydid[nameofperson/you]havewithself-care,suchaswashingordressing[yourself/himself/herself]?(1)None,(2)Mild,(3)Moderate,(4)SevereExtreme/CannotDo.Figure4providestheentropyanalysis,usingthesamecovariatesasthepreviousexample.Ascanbeseenbyeithermeasure,thesevignettesprovidelittlediscriminatorypower.Akeyreasonentropyissolowinthesedataisthatthevastmajorityofpeoplesurveyedfeeltheyhavelittletroublewithself-careandsothevignettesarefarfromtheirself-assessmentsandhencearerelativelyuninformative.However,thelowlevelsofentropyproducedbythesevignettesmightinsteadbeduetothepoorqualityofthevignettesthemselves.Forexample,thevignetteabovedescribesself-carewithrespecttoÔÔcleanliness,ÕÕÔÔdressing,ÕÕÔÔeating,ÕÕandÔÔbathing.ÕÕThesurveyquestionomitsallbutoneofthese(dressing)andaddsanothernotdirectlyreferencedinthevignette(washing).Inaddition,thevignettegivesareasonforthelevelofself-care(namely,backpain),whereasthereasonforanyÔÔdifÞcultyÕÕmentionedinthesurveyquestionisnotgiven.ItisevenpossiblethatsomerespondentsmayhaveinterpretedÔÔdifÞcultywithself-care,suchaswashingordressingÕÕasbeingunrelatedtothehealthcausesandinsteadreferringtopersonalhygieneandstyle.ConcludingRemarksThemethodsofferedhereareintendedtoevaluateandimprovetheinformationrevealedbysurveysusinganchoringvignettes.Thediversetypesofdataandsurveyquestionswe 0.20.40.60.81.01.21.4 0.20.40.60.81.01.21.4 Fig.3Estimatedbyknown(minimum)entropyforqualityofsleep.Theverticalaxisistheentropycalculatedfromthehistogramestimatedbyourcensoredorderedprobit.Thehorizontalaxisistheamountofentropyknowntoexistwithoutmakinganyassumptions.Thelistofvignettenumbersisplotted,withcombinationsofone,three,andÞvevignettesappearingingrayforclarity.ComparingIncomparableSurveyResponses havethusfaranalyzedseemtoindicatethatthesemethodscanrevealaconsiderableamountofinformationnototherwiseavailable,andsotheirusewouldappeartoberecommended.Researchersshouldalsokeepinmindthatanchoringvignettesweredesignedtocorrectforresponse-categoryDIF,notforallformsofDIF.Ifrespondentsunderstandthestemquestioninfundamentallydifferentways,thenanchoringvignettesandthemethodsde-scribedheremaynotÞxallinferentialproblems.Obviously,ifweinadvertentlyaskedhalfofourrespondentsthewrongsurveyquestion,wewouldgetnonsenseresults.Itshouldbenolessobviousthatifweaskasinglequestionofeveryonebuthalfoftherespondentsinterpretthequestioninamassivelydifferentway,methodologicalÞxesofalmostanykindwouldbeunlikelytoÞxtheproblem.Inourview,mostofthosewhoconductsurveysandusesurveydataintheirresearchneedtostartcorrectingfor,orevaluatingthepresenceof,DIFinresponsecategoriesusinganchoringvignettesandthemethodsdescribedherein.Withoutsuchanevaluation,manysurveyresultscouldbecalledintoquestion.Forsurveyresearchersandmethodologists,theremainingquestionishowalsotoaddressotheraspectsofinterpersonalincompara-bilityasidefromthatduetoresponse-categoryDIF.Golan,Amos,GeorgeJudge,andDougMiller.1996.Maximumentropyeconometrics:Robustestimationwithlimiteddata.London:JohnWileyandSons. 0.20.40.60.81.01.21.4 0.20.40.60.81.01.21.4 Fig.4Estimatedbyknown(minimum)entropyforself-careability.Theverticalaxisistheentropycalculatedfromthehistogramestimatedbyourcensoredorderedprobit.Thehorizontalaxisistheamountofentropyknowntoexistwithoutmakinganyassumptions.Thelistofvignettenumbersisplotted,withcombinationsofone,three,andÞvevignettesappearingingrayforclarity.GaryKingandJonathanWand