Information Systems Frontiers Kluwer Academic Publishers - PDF document

InformationSystemsFrontiers5:1,107Ð119,20032003KluwerAcademicPublishers.ManufacturedinTheNetherlands. CombinatorialInformationMarketDesignRobinHansonDepartmentofEconomics,GeorgeMasonUniversity,MSN1D3,CarowHall,FairfaxVA22030,USAE-mail:rhanson@gmu.edu(http://hanson.gmu.edu) Informationmarketsaremarketscreatedtoaggregate Tradersmustalsocoordinateonwhentheywillagreetotrade,sinceoffersthatwaitlongbeforebeingac-ceptedsufferadverseselectionfromnewpublicinfor-mation.Thustradersusuallyonlyexpectsubstantialtradingactivityinacertainlimitedsetofassets,andhavelittlereasontooffertotradeotherassets.Tradercoordinationcanbeaidedsomewhatbycallmarkets,wherealltradeshappensatpre-denedmoments,andbycombinatorialmatchingmarkets,whichsearchforcombinationsofoffersthatcanbematched.Butsuchaidcanonlygosofar.Considerthecasewhereasinglepersonknowssomethingaboutanevent,andeveryoneelseknowsthattheyknownothingaboutthatevent.Inthiscase,standardinformationmarketsbasedonthateventsimplycannotacquirethispersonsinformation.Standardinformationmarketsalsosufferfromanirrationalparticipationproblem.Oncerationalagentshavehedgedtheirrisksregardingtheeventscoveredbyaninformationmarket,theyshouldnotwanttotradewitheachother.Eveniftheyhavesubstantialprivateinformationnotreectedinpreviousmarketprices,agentswhocareonlyaboutwhattheycanbuywiththeassetstheymightgainmustrealizethatthegainsofsomecanonlycanfromthelossesofothers(MilgromandStokey,1982;Aumann,1976).Nowrealmarketsdoshowsurprisinglyhighlevelsofspeculativeactiv-ity,perhapsduetowidespreadirrationality,ortopeopleusingbetstosignaltheirexpertisetoobservers(asinbarbets).Butitmaynotbepossibletoinducemuchmoresuchactivity,inordertosupportnewinformationmarketswithouttakingawayfromothermarkets.Longbeforetherewereinformationmarkets,however,therewerescoringrules,designedtoob-taininformationbyelicitingprobabilitydistributionsfromindividuals.Scoringrulesdonotsufferfromir-rationalparticipationorthinmarketproblems;theyhavenotroubleinducingonepersontorevealinfor-mation.Theyinsteadsufferfromathickmarketprob-lem,namelyhowtoproduceasingleconsensuses-timatewhendifferentpeoplegivedifferingestimates.Thispaperconsidersanewtechnology,marketscoring,whichcombinestheadvantagesofinformationmarketsandscoringrules.Whendealingwithasingleperson,amarketscoringrulebecomesasimplescor-ingrule,yetwhendealingwithagroupofpeople,itbecomesanautomatedmarketmakersupportingtradesbetweenthosepeople.Thusamarketscoringrulebasi-callysolvesboththethinmarketandtheirrationalpar-ticipationproblemswithinformationmarkets,aswellasthethickmarketproblemofscoringrules.Givensomebasesetofafter-the-fact-veriablevari-ables,givenpatronswillingtopayamodestper-variablecosttoinduceinformationrevelationonthosevariables,andsettingasidecomputationallimitations,marketscoringrulesallowpeopletorevealwhattheyknowbytradingoneventdenedintermsofcom-binationsofthosebasevariablevalues.So,forexam-ple,givenbinaryvariables,peoplecouldtradeonanyofthe2possiblestatesor2possibleevents.Amarketscoringrulealwayshasacompleteconsensusprobabilitydistributionovertheentirestatespace,adistributionthatanyonecanchangeanypartofatanytime.Andrisk-neutralagentsshouldexpecttoprofromsuchchangeswhenevertheirownbeliefsdifferinanywayfromthisconsensusdistribution.Ofcoursethereareinfactlimitsonwhatwecancompute.Soafterreviewingexistinginformationag-gregationtechnologiesandthenewtechnologyofmar-ketscoringrules,thispaperwillgoonconsiderseveralimplementationissuesraisedbythisnewtechnology.PreviousTechnologiesofInformationAggregationThegeneraltaskweconsiderinthispaperistoin-ducepeopletoacquireandrevealinformationrele-vanttoestimatingcertainrandomvariables.Thesevari-ablesmayincludenaturaloutcomeslikeearthquakesortheweather,economicstatisticslikeGDP,andpo-liticaloutcomeslikeelections,warsandrevolutions.Wemayalsowantconditionalestimates,suchasthechanceofwargiventhatGDPfalls,andespeciallydecision-contingentestimatessuchasofthechanceofwargiventhatweelectaparticularpersonforpresi-dent,thestockpriceofacompanygiventhatwedumpthecurrentCEO,orapatientslifespangiventhatsheadoptsaparticularcourseoftreatment.Suchdecision-contingentestimatescangiveusrelativelydirectadviceaboutwhatchoicestomake(Hanson,1999).Wewillnotassumethatweknowwhichpeoplehavewhichareasofexpertize,butwewillassumethatwecansomedayverifythevariablevalueswithlittlecon-troversyafterthefact.Ideallywewouldlikeasinglecompleteconsistentprobabilitydistributionovertheentirestatespaceofallvariablevaluecombinations,embodyingeverythingthateveryoneknows,aswellaseverythingtheycouldlearnwithamodesteffort.Inpractice,wemayhavetoacceptcompromises. CombinatorialInformationMarketDesignScoringRulesTodate,therehavebeentwomainapproachestotheinformationaggregationtaskwehaveoutlined,scor-ingrulesandinformationmarkets.Inordertoelicitarisk-neutralpersonsprobabilitydistributionoverallstates1),onecanaskhimtoreportadistribution,andthenifthetruestateturnsouttobegivehimarewardaccordingtoaproperscoringrulec)thatsatisesanincentivecompatibilityconstraintargmaxandarationalparticipationconstraint(Therewardfornotparticipatingissetheretozero.)Giventheseassumptionsandconstraints,thepersonwillwanttoset.Scoringrulesalsogiveagentsincentivestoacquireinformationtheywouldnototh-erwisepossess(Clemen,2002).TheseincentivescomeattheexpenseofapatronwhoagreestopaytherewardIn1950Brierintroducedthequadraticscoringrule(Brier,1950)andin1952Goodfollowedwiththelogarithmicscor-ingrule(Good,1952)Thelogarithmicruleistheonlyrulethatcanbeusedbothtorewardanagentandtoevaluatehisper-formanceaccordingtostandardstatisticallikelihoodmethods(Winkler,1969).Scoringruleshavelongbeenusedinweatherforecasting(MurphyandWinkler,1984),economicforecasting(OCarroll,1977),stu-denttestscoring,economicsexperiments,riskanalysis(DeWispelare,Herren,andClemen,1995),andtheen-gineeringofintelligentcomputersystems(DruzdzelandvanderGaag,1995).Whilescoringrulesinprinciplesolveincentiveproblemswithelicitingprobabilities,manyotherprob-lemshaveappearedinpractice.Forexample,peopleareoftenhesitanttostateprobabilitynumbers,soproba-bilitywheelsandstandardwordmenusareoftenused.Also,correctionsareoftenmadeforhumancognitivebiases.Intheory,risk-aversion,uncommonpriors,andstate-dependentutilitycanalsomakeithardtoinfertheinformationpeoplehavefromtheestimatestheymake(KadaneandWinkler,1988),thoughinpracticethesearenotusuallyconsideredtobebigproblems.Whentheyareproblems,intheoryriskaversioncanbedealtwithbypayinginlotterytickets(Smith,1965;Savage,1971),anduncommonpriorsandstate-dependentutil-itycanbedealtwithbyhavingpeoplerstplayacertainlotteryinsurancegame(Hanson,2002a).Anotherproblemisthat,ingeneral,thenumberofstatesisexponentialinthenumberofvariables,makingithardtoelicitandcomputewithprobabilitydistribu-tionswithmanyvariables.Thisproblemisoftendealtwithbycreatingasparsedependencenetwork,suchasaBayesnet(Pearl,1988;Jensen,2001;PennockandWellman,2000),relatingthevariables.Adependencenetworkamongthevariablessaysthateachvariableis,givenitsneighborvariables,conditionallyindepen-dentofallothervariables.Thusbyelicitingasparsenetworkofdependencies,onecandrasticallyreducecomputationalcomplexity(moreonthisbelow).Whilelittleattentionisusuallypaidtoincentivecompatibil-itywhenelicitingsuchnetworkstructures,theyoftenseemuncontroversial.Onebigproblemwithscoringrulesremainslargelyunsolved,however.Whendifferentpeopleareaskedtoestimatethesamerandomvariable,theycanandoftendogivedifferentestimates.Yetwhatwereallywantisasingleestimatethataggregatesinformationfromdif-ferentpeople.Unfortunately,theliteratureoni.e.,constructingasinglepooledprobabil-itydistributionfromasetofindividualdistributions,ismostlydiscouraging(GenestandZidek,1986).Forexample,itturnsoutthatanytwoofthefollowingthreeapparentlyreasonableconditionsimplyadictator,i.e.,thatthepooleddistributionisequaltooneoftheindi-vidualdistributions:1.Iftwoeventsareindependentineachindividualdistribution,theyareindependentinthecommondistribution.2.Thepoolingprocedurecommuteswithupdatingthedistributionswithnewinformation.3.Thepoolingprocedurecommuteswithcoarseningthestatespace(e.g.,droppingavariable).Sincepoolingopinionswellishard,theusualpracticeinbuildinglargeprobabilitydistributionsistochooseasingleexperttospecifyparametersforeachvari-able.Forexample,asinglepersonmightbeassignedtoestimatetheweatherinagivengeographicarea. InformationMarketsInformationmarketscanovercomemanyofthelimita-tionsofscoringrulesasinformationaggregationmech-anisms.Likescoringrules,informationmarketsgivepeopleincentivestobehonest,becauseeachcontribu-tormustputhismoneywherehismouthsis.Insuchmarkets,peopletypicallytradeassetsoftheformPays$1ifeventandthemarketpriceforsuchassetsisinterpretedasaconsensusestimateofTheapproachesdescribedabovefordealingwithnum-bershyness,risk-aversion,andstate-dependentutilitycanalsobeusedwithinformationmarkets.Incontrasttoscoringrules,however,informationmarketscancombinepotentiallydiverseopinionsintoasingleconsistentprobabilitydistribution.Theinfor-mationpoolingproblemwithscoringrulesarisesbe-causeagivenprobabilitydistributionoversomelimitedsetofvariablescanarisefrommanydifferentinfor-mationsets,andsoonecannotdetermineanagentinformationfromhisprobabilityestimates.Rationalagentswhoarerepeatedlymadeawareofeachotherestimates,however,shouldconvergetoidenticales-timates,atleastgivencommonpriors(GeanakoplosandPolemarchakis,1982;Hanson,1998,2003).Infor-mationmarketscanusesuchrepeatedinteractiontoproducecommonestimatesthatcombineavailablein-formation,avoidingtheopinionpoolingproblem.Noknowledgeofwhoismoreexpertonwhattopicsisre-quired,andcorrectionsforcognitivebiascanevenbelefttomarketparticipants,whocanprotbymakingsuchcorrections.Markettradersself-selecttofocusonthetopicswheretheybelievetheyaremostexpert,andthosewhoaremistakenabouttheirareasofexpertizearepunishednancially.Aswasdiscussedintheintroduction,however,stan-dardinformationmarketssufferfrombothirrationalparticipationandthinmarketproblems.Oncerationalagentswhocareonlyaboutwhattheycanbuywiththeassetstheymightwinhavehedgedtheirrisks,theyshouldnotwanttomakespeculativetradeswitheachother,evenwhentheyknowthingsthatothersdonot(MilgromandStokey,1982;Aumann,1976).Andbe-causetradersmustcoordinateontheassetstotradeandonwhentoagreetotrade,tradersusuallyonlyexpectsubstantialtradingactivityinacertainlimitedsetofassets,andhavelittlereasontooffertotradeotheras-sets.Forexample,whenonepersonknowssomethingaboutacertainvariable,andeveryoneelseknowsthattheyknowlittleaboutthatvariable,standardinforma-tionmarketstradingassetsbasedonthatvariablewillnotrevealwhatthispersonknows.MarketScoringRulesFig.1illustratessomeperformanceissueswithscor-ingrulesandinformationmarkets.Wheninformationmarketsarethick,theiraccuracyshouldincreasewiththenumberoftraderswhofocusoneachasset.Asthenumberoftradersperassetfallsnearone,however,thethinmarketproblemshouldeliminatetradingac-tivity.Scoringrules,incontrast,canproduceestimatesinsuchcases,thoughoneexpectsestimateaccuracytofallwiththenumberofestimatesoneasksofanyoneperson.Whenusingscoringrulesinthethickmarketcase,however,wheremanypeopleareaskedtoestimatethesameparameter,opinionpoolingproblemsshouldmakethemlessaccuratethaninformationmarkets.Marketscoringrulescancombinetheadvantagesofscoringrulesandstandardinformationmarkets.Theyshouldthusproduceanaccuracylikethatofinforma-tionmarketswhenmanypeoplemakethesamekindofestimates,andlikethatofscoringruleswhenonlyonepersonmakeseachestimate.Marketscoringrulesare,inessence,sequentiallysharedscoringrules.Any-onecanatanytimechoosetousesucharule,i.e.,bepaidaccordingtothatrule,iftheyagreetopayoffthelastpersonwhousedtherule.Soifonlyonepersonusesamarketscoringrule,itineffectbecomesasim-plescoringrule.Butifmanypeopleusethatmarket .001.01.1110Estimates per trader Market Scoring Rules Simple Info Markets thin market Fig.1.Comparingmechanisms. CombinatorialInformationMarketDesignscoringrule,itineffectbecomesanautomatedmarketmakerfacilitatingtradesbetweenthosepeople.Andsinceeachuserpaysofftheprevioususer,themar-ketscoringrulespatronneedonlypayoffthelastuser.Moreformally,amarketscoringrulealwayshasacurrentprobabilitydistribution,whichisequaltothelastreportthatanyonehasmadetothisrule.Anyonecanatanytimeinspectthisdistribution,orchangeanypartofitbymakinganewreport.Ifsomeonechoosestogiveareportattime,andthentheactualstateeventuallyturnsouttobe,thispersonwillbepaidareward)issomeproperscoringrule,andisthelastreportmade.Sincethisamountcanbenegative,thepersonmustshowanabilitytopaythisifneeded,suchasbydepositingorescrowinganamountSincethepersongivingreportcannotchangethepreviousreport,hemaximizestheexpectedvalue)bymaximizingtheexpectedvalueof),andsowantstohonestlyreporthisbeliefsherewheneverhewouldforasimplescoringrule.eversomeonesbeliefsdifferfromthecurrentdistribu-,heexpectstoprotonaveragefrommakingareport,forthesamereasonthatsomeonewhohadmis-takenlymadethewrongreportwouldwanttocorrectsuchamistakebeforeitbecameofWhenapersonchangesamarketscoringruledistri-bution,hemustraisesomeprobabilityvalueslowerothers.Ifthetruestateturnsouttobeonewhereheraisedtheprobability,hewillgain,butifthetruestateturnsouttobeonewhereheloweredtheprobabil-ity,hewilllose.Thispersonisthusinessencemakingabetwiththemarketscoringruleregardingthetruestate.Sinceonecouldchoosetochangethedistribu-byonlyatinyamount,andonecouldundosuchabetbyreversingthechange,thesetinybetsareallfairbetsattheprobabilitiesSincepeoplearefreetochangeanycombinationofprobabilitiesin,andsincebetsarewhatmakethoseprobabilitieschange,amarketscoringruleisinessenceanautomatedinventory-basedmarketmakerwhostandsreadytomaketinyfairbetsatitscurrentThismarketmakeralsostandsreadytomakeanylargebetsthatcanbeconstructedfromtinybets.Theonlycatchisthatthepriceschangeastinybetsaremade.Wecansummarizeallthisbysayingthateachmar-ketscoringruleinessencehasanetsalessofarvector,whereeachsayshowmanyunitshavebeensoldofassetsoftheformPays$1ifthestateiscurrentunitpriceforatinyamountofsuchanassetis,andthesepriceschangeaccordingtoapricefunc-),whichisinessenceageneralizedinverseofthescoringrulefunction).Forexample,forthelog-arithmicscoringrule),thepricefunctionistheexponential ndthepriceforanynon-tinybundleofassets,onemustintegratethepriceasitchangesacrosssalesoftinybundles.Ifthesaleshistoryis),andthisnewsalestartsat andendsat,thenthetotalamountofthissaleis ),andthetotalpriceforthissaleis )isafunction,thisintegralisindependentofthesalespathbetween )andInmanyexistingmarkets,alltradesaremadewithoneorafewcentralmarketmakers.Theseactorsal-wayshavepublicofferstobuyortosell,andupdatethesepricesinresponsetotrades.Humanmarketmak-ershavebeenfoundtodoaswellasthestandarddou-bleauctionmarketforminaggregatinginformation(KrahnenandWeber,1999).Somemarkets,suchastheHollywoodStockExchange(www.hsx.com),useautomatedmarketmakerstollthisrole.Typically,eachassethasasingleautomatedmarketmakerwhichdealsonlyinthatasset.Amarketscoringruleislikethis,exceptthatitisasingleautomatedmarketmakerthatdealsinalloftheassetsassociatedwithanentirestatespace.So,forexample,givenbinaryvariables,asinglemarketscoringrulecanmaketradesonanyofthe2possiblestates,oranyofthe2possibleevents(i.e.,setsofstates).Aswithordinarymarketmakers,abstractionscanbedevisedtoallowuserstomoreeasilyfollowvariousstandardtradingscenarios.Forexample,alimitordercanspecifyanamountandaboundaryinspace,andmeanthatoneiswillingtospenduptoagivenamounttokeepfromcrossingthisboundary. Sincethepatronofamarketscoringrulemustonlypaythelastuser,histotalexpectedlossdependsonlyonhowinformativethatlastreportis,relativetotheinitialreporttherulewasstartedwith.Thislossisotherwiseindependentofthenumberofusers.Tominimizehisexpectedloss,thepatronshouldsettheinitialreporttohisinitialbeliefs,inwhichcasehisworstcaseexpectedlossiswhere(1whenandzerootherwise.Thisworstcaseappliesifineachstate,thelastreportissurethatthisisthetruestate.Forthelogarithmicscoringrule,thisworstcaseexpectedlossis(theentropyofthedistributionHowmuchmoredoesitcosttohaveasinglemarketmakerwhichcoverstheentirestatespace,relativetojusthavingamarketmakerforeachvariable?AmarketscoringrulethatcoveredonlyasinglevariablewithpossiblevalueswouldbeanautomatedmarketmakerforallassetsoftheformPays$1ifthisvariablehasvalueForalogarithmicrule,theworstcaselosswouldbe(times)theentropyofthemarginaldistributionoverthevalues.Ingeneralthesumoftheentropiesofthemarginaldistributionsofeachvariableisnoless,andusuallymore,thantheentropyofthejointdistributionoverallvariables.Thus(settingasidecomputationalcosts)itcostsnomoretofundanautomatedmarketmakertotradeintheentirestatespacethanitcoststofundautomatedmarketmakerslimitedtoeachvariable.Logarithmicmarketscoringrulesalsohavemodu-larityadvantages.Itseemsgoodtominimizetheex-tenttowhichpeoplemakingbetsonsomedimen-sionsoftheprobabilitydistributionregularlycauseunintendedchangestootherunrelateddimensionsofthedistribution.Forexample,whenonebelievesthat,oneexpectstogainbygivingupPays$1ifBintradeforoneunitofPays$1ifAandB.Andbymakingthistradeonetakesnoriskregardingwhethertheeventistrue,sinceonetradessomeassetswhichassumeforotherassetswhichalsoassume.Whileitseemsappropriateforsuchtradestochangethevalueof)inthedistribution,itdoesnotseemappropriateforthemtochangethevalueof).Requiringthiscondi-tiontoholdforallsuchtrades,however,isequiva-lenttorequiringthelogarithmicmarketscoringrule(Hanson,2002b).Givensuchaconditionaltrade,thelogarithmicrulepreservesnotonly),butforanyeventitalsopreservestheconditionalprobabilities),andandnotandthevariableindependencerelations),and),foravalueavalueof,andavalueof.(Forvari-,wesay)holdsfor)forallvalues,andRepresentingVariablesMarketscoringrulesweredescribedaboveintermsofprobabilitydistributionsoverstatesnedintermsofcombinationsofvariablevalues.Buthowshouldwedenethosevariablevalues?Imaginethatwehavetworeal-valuedrandomvari-,suchasGDPchangeandimmigration.OneusermightwanttoexpresshisopiniononhowmuchheexpectsGDPtochange,whileanotherusermightwanttoexpresshisopinionthatimmigrationwillbehighifGDPrisessomewhat.Ifwewanttoletboththeseusersexpresstheiropinions,butwealsowanttodescribeGDPintermsofasmallnumberofassets,tolimitcomputationalcomplexity,whatshouldthoseassetsbe?AuserwhowantstouseGDPasacondition,asinifGDPrisessomewhat,mightpreferdigitaloptionassetsdenedintermsofwhetherisinagivenregion,suchasPays$1ifPays$1ifPays$1ifAuserwhowantstoexpresshisopinionontheexpectedvalueofGDPchange,how-ever,mightpreferassetssuchasPays$Pays$(1wherewehavedenedarescaledvariable øxx whichiszeroupto ,isoneabove,andmoveslinearlywithinbetween.Whilescoringrulepayoffsweredenedaboveonlywhenagivenstateisfoundtobetruewithcertainty,wecaneasilyreinterpretthatframeworktoincludesuchlinearassets.Scoringrulescaninsteadpayoffaccord-ingtoanynalassignmentInessence,eachassetoftheformPays$1ifthepays$.Inthiscase,peopleshouldwant CombinatorialInformationMarketDesigntoreporttheirexpectationsfor,asinnai].Whenindividualvariablesareexpressedlinearly,suchaswithvalueassignmentsand1,valueassign-mentsforvariablecombinationscanbeproducts,as,(1),and(1).Thatis,whenthereal-valuedrandomvariablehasvalueletacorrespondingdiscretevariablevaluehaveas-),sothat1forall.Wecanthenlettheassignmentofstate)isthevalueofdiscretevariableinstate,andGivenlinearvariablevalueassignments,ausercouldinasensebetthatimmigrationwillbehighgiventhatGDPchangeishigh,i.e.,thatwillbehigh,giventhatishigh,bygivingupunitsofPaysforoneunitofPays$wheneverhebelieveddöxöy]E[öx].Formostpeople,however,thisseemsanunnaturalwaytodescribetheirconditionalexpectations.TohaveasingleassetrepresentationthatcanletpeopleexpressopinionsontheexpectedchangeinGDP,letpeopleexpressopinionswhichuseGDPrisessomewhatasacondition,wemightusesawtoothassets,suchasillustratedinFig.2.Eachofasetof1pointswouldhaveamatchingassetwithanalassignment,sothatnomatterwhathappens.When,thenandtheotherarezero,andifandtheothersarezero.Ifforanythenthat1andtheothersarezero.Otherwisetheassign-mentvarieslinearlybetweenthesevalues.Thatis,foranythereissome,andwe,andallotherstozero. 10 x0 x1 x2 x3 Asset AssignmentReal-valued Random VariableFig.2.Sawtoothassetrepresentation.Asawtoothassetonlyhasvaluewhentherealvari-aparticularvalue.Thussuchassetscanbeusedasacondition,asinifGDPrisessome-YetlinearcombinationsofsawtoothassetscanalsoreproduceassetslikePays$andtheycandothisfor equaltoanypairofvalues.ThusiftheGDPchangeandimmigrationvariablesarebothrepresentedbysawtoothassets,userscannaturallybetonboththeexpectedvalueofGDPchange,andontheexpectedvalueofimmigration,giventhatGDPchangeissomewhathigh.Notethatthestatespaceforavariablecanbereontheywhenrealvariablesarerepresentedeitherbysawtoothassetsordigitaloptions.Bothrepresentationsrelyonasetofcutpoints,andnewcutpointscanbeinsertedatanytime,asitisstraightforwardtotranslateassetsdenedintermsoftheoldcutpointsintoassetsnedintermsofthenewcutpoints.However,sinceaddingcutpointsmakesitpossibleforthelastreporttobemoreinformativeabouttheactualstateoftheworld,addingcutpointscanincreasetheexpectedcosttothepatrontopayforamarketscoringrule.Newvariablescanalsobeeasilyaddedonthey,ifapatroniswillingtopayfortheextracost.ComputationalIssuesWhenthereareonlyadozenorsovariables,eachwithahandfulofvalues,onecanexactlycomputeamar-ketscoringrule.Acentraldatastructurecanstorethecurrentprobabilityexplicitlyforeverystate,andprobabilitiesofeventscanbedirectlycalculatedviasums,asin.Asimilardatastructureperusercandescribehisassets,bygivinghispayoffineachstate.Thatis,foreachuseronestoresanum-foreachstate,whichdescribesanassetofthePays$cifstateIfauseracquiresanassetPays$cifeventAhappens,thisassetcanbestoredbyincreasingbytheamountforeachstateThisrepresentationofassetsallowsuserstomaximallyreuseassetstheyacquiredinpreviousbetstosupportfuturebets.Whenvalueofvariablehasprobability,andthenisveriedtohaveassignment,theentiredistri-butioncanbeupdatedtoforeachstate.Whenallthevaluesofthisvari-ableareassigned,thestatespacecanbecoarsenedtoeliminatethisvariable,bymergingallstateswhichdif-feronlyintheirvalueofthisvariable.Inthismerging process,theprobabilityofthenewstateisthesumoftheprobabilitiesoftheoldstates,andtheassetshold-ingsforeachpersoninthenewstateisanaverageoftheirassetholdingsintheoldstates,weightedbytheConsideralogarithmicmarketscoringrulebased),withaninteger.ThepatronofthisrulecanavoidroundofferrorlossesbystoringtheeachuserholdsofassetPays$1ifstateasexponentials.Sowhenausermakesathatisamultipleof,wecansetallhisinitialholdingstotheinteger.Ifprobabil-arealwaysrationalnumbers,thenwheneverauserstartsbyholding,andthenchangesaprobability,wecanexactlycomputehisnewhold-ingtobetherational).Andifweneverallowausertomakeachangethatwouldproduceany1(inastatewhere0),thenuserscannevergobankrupt.Wecanalsoallowausertowithdrawcashbyspecifyingathatisamultipleof.Wejustset,andgivehimcash(Asampleimplementation,inCommonLisp,isavail-ableathttp://hanson.gmu.edu/mktscore-prototype.Beyondadozenorsovariables,eachwithahandfulofvalues,theaboveapproachquicklybecomesinfea-sible.Afterall,whenvariableseachhavevalues,therearepossiblestates,and2events.Whenismorethanabillionorso(e.g.,thirtybinaryvariables),theaboveapproachbecomesinfeasibleontodayscomputers.Inthiscasesomeotherapproachesmustbeusedtomaintainaconsistentsetofcurrentmarketprices,tomaintainadescriptionoftheassetseachpersonholds,andtodeterminewhenausersassetscanallowthemtobackupagivennewbet.LimitingtheProbabilityDistributionOneapproachtodealingwithenormousstatespacesistochooseaparticularfamilyofprobabilitydistribu-tions,andlimituserstochoosingdistributionswithinthatfamily.Betsmustchangetheprobabilitydistribu-tionfromonememberofthisfamilytoanother,anduserassetschangesmustbeconsistentwiththatcon-straint.Forexample,thefamilyofBayeslineardistri-butionshassomeattractivecomputationaladvantages(Goldstein,1987).WewillhereelaborateonBayesnets(Pearl,1988;Jensen,2001),however,becauseoftheirpopularity.InaBayesnet,variablesareorganizedbyadirectedgraphinwhicheachvariablehasasetofvariables.Theprobabilityofanystatecanthenbewrittenas)isthevalueofvariableinstate.Ifeachvariablepossiblevalues,thisdistributioncanthenbestoredintables,withonetableofsizeforeachvariable.Thuswhenthenetworkissparse,meaningthateachvariablehasonlyafewparents,thereareonlyafewparameterstospecifyandstorepervariable,itisstraightforwardtocomputetheprobabilityofanygivenstate,anditiseasytocalculateorchangetheprobabilityofavariablevalue,conditionalonthevaluesoftheparentsofthatvariable.AmarketscoringrulewhichrepresenteditsprobabilitiesintermsofaBayesnetcouldthuseasilysupportbetswhichchangetheseparticularconditionalButwhataboutsupportingbetsonotherconditionalprobabilities,orontheunconditionalprobabilitiesofvariablevalues?Inordertosupportsimplebetsonvari-ablevalues,onewantstobeabletocalculatetheun-conditionalprobabilitydistributionoverthevaluesofavariablegiventhecurrentnetwork,andonewantstoknowhowtochangethatnetworktobeconsistentwithanewdesireddistribution.ItturnsoutthatallthisandmoreispossibleiftheBayesnethappenstobesingly-connected,sothatthereisatmostonepathcon-nectinganytwovariables.Inthiscase,onecanchangetheunconditionaldistributionofanyvariable,andthenpropagatethosechangesacrossthenetwork,toexactlyupdatetheunconditionaldistributionsofothervari-ables.Conditionaldistributionsofonevariablegivenanothercanbeexactlycalculated,suchasbyprovision-allychangingonevariableandpropagatingthatchangethroughtotheothervariable.Also,sinceonecanmergeasetofvariables,eachwithvalues,intoasinglevariablewithvalues,onecanalsoapplythisapproachtoBayesnetsthatarebutnotexactlyThusamarketscoringrulewhichrepresenteditsprobabilitiesintermsofanearlysingly-connectedBayesnetcouldsupportbetswhichchangeanyunconditionalvariableprobabilities,andanycondi-tionalprobabilitiesbetweenpairsofvariables.Such CombinatorialInformationMarketDesignamarketscoringrulecouldevensupportbetswhichchangethenetworkstructure,aslongasthosechangeskeepthenetworknearlysingly-connected.Unfortu-nately,whiletheBayesnetsthatseemplausibletopeopleareusuallysparse,withonlyafewconnec-tionspervariable,theyareusuallynearlysingly-connected.SoifaBayes-net-basedmarketscoringrulealloweduserstocreateanysparseBayesnettheywished,itwouldtypicallybeinfeasibletoexactlycalculatemostunconditionalvariableprobabilities,andmostconditionalprobabilitiesrelatingvariableThereareofcoursemanywaystocomputeapproxi-mationstotheseprobabilities.Soonemightbetemptedtoendowamarketscoringrulewithsuchanapprox-imationalgorithm,andthenletitacceptfairbetsattheapproximateprobabilitiesdeterminedbythatalgo-rithm.Thisapproach,however,mightturnthemarketscoringrulepatronintoamoneypump.Thisisbecausetheaccuracyoftheseapproximationsvariesincom-plexwayswiththecontext.Ifanyusercould,foranypartoftheprobabilitydistribution,ndanysystematicpatterninwhentheofcialapproximationmadeover-estimatesvs.under-estimates,hecouldinprincipleusethatpatterntomakemoneyviaarbitrage.Toarbitrage,hewouldmakesimplebetsonewayattheapproximateprobabilities,andmakesomecombinationofotherbetsatthemorebasicprobabilities.Ofcourseitmaybehardndsuchpatternsandbetcombinationsthatexploitthem,andtheamountsgainedmaybesmallcomparedtotheeffortrequired.Also,amarketscoringrulewithasmartmanagermightlimitlossesbydetectingsuchactivity.Potentialmarketscoringrulepatronsmightndtheseconsiderationssufcientlyreassuring,however.Inordertoguaranteethatamarketscoringrulepa-tronlosesnomorethanagivenamountofmoneywhilesupportinganindenitenumberoftrades,onemightadoptapolicyofonlyallowingbetsonprobabilitiesthatonecanexactlycompute.Thispolicy,however,couldpreventmostofthetradesthatuserswouldbemostinterestedin.OverlappingMarketMakersAnotherapproachtodealingwithenormouspotentialstatespacesistohaveseveralmarketscoringrules,eachofwhichdealtexactlywithsomepartofthesametotalstatespace.Forexample,amarketscoringrulethatrepresenteditsprobabilitiesviaageneralsparseBayesnetmightbecombinedwithmarketscoringrulesthateachdealtonlyintheunconditionalprobabilitiesofasinglevariable.TheniftheunconditionalprobabilitiesofavariablewerenotconsistentwiththeBayesnetprobabilities,userswhodiscoveredthisinconsistencycouldprotthereby,buttheirprotwouldbebounded.Afterall,ifeachmarketscoringruleisexactlycom-puted,wecanexactlyboundthelossofeachrule,andhenceboundthetotalloss.Perhapsthesimplestapproachofthissortwouldconsistofasetofmarketscoringrulesthateachusedacompletelygeneralprobabilitydistributionoversomesubsetofthevariables.Userscouldthendi-rectlymakeanybetsandchangeanyprobabilityesti-mates,aslongasthesetofvariablesusedinthatbetorprobabilityestimatewasasubsetofthesetofsomemarketmaker.Suchbetsorchangeswouldjustbeimplementedbydealingsimultaneouslyanddirectlywithallofthemarketmakersthatoverlapthatset,i.e.,bytradingwithBetweenuser-initiatedtradesofthissort,thesystemcouldbringthemarketmakersintogreaterconsistencywitheachotherbyseekingoutarbitrageopportuni-ties,i.e.,setsoftradeswhichproduceapurecashnetLetuscalltwomarketmakersneighborsiftheyhavevariablesincommon,i.e.,ifisnon-empty.Arbitrageopportunitieswouldexistbetweenanytwoneighboringmarketmakersiftheyhaddif-ferentprobabilitydistributionsoverthesetofvariablestheyshared.(Forlogarithmicmarketmakers,arbitrag-ingthisdifferencewouldchangetheprobabilitydis-tributionsoverthesesharedvariablestobeanormal-izedgeometricmeanofthedistributionsofthemarketmakers.)Thusallthemarketmakerscouldbemadeconsistentwitheachotherviawavesofarbitragepass-ingthoughanetworkofmarketmakers.Neighbor-ingmarketmakerswouldbearbitraged,andifthisproducedalargeenoughchange,neighborsofthoseneighborswouldalsobearbitraged,andsoon.Oncethisprocessstopped,userswouldinessencebeinvitedtoprotbyndingandcorrectinganyremainingin-consistencies.Thisprotmightbelargeiftheremain-inginconsistencieswerelarge,butitwouldbestrictlyboundedbythetotalsubsidyofferedbyallthemarketmakers.Underthissortofapproach,ifonecouldanticipatethesortsofvariablesthatuserswouldbelikelytowant torelatetooneanother,onemightconstructanappro-priatesetofmarketmakerstosupportthoseusers.Eachrulewouldbebasedonatypeofprobabilitydistributionwhoseconsistencyacrosstheentirestatespacecouldbeguaranteed,andeachrulewouldonlyacceptbetsonprobabilitiesthatitcouldexactlycomputefromthisdis-tribution.Therecouldberulesthatcoveredsinglevari-ables,rulesforallvaluecombinationsforcertainsetsofadozenvariables,rulesfornearly-singly-connectedBayesnets,rulesforgeneralsparseBayesnets,rulesforsparselinearcovariancematrices,andrulesforlo-gisticregressions.Marketscoringrulepatronsmightwanttherstcrackatarbitraginganyinconsistenciesbetweenprobabilitiesfromdifferentrules,ifthosepa-tronsarewillingtotakeontherisksrequired.Butafterthis,userswouldbeinvitedtoprotbyxinganyre-maininginconsistencies.AvoidingBankruptcyEvenwhenprobabilitydistributionsareexactlyconsis-tent,usersmightstillturnamarketscoringrulepatronintoamoneypumpbyplayingheadsIwin,tailsIThatis,usersmightmakebetsforwhichtheyarepaidwhentheywin,butwhichtheycannotpaywhentheylose.Toavoidthis,amarketscoringruleshouldcomparethepricechangesthatauserpro-posestotheassetsthatheoffersascollateraltocoverhisproposedbet.Specically,alogarithmicrulebased)couldtreatanysetofcollateralas-setsasinprincipledescribableintermsofamountsheldofeachstateassetPays$1ifstateIfamarketscoringrulecandeterminethatinallstates thenitcansafelyapproveachangefromOfcoursethebestwaytomakesuchadetermina-tionwilllikelydependonthetypeofdistributionhandledbythismarketscoringrule,andonthetypeofassetsofferedascollateral.Thisdeterminationcanbeveryeasywhenthecollateralofferedisassetswithstate-independentvalue.Ontheotherhand,thisdeterminationmaygetveryhardwhenthecollat-eralistheresultofpreviousbetsmadewithrespecttoratherdifferentdistributionsortypesofdistributions.Forexample,ifthestructureofaBayesnetchangedsubstantiallyovertime,thenauserwhooncemadeabetwithrespecttoanoldversionofthenetmayndithardtousethatbetascollateraltosupportanewbet.Whileitmightbetemptingtorequirecashascollat-eralforallbets,suchapolicywouldgreatlydiscourageusersfrommakingsmallchanges,i.e.,changestotheprobabilitydistributionwhichcoveredonlysmallpartsofthestatespace.Thisisbecausesmallchangeswouldhaveasimilaropportunitycosttolargechanges,intermsofassetswhichcannolongerbeusedtosupportbets,eventhoughthepotentialpayofffromsuchbetsismuchsmaller.Sinceitisoftennothardtoverifythatotherkindsofassetscanserveascollateral,insistingoncashseemsmuchtoosevereaconstraint.Marketscoringrulesthusneedanon-trivialpolicyregardinghowmuchcomputationtheyarewillingtoundergotodeterminethatanygivencollateralwillcoveranygivenbet.Thispolicycanofcoursedependonthekindofchangesproposedandcollateraloffered.Sinceitisofteneasiertocheckproofsthantocreatethem,onepossiblepolicyisthatinmorecomplexcasestheusermustprovideaofcoverage,aproofthatthemarketscoringruleneedonlycheck.Thefactthatchangestoprobabilitydistributionscanmakeitharderforoldbetstobeusedascollateraltocovernewbetsmeansthatuserswhomakebetscancauseasubstantialexternalityonprevioususers.Forexample,inasingly-connectedBayesnetmarketscor-ingrule,changingtheunconditionaldistributionofavariablewillhavelittleimpactonprevioususers,butchangingthestructureofthenetworkcanhavealargeimpact.Auserwhomadeabetabouttheconditionalprobabilitiesassociatedwithagivenlinkmighteas-ilylaterundothatbetifthatlinkstillremainedinthecurrentnetwork,butifthelinkwasgonetheymightneedtorstadditbackin.Andiftherewerebindingcomplexitylimitsonthenetwork,thismayrequiretheeliminationofsomeotherlinksinthenetwork.Thusprevioususerscanbeharmedbytakingawaylinksthattheyhavemadebetson,andcanbebenetedbyreplac-ingsuchlinks.Similareffectscomeifuserscanaddordeletecutpointsfromadigitaloptionorsawtoothrep-resentationofareal-valuedrandomvariable.Thelawandeconomicsperspectiveseemsanappro-priateframeworktoconsiderthisexternality(CooterandUlen,2000).Ifitwereeasyforeachnewusertonegotiatewitheffectedprevioususers,thenwecouldjustclearlyassignapropertyrightforthemtotrade.Thisseemsdifcult,however,bothbecausetherewilloftenbemanyprevioususers,andbecauseofthepossi-bilityofstrategicallythreateningtobecomeanewuser. CombinatorialInformationMarketDesignWhennegotiationsaredifcult,wemustguessatwhatrightsthepartieswouldassign,iftheycouldnegotiate.Ifthepartiescouldnegotiate,wouldtheyagreetotaxorsubsidizecertainprobabilitychanges,becauseofthenetexternalityduetothosechanges?Fortopicswhereweexpecttoolittleinformationag-gregation,suchasduetofree-ridingproblemsamongpotentialpatrons,weexpectuserstoproduceanetpos-itiveinformationaggregationexternality,onaverage.Incontrast,fortopicswhereweexpecttoomuchin-formationaggregation,suchduetowastefulsignaling,weexpectanetnegativeinformationaggregationex-ternality,onaverage.Withsomewaytodistinguishparticularusesfromothers,intermsoftherelativemagnitudesofbenetsandcosts,weshouldwanttosubsidizeusersintherstcaseandtaxtheminthesecondcase.Whilestructuralchangesdoseemtohavemorepotentialfornegativeimpactonprevioususers,whenthereisanetpositiveinformationaggregationex-ternalitysuchchangesalsoseemtohavemorepotentialforapositiveimpactviaproducingabetterprobabilityIntheabsenceofamoredetailedanalysiswhichcanbetterpinpointwhichsortsofchangesshouldbedis-couraged,relativetootherchanges,thesimplestpolicyseemsbest.Letusersmakewhateverstructuralchangestheylike,iftheycanaffordthebetsandtheresult-ingstructuralcomplexityisacceptable,andleaveituptoprevioususerstomitigatetheexternalitiessuchchangesproduceonthem.Userscanmitigatesuchex-ternalitiesbynotwaitingtoolongbeforeundoingbets,avoidingbettingonstructuralchangestheydonotex-pecttolast,andseekingsynergiesamongthebetswhichmaketomakeiteasiertoprovethatearlierbetscanserveascollateralforlaterbets.Ingeneralintheworld,theintroductionofnewprod-uctsbenetsthosewhousethatnewproduct,andothersassociatedwiththosebeneciaries,butthisactcanalsoproducenegativeexternalitiesonthosewhoaretiedtotheoldproducts.Whilethisisashame,onnetitstillseemsbettertousuallyencouragenewproducts.Un-tilwecantellwhichnewproductsarenetharms,weshouldjustembracethemall.Informationmarketsseemtodowellataggregatingin-formationinthethickmarketcase,wheremanytradersallestimatethesamevalue.Buttohavecombinato-rialinformationmarkets,whichestimateaprobabilitydistributionoverallcombinationsofvaluesofmanyvariables,wewanttobetterdealwiththethinmarketandirrationalparticipationproblems.Simplescoringrulesavoidtheseproblems,butsufferfromopinionpoolproblemsinthethickmarketcase.Marketscor-ingrulesavoidalltheseproblems.Afterreviewinginformationmarketsandscoringrules,andpresentingmarketscoringrules,thispaperhasconsideredseveralimplementationissueswithus-ingmarketscoringrulesascombinatorialinformationmarketmakers.Wesawthatsawtoothassetscansup-portbothunconditionalestimatesofexpectedvalues,andconditioningonavariablebeingsomevalue.Wesawthatbyusingseveralmarketscoringrulesonthesamestatespace,eachinternallyconsistentbutwithin-consistenciespossiblebetweentherules,onecanavoidbecomingamoneypumpduetothedifcultyofcom-putingconsistentprobabilities,whilestillallowingbetsonmostprobabilitiesofinterest.Wesawthatmarketscoringrulemanagersshouldwanttomakesomeef-fortstoallowpastbetstobeusedascollateralforfu-turebets,andthatitremainsanopenquestionjusthowmucheffortisappropriate.Finally,wesawthatwhilebetsthatproducestructuralchangescanhaveaneg-ativeexternalitiesonthosewhohavemadebetsus-ingolderstructures,onnetitseemsbettertousuallyleavethemitigationofthisexternalitytothosewitholdAsummaryoftheproposeddesignisasfollows:1.Thereareseverallogarithmicmarketscoringrules,eachofwhichmaintainsadistributionofacertaintype,suchasanearlysingly-connectedBayesnet.2.Patronschooseasetofvariablestopatronize.Vari-ablescanbeaddedandrenedonthey,iffurtherfundingisavailable.3.Eachreal-valuedvariableisrepresentedbyadis-cretevariable,whosevaluesareassignedpayoffsaccordingtosawtoothfunctions)describedearlier.4.Patronschooseaninitialdistributionforeachmar-ketscoringrule.5.Patronschooseasubsidylevelforeachmarketscoringrule.Theexpectedlossofpatronshasaguaranteedbound,andwithuniforminitialdistri-butions,theexactlosshasaguaranteedbound.6.Anyusercanatanytimemakeachangetoanyofthesedistribution,bychangingdistribution intoanotherdistributionofthesametype.Suchchangesarecalculatedexactly,withoutroundofferror.7.Inbetweenuserchangestoindividualmarketscor-ingrules,arbitrageopportunitiesbetweensuchrulesarefoundandexploited,reducingtheinconsisten-ciesbetweentheirprobabilitydistributions.8.Considerauserwhohassofaronnetdeposited,andhassofarmadeasetofchanges)tovariousscoringrules,wherewede.Foralogarithmicmarketscoringrule,suchausercanmakeonemorechange,andwith-drawcash,ifhecanprovethat,consideringallpastchangesandthisproposednewchange,foreachstatewherenow0wehave9.Wheneveravariablesvaluesareassigned,stateprobabilitiesarereweightedaccordingtothisas-signment,andthestatespaceiscoarsenedtoelimi-natethisvariable.Assetholdingsbecomeaweightedaverageofassetholdingsintheoldstates.Themaincategoryofissuesnotdealtwithinthispaperisthatofuserinterfaces.Usersneedtobeabletobrowseaprobabilitydistributionoverthesetofallvariablevaluecombinations,toidentifytheestimates,suchasprobabilitiesandexpectedvalues,whichtheythinkareinerror.Userthenneedtobeabletochoosevaluesforthoseestimates,andseewhethertheyhaveenoughcollateraltocoverthebetsrequiredtomakethesechanges.Usersalsoneedtobeabletoseehowmuchrisktheyhavealreadyacquiredalongthesedi-mensions,sotheycanjudgehowmuchmoreriskisappropriate.Whilereasonableapproachesareknowninmanycases,itisfarfromclearwhatapproachesaremosteffectiveandrobustacrossawiderangeofthemostinterestingcases.AcknowledgmentsFortheircomments,Ithankparticipantsofthe2002conferenceonInformationDynamicsintheNet-workedSoceity,participantsofthe2002InformalSeminaronProgressinIdeaFuturesTechnology,HassanMasum,RaviPandya,andespeciallyDavid1.Forascoringrulethatisnotproper,thisargmaxcontainsotherdistributionsbesides.Inthispaper,weconsideronlyproperscoringrules.2.Forrewardsthatareintheformofbraggingrights,thepa-tronisinessencewhateveraudienceisimpressedbysuch3.Procedureswhentheresultisindependentoftheorderofapplyingthem.4.Notealsothatifmuchtimepassesbeforestateswillbeveri-edandpaymentsmade,tradesareinessenceoffutureassets.Thusifinterestratesarepositive,fewerrealresourcesneedbedepositednowtoguaranteesuchanabilitytopay.Forexam-ple,ifthefutureassetstradedweresharesinastockindexfund,onecoulddepositorescrowthesamenumberofsharesnow.5.Thisanalysisignoresthestrategicconsiderationthatonesreportmayinuencethereportsthatothersmakelater,whichcouldtheninuenceonesprotsfrommakingyetmorereportsafterthat.Butifthisprocesseverreachesanequilibriumwherepeopletexpectfurtherchanges,theequilibriummarketscoringruledistributionshouldbewhateachpersonwouldreporttoasimplescoringrule.6.Ithaslongbeenknownthatinonedimension,anagentusingascoringruleisequivalenttohischoosingaquantityfromacon-tinuousofferdemandschedule(Savage,1971).Thisequivalencealsoholdsinhigherdimensions.7.Thisisthesamefactthatinphysicsensuresthatthetotalentropyoftwocorrelatedsystemsislowerthanthesumoftheentropiesofeachsystemconsideredseparately.8.Userscouldalsowithdrawnon-multipleamounts,iftheywerewillingtosufferaroundingdowntocovertheworstpossibleroundofferrorincalculatingthelogarithm.9.Technically,thedirectednetworkofaBayesnetcanbetrans-latedintoanundirectednetworkbyaddinglinksbe-tweentheparentsofeachvariable.Theofthisundi-rectednetworkarethenthelargestsetsofnodesthatarealllinkedtoeachother.Byaddingmorelinkstothisnetwork,thesecliquescanbeorganizedintoajointree,whereforev-erypairofcliques,theirintersectionisasubsetofeverycliqueonthetreepathbetweenthesetwocliques.Thecosttoconsis-tentlyupdatethisnetworkisroughlyproportionaltothenum-berofvariables,timesafactorexponentialinthesizeofthecliques.(ThecomputationalcomplexityofndingthecliquesofanetworksisNP-completeintheworstcase,however(Cooper,ReferencesAumannR.Agreeingtodisagree.TheAnnalsofStatisticsBergJ,NelsonF,RietzT.Accuracyandforecaststandarderrorofpredictionmarkets.Tech.rep.,UniversityofIowa,CollegeofBusi-nessAdministration,2001.BrierGW.Vericationofforecastsexpressedintermsofprobability.MonthlyWeatherReview CombinatorialInformationMarketDesignChenK-Y,PlottCR.Informationaggregationmechanism:Concept,design,andimplementationforasalesforecastingproblem.Tech.rep.,CaliforniaInstituteofTechnology,1998.ClemenRT.Incentivecontractsandstrictlyproperscoringrules.TestCooperGF.ThecomputationalcomplexityofprobabilisticinferenceusingBayesbeliefnetworks.cialIntelligenceCooterR,UlenT.LawandEconomics(3rdedn.),NewYork:Addison-Wesley,2000.DeWispelareAR,HerrenLT,ClemenRT.Theuseofprobabilityelicitationinthehigh-levelnuclearwasteregulationprogram.ternationalJournalofForecastingDruzdzelMJ,vanderGaagLC.Elicitationofprobabilitiesforbeliefnetworks:Combiningqualitativeandquantitativeinformation.In:UncertaintyinArticialIntelligence,1995:141FiglewskiS.Subjectiveinformationandmarketefciencyinabettingmarket.JournalofPoliticalEconomyGeanakoplosJ,PolemarchakisH.Wecantdisagreeforever.JournalofEconomicTheoryGenestC,ZidekJ.Combiningprobabilitydistributions:Acritiqueandannotatedbibliography.StatisticalScienceGoldsteinM.SystematicanalysisoflimitedbeliefspeciGoodIJ.Rationaldecisions.JournaloftheRoyalStatisticalSociety.SeriesB(Methodological)HansonR.Consensusbyidentifyingextremists.TheoryandDecisionHansonR.Decisionmarkets.IEEEIntelligentSystemsHansonR.Elicitingobjectiveprobabilitiesvialotteryinsurancegames.Tech.rep.,GeorgeMasonUniversityEconomics,2002a.(http://hanson.gmu.edu/elicit.pdf).HansonR.Logarithmicmarketscoringrulesformodularcom-binatorialinformationaggregation.Tech.rep.,GeorgeMasonUniversityEconomics,2002b.(http://hanson.gmu.edu/mktscore.pdf).HansonR.Disagreementisunpredictable.EconomicsLetters(http://hanson.gmu.edu/unpredict.pdf).JensenFV.BayesianNetworksandDecisionGraphs.NewYork:Springer,2001.KadaneJ,WinklerR.SeparatingprobabilityelicitationfromJournaloftheAmericanStatisticalAssociationKrahnenJP,WeberM.Doesinformationaggregationdependonmar-ketstructure?Marketmakersvs.doubleauction.ZeitschriftfurWirtschaftsundSozialwissenschaftenLoAW.MarketEfciency:StockMarketBehaviourinTheoryandPractice.Elgar,Lyme,1997.MilgromP,StokeyN.Information,tradeandcommonknowledge.JournalofEconomicTheoryMurphyAH,WinklerRL.Probabilityforecastinginme-terology.JournaloftheAmericanStatisticalAssociationCarrollFM.Subjectiveprobabilitiesandshort-termeconomicforecasts:Anempiricalinvestigation.AppliedStatisticsPearlJ.ProbabilisticReasoninginIntelligentSystems:NetworksofPlausibleInference.SanMateo,California:MorganKauffmann,PennockDM,GilesC,NielsenF.Therealpowerofarticialmarkets.PennockDM,WellmanMP.CompactsecutiriesmarketsforParetooptimalreallocationofrisk.In:ProceedingsoftheSixteenthConferenceonUncertaintyinArticialIn-telligence,SanFrancisco:MorganKaufmann,2000:481(http://rome.exp.sis.pitt.edu/UAI/Abstract.asp?articleIDRollR.Orangejuiceandweather.TheAmericanEconomicReviewSavageLJ.Elicitationofpersonalprobabilitiesandexpecta-JournaloftheAmericanStatisticalAssociationSmithCAB.Personalprobabilityandstatisticalanalysis.JournaloftheRoyalStatisticalSociety.SeriesA(General)WinklerRL.Scoringrulesandtheevaluationofprobabil-ityassessors.JournaloftheAmericanStatisticalAssociationRobinHansonisanassistantprofessorofeconomicsatGeorgeMasonUniversity.In1998RobinreceivedhisPh.D.insocialsciencefromtheCaliforniaInsti-tuteofTechnology,andwasafterwardaRobertWoodJohnsonFoundationhealthpolicyscholarattheUniversityofCaliforniaatBerkeley.Earlier,in1984,RobinreceivedamastersinphysicsandamastersinthephilosophyofsciencefromtheUniversityofChicago,andthenspentnineyearsresearchingarticialintelli-gence,Bayesianstatistics,andhypertextpublishingatLockheed,NASA,andindependently.RobinhaspublishedinCATOJournal,Communi-cationsoftheACM,EconomicsLetters,Economet-rica,Extropy,IEEEIntelligentSystems,InformationSystemsFrontiers,InternationalJointConferenceoncialIntelligence,JournalofEvolutionandTech-nology,JournalofPublicEconomics,SocialEpiste-mology,SocialPhilosophyandPolicy,andTheoryandRobinhasunusuallydiverseresearchinterests,withpapersonspatialproductcompetition,healthincentivecontracts,groupinsurance,explainingproductbans,evolutionarypsychologyofhealthcare,bioethics,voterincentivestobecomeinformed,Bayesianstatisticsandcation,agreeingtodisagree,self-deceptionindisagreement,informationaggregationviaspeculativemarkets,governancebasedonsuchmarkets,probabil-ityelioitation,combinatorialinformationmarketmak-ers,wiretaps,imagereconstruction,thehistoryofsci-enceprizes,reversiblecomputation,theoriginoflife,verylongtermeconomicgrowth,growthgivenma-chineintelligence,andinterstellarcolonization.