Collective Revelation: A Mechanism for Self-Verified, - PowerPoint Presentation

Slide1

Collective Revelation: A Mechanism for Self-Verified,Weighted, and Truthful Predictions

Sharad Goel, Daniel M. Reeves, David M. Pennock

Presented by: Nir ShabbatSlide2

Outline

IntroductionBackground and Related WorkThe SettingsA Mechanism For Collective RevelationThe Basic MechanismA General Technique for Balancing Budgets

SummarySlide3

Introduction

In many cases, a decision maker may seek to elicit and aggregate the opinions of multiple experts.Ideally, would like a mechanism that is:Incentive Compatible - Rewards participants to be

truthful

Information Weighted

-

Adjusts for the fact that some experts are better informed than

others

Self-Verifying

-

Works without the need for objective, “ground truth”

observations

Budget Balanced

-

Makes no net transfers to

agentSlide4

Background and Related Work

Proper Scoring Rules – Rewards the agent by assessing his forecast against an actual observed outcome.Agents are incentivized to be truthful.For example, using the Brier scoring rule:

We

get: Slide5

Background and Related Work

Peer Prediction Methods – Can be Incentive Compatible and Self-Verifying.For example, the Bayesian truth serum

asks agents to report both their own prediction and their prediction of other agents’ predictions.

Relies on the general property of information- scores, that a truthful answer constitutes the best guess about the most “surprisingly common” answer.Slide6

Background and Related Work

With the appropriate reward structure , this framework leads to truthful equilibria:

iff response of agent

is

- prediction of player

for the frequency of

- geometric average of predictions (or opinions) and meta predictions

 Slide7

Background and Related Work

Predictions Markets – encourage truthful behavior and automatically aggregate predictions from agents with diverse information.For example, a prediction market for U.S. presidential elections. Agents buy and sell assets tied to an eventual Democratic or Republican win. Each share pay $1 if the corresponding event occurs.Slide8

Background and Related Work

Delphi Method – generates consensus predictions from experts generally through a process of structured discussion.In each round, each expert anonymously provides his forecast and reasons, at the end of the round, market maker summarize the forecasts and reasons.Process is repeated until forecasts converge.Slide9

Background and Related Work

Competitive Forecasting – elicits confidence intervals on predictions, thereby facilitating information weighting. Like prediction markets, competitive forecasting rewards accuracy, though is

not rigorously incentive compatible and relies on

benchmarking against

objective measurements.Slide10

Background and Related Work

Incentive Compatible

Information Weighted

Self-Verifying

Proper Scoring

Rules

●

Prediction Markets

○

●

Peer Prediction

○

●

Delphi Method

●

Competitive Forecasting○

●Polls●Collective Revelation

○

●

●Slide11

The Settings

Common prior – Agents & Market Maker have a common prior on the distribution of

Independent Private Evidence

– Each agent

privately observes

independent realization of the random variable

(independent both of each other and across agents).

Rationality

– Agents update their beliefs via Bayes’ rule.

Risk neutrality

– Agents act to maximize their expected payoff.

 Slide12

A Mechanism For Collective Revelation

Agents are scored against one another’s private information (Self Verification).Agents report their subjective expectation of

, and their revised estimate in light of new hypothetical evidence (

HFS

).

From these 2 reports the subjective posterior, private information and also this agent’s prediction of other agents’ prediction is constructed.

 Slide13

A Mechanism For Collective RevelationSlide14

The Basic Mechanism

LEMMA 1

An agent has subjective distribution

for

It is possible to elicit the k

th

moment of

via a proper scoring rule that pays based on the outcome

iff

 Slide15

The Basic Mechanism

Simply put, scoring against a single Bernoulli observation can only reveal the agent’s expectation and not the uncertainty of it’s prediction (as quantified by the variance).So in order for a mechanism to be Information-Weighted

(for Bernoulli outcome) it must score agents against

multiple observations

.Slide16

The Basic Mechanism

For example, say we want to elicit an agent’s prediction regarding the distribution of

.

We want to reward the agent using a proper scoring rule based on a single outcome

:

To elicit the confidence, we

must

use another outcome

. For example, like this:

 Slide17

The Basic Mechanism

The Beta distribution is the “conjugate prior” of the Bernoulli distribution.That is, given

with a prior

the posterior distribution of

, given

independent trials out of which

are successful, is:

 Slide18

The Basic Mechanism

LEMMA 2Let

a prior on

Fix

(trials) and

(successes) integers

Define:

That is:

 Slide19

The Basic Mechanism

LEMMA 2 (Cont’d)Then:

is a bijection from

to

And

 Slide20

The Basic Mechanism

LEMMA 3Let

Let

a prior on

Let

Define:

 Slide21

The Basic Mechanism

LEMMA 3 (Cont’d)Then:

is a bijection from

to

And:

 Slide22

The Basic Mechanism

THEOREM 1Consider a Bayesian game according to the settings shown before with

players,

.

Let

a common prior on

Let

be the posterior dist. of agent

for

after updating according to its private info.

Fix

and

integers

 Slide23

The Basic Mechanism

THEOREM 1 (Cont’d)Suppose that each agent

plays action

where

 Slide24

The Basic Mechanism

THEOREM 1 (Cont’d)Define:

Where

is projection into the k

th

component.

 Slide25

The Basic Mechanism

THEOREM 1 (Cont’d)For arbitrary constants

, set the reward of agent

to be:

 Slide26

The Basic Mechanism

THEOREM 1 (Cont’d)Then:(*)is a strict Nash Equilibrium.Slide27

The Basic Mechanism

PROOFFix attention on agent , and suppose all agents

play according to (*).

By Lemma 2 -

parameters of

.

 Slide28

The Basic Mechanism

PROOF (Cont’d)Let

be private observations of agent

. Then:

Consequently:

 Slide29

The Basic Mechanism

PROOF (Cont’d)In particular,

and

If

plays according to (*) then:

Since we are using the Brier Proper scoring rule, strategy (*) maximizes

’s expected reward.

 Slide30

The Basic Mechanism

PROOF (Cont’d)Moreover, since

is an injection, this is

’s unique best response, and so strategy (*) is a strict Nash equilibrium.

 Slide31

The Basic Mechanism

CORROLARYSuppose all agents play the equilibrium strategy.Define:Slide32

The Basic Mechanism

CORROLARY (Cont’d)Define the aggregate information-weighted prediction as:

Let

be the posterior dist. Resulting from the cumulative private evidence of all agents.

Then

.

 Slide33

A General Technique for Balancing Budgets

The idea is to reward agents via a Shared scoring rule.

A simple application would be:

And this is clearly budget balanced since

 Slide34

A General Technique for Balancing Budgets

In our setting, the “observations”

are determined precisely by agents’ reports – simple application won’t work!

The solution proposed is to decouple

scoring

from

benchmarking

, e.g. agent

is rewarded according to its performance relative to a set of agents whose score cannot be affected by agent

’s action.

 Slide35

A General Technique for Balancing Budgets

DEFINITIONLet

be a Bayesian game with

players

A

-projective family of

is a family of games

such that for each

with

,

is a Bayesian game restricted to the players

that preserves type spaces, action spaces, and players’ beliefs about types. In particular,

is determined by a family of reward functions

that specifies the expected reward for each player in

resulting from any given strategy profile of those players.

 Slide36

A General Technique for Balancing Budgets

THEOREM 2G is a Bayesian game with n players

is a (strict) Nash equilibrium of G

Suppose

is a k-projective family of G

such that

Suppose that for each

,

is a (strict) Nash equilibrium for

.

 Slide37

A General Technique for Balancing Budgets

THEOREM 2 (Cont’d)Then for any constant

, there are player rewards

for the n-player game G such that:

For any strategy profile s,

The original equilibrium q is still a (strict) Nash equilibrium for the modified game

In particular, by setting

, one can alter the rewards so that the game G is strongly budget balanced.

 Slide38

A General Technique for Balancing Budgets

PROOFFor

, denote:

- reward function of player

in

- reward function of player

in the game

- reward of player

in game

, when

is restricted to game

.

 Slide39

A General Technique for Balancing Budgets

PROOF (Cont’d)Denote the player sets

where the players wrap around for

.

Consider the modified rewards for

:

 Slide40

A General Technique for Balancing Budgets

PROOF (Cont’d)Summing over all players we get:Slide41

A General Technique for Balancing Budgets

PROOF (Cont’d)Further more:

And since

, we have

, and so

for

.

 Slide42

A General Technique for Balancing Budgets

PROOF (Cont’d)Consequently, if

is the strategy of player

and

is the strategy profile of all other players, then:

Where

is a function that does not depend on

.

 Slide43

A General Technique for Balancing Budgets

PROOF (Cont’d)Since

is a (strict) Nash equilibrium for

:

And so, using

the new reward functions, the original equilibrium

is still a (strict) Nash equilibrium.

 Slide44

Summary

We’ve seen a mechanism, i.e. Collective Revelation, for aggregating experts opinions which is:Incentive CompatibleInformation WeighedSelf-VerifyingWe’ve seen a general technique for constructing budget balanced mechanisms that applies both to collective revelation and to past peer-predictions method.Slide45

THE END