ECE700.07: Game Theory with - PowerPoint Presentation

1. ECE700.07: Game Theory withEngineering ApplicationsSeyed Majid ZahediLecture 5: Games in Extensive Form

2. OutlinePerfect information extensive form gamesSubgame perfect equilibriumBackward inductionOne-shot deviation principleImperfect information extensive form gamesReadings:MAS Sec. 5, GT Sec. 3 (skim through Sec. 3.4 and 3.6), Sec. 4.1, and Sec 4.2 

3. Extensive Form GamesSo far, we have studied strategic form gamesAgents take actions once and simultaneously Next, we study extensive form gamesAgents sequentially make decisions in multi-stage gamesSome agents may move simultaneously at some stageExtensive form games can be conveniently represented by game trees

4. Example: Entry Deterrence GameEntrant chooses to enter market or stay outIncumbent, after observing entrant’s action, chooses to accommodate or fightUtilities are given by at leaves for each action profile (or history) denotes utility of agent 1 (entrant) and denotes utility of agent 2 (incumbent) 

5. Example: Investment in Duopoly Agent 1 chooses to invest or not investAfter that, both agents engage in Cournot competitionIf agent 1 invests, then they engage in Cournot game with and Otherwise, they engage in Cournot game with   

6. Finite Perfect-Information Extensive Form GamesFormally, each game is tuple is finite set of agents is set of actions available to agent is set of choice nodes (internal nodes of game tree) is set of terminal nodes (leaves of game tree) is agent function, which assigns to each choice node set of agents is action function, which maps choice nodes to set of actions available to agent is successor function, which maps choice nodes and action profiles to new choice or terminal node, such that if , then and is utility function, which assigns real-valued utility to agent at terminal nodes 

7. History in Extensive Form GamesLet be set of all possible stage nodes in game’s tree initial history stage 0 action profile history after stage 0 stage 1 action profile history after stage 1 history after stage If number of stages is finite, then game is called finite horizon gameIn perfect information extensive form games, each choice (and terminal) node is associated with unique history and vice versa 

8. Strategies in Extensive Form GamesPure strategies for agent is defined as contingency plan for every choice node that agent is assigned toExample:Agent 1’s strategies: Agent 2’s strategies: For strategy profile , outcome is terminal node  

9. Randomized Strategies in Extensive Form GamesMixed strategy: randomizing over pure strategiesBehavioral strategy: randomizing at each choice nodeExample:Give behavioral strategy for agent 1L with probability 0.2 and L with probability 0.5Give mixed strategy for agent 1 that is not behavioral strategyLL with probability 0.4 and RR with probability 0.6 (why this is not behavioral?)2,45,33,21,00,1Agent 1Agent 2Agent 1Agent 2LRLRLRLR

10. Example: Sequential Matching PenniesConsider following extensive form version of matching pennies How many strategies does agent 2 have?Extensive form games can be represented as normal form gamesWhat will happen in this game?  Agent 2Agent 1HHHTTTTHHeads(-1, 1)(-1, 1)(1, -1)(1, -1)Tails(1, -1)(-1, 1)(-1, 1)(1, -1)

11. Example: Entry Deterrence Game Consider following extensive form gameWhat is equivalent strategic form representation?Two pure Nash equilibrium: (In, A) and (Out, F)Are Nash equilibria of this game reasonable in reality?(Out, F) is sustained by noncredible threat of Entrant IncumbentEntrantAFIn(2, 1)(0, 0)Out(1, 2)(1, 2)

12. SubgamesSuppose that represents set of all nodes in ’s game treeSubgame of consists of one choice node and all its successorsRestriction of strategy to subgame is denoted by Subgame can be analyzed as its own game Example: sequential matching penniesHow many subgame does this game have?Given that game itself is also considered as subgame, there are three subgames 

13. Matrix Representation of Subgames Agent 2Agent 1LLLRRLRRLL2, 42, 45, 35, 3LR2, 42, 45, 35, 3RL3, 21, 03, 21, 0RR3, 20, 13, 20, 1 Agent 2Agent 1***L1, 0*R0, 1 Agent 2Agent 1*L*R*L3, 21, 0*R3, 20, 1 Agent 2Agent 1L*R***2, 45, 32,45,33,21,00,1Agent 1Agent 2Agent 1Agent 2LRLRLRLR

14. Subgame Perfect Equilibrium (SPE)Profile is SPE of game if for any subgame of , is NE of  Loosely speaking, subgame perfection will remove noncredible threatsNoncredible threads are not NE in their subgamesHow to find SPE?One could find all of NE, then eliminate those that are not subgame perfectBut there are more economical ways of doing it  

15. Backward Induction for Finite Games(1) Start from “last” subgames (choice nodes with all terminal children)(2) Find Nash equilibria of those subgames(3) Turn those choice nodes to terminal nodes using NE utilities (4) Go to (1) until no choice node remains[Theorem] Backward induction gives entire set of SPE

16. SPE of Extensive Form Game and NE of Subgames(RR, LL) and (LR, LR) are not subgame perfect equilibria because (*R, **) is not an equilibrium(LL, LR) is not subgame perfect because (*L, *R) is not an equilibrium, *R is not a credible threat1,03,22,43,2 Agent 2Agent 1LLLRRLRRLL2, 42, 45, 35, 3LR2, 42, 45, 35, 3RL3, 21, 03, 21, 0RR3, 20, 13, 20, 1 Agent 2Agent 1***L1, 0*R0, 1 Agent 2Agent 1*L*R*L3, 21, 0*R3, 20, 1 Agent 2Agent 1L*R***2, 45, 32,45,33,21,00,1Agent 1Agent 2Agent 1Agent 2LRLRLRLR

17. Example: Stackleberg Model of CompetitionConsider variant of Cournot game where firm 1 first chooses , then firm 2 chooses after observing (firm 1 is Stackleberg leader)Suppose that both firms have marginal cost and inverse demand function is given by , where , and  Solve for SPE by backward induction starting firm 2’s subgameFirm 2 chooses Firm 1 chooses  

18. Example: Ultimatum GameTwo agents want to split dollars1 offers 2 some amount If 2 accepts, outcome is If 2 rejects, outcome is  What is 2’s best response if ?Yes What is 2’s best response if ?Indifferent between Yes or NoWhat are 2’s optimal strategies?(a) Yes for all  (b) Yes if , No if       Agent 1  Agent 2YesNo

19. SPE of Ultimatum Game What is 1’s optimal strategy for each of 2’s optimal strategies?For (a), 1’s optimal strategy is to offer  For (b), If agent 1 offers , then her utility is If she wants to offer any , then she must offer This optimization does not have any optimal solution!No offer of agent 1 is optimal!Unique SPE of ultimatum game is: “Agent 1 offers , and agent 2 accepts all offers” 

20. Modified Ultimatum GameIf is in multiples of cent, what are 2’s optimal strategies?(a) Yes for all  (b) Yes if , No if  What are 1’s optimal strategies for each of 2’s?For (a), offer For (b), offer What are SPE of modified ultimatum game?Agent 1 offers , and agent 2 accepts all offers Agent 1 offers , and agent 2 accept all offers except  Show that for every , there exists NE in which 1 offers What is agent 2’s optimal strategy? 

21. limitation of Backward InductionIf there are ties, how they are broken affects what happens up in treeThere could be too many equilibriaAgent 1Agent 2Agent 23,22,34,10,10.876550.123451/21/2

22. Example: Bargaining Game Two agents want to split dollarFirst, 1 makes her offerThen, 2 decides to accept or rejectIf 2 rejects, then 2 makes new offerThen, 1 decides to accept or reject Let with denote allocations in 1st roundLet with denote allocations in 2nd round   Agent 1  Agent 2YesNoAgent 2     Agent 1YesNo

23. Backward Induction for Bargaining GameSecond round is ultimatum game with unique SPEAgent 2 offers , and agent 1 accepts all offers What is 2’s optimal strategy in her round 1’s subgame?(a) If , reject(b) If , accept, and reject otherwiseWhat are 1’s optimal strategies in round 1 for each of 2’s?For both (a) and (b), agent 1 is indifferent between all strategiesAgent 1’s weakly dominant strategy is to offer How many SPE does this game have?Infinitely many! In all SPE, agent 2 gets everything Last mover’s advantage: In every SPE, agent who makes offer in last round obtains everything 

24. Example: Discounted Bargaining GameSuppose utilities are discounted every round by discount factor,  What is unique SPE of (1)?2 offers and 1 accepts all offersWhat are optimal strategies in (2)?(a) Yes if , No otherwise(b) Yes if , No otherwiseWhat are optimal strategies in (3)?For (a), offer For (b), there is no optimal strategy   Agent 1  Agent 2YesNoAgent 2     Agent 1YesNo(1)(2)(3)

25. Unique SPE of Discounted Bargaining GameWhat are SPE strategies?Agent 1’s proposes Agent 2 only accepts proposals with Agent 2 proposes after any history in which1’s proposal is rejectedAgent 1 accepts all proposals of Agent 2What is SPE outcome of game?Agent 1 proposes Agent 2 acceptsResulting utilities are Desirability of earlier agreement yields positive utility for agent 1 

26. Stahl’s Bargaining Model (for Finite Horizon Games)2 rounds: 3 rounds: 4 rounds: 5 rounds: 2k rounds: 2k+1 rounds: Taking limit as , we see that agent 1 gets at SPE 

27. Rubinstein’s Infinite Horizon Bargaining ModelSuppose agent can alternate offers forever There are two types of outcome to considerAt round , one agent accepts her offer Every offer gets rejected: This is not finite horizon game, backward induction cannot be usedWe need different method to verify any SPE 

28. One-Shot Deviation Principle One-shot deviation from strategy means deviating from in single stage and conforming to it thereafterStrategy profile is SPE if and only if there exists no profitable one-shot deviation for each subgame and every agentThis follows from principle of optimality of dynamic programming  

29. SPE for Rubinstein’s ModelRecall that in Stahl’s model, for , Is following strategy profile SPE?Agent 1 proposes and accepts if and only if Agent 2 proposes and accepts if and only if  

30. One-Shot Deviation Principle for Rubinstein’s ModelFirst note that this game has two types of subgames(1) first move is offer(2) first move is response to offerFor (1), suppose offer is made by agent 1If agent 1 adopts , agent 2 accepts, agent 1 gets If agent 1 offers , agent 2 accepts, and agent 1 gets If agent 1 offers , agent 2 rejects and offers , agent 1 accepts and gets  For (2), suppose agent 1 is responding to offer If agent 1 adopts , she accepts and gets If agent 1 rejects and offers in next round, agent 2 accepts, agent 1 gets For (2), suppose agent 1 is responding to offer If agent 1 adopts , she rejects and offers in next round, agent 2 accepts, agent 1 gets If agent 1 accepts, she gets Hence is SPE (in fact unique SPE, check GT, Section 4.4.2 to verify) 

31. Rubinstein’s Model for Symmetric AgentsSuppose that If agent 1 moves first, division is  If agent 2 moves first, division is First mover’s advantage is related to impatience of agentsIf , FMA disappears and outcome tends to  If , FMA dominates and outcome tends to  

32. Imperfect Information Extensive Form GamesIn perfect information games, agents know choice nodes they are inAgents know all prior actions Recall that in such games choice nodes are equal to histories that led to themAgents may have partial or no knowledge of actions taken by othersAgents may also have imperfect recall of actions taken by themselves

33. Example: Imperfect Information Sequential Matching PenniesAgent 1 takes actionAgent 2 does not see agent 1’s actionAgent 2 takes action, and outcome is revealedInformation set is collection of choice nodes that cannot be distinguished by agents whose turn it isSet of agents and their actions at each choice node in information set has to be the same, otherwise, agents could distinguish between nodesAgent 1HTAgent 2HHTT-1-111

34. Finite Imperfect-Information Extensive Form GamesFormally, each game is tuple is perfect information, extensive form game , where , is partition of such that if , then , and for all ,  

35. Example: Poker-Like GameWhat are agent 1’s strategies?What are agent 2’s strategies?How can we find NE of this game?Model game as normal form zero-sum gameEach cell represents expected utilities (nature’s coin toss)Eliminated (weakly) dominated strategiesSolve for (mixed strategy) NE   Agent 2Agent 1CCCFFCFFRR0, 00, 01, -11, -1RC0.5, -0.51.5, -1.50, 01, -1CR-0.5, 0.5-0.5, 0.51, -11, -1CC0, 01, -10, 01, -1NatureGive 1 KingGive 1 Jack50%50%Agent 1Agent 1RaiseRaiseCheckCheck2/31/31/32/3Agent 21-111callfoldcallfold1-212callfoldcallfoldAgent 2

36. Example: Kune Pokerhttps://justinsermeno.com/posts/cfr/

37. Imperfect Recall, Mixed vs Behavioral StrategiesConsider mixed strategiesWhat is NE of this game?(R,D) with outcome utilities (2,2)Consider behavioral strategiesWhat is 1’s expected utility if she does What is 1’s best response?What is NE of this game? Agent 1LRAgent 1LURD1,12,2100,1005,1Agent 2

38. Solving Extensive Form Games:Perfect vs Imperfect InformationIn perfect information games, optimal strategy for each subgame can be determined by that subgame alone (how backward induction works!)We can forget how we got hereWe can ignore rest of gameIn imperfect information games, this is not necessarily trueWe cannot forget about path to current nodeWe cannot ignore other subgames https://www.chess.com

39. ExampleIs always accommodating good strategy?No, leads to utility of -2.5 for incumbentIs always fighting good strategy?No, leads to utility of -1.5 for incumbentWhat should incumbent do?A with 3/8 probability and F with 5/8What if we swap 2 and -2?A with 7/8 probability and F with 1/8InInEntrantEntrantOutOut2NatureHeadsTails50%50%-2Incumbent-535-3AFAF

40. Subgame Perfection and Imperfect InformationThere are two subgames: game itself and subgame after agent 1 plays R(R, RR) is NE and SPEBut, why should 2 play R after 1 plays L/M?This is noncredible threatThere are more sophisticated equilibrium refinements that rule this outAgent 1Agent 2Agent 24, 10, 05, 11, 0Agent 23, 22, 3LMRLRLRLR

41. Questions?

42. AcknowledgementThis lecture is a slightly modified version of ones prepared byAsu Ozdaglar [MIT 6.254]Vincent Conitzer [Duke CPS 590.4]