15-446 Distributed Systems - PowerPoint Presentation

Slide1

15-446 Distributed SystemsSpring 2009

L-27 Social Networks and Other StuffSlide2

Overview

Social NetworksMultiplayer Games

Class Feedback Discussion

2Slide3

Background: Sybil Attack

Sybil attack: Single user pretends many fake/sybil identities

Creating multiple accounts from different IP addressesSybil identities can become a large fraction of all identities

Out-vote honest users in collaborative tasks

launch

sybil

attack

honest

malicious

3Slide4

Background: Defending Against Sybil Attack

Using a trusted central authorityTie identities to actual human beings

Not always desirableCan be hard to find such authority

Sensitive info may scare away usersPotential bottleneck and target of attackWithout a trusted central authorityImpossible unless using special assumptions [Douceur’02]

Resource challenges not sufficient -- adversary can have much more resources than typical user 4Slide5

SybilGuard Basic Insight:

Leveraging Social Networks

Undirected graphNodes = identitiesEdges =

strong trust E.g., colleagues, relatives

Our Social Network Definition

5Slide6

SybilGuard Basic Insight

malicious

user

honest

nodes

sybil

nodes

Observation: Adversary cannot create extra edges between honest nodes and sybil nodes

attack edges

n

honest users: One identity/node each

Malicious users: Multiple identities each (sybil nodes)

Sybil nodes may collude – the adversary

6Slide7

SybilGuard Basic Insight

honest

nodes

sybil

nodes

Dis

-proportionally

small cut disconnecting a large number of identities

But cannot search for such cut brute-force…

7Slide8

Goal of Sybil Defense

Goal: Enable a verifier node to decide whether to

accept another suspect node

Accept: Provide service to / receive service from Idealized guarantee: An honest node accepts and only accepts other honest nodes

SybilGuard:Bounds the number of sybil nodes acceptedGuarantees are with high probabilityApproach: Acceptance based on random route intersection between verifier and suspect

8Slide9

Random Walk Review

pick random edge d

pick random edge c

pick random edge e

a

b

d

e

f

c

9Slide10

Random

1 to 1 mapping

between

incoming edge and outgoing edge

Random Route: Convergence

Using routing table gives Convergence PropertyRoutes merge if crossing the same edge

10

a

 d

b  a

c  b

d

 cd  ee  df  f

abcde

frandomized routing tableSlide11

Random Route: Back-traceable

Using 1-1 mapping gives Back-traceable Property

Routes may be back-traced

11

a

 d

b  a

c  b

d  cd

 ee  d

f  f

a

bc

defIf we know the route traverses edge e, then we know the whole routeSlide12

Random Route Intersection:

Honest Nodes

Verifier accepts a suspect if the two routes intersectRoute length w

:W.h.p., verifier’s route stays within honest regionW.h.p

., routes from two honest nodes intersect

sybil nodes

honest nodes

Verifier

Suspect

12Slide13

Random Route Intersection: Sybil Nodes

SybilGuard bounds the number of accepted

sybil nodes within g

*wg: Number of attack edgesw

: Length of random routesNext …Convergence property to bound the number of intersections within gBack-traceable property to bound the

number of accepted sybil nodes per intersection within w13Slide14

Bound # Intersections Within

gConvergence: Each attack edge gives one intersection

 at most

g intersections with g attack edges

sybil nodes

honest nodes

Verifier

Suspect

must cross attack edge to intersect even if sybil nodes do not follow the protocol

same

intersection

Intersection =

(node, incoming edge

14Slide15

Bound # Sybil Nodes Accepted per Intersection within

w

Back-traceable: Each intersection should correspond to routes from

at most

w

honest nodesVerifier accepts at most w nodes per intersectionWill not hurt honest nodes

Verifier

for a given intersection

15Slide16

Summary of SybilGuard Guarantees

Power of the adversary:

Unlimited number of colluding sybil nodes

Sybil nodes may not follow SybilGuard protocol W.h.p., honest node accepts ≤

g*w sybil nodesg: # of attack edgesw: Length of random route

If SybilGuard bounds # accepted sybil nodes within

Then apps can do

n

/

2

byzantine consensus

n

majority voting

not much larger than neffective replication16Slide17

Overview

Social NetworksMultiplayer Games

Class Feedback Discussion

17Slide18

Individual Player’s View

Interactive environment (e.g. door, rooms)

Live ammoMonsters

PlayersGame state

18Slide19

High-Speed, Large-Scale, P2P:

Pick 2

Many console games are peer hosted to save costs

Limits high-speed games to 32 players

1000+ player games need dedicated servers

High-speed

Large-scale

P2P

Question

: Can we achieve all 3?

19Slide20

P2P

Internet

Primary object

Local View

Replica objects

20Slide21

High-Speed

Internet

Local View

Inter-object writes

must be reflected

very quickly

Primary object

Replica objects

21Slide22

High-Speed

Internet

Local View

Replica objects

20 updates/sec

≈ 16 kbps per player

Delay must be < 150ms

[

Beigbeder

‘04]

Primary object

22Slide23

Large-Scale

Internet

Bandwidth per Player

(Mbps)

23Slide24

Area-of-Interest (AOI) Filtering

Large shared world

Composed of map information, textures, etc

Populated by active entities: user avatars, computer AI’s, etcOnly parts of world relevant to particular user/player

Player 1

Player 2

Game World

24Slide25

Object Model

Game world treated as collection of objectsSingle primary copy for each object

Maintained at a single owner node

Serialization point for updatesDetermines actions/behavior of objectSecondary replicas

Primary object may need to examine other objects to determine actionNeed low latency access to object  must replicate object in advanceHow to find set of objects to replicate?  hardest part

25Slide26

Object Discovery –Solution

Use publish-subscribe to “register” long-lived distributed lookupsPublications created each time object is updated (or periodically when no update is done)

Publication contents = state of object

Player

x

≥

50

x ≤ 150y ≥ 150

y ≤ 250Interests

x 100

y 200Events

(100,200)

(150,150)(50,250)ArenaVirtual World26Slide27

Publish-Subscribe (PubSub) Overview

Key feature

 subscription languageSubject/channel-based subscriptions (e.g. all publications on the IBM stock channel)

Rich database-like subscription languages (e.g. all publications with name=IBM and volume > 1000)

State-of-the-artScalable distributed designs with channel-based subscriptionsUnscalable distributed designs with rich subscriptionsGoal  scalable distributed design with “richer” subscriptionsExample: x ≤ 200 & y < 100

 Support for multi-dimensional range predicates

Subscription

Publications

Publishers produce

events

Subscribers register their

interests

27Slide28

Using DHTs for Range Queries

No cryptographic hashing for key

 identifier

Query: 6

 x

 13key = 6  0xabkey = 7

 0xd3…key = 13  0x12

0x00

0x70

0xf0

0x30

0xb0

0x20

0xc0

0xd0

0x100xe00xa0

0x90

0x800x400x500x60

Query: 6  x  13

28Slide29

Using DHTs for Range Queries

Nodes in popular regions can be overloaded

Load imbalance!

29Slide30

DHTs with Load Balancing

Mercury load balancing strategyRe-adjust responsibilities

Range ownerships are skewed!

30Slide31

DHTs with Load Balancing

0x00

0x30

0xa0

0xb0

0xd0

Finger pointers

get skewed!

Popular

Region

0x90

0x80

0xf0

0xe0

Each routing hop may not reduce node-space by half!

 no log(n) hop guarantee

31Slide32

Ideal Link Structure

0x00

0x30

0xa0

0xb0

0xd0

Popular

Region

0x90

0x80

0xf0

0xe0

32Slide33

Mercury

Values

Nodes

If we had the above information…

For finger

i

Estimate value v for which

2i th node is responsible

Need to establish links based on node-distance

4

8

v

4

v

8

33Slide34

Mercury

Values

Nodes

Need to establish links based on

node-distance

4

8

v

4

v

8

Values

Node-density

Piece-wise linear approximation

Histogram

34Slide35

Histogram Maintenance

0xf0

0x00

0x70

0xa0

0xb0

0xd0

0x90

0x80

0xe0

0x30

Request sample

(Range, density)

Measure node-density locally

Gossip about it!

Values

Node-density

(Range, density)(Range, density)

35Slide36

Load Balancing

Basic idea: leave-rejoin

Steps

Find average, check if heavy or light

Light nodes perform a leave and rejoin

0

10

20

25

35

45

60

65

707585Average

HeavyLight

1572.5

Load histogram

Load

36Slide37

M

ERCURY

Routing [Sigcomm 2004]

Send subscription to

any one

attribute hubSend publications to

all attribute hubsTunable number of long links  can range from full-mesh to DHT-like[0, 80)

[210, 320)

50 ≤ price

≤ 150150 ≤ volume ≤ 250

price 100volume 200

H

price

[240, 320)[80, 160)[160, 240)Hvolume

[0, 105)[105, 210)SubscriptionPublication

Rendezvous point37Slide38

Object Discovery – Basic Design

Use publish-subscribe to “register” long-lived distributed lookupsPublications created each time object is updated (or periodically when no update is done)

Publication contents = state of object

Player

x

≥

50

x ≤ 150y ≥ 150

y ≤ 250Interests

x 100

y 200Events

(100,200)

(150,150)(50,250)ArenaVirtual World38Slide39

Prefetching and Persistence

Basic design has problemsCollecting/updating existing state is too slow

High subscription update rate

1st fix 

use Mercury only for object discovery and not state update2nd fix  persistent publications/subscriptionsPublication lifetimes  subsequent subscriptions would immediately trigger transmissionSubscription lifetimes  enabled soft-state approach to subscriptions

3rd fix  predict future subscriptionsCreates a new hybrid of persistent storage and pubsub

39Slide40

Colyseus Architecture Overview

Object Location

Replica Management

1. Specify Predicted Interests:

(5 < X < 60 & 10 < y < 200) TTL 30sec

2. Locate Remote Objects: P3 on s2, P4 on s2

3. Register Replicas: R3 (to s2), R4 (to s2)

4. Synch Replicas: R3, R4

Mercuryserver s2

R5

R3

R4

P1P2

Object Store

server s1P3P4Object Placement

5. Infer Interests: R3, R4, P2

5. Optimize Placement: migrate P1 to server s2

P140Slide41

Demo

Synthetic game used

for evaluation

41Slide42

42

View Inconsistency

Observations:

1. View inconsistency is small and gets repaired quickly

2. Missing objects on the periphery

no delay100 ms delay400 ms delaySlide43

Area-of-Interest (AOI) Filtering

Only receive updates from players in your AOI

Colyseus

[Bharambe ‘06]

VON [Hu ‘06]SimMUD [

Knutsson ’04]Problems:Open-area maps, large battlesRegion populations naturally follow a power-law[

Bharambe ‘06, Pittman ‘07]

Requirement

: ~1000 players in

same

AOI

Low population

High population

Quake 3 region popularity

43Slide44

Smoothing Infrequent Updates

Send

guidance

(predictions)

instead of state updatesGuidable AI extrapolates transitions between pointsE.g., game path-finding code

guidance

guidance

Actual path

?

Problem

: Predictions are not always accurate

Interactions appear inconsistent

Jarring if player is paying attention

Replica

object

44Slide45

Donnybrook: Interest Sets

Intuition

: A human can only focus on a constant number of objects at once

[Cowan ‘01, Robson ‘81]Only need a constant number of high-accuracy replicasInterest Set

: The 5 players that I am most interested inSubscribe to these players to receive 20 updates/secOnly get 1 update/sec from everyone else

Me

My Interest Set

45Slide46

Donnybrook: Interest Sets

How to estimate human attention?

Attention(

i

) = how much I am focused on player i

d

1

d

2

θ

1

θ

2

Attention(i) =fproximity(di)faim(θi)finteraction-recency(ti)+

+Player 1Player 246Slide47

= Interest Set

Not in Interest Set

47Slide48

48Slide49

49Slide50

50Slide51

Existing Distributed File Systems

Each server responsible for pseudo-random range of ID spaceObject are given pseudo-random IDs

150-210

211-400

401-513

324

987

110

51Slide52

Preserving File Locality

Use Mercury to store data

Bill

Bob

Docs

1

1

2

6

1

bid

0

…

6

1

bid

1

…61

bid2…useridpath encode

blockid

67570-600601-660

661-700

52Slide53

Benefits

Order preserving placement achieves availability within one ‘9’ of the optimal placement.

Order preserving placement can speedup task latency by 60-100%.

Larger networks see greater speedup.

Availability Improvement

Request Speedup

53Slide54

Continuous Load Balancing

Mercury can balance file system load better than existing systems even under very skewed write patterns

54Slide55

Overview

Social NetworksMultiplayer Games

Class Feedback Discussion

55Slide56

Discussion

Lecture topics?Balance of networking/dist sys/ubiq?

Research/adv topics/practical engineering?Text book vs. papers vs. lectures

ProjectsFree form vs. proposedAndroid vs. no Android?

56