Logic and Lattices for Distributed Programming - PowerPoint Presentation

Slide1

Logic and Lattices for Distributed Programming

Neil Conway, William R. Marczak, Peter Alvaro, Joseph M. HellersteinUC Berkeley

David MaierPortland State UniversitySlide2

Distributed Programming:

Key ChallengesAsynchronyPartial

FailureSlide3
Slide4

Dealing with Disorder

Enforce global orderPaxos, Two-Phase Commit, GCS, …“Strong Consistency”

Tolerate disorder

Programmer must ensure correct behavior for

many possible network orders

“

Eventual Consistency

”

Typical goal: replicas converge to same final

stateSlide5

Dealing with Disorder

Enforce global orderPaxos, Two-Phase Commit, GCS, …“Strong Consistency”

Tolerate disorder

Programmer must ensure correct behavior for

many possible network orders

“

Eventual Consistency

”

Typical goal: replicas converge to same final

stateSlide6

Goal:

Make it easier to writeprograms on top of

eventual consistency Slide7

This Talk

Prior WorkConvergent Modules (CRDTs)Monotonic Logic (CALM)BloomLCase StudySlide8

Read: {Alice, Bob}

Write: {Alice, Bob, Dave}Write: {Alice, Bob, Carol}

Students

{Alice, Bob, Dave}

Students

{Alice, Bob, Carol}

Client

0

Client

1

Read: {Alice, Bob}

Students

{Alice, Bob}

How to resolve?

Students

{Alice, Bob}Slide9

Problem

Replicas perceive different event orders

Goal

Same final state at all replicas

Solution

Use commutative operations (“merge functions”)Slide10

Students

{Alice, Bob, Carol, Dave}Students

{Alice, Bob, Carol, Dave}

Client

0

Client

1

Merge = Set UnionSlide11

Commutative Operations

Common design patternFormalized as CRDTs: Convergent and Commutative Replicated Data

TypesShapiro et al., INRIA (2009-2012)Based on join semilatticesSlide12

Lattices

hS,t,?i is a bounded join semilattice

iff:S is a sett

is a binary operator (“least upper bound”)Associative, commutative, and idempotentInduces a partial order on

S

:

x

·

S

y

if

x

t

y

=

y

Informally, “merge function” for elements of

S

?

is the “least” element in

S

8

x 2 S: ? t x

= x12Slide13

Time

Set(LUB = Union

)

IncreasingInteger

(LUB =

Max

)

Boolean

(LUB =

Or

)Slide14

Client

0Client1

Students

{Alice, Bob, Carol, Dave}

Students

{Alice, Bob, Carol, Dave}

Teams

{<Alice, Bob>}

Teams

{<Alice, Bob>}

Read: {Alice, Bob, Carol, Dave}

Read: {<

Alice,Bob

>}

Write: {<

Alice,Bob

>, <

Carol,Dave

>}

Teams

{<Alice, Bob>,

<Carol, Dave>

}

Remove: {Dave}

Students

{Alice, Bob, Carol}

Replica Synchronization

Students

{Alice, Bob, Carol}

Teams

{<Alice, Bob>,

<Carol, Dave>

}

Teams

{<Alice, Bob>,

<Carol, Dave>

}

Teams

{<Alice, Bob>,

<Carol, Dave>

}Slide15

Client

0Client1

Students

{Alice, Bob, Carol, Dave}

Students

{Alice, Bob, Carol, Dave}

Teams

{<Alice, Bob>}

Read: {Alice, Bob, Carol}

Read: {<

Alice,Bob

>}

Teams

{<Alice, Bob>}

Remove: {Dave}

Students

{Alice, Bob, Carol}

Replica Synchronization

Students

{Alice, Bob, Carol}

Nondeterministic Outcome!

Teams

{<Alice, Bob>}

Teams

{<Alice, Bob>}Slide16

Problem:

Composition of CRDTs canresult in non-determinismSlide17

Possible Solution:

Encapsulate all distributedstate in a single CRDTHard to design,verify, and test

Doesn’t scale with application sizeSlide18

Goal:

Design a language that allowssafe composition of CRDTsSlide19

Solution: … Datalog?

Concurrent work: distributed programming using DatalogP2 (2006-2010)Bloom (2010-2012)Monotonic logic: building block for convergent distributed programsSlide20

Monotonic Logic

As input set grows, output set does not shrink“Retraction-free”Order independente.g., map, filter, join, union, intersectionNon-Monotonic LogicNew inputs might retract previous outputsOrder sensitivee.g., aggregation, negationSlide21

Monotonicity and Determinism

Agents learn strictly more knowledge over time

Different learning order, same final outcome

Result:

Program is deterministic!Slide22

C

onsistencyAsLogicalMonotonicity

CALM AnalysisAll

monotone programs are deterministic

Simple syntactic test for monotonicity

Result:

Whole-program

static analysis

for

eventual consistencySlide23

Problem:

CALM only applies toprograms over growing setsVersion Numbers

Timestamps

Threshold TestsSlide24

Quorum Vote

A coordinator accepts votes from agentsCount # of votesWhen Count(Votes) > k, send “success” messageSlide25

Quorum Vote

A coordinator accepts votes from agentsCount # of votesWhen Count(Votes) > k, send “success” message

Aggregation isnon-monotonic!Slide26

CRDTs

Limite

d scope(single object)

Flexible types

(any lattice)

CALM

Whole program

analysis

Limited types (only

sets)

Bloom

L

Whole program

analysis

Flexible types (any lattice

)Slide27

BloomL Constructs

Organization

Collection of agents

Communication

Message passing

State

Lattices

Computation

Functions

over latticesSlide28

Monotone Functions

f : ST is a monotone function iff8a,b

2 S : a ·S

b ) f(a)

·

T

f(b)

28Slide29

Time

Set(LUB =

Union)Increasing

Integer

(LUB =

Max

)

Boolean

(LUB =

Or

)

size()

>= 5

Monotone function from

set



increase-

int

Monotone function

from

increase-

int



booleanSlide30

Quorum Vote in BloomL

QUORUM_SIZE = 5

RESULT_ADDR =

"example.org

"

class

QuorumVote

include

Bud

state

do

channel

:

vote_chn

,

[

:

@

addr

,

:

voter_id

]

channel

:

result_chn

,

[

:@

addr]

lset :

votes

lmax

:

vote_cnt

lbool

:

got_quorum

end

bloom

do

votes

<=

vote_chn

{

|

v

|

v

.

voter_id

}

vote_cnt

<=

votes

.

size

got_quorum

<=

vote_cnt

.

gt_eq(

QUORUM_SIZE

)

result_chn

<~

got_quorum

.

when_true

{

[

RESULT_ADDR

]

}

end

end

Monotone function: set

!

max

Monotone function: max

!

bool

Threshold test on

bool

(monotone)

Lattice state declarations

30

Communication interfaces

Accumulate votes

into set

Annotated Ruby class

Program state

Program logic

Merge function for set lattice

Monotonic



CALMSlide31

BloomL Features

Generalizes logic programming to latticesIntegration of relational-style queries and functions over latticesEfficient incremental evaluation schemeLibrary of built-in latticesBooleans, increasing/decreasing integers, sets, multisets, maps, …API for defining custom latticesSlide32

Case Studies

Key-Value StoreObject versioning via vector clocksQuorum replication

Replicated Shopping Cart

Using custom lattice types to encode domain-specific knowledgeSlide33

Case Studies

Key-Value StoreObject versioning via vector clocksQuorum replication

Replicated Shopping Cart

Using custom lattice types to encode domain-specific knowledgeSlide34

Case Study: Shopping Carts

34Slide35

Case Study: Shopping Carts

35Slide36

Case Study: Shopping Carts

36Slide37

Case Study: Shopping Carts

37Slide38

Perspectives on Shopping

CRDTsIndividual server replicas convergeBloomCheckout is non-monotonic  requires distributed coordinationBuilt-in BloomL lattice typesCheckout is not a monotone function of any of the built-in latticesSlide39

Observation:

Once a checkoutoccurs, no more shopping actionscan be performedSlide40

Observation:

Each client knowswhen a checkout can beprocessed “safely”Slide41

Monotone Checkout

OPS = [1]IncompleteOPS = [2]

Incomplete

OPS = [3]Incomplete

OPS = [1,2]

Incomplete

OPS = [2,3]

Incomplete

OPS = [1,2,3]

Complete

41Slide42

Monotone Checkout

42Slide43

Monotone Checkout

43Slide44

Monotone Checkout

44Slide45

Monotone Checkout

45Slide46

Shopping TakeawaysCheckout summary

is a monotone function of client’s activitiesCustom lattice type captures application-specific notion of “forward progress”“Unsafe” state hidden behind ADT interfaceSlide47

Recap

How to build eventually consistent systemsWrite disorderly programsDisorderly stateLatticesDisorderly computationMonotone functions over lattices

BloomLType system for deterministic behaviorSupport for custom lattice typesSlide48

Thank You!

http://www.bloom-lang.netSlide49

Backup SlidesSlide50

Strong Consistency in Industry

“… there was a single overarching theme within the keynote talks… strong synchronization of the sort provided by a locking service must be avoided like the plague… [the key] challenge is to find ways of transforming services that might seem to need locking into versions that … can operate correctly without locking.”

-- Birman et al.,“Toward a Cloud Computing Research Agenda” (LADIS, 2009)

50Slide51

Bloom Operational Model

51Slide52

QUORUM_SIZE

= 5

RESULT_ADDR

= "

example.org

"

class

QuorumVote

include

Bud

state

do

channel

:

vote_chn

,

[

:

@

addr

,

:

voter_id

]

channel

:

result_chn

,

[

:@addr

] table :votes

, [:

voter_id

]

scratch

:

cnt

,

[]

=>

[

:

cnt

]

end

bloom

do

votes

<=

vote_chn

{

|

v

|

[

v

.

voter_id

]

}

cnt

<=

votes

.

group(nil, count(

:

voter_id

))

result_chn

<~

cnt

{

|

c

|

[

RESULT_ADDR

]

if

c

>=

QUORUM_SIZE

}

end

end

Quorum Vote in Bloom

Communication

Persistent Storage

Transient Storage

Accumulate votes

Send message when quorum reached

Not (set) monotonic!

52

Count votes

Annotated Ruby class

Program state

Program logicSlide53

Built-in Lattices

NameDescription?

a t b

Sample Monotone Functions

lbool

Threshold

test

false

a

∨

b

when_true

()

!

v

lmax

Increasing

number

1

max(

a,b

)

gt

(n)

!

lbool

+(n)

!

lmax

-(n) !

lmaxlmin

Decreasing number−

1min(a,b)

lt

(n)

!

lbool

lset

Set of values

;

a

[

b

intersect(

lset

)

!

lset

product(

lset

)

!

lset

contains?(v)

!

lbool

size()

!

lmax

lpset

Non-negative set

;

a

[

b

sum()

!

lmax

lbagMultiset of values;

a

[

b

mult

(v)

!

lmax

+(

lbag

)

!

lbag

lmap

Map from keys to lattice values

empty map

at(v)

!

any-

lat

intersect(

lmap

)

!

lmap

53Slide54

Failure Handling

Great question! Monotone programs handle transient faults very wellDeterministic  simple loggingCommutative, idempotent  simple recovery

Future work: “controlled non-determinism”Timeout code is fundamentally non-deterministic

But we still want mostly deterministic programsSlide55

Handling Non-Monotonicity

… is not the focus of this talk Basic alternatives:Nodes agree on an event order using distributed coordination (e.g., Paxos)Allow non-deterministic outcomesIf needed, compensate and apologize