Causal Reliable Broadcast Ali Ghodsi – UC Berkeley

slide1. Causal Reliable Broadcast Ali Ghodsi – UC Berkeley / KTH

alig(at)cs.berkeley.edu<br>
slide2. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 2 Motivation Assume we chat application
Whatever written is reliably broadcast to group

If you get the following chat output, is it ok?

UserX’s message caused UserY’s message,
UserY’s message caused UserZ’s message [UserZ] Ok
[UserY] Can we push back 1h?
[UserX] Let’s meet at 2pm?<br>
slide3. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 3 Motivation (2) Does uniform reliable broadcast remedy this? [d]

Causal reliable broadcast solves this
Deliveries in causal order!

Causality is same as happened-before relation by Lamport!<br>
slide4. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 4 Causality Recalled Let m1 and m2 be any two messages:
m1m2 (m1 causally precedes m2) if

C1 (FIFO order).
Some process pi broadcasts m1 before broadcasting m2

C2 (Network order).
Some process pi delivers m1 and later broadcasts m2

C3 (Transitivity).
There is a message m’ such that m1  m’ and m’  m2<br>
slide5. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 5 Causal Broadcast Interface Module:
Name: CausalOrder, instance co

Events
Request: co, Broadcast | m
Indication: co, Deliver | src, m

Property:
CB: If node pi delivers m1, then pi must have delivered every message causally preceding () m1 before m1

Is this useful? How can it be satisfied? [d]
It is only safety. Satisfy it by never delivering!<br>
slide6. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 6 Causality C1 (FIFO order).
Some process pi broadcasts m1 before broadcasting m2 p1 p2 p3 m1 m2 p1 p2 p3 m1 m2<br>
slide7. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 7 Causality (2) C2 (Network order).
Some process pi delivers m1 and later broadcasts m2 p1 p2 p3 m1 m2 p1 p2 p3 m1 m2<br>
slide8. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 8 Causality (3) C3 (Transitivity).
There is a message m’ such that m1  m’ and m’  m2 p1 p2 p3 m1 m2 m3 p1 p2 p3 m1 m2 m3<br>
slide9. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 9 Different Causalities Property:
CB: If node pi delivers m1, then pi must deliver every message causally preceding () m1 before m1
CB’: If pj delivers m1 and m2, and m1m2, then pj must deliver m1 before m2

What is the difference? [d]

Indeed, CB implies CB’ p1 p2 p3 m1 m2 m3 p1 p2 p3 m1 m2 m3 Violates CB and CB’ Violates CB, not CB’<br>
slide10. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 10 Reliable Causal Broadcast Interface Module:
Name: ReliableCausalOrder, instance rco

Events
Request: rco, Broadcast | m
Indication: rco, Deliver | src, m

Property:
RB1-RB4 from regular reliable broadcast
CB: If node pi delivers m, then pi must deliver every message causally preceding () m before m<br>
slide11. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 11 Uniform Reliable Causal Broadcast Module:
Name: UniformReliableCausalOrder, instance urco

Events
Request: urco, Broadcast | m
Indication: urco, Deliver | src, m

Property:
URB1-URB4 from uniform reliable broadcast
CB: If node pi delivers m, then pi must deliver every message causally preceding () m before m<br>
slide12. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 12 Reusing abstractions Reuse RB for CB

Use reliable broadcast abstraction to implement reliable causal broadcast

Use uniform reliable broadcast abstraction to implement uniform causal broadcast<br>
slide13. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 13 Towards an implementation Main idea
Each broadcasted message carries a history
Before delivery, ensure causality

First algorithm
History is set of all causally preceding messages<br>
slide14. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 14 Fail-Silent No-Waiting Causal Bcast Each message m carries ordered list of causally preceding messages in pastm

Whenever a node rbDelivers m
coDeliver causally preceding messages in pastm
coDelivers m
Avoid duplicates using delivered<br>
slide15. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 15 p1 p2 p3 m1 coB(m1) coD(m1) coD(m1) coB(m2) m2 [m1] coD(m2) rbD(m2) m2 [m1] coD(m2) coD(m1) coD(m2) m1 Execution (direct override)<br>
slide16. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 16 p1 p2 p3 m1 coB(m1) coD(m1) coD(m1) coB(m2) m2 [m1] coD(m2) rbD(m2) m2 [m1] coD(m2) coD(m1) coD(m2) m1 Execution (indirect override)<br>
slide17. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 17 Fail-silent Causal Broadcast Impl Implements:
ReliableCausalOrderBroadcast instance rco.

Uses: ReliableBroadcast instance rb.

upon event rb, Init do
delivered := ; past := nil

upon event rco, Broadcast | m do
trigger rb, Broadcast | (DATA, past, m)
past := append(past, <pi, m>) Append this message to past history<br>
slide18. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 18 Fail-silent Causal Broadcast Impl (2) upon event rb,Deliver | pi,(DATA, pastm , m) do
if mdelivered then
forall (sn,n)pastm do
if ndelivered then
trigger rco,Deliver|sn, n
delivered := delivered{n}
past := append(past, <sn,n>)
trigger rco,Deliver|pi,m
delivered := delivered{m}
past := append(past, <pi,m>) in ascending order deliver preceding messages append to history deliver current message append to history<br>
slide19. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 19 Correctness RB1-RB4 follow from use of RB
No creation and no duplication still satisfied
Validity still satisfied
Some messages might be delivered earlier, never later
Agreement directly from RB

CO by induction on prefixes of executions
It is vacuously true for empty executions
Assume it is true for all deliveries of a prefix
Then it is true for any extension with one event<br>
slide20. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 20 Improving the algorithm Disadvantage of algorithm is that the message size (bit complexity) grows

Useful idea
Garbage collect old messages

Implementation of GC
Ack receipt of every message m to all
Use perfect failure detector P
Determine with P when all correct nodes got message m
Delete m from past when all correct nodes got m<br>
slide21. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 21 GC implementation Uses: ReliableBroadcast instance rb, PerfectFD instance P

upon event rco, Init do
delivered := ; past := nil
correct := 
forall m: ack[m] := 

upon event P, crash | pi do
correct := correct \ {pi}

upon event mdelivered and selfack[m] do
ack := ack[m] U {m}
trigger rb, Broadcast | (ACK, m)

upon event rb, Deliver | pi, [ACK, m] do
ack := ack[m] U {pi}
if correctack[m] do
past := remove(past, <x, m>) book keeping of acks called upon coDeliver ack to all When received ack from all, GC m from any x<br>
slide22. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 22 Towards another implementation Main idea
Each broadcasted message carries a history
Before delivery, ensure causality

First algorithm
History is set of all causally preceding messages

Second algorithm [d]
History is a vector timestamp<br>
slide23. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 23 Fail-Silent Waiting Causal Broadcast Represent past history by vector clock (VC)

Slightly modify the VC implementation
At node pi

VC[i]: number of messages pi coBroadcasted

VC[j], ji: number of messages pi coDelivered from pj<br>
slide24. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 24 Fail-Silent Waiting Causal Broadcast Upon CO broadcast m
Piggyback VC and RB broadcast m

Upon RB delivery of m with attached VCm
compare VCm with local VCi
Only deliver m once VCm precedes VCi<br>
slide25. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 25 p1 p2 p3 b(m1) d(m1) b(m2) d(m2) d(m1) d(m2) d(m2) d(m1) (0,0,0) (1,0,0) m1(0,0,0) m1(0,0,0) (2,0,0) m2(1,0,0) m2(1,0,0) (1,0,0) (0,0,0) (0,0,0) (1,0,0) (2,0,0) (2,0,0) Execution hold m2<br>
slide26. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 26 Fail-Silent Waiting Causal Impl. Implements:
ReliableCausalOrderBroadcast, instance rco
Uses: ReliableBroadcast, instance rb

upon event rco, Init do
forall pi   do VC[i] := 0

upon event rco, Broadcast|m do
trigger rb,Broadcast|(DATA, VC, m)
VC[self] := VC[self] + 1
trigger rco,Deliver|self, m send m with VC VC has only increased, so RCO deliver<br>
slide27. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 27 Fail-Silent Waiting Causal Impl. (2) upon event rb,Deliver|pj, (DATA, VCm , m) do
if pj ≠ self then
pending := pending  (pj, (DATA, VCm, m))
deliver-pending()

procedure deliver-pending()
while exists x=(sm,(DATA,VCm,m))pending s.t. VCVCm do
pending := pending \ (sm, (DATA, VCm, m)
trigger rco,Deliver | sm, m
VC[ rank(sm) ] := VC[ rank(sm) ] + 1 put on hold Remove on hold deliver and increase local VC<br>
slide28. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 28 Delivery order isn’t same!
What is wrong? [d] p1 p2 b(m1) d(m1) d(m2) d(m2) d(m1) M1(0,0) m2(0,0) b(m2) Possible execution? (1,0) (1,1) (0,1) (1,1) (0,0) (0,0) Nothing, there is no causality.<br>
slide29. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 29 Other possible orderings Other common orderings
Single-source FIFO order

Total order

Causal order<br>
slide30. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 30 Single-Source FIFO order Intuitively
Msgs from same node delivered in order sent

For all messages m1 and m2 and all pi and pj,
if pi broadcasts m1 before m2, and if pj delivers m1 and m2, then pj delivers m1 before m2

Caveat
This formulation doesn’t require delivery of both messages<br>
slide31. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 31 Total Order Intuitively
Everyone delivers everything in exact same order

For all messages m1 and m2 and all pi and pj,
if both pi and pj deliver both messages, then they deliver them in the same order

Caveat
This formulation doesn’t require delivery of both messages
Everyone delivers same order, maybe not send order!<br>
slide32. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 32 a Execution Example (1) b single-source FIFO? totally ordered? causally ordered? yes no yes<br>
slide33. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 33 Execution Example (2) a b single-source FIFO? totally ordered? causally ordered? no yes no<br>
slide34. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 34 Execution Example (3) a b single-source FIFO? totally ordered? causally ordered? yes no no<br>
slide35. 3/14/13 Ali Ghodsi, alig(at)cs.berkeley.edu 35 Hierarchy of Orderings Stronger implies weaker ordering () best-effort reliable uniform reliable FIFO best-effort reliable FIFO uniform reliable FIFO causal best-effort reliable causal uniform reliable causal<br>