Byzantine Quorums and Hyperledger Fabric and a few

slide1. Byzantine Quorums and Hyperledger Fabricand a few more things on Byzantine Fault-Tolerance (BFT) and Blockchainsvirtually @EPFLNovember 16, 2020Dr. Marko VukolićIBM Research – Zurich<br>
slide2. Byzantine Fault Tolerance (BFT) 2 replica replica replica robust service replica<br>
slide3. Byzantine Fault Tolerance (BFT) Malicious behavior typically modeled with arbitrary faults

For historical reasons* arbitrary faults are also called Byzantine faults

Examples of arbitrary faults
Bugs
Intrusions
Rogue administrator(s) (e.g., in blockchain)

* L. Lamport, R. E. Shostak, M. C. Pease: The Byzantine Generals Problem. ACM Trans. Program. Lang. Syst. 4(3): 382-401 (1982) 3<br>
slide4. What is a Blockchain? A chain (sequence) of blocks of transactions
Each block consists of a number of (ordered) transactions
Blockchain establishes total order of transactions 4 #234 #235 #236 … #1 #0
Genesis block Node A Node E Node B Node D Node C Node F Consensus protocol ensures ledger replicas are identical* datastructure Network of
untrusted nodes<br>
slide5. This lecture 1st part (BFT basics)
Resilience lower bounds
BFT consensus and state-machine replication

2nd part (BFT and Blockchain in IBM)
Hyperledger Fabric
Mir-BFT<br>
slide6. Our playground 6<br>
slide7. Our playground - Model Byzantine faults
Processes that are not faulty are called correct

Asynchrony (incl. partitions, network failures)
Messages can be delayed for an arbitrary time

Concurrency
Multiple clients accessing replicated service concurrently 7<br>
slide8. How many replicas BFT needs? Assume we want to mask a threshold t of replica faults

Let’s focus on a simple case where t = 1 8<br>
slide9. How many replicas BFT needs? 9 t = 1 write(tweet) tweet = “hello” tweet = “hello” tweet = “” twitter bftwitter Model of bftwitter:
R/W Storage
(also called a R/W register)

2 operations
write(value)
read()  value

Read should return the last value written
Register is initialized to a default value<br>
slide10. How many replicas BFT needs? 10 t = 1 tweet = “hello” tweet = “hello” tweet = “” read()<br>
slide11. How many replicas BFT needs? 11 t = 1 tweet = “hello” tweet = “hello” tweet = “” read() tweet = “hello” ? tweet = “” tweet = “” tweet = “good bye” tweet = “good
bye” ?<br>
slide12. How many replicas BFT needs? To tolerate t Byzantine faults 3t replicas are not enough*
even for a simple twitter-like app (R/W storage)

As we will shortly see…
…3t+1 replicas are enough

The number of replicas may go down if we strengthen assumptions
synchrony, consistency level, etc…
We will come to that also…


* JP. Martin, L.Alvisi, M. Dahlin: Minimal Byzantine Storage. DISC 2002: 311-325 12<br>
slide13. 13 S M R State machine replication (SMR)<br>
slide14. Bad news straight away State machine replication needs consensus
Replicas need to agree on a total order of clients requests

Consensus is impossible in our playground (FLP)
Asynchrony + faults
Even if faults are crash-only
Michael J. Fischer, Nancy A. Lynch, Mike Paterson: Impossibility of Distributed Consensus with One Faulty Process J. ACM 32(2): 374-382 (1985)

Observe:
SMR is more general than a R/W storage service
R/W storage is possible in our playground (as we just saw…) 14<br>
slide15. The things are not that bad, after all Consensus is possible, despite (Byzantine) faults
Probabilistically, with probability 1
Deterministically, if the system expresses some amount of synchrony

So we change our playground
We assume eventual synchrony*
Messages cannot be delayed indefinitely

* Cynthia Dwork, Nancy A. Lynch, Larry J. Stockmeyer: Consensus in the presence of partial synchrony. J. ACM 35(2): 288-323 (1988) 15<br>
slide16. Eventual synchrony The system is asynchronous until some (unknown) point in time T

After T, the system is synchronous
messages are delivered within a (possibly unknown) period D

Assumption about the (real-world) system
Communication is not synchronous
But not completely asynchronous either
Allows us to circumvent the FLP impossibility 16<br>
slide17. BFT SMR protocols Castro/Liskov Practical BFT (PBFT)

How can we further improve it 17<br>
slide18. PBFT Seminal paper by Miguel Castro and Barbara Liskov
Practical Byzantine fault tolerance and proactive recovery. ACM Trans. Comput. Syst. 20(4): 398-461 (2002)

A BFT SMR implementation w.
eventual synchrony
3t+1 replicas (clearly optimal) 18<br>
slide19. PBFT components Common-case operation
Reconfiguration (view changes)
Garbage Collection
Recovery

This lecture focuses on the common-case (mostly) 19<br>
slide20. PBFT common case 20 client r1 (primary) r2 r3 r4 Invoke(signed_req) commit(req,reply) t = 1 If (order#,req) is commited by 2t+1 replicas
 (order#,req) is executed locally by a replica
At this point at least t+1 correct replicas agree on order Append order#
to req If (order#,req) is confirmed by 2t+1 replicas
 (order#,req) is committed locally by a replica<br>
slide21. PBFT common-case: Summary Primary/leader based
Client’s send requests to a primary replica
Primary/leader imposes the order on other replicas (a 3-phase subprotocol)
At the end, correct replicas agree on the order of reqs
Replicas execute the request
guarantees that correct replicas execute reqs in the same order

Replicas then send reply to the client
Client acts on the reply if it is confirmed by at least t+1 replicas

Designed to work when the system is synchronous
With eventual synchrony, this will eventually happen 21<br>
slide22. Why 3 phases for agreement? 22<br>
slide23. What if we had a single phase? 23 client r1 (primary) r2 r3 r4 Invoke(signed_req) t = 1 client Invoke(signed_req2) No agreement
on order! Primary assigns the same
order# to different reqs<br>
slide24. What if we had 2 phases only? 24 client r1 (primary) r2 r3 r4 Invoke(signed_req) If (order#,req) is confirmed by 2t+1 replicas
 (order#,req) is executed locally by a replica Can we really do this? Seems ok… It also did for NEO blockchain designers
A majority of correct replicas seem to agree on order…<br>
slide25. However, there are issues… 25 client r1 (primary) r2 r3 r4 Invoke(signed_req) Let’s consider t=2  i.e., we have a total of 7 replicas
Consider 2 executions, in both primary is malicious and r2 executes the request r5 r6 r7 Invoke(signed_req2) client Invoke(signed_req) Invoke(signed_req2) ex1 ex2<br>
slide26. However, there are issues… 26 client r1 (primary) r2 r3 r4 Invoke(signed_req) Only r1, r2 and r3 can distinguish ex1 from ex2
r4 and r5 receive (order#,req) in both ex1 and ex2
r6 and r7 receive (order#,req2) in both ex1 and ex2 r5 r6 r7 Invoke(signed_req2) client Invoke(signed_req) Invoke(signed_req2) ex1 ex2<br>
slide27. Issue with 2 phases Assume no other message is delivered while r1 is the primary
due to asynchrony

When this occurs, PBFT elects a new primary
Say r7 is the new primary

Moreover, in both executions
r1 and r2 become temporarily partitioned from other replicas
r3 is malicious and pretends have not received any message from r1 27<br>
slide28. Issue with 2 phases 28 client r1 (primary) r2 r3 r4 Invoke(signed_req) r7 asks all replicas: what happened while r1 was the primary? r5 r6 r7 Invoke(signed_req2) client Invoke(signed_req) Invoke(signed_req2) ex1 ex2<br>
slide29. Issue with 2 phases 29 client r1 (primary) r2 r3 r4 Invoke(signed_req) r1,r2 do not reply… (asynchrony)
r3 (malicious): I did not see anything…
r4,r5: we saw (order#,req) from r1
r6,r7: we saw (order#,req2) from r1 r5 r6 r7 Invoke(signed_req2) client Invoke(signed_req) Invoke(signed_req2) ex1 ex2<br>
slide30. Issue with 2 phases 30 client r1 (primary) r2 r3 r4 Invoke(signed_req) r5 r6 r7 Invoke(signed_req2) client Invoke(signed_req) Invoke(signed_req2) ex1 ex2 Since PBFT is a deterministic protocol r7 picks 1 of the values and assigns order# to it
In both execution this value is the same
In one of them, some correct replica (r4-r7) will execute a request different from r2<br>
slide31. PBFT - conclusions The protocol is involved
We tried to see why we need 3 phases in the common case

But it works! 31<br>
slide32. Elli Androulaki, Artem Barger, Vita Bortnikov, Christian Cachin, Konstantinos Christidis, Angelo De Caro, David Enyeart, Christopher Ferris, Gennady Laventman, Yacov Manevich, Srinivasan Muralidharan, Chet Murthy, Binh Nguyen, Manish Sethi, Gari Singh, Keith Smith, Alessandro Sorniotti, Chrysoula Stathakopoulou, Marko Vukolić, Sharon Weed Cocco and Jason Yellickhttps://dl.acm.org/doi/10.1145/3190508.3190538 a Distributed Operating System for Permissioned Blockchains<br>
slide33. What is a Blockchain? A chain (sequence, typically a hash chain) of blocks of transactions
Every block consists of a number of (ordered) transactions
Blockchain establishes total order of transactions 33 #234 #235 #236 … #1 #0
Genesis block Node A Node E Node B Node D Node C Node F Consensus protocol ensures ledger replicas are identical* datastructure Network of
untrusted nodes<br>
slide34. Blokchain transactions and distributed applications Bitcoin transactions
simple virtual cryptocurrency transfers

Transactions do not have to be simple nor related to cryptocurrency
Distributed applications
smart contracts (Ethereum) or chaincodes (Hyperledger Fabric)

A smart contract is an event driven program, with state,
which runs on a replicated, shared ledger [Swanson2015]
“Smart contract”  (replicated) state machine 34<br>
slide35. Are Blockchains the same as SMR? SMR = State-Machine Replication [Lamport 78, countless follow-up papers]
Well, not really…

The main difference 35 SMR approach

single trusted application Blockchain smart-contracts

Multiple applications

Not (necessarily) trusted!
Developed by third party application developers<br>
slide36. Blockchain evolution (2009-present) 36 2009 Blockchain 1.0 A hard-coded cryptocurrency application
Limited stack-based scripting language
Native cryptocurrency (BTC)
Proof-of-Work-consensus
Permissionless blockchain system 2014 Blockchain 2.0 Distributed applications (smart contracts)
Domain-specific language (Solidity)
Native cryptocurrency (ETH)
Proof-of-Work-consensus
Permissionless blockchain system 2017 Blockchain 3.0 Distributed applications (chaincodes)
Different general-purpose languages (e.g., golang, Java)
No native cryptocurrency
Modular/pluggable consensus
Permissioned blockchain system
Multiple instances/deployments<br>
slide37. Blockchain SOTA follows order-execute architecture Order transactions using Proof-of-Work (PoW) or BFT consensus
Execute transactions at each node
Order/execute architecture is found in many SMR systems
Active state machine replication [Schneider90]
Paxos and co. and vast majority of BFT 37<br>
slide38. Order-Execute in Permissionless Blockchains Step 1: PoW block “mining”

Step 2: Gossip block #237 across the network

Step 3: Execution (tasks at every miner )
Execute transactions in the block
Verify hash of Block #237 < DIFFICULTY 38 #234 … #235 #236 A =hash of block #236
B = Root hash of
Merkle tree of tx
hashes
C = nonce

Block #237 Transactions
(payload) Miner tasks
Pre-execute transactions in the block
Find nonce such that
h: hash of Block #237
h = SHA256(A||B||C) < DIFFICULTY #234 … #235 #236 A =hash of block #236
B = Root hash of
Merkle tree of tx
hashes
C = nonce

Block #237 Transactions
(payload) miner<br>
slide39. Order-Execute in Permissioned Blockchains 39 Node A (leader) Node B Node C Node D Tx1 Tx2 Tx3 Tx4 Block #237 … #234 #235 #236 Tx1 Tx2 Tx3 Tx4 Block
#237 example:
PBFT [Castro/Liskov02] Execute tx<br>
slide40. Hyperledger Fabric – key requirements No native cryptocurrency

Ability to code distributed apps in general-purpose languages

Modular/pluggable consensus 40 Satisfying these requirements required
a complete overhaul of the (permissioned) blockchain design! end result
Hyperledger Fabric v1+
http://github.com/hyperledger/fabric
Open source, Apache 2.0 license<br>
slide41. Blockchain 2.0 (ORDER  EXECUTE) architecture issues Sequential execution of smart contracts
long execution latency blocks other smart contracts, hampers performance
DoS smart contracts (e.g., infinite loops)
How Blockchain 2.0 copes with it:
Gas (paying for every step of computation))
Tied to a cryptocurrency
Non-determinism
Smart-contracts must be deterministic (otherwise – state forks)
How Blockchain 2.0 copes with it:
Enforcing determinism: Solidity DSL, Ethereum VM
Cannot code smart-contracts in general-purpose language (Java, golang, etc)
Inflexible consensus: Consensus protocols are hard-coded
Confidentiality of execution: all nodes execute all smart contracts
Inflexible trust models: consensus trust model becomes also application trust model 41<br>
slide42. Existing blockchains’ architecture

input tx tx against smart contracts
Hyperledger Fabric v1 architecture
EXECUTE  ORDER  VALIDATE Hyperledger Fabric v1 architecture in one slide 42 Application consists of two components:
Chaincode (execution code)
Endorsement policy (validation code)<br>
slide43. Hyperledger Fabric v1 Transaction flow client (C) endorsing
peer (EP1) endorsing
peer (EP2) endorsing
peer (EP3) Ordering service (consensus) orderers Simulate tx execution
Produce r/w sets
Sign TX-ENDORSED Collect endorsement
(“sufficient” no. of
TX-ENDORSED Msgs) 1 2 4 3 1 <PROPOSE, clientID, chaincodeID, txPayload, timestamp, clientSig> 2 <TX-ENDORSED, peerID, txID, chaincodeID, readset, writeset> 3 BROADCAST(blob) 4 DELIVER(seqno,prevhash,block) Total order semantics
(ordering service) broadcast(endorsement)<br>
slide44. Hyperledger Fabric v1 Transaction flow client (C) endorsing
peer (EP1) endorsing
peer (EP2) endorsing
peer (EP3) Ordering service (consensus) orderers Simulate tx execution
Produce r/w sets
Sign TX-ENDORSED Collect endorsement
(“sufficient” no. of
TX-ENDORSED Msgs) 1 2 4 3 1 <PROPOSE, clientID, chaincodeID, txPayload, timestamp, clientSig> 2 <TX-ENDORSED, peerID, txID, chaincodeID, readset, writeset> 3 BROADCAST(blob) 4 DELIVER(seqno,prevhash,block) (committing)
peer (CP4) (committing)
peer (CP5) 4 Total order semantics
(ordering service) broadcast(endorsement)<br>
slide45. Hyperledger Fabric v1 Transaction flow client (C) endorsing
peer (EP1) endorsing
peer (EP2) endorsing
peer (EP3) Ordering service (consensus) orderers Simulate tx execution
Produce r/w sets
Sign TX-ENDORSED Collect endorsement
(“sufficient” no. of
TX-ENDORSED Msgs) 1 2 4 3 1 <PROPOSE, clientID, chaincodeID, txPayload, timestamp, clientSig> 2 <TX-ENDORSED, peerID, txID, chaincodeID, readset, writeset> 3 BROADCAST(blob) 4 DELIVER(seqno,prevhash,block) (committing)
peer (CP4) (committing)
peer (CP5) 4 Total order semantics
(ordering service) broadcast(endorsement)<br>
slide46. Hyperledger Fabric v1 Transaction flow client (C) endorsing
peer (EP1) endorsing
peer (EP2) endorsing
peer (EP3) Ordering service (consensus) orderers Simulate tx execution
Produce r/w sets
Sign TX-ENDORSED 1 2 4 3 1 <PROPOSE, clientID, chaincodeID, txPayload, timestamp, clientSig> 2 <TX-ENDORSED, peerID, txID, chaincodeID, readset, writeset> 3 BROADCAST(blob) 4 DELIVER(seqno,prevhash,block) (committing)
peer (CP4) (committing)
peer (CP5) 4 Validate(endorsement,
End. Policy)
Validate(readset vers)
Commit tx Validate(endorsement,
End. Policy)
Validate(readset vers)
Commit tx Total order semantics
(ordering service) Sufficiently enough to satisfy
Endorsement
Policy (EP) Collect endorsement
(“sufficient” no. of
TX-ENDORSED Msgs) local FS
+<br>
slide47. Distributed applications in Fabric Execution code (a.k.a. chaincode)
Execute untrusted chaincode before consensus
Non-deterministic chaincode tolerated
EXECUTEORDERVALIDATE: non-deterministic tx are not guaranteed to be live
ORDEREXECUTE: non-deterministic tx are not guaranteed to be safe (forks)
Validation code (a.k.a. endorsement policy)
Deterministic(!), executed post-consensus
Deployed by a set of administrators (e.g., majority of nodes on the network)
Instantiated by chaincode
Examples
K out of N chaincode endorsers need to endorse a tx
Alice OR (Bob AND Charlie) need to endorse a tx
Fabcoin – Bitcoin-inspired UTXO authority-minted cryptocurrency for Fabric
Customized validation code 47 Fabric mixes
passive and active replication
into hybrid replication<br>
slide48. Mir-BFT (https://arxiv.org/pdf/1906.05552.pdf)
is a robust BFT total order protocol
with highest throughput on WANs
with up to 100 (and probably more) nodes

Suitable for Permissioned (e.g., Hyperledger Fabric)
and Proof-of-Stake Permissionless Blockchains What BFT for Fabric ordering service?<br>
slide49. Understanding bottlenecks: PBFT – Common case protocol … #234 #235 #236 Tx1 Tx2 Tx3 Tx4 Block
#237 [Castro/Liskov02] Deliver batch Pre-prepare Node 0 (Primary) Node 1 Node 2 Node 3 Prepare Commit<br>
slide50. Understanding bottlenecks: PBFT – Common case protocol … #234 #235 #236 Tx1 Tx2 Tx3 Tx4 Block
#237 [Castro/Liskov02] Deliver batch Pre-prepare Node 0 (Primary) Node 1 Node 2 Node 3 Prepare Commit Small messages Very large message Batch #237 Batch #237<br>
slide51. Understanding bottlenecks: PBFT<br>
slide52. node 0 node 1 node 2 node 3 Mir: Robust Scaling of Classical BFT 52 PBFT: Single leader Mir: Parallel leaders<br>
slide53. Main Challenge with Multiple Leaders: Request duplication Node 0 Node 1 Node 2 Node 3 CLIENT Tx1 Tx1 1 2 … #1 #2<br>
slide54. Mir BFT Total Order Broadcast protocol: The main principles Multiple leaders
Have multiple leaders propose requests in parallel (vs PBFT single leader)
Sharding block/batch sequence numbers across leaders, multiplexing several single leader instances
Prevents request duplication
Prevent duplicates using rotating assignment of partitioned hashspace
Incrementally built on PBFT/Aardvark
Mir is a strict generalization of PBFT/Aardvark: Critical for easier reasoning about correctness
Changes only to PBFT/Aardvark leader(s) election part
Asynchronous* & optimally resilient: n=3f+1 nodes in total up to f can be Byzantine
Implementation & Optimizations
Parallelized networking and implementation
Maintain multiple gRPC connections between each pair of nodes (OSNs)
Signature verification sharding (SVS) optimization
make sure at least one correct node verifies client sig, as opposed to all nodes)<br>
slide55. Preventing Transaction Duplication Mir Transaction Sharding principles
Partition transactions into “buckets” using a cryptographic hash function H
Each transaction is mapped to a unique bucket

Each leader has a different active bucket
Leader proposes only transactions from an active bucket

Periodic active bucket re-assignment (“rotation”)<br>
slide56. Preventing Transaction Duplication: Bucket assignement Mir Transaction Sharding principles
Partition transactions into “buckets” using a cryptographic hash function H
Each transaction is mapped to a unique bucket

Each leader has a different active bucket
Leader proposes only transactions from an active bucket

Periodic active bucket re-assignment (“rotation”) 0 FF…F #12 #13 hashspace #12 #13 #12 #12 #13 #13 #13 #12 Nodes
(leaders) Clients’
requests
(txs) Active bucket at a node Inactive bucket at a node<br>
slide57. Preventing Transaction Duplication: Bucket Rotation Mir Transaction Sharding principles
Partition transactions into “buckets” using cryptographic hash function H
Each transaction is mapped to a unique bucket

Each leader has a different active bucket
Leader proposes only transactions from an active bucket

Periodic active bucket re-assignment (“rotation”) 0 #12 #13 hashspace Nodes
(leaders) Clients’
requests
(txs) Active bucket at a node Inactive bucket at a node #13 #13 #13 #13 FF…F<br>
slide58. Growing and Shrinking Leader Set (see paper for details) Primary of an epoch announces (reliably broadcasts) the epoch leader set
Subject to constraints
Stable epoch: Number of leaders equal to number of nodes
Unbounded epoch duration, moving to recovery epoch only in case of faults/partitions
Periodic bucket rotation (to protect against censorship attacks)
Recovery epoch: Number of leaders is smaller than n
Recovery epoch is limited duration (measured in number of blocks)
Gracious epoch change
If “Things are good” (a complete recovery epoch e)
Then number of leaders in epoch e+1 does not reduce
Leader set grows if the new epoch primary perceives more nodes as alive (until we reach a stable epoch)
Ungracious epoch change
Only in case of faults/partitions
The size of the leader set reduces<br>
slide59. Signature Verification Sharding (SVS) optimization Batch #237 Batch #237 Validators for #237 59 Node 0 Node 1 Node 2 Node 3 Client Tx1<br>
slide60. Performance Evaluation Setup IBM Cloud (Softlayer)
Baselines
PBFT/Aardvark (on Mir codebase for fair comparison)
Chain (best-case, fault-free only, optimistic Chain BFT replication), Aublin et al. TOCS 2015
Honeybadger (HB) BFT (CCS 2016)
Libra HotStuff (PODC 2019)
WAN
(up to) 16 datacenters across the world
32 vCPU / 32 GB RAM 2.0Ghz VMs
1Gbps links nominal full duplex bandwidth
Up to 100 nodes
500 byte transactions (Bitcoin size) & 3500 bytes transactions (Fabric size)
LAN (see the paper) 1 3 2 Chain<br>
slide61. WAN Scalability (3500B requests – typical Hyperledger Fabric tx size)<br>
slide62. WAN throughput scalability (Bitcoin size txs. - 500 bytes)<br>
slide63. Impact of Duplication Prevention (500B requests)<br>
slide64. Latency, Impact of Bucket Rotation and SVS (16 nodes, WAN, 500B) typical Mir-BFT latency<br>
slide65. Summary Mir-BFT
Excellent scalability (at least up to 100 nodes)
Best performance to date on WANs
Robust to performance attacks

Next steps: Mir-BFT as a production-ready BFT library
watch https://github.com/IBM/mirbft The paper is available on arXiV
https://arxiv.org/pdf/1906.05552.pdf<br>
slide66. Thank you!<br>