Lock server for distributed applications 1 Dennis Kafura CS5204 Operating Systems Overview Purpose Client synchronization coursegrain locking leader election Environment information ID: 185718
Download Presentation The PPT/PDF document "Chubby" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Chubby
Lock server for distributed applications
1
Dennis Kafura – CS5204 – Operating SystemsSlide2
Overview
PurposeClient synchronization course-grain locking
leader electionEnvironment informationnaming
metadata
Goals
High reliability/availabilityThousands of clients
Dennis Kafura – CS5204 – Operating Systems
2
leader electionclient-master discoverymaster-server discoverymetadata storage
BigTable
GFS
Chubby
storage
leader election
metadata storageSlide3
Design
Service not a library (for consensus)Easier to retrofit into evolving applications
Need environment information, not just consensusMore familiar interface (at least to their users)Eases burden of maintaining a client quorum for consensus
Services
Course grain locks (that survive failures)
Low acquisition rateHeld for considerable time
Small filesWhole file read/writeLimited to 256Kb
Consistent client-side cachingEvent notification (of changes)Access control
Dennis Kafura – CS5204 – Operating Systems3Slide4
Structure
Library provides API for client applicationsChubby cellreplicas (identical, maintain copies of database)
election of a master via consensus protocol master lease: period of time for which master is elected
can be renewed
Dennis Kafura – CS5204 – Operating Systems
4
name
contents
lock
file or name service
simple lock serviceSlide5
Files
NamingTree of directories/filesExample: /
ls/foo/wombat/pouch/
ls
: lock service
/foo: name of chubby cell
/wombat/pouch: file on named cellSub-trees may be served from different Chubby mastersFiles cannot move between directories
No directory modified timesNo path-dependent access controlNo symbolic or hard links
Does not reveal last-access timesDennis Kafura – CS5204 – Operating Systems
5Slide6
File, Handles, Sequencers
Dennis Kafura – CS5204 – Operating Systems
6
ACL
read file
write file
change file names
instance
content generation
lock generation
ACL generation
content checksum
file
metadata
mode
status (64 bit numbers)
lock
handle
refers to
sequence number
mode
check digits
sequencer
name
mode
lock generation number
returned by open()
to client
opaque byte-string
passed to server
server checks validity
generatesSlide7
Locks and Ordering
Locksmodes: read (shared), write (exclusive)
permissions: requires write permission to fileare advisory: the lock associated with a file does NOT control access to the file’s contentsOrdering
problem
after acquiring a lock and issuing a request (R1) a client may fail
another client may acquire the lock and issue its own request (R2)
R1 arrive later at the server and be acted upon (possible inconsistency)solutionclient requests sequencer for lock (sequence contains lock generation number)
client passes sequence to server along with requestserver verifies that sequencer is still validserver can maintain session with Chubby and cache sequencer
informaitonDennis Kafura – CS5204 – Operating Systems
7Slide8
Locks and election
clients in a distributed applications using Chubby may elect a primary using the Chubby interfacesteps in primary electionclients open a lock file and attempt to acquire the lock in exclusive mode
one client succeeds (becomes the primary) and writes its name/identity into the fileother clients fail (become replicas) and discover the name of the primary by reading the file
primary obtains sequencer and passes it to servers (servers can insure that the elected primary is still valid)
Dennis Kafura – CS5204 – Operating Systems
8Slide9
Operations
Dennis Kafura – CS5204 – Operating Systems
9
master
Paxos
read
...
write
election
replica
write
Paxos
replica
write
Paxos
write
write
reads:
served by master alone
creates potential performance bottleneck (
client side caching)
writes:
acknowledged after consensus by a majority of replicas
insures all (non-failed) replicas have identical databases (and can become master in case the current master fails)Slide10
Caching - Invalidation
Clients maintain cached copy for reading; server maintains list of clients with a copyOn write attempt, server invalidates cached copies
When invalidated, clients discard cached copy and acknowledgeWhen acknowledgments received, server update cache list and data (F)
Dennis Kafura – CS5204 – Operating Systems
10
cache
F
read
C1
cache
F
read
C2
cache
F
write
C3
F
C1
C2
C3
server
1. request
2. invalidate
3.discard F
4. acknowledge
5. Remove C1,C2 from list
6. update FSlide11
Leases
Lease a promise by one party to abide by an agreement for a given interval of time unless the promise is explicitly revoked
may be renewed or extended by either partyat expiration of interval, agreement no longer validUses in Chubby
master lease (role oriented) – interval during which one replica holds the right to serves as the cell master
session lease (resource oriented) – interval during which client’s cached items (handles, locks, files) are valid
Leases contribute to reliability/fault tolerance
at expiration of lease held by a failed party, the non-failed party knows the state of the resource associated with the lease
Dennis Kafura – CS5204 – Operating Systems
11Slide12
Session Leases and KeepAlive
Messages(1) lease requested by client via RPC; master accepts lease request by not returning
(2) master returns to indicate near-term expiration of lease(3) client immediately re-issues lease request to extend lease interval
Dennis Kafura – CS5204 – Operating Systems
12
1
lease M1
lease C1
3
2
4
lease M2
lease C2
6
5
(4) lease requested by client via RPC; master accepts lease request by not returning
(5) master returns
piggybacking cache invalidation notification; client discards invalidated cached objects;
(6) client re-issues lease request to extend lease interval
MASTER
CLIENTSlide13
Master lease management
client maintains local lease: a conservative estimate of master’s lease intervalif client’s local lease expires,
client disables cache (session is in jeopardy)sends “
jeopardy”
event to application
application can suspend activityclient attempts to exchange
KeepAlive messages with master during grace periodsucceed : re-enables cache; send s“safe
” event to applicationfail: discards cache; sends “expired” event to applicationmaster fail-over shown in figure
Dennis Kafura – CS5204 – Operating Systems13Slide14
Scaling
reducing communication with the mastercreate an arbitrary number of Chubby cells to divide the load of numerous separate applications
increase the lease time to process fewer KeepAlive messagesclient caching of file data, metadata, absence of files, and open handles to reduce contact with server
use protocol-conversion servers
use proxies
handles KeepAlive and read requests
introduces another point of failurepartition name space
Dennis Kafura – CS5204 – Operating Systems
14Slide15
Usage
most files are smallpopular as a name serverRPC traffic dominated by
KeepAlive
Dennis Kafura – CS5204 – Operating Systems
15