Albatross: Lightweight Elasticity in Shared - PowerPoint Presentation

Albatross: Lightweight Elasticity in Shared Storage Databases for the Cloud using Live Data Migration Sudipto Das1, Shoji Nishimura2, Divyakant Agrawal1, and Amr El Abbadi11Computer Science, UC Santa Barbara2NEC Laboratories, Japan Sponsors:

Web replacing DesktopSudipto Das {sudipto@cs.ucsb.edu} 2

3 Paradigm shift in Infrastructure Sudipto Das {sudipto@cs.ucsb.edu}

Cloud application platforms Serve thousands of applications (tenants)AppEngine, Azure, Force.comTenants are (typically)SmallSLA sensitiveErratic load patternsSubject to flash crowdsSupport for multitenancy is criticalOur focus: DBMSs serving these platformsSudipto Das {sudipto@cs.ucsb.edu}4

Optimize operating cost “Put $ in the y-axis”Traditional systems only optimize performanceIncentive to aggressively consolidate tenant databasesSudipto Das {sudipto@cs.ucsb.edu}5 Aggressive consolidation How to deal with surge in load?

Elasticity in the Database tier Database tier Sudipto Das {sudipto@cs.ucsb.edu} Load Balancer Application/Web/Caching tier 6

Live database migration Migrate a tenant (or database partition) in a live systemA critical operation to support elasticityOptimize operating costResource orchestration in multitenant systemsDifferent fromMigration between software versionsMigration in case of schema evolutionThis paper: Decoupled storage architecturesShared nothing: see our SIGMOD 2011 paper 7Sudipto Das {sudipto@cs.ucsb.edu}

Decoupled storage abstraction August 31, 2011Sudipto Das {sudipto@cs.ucsb.edu}8DBMS NodesQuery RouterController Tenant Cell (TM + DM) Tenant Cell (TM + DM) Tenant Cell (TM + DM) Cached DB State Transaction State Log Manager DBMS Node Tenant Cell Network Attached Storage (NAS) Tenant Applications

VM migration for DB elasticity One tenant-per-VM Pros: allows fine-grained load balancingCons Performance overheadPoor consolidation ratio [Curino et al., CIDR 2011]Multiple tenants in a VMPros: good performanceCons: Migrate all tenants Coarse-grained load balancingSudipto Das {sudipto@cs.ucsb.edu}9 Hypervisor VM VM VM Hypervisor VM

Live database migration Multiple tenants share the same database processShared process multitenancyMigrate individual tenants on-demand in a live systemVirtualization in the database tierStraightforward solutionStop serving tenant at the source Migrate to destinationStart serving at the destinationExpensive!10Sudipto Das {sudipto@cs.ucsb.edu}

Migration cost measures Service un-availabilityTime the tenant is unavailableNumber of failed requests Number of operations failing/transactions aborting Performance overheadImpact on response timesAdditional data transferred11Sudipto Das {sudipto@cs.ucsb.edu}

Migrating a live tenant databaseHow to ensure no transaction aborts? How to minimize performance impact?Nodes can fail during migrationHow to guarantee correctness during failures?Transaction atomicity and durabilityRecover migration after failureTransactions execute during migrationHow to guarantee serializability?Transaction correctness equivalent to normal operationSudipto Das {sudipto@cs.ucsb.edu}Why is live DB migration hard?12

Our approach: Albatross Migrate DB cache and transaction stateMake a snapshot of the cache and migrateSource continues serving transactions  destination lags sourceIteratively copy the database cacheCopy transaction state in final stepTransactions resume at destinationDestination start “warm”Low performance impact and no aborted transactionsMigrate transactions on-the-flyTransactions start at source and complete at destinationSudipto Das {sudipto@cs.ucsb.edu} 13

Normal operation Sudipto Das {sudipto@cs.ucsb.edu}14 Tenant/DB Partition Transactions Source Destination T 11 , T 12 , T 13 , … Cache 172 183 124 231 123 142 182 192 197 130 149 84

Phase I: Begin migration Sudipto Das {sudipto@cs.ucsb.edu}15 Tenant/DB Partition Transactions Source Destination Tenant/DB Partition T 11 , T 12 , T 13 , … Cache 172 183 124 231 123 142 182 192 197 130 149 84 Snapshot of the cache 172, 124, 231, 123, 142, 182, 192, 130, 84

Phase I: Begin migration Sudipto Das {sudipto@cs.ucsb.edu}16 Tenant/DB Partition Transactions Source Destination Tenant/DB Partition T 11 , T 12 , T 13 , … Track changes 172 183 124 231 123 142 182 192 197 130 149 84 123, 142, 182, 192, 130, 84 172 124 231 172 124 231

Phase I: Begin migration Sudipto Das {sudipto@cs.ucsb.edu}17 Tenant/DB Partition Transactions Source Destination Tenant/DB Partition T 11 , T 12 , T 13 , … Track changes 172 183 124 231 123 142 182 192 197 130 149 84 172 124 231 123 142 182 192 130 84 287 105

Phase II: Iterative phase Sudipto Das {sudipto@cs.ucsb.edu}18 Tenant/DB Partition Transactions Source Destination Tenant/DB Partition T 2 1 , T 2 2 , T 2 3 , … Track changes 172 183 124 231 123 142 182 192 197 149 172 124 231 123 142 182 192 130 84 287 105 -84 -130, 105, 287, 183, 142, 192, 197, 149

Phase II: Iterative phase Sudipto Das {sudipto@cs.ucsb.edu}19 Tenant/DB Partition Transactions Source Destination Tenant/DB Partition T 2 1 , T 2 2 , T 2 3 , … Track changes 172 183 124 231 123 142 182 192 197 149 172 124 231 123 142 182 192 130 84 287 105 287 105 183 197 149 212 3

Phase III: Atomic handover Sudipto Das {sudipto@cs.ucsb.edu}20 Tenant/DB Partition Transactions Source Destination Tenant/DB Partition T 31 , T 32 , T 33 , … 183 124 231 123 142 182 192 149 172 124 231 123 142 182 192 287 105 287 105 183 197 149 212 3 -172, -197, 3 , 212, 123 142, 231 T 31 , T 32 , T 33 , … 3 212

Normal operation Sudipto Das {sudipto@cs.ucsb.edu}21 Tenant/DB Partition Transactions Source Destination Tenant/DB Partition T 31 , T 32 , T 33 , … 183 124 231 123 142 182 192 149 172 124 231 123 142 182 192 287 105 287 105 183 197 149 212 3 T 31 , T 32 , T 33 , … 3 212

Correctness Safety: Exactly one owner for a tenantAtomic handover protocol: guarantees safety of ownership handoverSerializability: Transaction state copied during migration  guarantees serializabilityDurability: Changes from aborted transactions are not persistedChanges from committed transactions are never lostTransaction logs synchronized to guarantee durabilityIndependent recovery in most casesSudipto Das {sudipto@cs.ucsb.edu}22

Implementation Implemented in ElasTraS, a scalable multitenant database systemAppend-only storage layoutSeparate read and write cachesOptimistic concurrency controlStorage layout consists of SSTablesAn SSTable is a collection of blocks with in an internal indexRead cache is a set of the blocksSnapshot: Read the block ids cachedChanges maintained incrementallyTermination  copy read and write sets of transactions and flush write cache Sudipto Das {sudipto@cs.ucsb.edu} 23

EvaluationEvaluated against two baseline techniques Stop and Migrate (S&M)Stop serving tenant at sourceFlush changesMigrate control to destination and restartFlush and Migrate (F&M)Flush changes while continuing to serve tenantFinal stop and migrateEvaluated using two benchmarksTPC-C and YCSBSudipto Das {sudipto@cs.ucsb.edu}24

Sudipto Das {sudipto@cs.ucsb.edu}Experimental methodology Metadata Default transaction parameters: 10 operations per transaction 80% Read, 15% Update, 5% Inserts Workload: 12K Txns Hardware: 2.4 Ghz Intel Core 2 Quads, 8GB RAM, 7200 RPM SATA HDs with 32 MB Cache, Gigabit ethernet System Controller Migrate Default DB Size: 1 GB Default Cache size: 250 MB 25

Results overview Unavailability windowAlbatross: 300-800msS&M: 2-4 second unavailability, F&M: 200-500msFailed requestsAlbatross: ZeroS&M and F&M: hundredsIncrease in transaction latencyAlbatross: 15-30%Negligible performance impact during migrationS&M and F&M: 200-400% increase in latencyData transferred: Albatross:1.3-1.6 times database cacheS&M and F&M: approximately the size of the cacheSudipto Das {sudipto@cs.ucsb.edu} 26

Impact on latencySudipto Das {sudipto@cs.ucsb.edu} 27Minimal impact on latency as a result of migration

Unavailability windowSudipto Das {sudipto@cs.ucsb.edu} 28

Failed requestsSudipto Das {sudipto@cs.ucsb.edu} 29

Adversarial scenarioSudipto Das {sudipto@cs.ucsb.edu} 30Working set does not fit in cache

Adversarial scenarioSudipto Das {sudipto@cs.ucsb.edu} 31Continues to have minimal performance impact

More experimentsSudipto Das {sudipto@cs.ucsb.edu} 32

HighlightsLive database migration critical for lightweight elasticity as a first class notion Albatross – low cost live migration for shared storage architecturesNo transactions aborted  No service interruption<15% increase in transaction latency immediately after migrationGuaranteed safety in the presence of failuresSudipto Das {sudipto@cs.ucsb.edu}33

Thank you! sudipto@cs.ucsb.eduhttp://www.cs.ucsb.edu/~sudiptohttp://www.cs.ucsb.edu/~dsl

Back-up Sudipto Das {sudipto@cs.ucsb.edu}35

Unused resources Challenge: Lightweight Elasticity Provisioning on-demand and not for peak Optimize operating cost! Traditional Infrastructures Deployment in the Cloud Demand Capacity Time Resources Demand Capacity Time Resources Slide Credits: Berkeley RAD Lab 36 Sudipto Das {sudipto@cs.ucsb.edu}

Albatross Sudipto Das {sudipto@cs.ucsb.edu}37 Tenant/DB Partition Persistent Image DBMS Node Source Destination Tenant/DB Partition Cached DB State Transaction State Cached DB State Transaction State

AlbatrossAugust 31, 2011 Sudipto Das {sudipto@cs.ucsb.edu}38 Finalize Migration Stop serving Tenant at N srcSynchronize cacheMigrate transaction stateTransfer ownership to N dst Ownership Source ( N src ) Destination ( N dst ) Time 1. Begin Migration 2 . Iterative Copying 3. Atomic Handover Synchronize and Catch-up Track changes to DB State at N src Iteratively synchronize state changes Initiate Migration Snapshot cache at N src Initialize tenant at N dst N src continues executing transactions Steady State Steady State

Impact on latencySudipto Das {sudipto@cs.ucsb.edu} 39

Impact on throughputSudipto Das {sudipto@cs.ucsb.edu} 40Minimal impact on throughput both during and after migration