ioSnap TM Sriram Subramanian Swami Sundararaman Nisha Talagala Andrea ArpaciDusseau Remzi ArpaciDusseau Copyright 2014 Fusionio Inc All rights reserved ID: 405693
Download Presentation The PPT/PDF document "Snapshots in a Flash with" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Snapshots in a Flash with ioSnapTM
Sriram Subramanian, Swami Sundararaman, Nisha Talagala,Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau
Copyright © 2014 Fusion-io, Inc. All rights reserved.Slide2
10
10
Snapshots OverviewPoint-in-time representation of the state of a storage device
Used for backup/recovery, audit trails
Implemented at different levels
Device, block layer, LVM, file systems, applications, databases
Use Copy-on-Write (CoW) or Redirect-on-Write (RoW)
Copyright © 2014 Fusion-
io, Inc. All rights reserved.
2
10
10
10
Existing Data
Update Data
Existing Data
Existing Data
Update DataSlide3
Why Rethink Snapshots for Flash?
Flash is revolutionizing storage systems Accelerating data centers, enterprise apps, desktopsNatural fit for supporting snapshots
Redirect-on-Write: never overwrite existing dataLog-structuring: data ordered on their creation time (almost)
Rate of data-change is higher for Flash devices
e
.g., multi-threaded 8KB IOs, TB capacity device
Flash
: 30K IOPS and device fills in ~
1 hourHDD : 500 IOPS and device fills in ~3 days
Copyright © 2014 Fusion-io, Inc. All rights reserved.
3Slide4
Block Driver
Why Implement Snapshots in Block Layer?
Block Driver
Disk-based Systems
Free space
Namespace
journaling
Mapping
File Systems
Free space
Atomic write
Mapping
Dynamic
provisioning
Flash-based Systems
Block Driver
Free space
Namespace
journaling
Mapping
File Systems
Copyright © 2014 Fusion-io, Inc. All rights reserved.
4Slide5
Snapshots in Flash Challenges
Users are sensitive to performance variabilityNeed predictable performance all the time NAND flash has low endurance & inefficient in-place writesIn-place updates of reference counts not possibleData could be moved due to a variety of reasons
Cannot waste storage space for storing snapshot metadata$/GB is high and need to keep costs low
Copyright © 2014 Fusion-io, Inc. All rights reserved.
5Slide6
ioSnapTM Overview
First flash-aware snapshotting systemLeverages RoW in FTL to support snapshot operations
Proposes usage of epochs in FTL to maintain log-time orderingSupports large number of
writable
snapshots (2
16
)Embraces rate-limiting
to minimize performance implicationsPerformance Results
(prototype built into Fusion-
io VSL driver)Instantaneous snapshot creation/deletion (~
50usec, 4k metadata)Matches vanilla read/write performance numbers Provides predictable performance for foreground IOs
Copyright © 2014 Fusion-io, Inc. All rights reserved.6Slide7
Outline
IntroductionioSnapTM DesignEvaluation
Conclusions Copyright © 2014 Fusion-io, Inc. All rights reserved.7Slide8
Design Goals and Assumptions
GoalsNegligible impact on foreground performancePredictable foreground performanceMinimal space overheads for snapshot metadataIntegrate with existing FTLs
AssumptionsSnapshots are primarily used for backup, disaster recoveryCreates/deletes are common operationsAccesses/activates are rare operations
Copyright © 2014 Fusion-io, Inc. All rights reserved.
8Slide9
Relevant components of Fusion-
io VSLTM
Organizes NAND Flash into segments & maintains a log NAND Flash Modules
Mapping Layer (FTL)
LBA NAND ADDR
Free Space Mgmt.
Validity of blocks
Segment cleaner
01001100
00000000
Segment 1
Segment N
Validity Bitmap
Segment N
Segment 1
Virtual Storage Layer (VSL
T
M
)
Crash Recovery
Reconstruct
forward and validity map
Copyright © 2014 Fusion-io, Inc. All rights reserved.
9Slide10
Segment Boundary
Creating / Deleting Snapshots in Flash
Key concept:
Infinite size log
S
1
S
2
S
3
S
N
S
napshot
Deletion is similar to creation – simply write a deletion note
C
reation is just writing a snapshot create note in the log
Copyright © 2014 Fusion-io, Inc. All rights reserved.
10
Leverage time ordering of data in a log to create snapshots
Segment 1
Segment 2
Segment N
Snapshot creation/deletion is fast &
negligible (fixed) metadataSlide11
Well What About Segment
Cleaner?
10
20
30
10
40
60
S1
Active
Segment Boundary
Snapshot
10
20
30
10
40
60
Active
70
S1
Active
S1
Active
10
1
10
2
40
2
60
2
Active
S1
Active
S1
Active
40
2
Epoch 1
Epoch 2
LBA
Epoch
60
2
10
1
20
1
70
2
30
1
10
2
20
1
30
1
Epoch:
notion of period of time
Copyright © 2014 Fusion-io, Inc. All rights reserved.
11
Epochs help preserve log-time ordering Slide12
Managing Liveness of blocks
Issue: snapshots indirectly increase the reference countValidity bitmap with a single bit doesn’t workPossible solutions: Maintain more bits/block (216 snapshot implies 16x increase in bitmaps
)Selectively maintain per sub-segment bitmap for snapshotsOnly create a bitmap if a snapshot has (or modified) data in it Insight: determine if a given block has at least
one
reference to it
Copyright © 2014 Fusion-io, Inc. All rights reserved.
12Slide13
1
0
0
0
1
1
1
1
1
0
0
0
1
1
1
1
Preserving
L
iveness
Via CoW Validity
B
itmaps
Epoch 1
Segment Boundary
Snapshot
10
1
20
1
30
1
40
1
60
1
1
0
0
0
1
1
1
1
Epoch 1
10
1
20
1
30
1
40
1
60
1
10
2
1
0
0
0
1
1
1
1
1
1
0
0
0
1
1
1
Bits needed to be flipped
Validity Map
CoW
Epoch 2
Epoch 1
Epoch 1
Epoch 2 (step1)
Epoch 2 (step2)
Copyright © 2014 Fusion-io, Inc. All rights reserved.
13Slide14
Snapshot-Aware Segment Cleaner
Epoch 1
10
1
20
1
30
1
10
1
60
1
1
0
0
0
0
1
1
1
Epoch 1
10
2
60
2
70
2
0
1
1
1
0
1
1
0
Epoch 2
Merge Bitmaps
0
1
1
1
Valid Block
Invalid
Block
Epoch 2
Epoch 1
60
1
1
0
0
0
1
1
1
Epoch 1
10
2
60
2
70
2
0
1
1
1
1
1
0
Epoch 2
Epoch 2
20
1
30
1
10
1
Cleaned Segment
Epoch 1
Copyright © 2014 Fusion-io, Inc. All rights reserved.
14
Segment Boundary
Snapshot
Segment cleaner preserves log-time ordering Slide15
Design Summary
Leverage implicit time ordering in the LogEpochs preserve log-time ordering even with a cleanerSub-segment-level bitmaps to track validity of blocksLazily update reference counts for snapshotted blocksSnapshot-aware cleaner preserves log-time and block validitySnapshot management
Background snapshot activationRate-limiting to provide predictable foreground performance Copyright © 2014 Fusion-io, Inc. All rights reserved.
15Slide16
Outline
IntroductionioSnapTM Design
EvaluationConclusions Copyright © 2014 Fusion-io, Inc. All rights reserved.
16Slide17
Evaluation
What’s the impact on user IOs in the absence of snapshots?Snapshot creation/deletion time? Implications on user IO?How does it compare with existing snapshotting systems?Implications of a snapshot-aware segment cleaner?
How long does it talk to activate a snapshot?Implications on the crash recovery mechanism?
Setup:
quad core
I
ntel i7 processor, 1.2TB NAND flash, 12GB RAM, Linux 2.6.35, older generation of Fusion-
io VSL driver, 4K Block sizes
Copyright © 2014 Fusion-io, Inc. All rights reserved.
17Slide18
Performance of Regular operations
Experiment: 4K read/writes using two threads. 16GB of data read/written. Asynchronous writes to device.
Snapshot support does not impact regular operationsCopyright © 2014 Fusion-io, Inc. All rights reserved.
18
Bandwidth (MB/s)Slide19
Snapshot creation / Deletion
Always 50us latency and 4K metadata (independent of data
)Impact on foreground operations?
W
rite
512 byte blocks to arbitrary logical addresses (total data of 3GB)
.
Create a snapshot
and then continue writing data to random locations (of 8 MB).
Copyright © 2014 Fusion-io, Inc. All rights reserved.
19
Validity bitmap CoWSlide20
Comparison with BTRFS (1)
Impact on foreground latency upon snapshot creationAround 8 GB of sequential writes followed by a random workload interspersed by a snapshot every 5 sec
Snapshots in ioSnap does not impact foreground latency
Copyright © 2014 Fusion-io, Inc. All rights reserved.
20Slide21
Comparison with BTRFS (2)
Impact of snapshots on sustained bandwidthAround 200 GB of sequential writes followed by a random workload interspersed by a snapshot once every 15 sec
Snapshots in ioSnap does not impact sustained bandwidth
Copyright © 2014 Fusion-io, Inc. All rights reserved.
21Slide22
Outline
IntroductionioSnapTM
DesignEvaluationConclusions
Copyright © 2014 Fusion-io, Inc. All rights reserved.
22Slide23
Conclusions
“Make everything as simple as possible, but not simpler.” – Albert EinsteinFlash is revolutionizing the storage industryRethink data services to leverage flash’s capability & performance
ioSnap: first flash-aware snapshotting systemLeverages RoW capability in FTLs to implement snapshotsProposes usage of
epochs
to preserve log-time ordering
L
ow
-overhead instantaneous snapshots
(performance & storage)Embraces
rate-limiting to minimize impact to foreground traffic Activations are slow & can be 10s of sec for a TB size snapshot
Copyright © 2014 Fusion-io, Inc. All rights reserved.
23Slide24