Coding for Modern Distributed Storage Systems:
Description: Coding for Modern Distributed Storage Systems: Part 1. Locally Repairable Codes Parikshit Gopalan Windows Azure Storage, Microsoft. The data deluge Problem Statement: We generate insane amounts of digital data. We expect it to be stored
Related Topics
Download Presentation
"Coding for Modern Distributed Storage Systems:" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Coding for Modern Distributed Storage Systems: Part 1. Locally Repairable Codes Parikshit Gopalan
Windows Azure Storage, Microsoft.<br>
slide2. The data deluge … Problem Statement:
We generate insane amounts of digital data.
We expect it to be stored reliably and accessible anytime, anywhere.
And for free!
Total data in the cloud is of the order of few hundred Exabytes
A terabyte hard drive costs $100.
Even storing raw data costs hundreds of millions.
Hardware is no longer cheap.
Currently, data centers consume up to 3 percent of all global electricity production while producing 200 million metric tons of carbon dioxide.<br>
slide3. Data Storage: the basics Goal: Tolerate one disk failure.
Duplication
2X overhead.
Quick recovery.
Simple XOR [RAID5]
Treat each disk as a bit vector.
1.2X overhead.
Slower recovery.<br>
slide4. Data Storage: the basics Goal: Tolerate two disk failures.
Triplication
3X overhead.
Quick recovery.
[6,4] Reed Solomon Code [RAID6]
1.5X overhead.
Slower recovery.
Need a larger field: each disk is a byte-vector.<br>
slide5. Data storage: the basics<br>
slide6. Data storage: the basics<br>
slide7. Degraded Reads Typical failure scenario: a single disk fails or is prohibitively slow.<br>
slide8. Degraded Reads Typical failure scenario: a single disk fails or is prohibitively slow.<br>
slide9. Disk Failure Typical failure scenario: a single disk fails and needs to be replaced.<br>
slide10. Can we do better? Regenerating Codes [Dimakis-Godfrey-Wu-Wainwright-Ramachandran’10]
Metric: Network bandwidth.
Optimize the amount of data communicated to repair a single lost node. Locally Repairable Codes [G.-Huang-Simitci-Yekhanin’12]
Metric: Number of disk reads.
Optimize the number of disk reads needed to repair a single lost node.<br>
slide11. Part 1 of this Tutorial: LRCs Part 1.1: Locality
Locality of codeword symbols.
Rate-distance-locality tradeoffs.
Part 1.2: Reliability
Beyond minimum distance: Maximum recoverability.
Constructions of Maximally Recoverable LRCs.<br>
slide12. Locality [Chen-Huang-Li’07, Oggier-Datta’11, G.-Simitci-Huang-Yekhanin’12, Papailiopoulos-Luo-Dimakis-Huang-Li’12]<br>
slide13. Locally Decodable/Testable Codes<br>
slide14. Codes with data locality<br>
slide15. Codes with all-symbol locality<br>
slide16. Rate-distance-locality tradeoffs<br>
slide17. Rate-distance-locality tradeoffs<br>
slide18. Rate-distance-locality tradeoffs<br>
slide19. Rate-distance-locality tradeoffs<br>
slide20. Explicit codes with all-symbol locality.<br>
slide21. Rate-distance-locality tradeoffs<br>
slide22. Generalizations Non-linear codes
[Papailiopoulos-Dimakis, Forbes-Yekhanin].
Vector codes
[Papailoupoulos-Dimakis, Silberstein-Rawat-Koyluoglu-Vishwanath, Kamath-Prakash-Lalitha-Kumar]
Codes over bounded alphabets
[Cadambe-Mazumdar]
Codes with short local MDS codes
[Prakash-Lalitha-Kamath-Kumar, Silberstein-Rawat-Koyluoglu-Vishwanath]
Codes with local Regeneration
[Silberstein-Rawat-Koyluoglu-Vishwanath, Kamath-Prakash-Lalitha-Kumar…]<br>
slide23. Towards locally decodable codes<br>
Windows Azure Storage, Microsoft.<br>
slide2. The data deluge … Problem Statement:
We generate insane amounts of digital data.
We expect it to be stored reliably and accessible anytime, anywhere.
And for free!
Total data in the cloud is of the order of few hundred Exabytes
A terabyte hard drive costs $100.
Even storing raw data costs hundreds of millions.
Hardware is no longer cheap.
Currently, data centers consume up to 3 percent of all global electricity production while producing 200 million metric tons of carbon dioxide.<br>
slide3. Data Storage: the basics Goal: Tolerate one disk failure.
Duplication
2X overhead.
Quick recovery.
Simple XOR [RAID5]
Treat each disk as a bit vector.
1.2X overhead.
Slower recovery.<br>
slide4. Data Storage: the basics Goal: Tolerate two disk failures.
Triplication
3X overhead.
Quick recovery.
[6,4] Reed Solomon Code [RAID6]
1.5X overhead.
Slower recovery.
Need a larger field: each disk is a byte-vector.<br>
slide5. Data storage: the basics<br>
slide6. Data storage: the basics<br>
slide7. Degraded Reads Typical failure scenario: a single disk fails or is prohibitively slow.<br>
slide8. Degraded Reads Typical failure scenario: a single disk fails or is prohibitively slow.<br>
slide9. Disk Failure Typical failure scenario: a single disk fails and needs to be replaced.<br>
slide10. Can we do better? Regenerating Codes [Dimakis-Godfrey-Wu-Wainwright-Ramachandran’10]
Metric: Network bandwidth.
Optimize the amount of data communicated to repair a single lost node. Locally Repairable Codes [G.-Huang-Simitci-Yekhanin’12]
Metric: Number of disk reads.
Optimize the number of disk reads needed to repair a single lost node.<br>
slide11. Part 1 of this Tutorial: LRCs Part 1.1: Locality
Locality of codeword symbols.
Rate-distance-locality tradeoffs.
Part 1.2: Reliability
Beyond minimum distance: Maximum recoverability.
Constructions of Maximally Recoverable LRCs.<br>
slide12. Locality [Chen-Huang-Li’07, Oggier-Datta’11, G.-Simitci-Huang-Yekhanin’12, Papailiopoulos-Luo-Dimakis-Huang-Li’12]<br>
slide13. Locally Decodable/Testable Codes<br>
slide14. Codes with data locality<br>
slide15. Codes with all-symbol locality<br>
slide16. Rate-distance-locality tradeoffs<br>
slide17. Rate-distance-locality tradeoffs<br>
slide18. Rate-distance-locality tradeoffs<br>
slide19. Rate-distance-locality tradeoffs<br>
slide20. Explicit codes with all-symbol locality.<br>
slide21. Rate-distance-locality tradeoffs<br>
slide22. Generalizations Non-linear codes
[Papailiopoulos-Dimakis, Forbes-Yekhanin].
Vector codes
[Papailoupoulos-Dimakis, Silberstein-Rawat-Koyluoglu-Vishwanath, Kamath-Prakash-Lalitha-Kumar]
Codes over bounded alphabets
[Cadambe-Mazumdar]
Codes with short local MDS codes
[Prakash-Lalitha-Kamath-Kumar, Silberstein-Rawat-Koyluoglu-Vishwanath]
Codes with local Regeneration
[Silberstein-Rawat-Koyluoglu-Vishwanath, Kamath-Prakash-Lalitha-Kumar…]<br>
slide23. Towards locally decodable codes<br>