●ȃvԂa– Mऊih– - PDF document

●ȃvԂa– Mऊih– NTT C༐er Sጂce LȐᘗ 9 Avgvᘅ, 2010 Shᄑጠचℇ​sh਋ငhᄠ shoਂgᄇsyshem Shᄑጠचℇ​sh਋ငhᄠ shoਂgᄇsyshem ⌉r QEMU ⌉r QEMU Cओyrighh © 2010 NTT Cऊጉr–hiन I––S e⠪ଊonmenh Mअiv–hiन Mअiv–hiन VMs Hosh m–chines There is no open sovrce shorȚe syshem which fihs for ⤂Ȓ environmenh liae Amȃon EBS B⸉ca level vo⸄mes - Sc–l–bilihy - ReliȐilihy - M–nȚeȐiliԏ Reqviremenhs for shorȚe syshem Cओyrighh © 2010 NTT Cऊጉr–hiन 㐟༇Ȩअhᄊ shor–gᄇs༖hᄢ? 㐟༇Ȩअhᄊ shor–gᄇs༖hᄢ? Why noh SAఇshoਂge? L–rge ጊओriehȊ༇ᘅऊ–ge ᘏᘅem iᘇhउ e㘓enᘋ⨑ Sh–re ᘅऊȚe coЮ ထ – ᘋ⠚le ጉi⠅ ण fȋlvre Why noh dishribvhed file syshems? (e.g. Ceph, ᔄsher) Cढጮe㘇cनfigЊ–ԋन Ȑऄԇ clvᘅer memထrᘟip Sᤌ shor–ge FC sⴋhch VMs Dishribvhed file syshem ehherneh swihch VMs Hosh m–chines d–h– ser⨑rs meh–〠–h– ser⨑rs Hosh m–chines Cओyrighh © 2010 NTT Cऊጉr–hiन Shᄑጠच Shᄑጠच Ehhernᄅ swihᐟ Sc–lȐilihy ScȮes ho 1000 nodes ࠂnȚeȐilihy Avhनढऄs Dyn–mic mᄢbᄊship Adv–nced vमvmᄇ m–nipvl–hion Re⸋Ȑi⸋hy D–Ԃ replicȅion No SPOF ●–eevhav܇MoriNhTrNCioMNo–rM hἑ਑ iᘇno cenԊ–l ⠉de svch –s – meԂ-d–Ԃ seਪer Vࠖ Hखh m–ᐟinᄖ Cओyrighh © 2010 NTT Cऊጉr–hiन Dᄖign: nअ gᄨᄊ–l filᄇs༖hᄢ Dᄖign: nअ gᄨᄊ–l filᄇs༖hᄢ We hȪe simplified hhe design significȨhly ᤽I iᘇdeᘋg⠑ ᘓecific hइ␥MU We cȨnअ vᘑ ᘟeeጠच –ᘇ– file ᘏᘅem One ⨉lvme cȨ ထ ȅԂcἑ hइनl༇नe VM –h नce Hosh m–chine disa QEMU Gvesh OS Gvesh OS sheep 㜖er⨑r process) sheep disa sheep disa Commvnic–he ⴋhh ohher node sheeps disa I/O Cओyrighh © 2010 NTT Cऊጉr–hiन g–hew–y shor–ge ser⨑r shor–ge server Shᄑጠच cढጉnᄨhs Shᄑጠच cढጉnᄨhs corsync corsync corsync VM qemv VM qemv VM qemv g–heⴂy shor–ge server g–heⴂy 㸐jᄔh shऊ–ge Clvshᄊ m–n–gemᄨh Cओyrighh © 2010 NTT Cऊጉr–hiन Objᄔh shor–ge Objᄔh shor–ge Sԉrᄖ fle㘋blᄰsଃᄠ d–Ԃ wiԟ – vniqve ⤺ (objᄔԖ) Cl଑nԖ don'ԇc–rᄇ–bovԇwhᄊᄇԉ sԉrᄇobjecԖ Two aନds of objᄔԖ in Shᄑpdog One wriher, नe re–der No wriher, mvlhip⸑ re–ders Objeᐅ w਋he ਑– Objeᐅ Cओyrighh © 2010 NTT Cऊጉr–hiन Hभ ho shऊᄇvolvmᄖ? Hभ ho shऊᄇvolvmᄖ? Vo⸄mes Ȋe divided inho 4 ࠯ dȅȇob䌑chs A⸮ocȅion h–ble is shored ho VD⤇ob䌑ch Vo⸄me ●–ev●h–a VDI Objech VD⤇Objech Objech sԉr–ge D–h– Objechs Cओyrighh © 2010 NTT Cऊጉr–hiन Sn–጖hअ Sn–጖hअ Copy VDI 㸐jeᐅᬇ–nd m–ae –lloᐂhed d–h– objeᐅs ਑–d〉nly ☓d–hing ਑–d〉nly objeᐅs ᐂvses ᐉpy〉n〭਋he VD⤇Objech VDI Objech 㜖n–pshoh) VDI Objech VD⤇Objech 㜖n–pshoh) VD⤇Objech Crᄂhᄇsn–pshअ Copy-न-wrihe 20 23 10 11 13 10 11 13 10 11 13 10 11 13 11 10 Cओyrighh © 2010 NTT Cऊጉr–hiन 㐟ᄊᄇhइshorᄇo၃ᄔhs? 㐟ᄊᄇhइshorᄇo၃ᄔhs? We vse consishenh hȖhing ho decide which node ho shore objechs EȔh nठe iᘇ–lᘉ ጮ–ced न hhe ri⠚ Ƞ​hion ऊ remप–l ण ⠉‑ᘇ eᘇnअ ᘋgnific–nԮ༇chȨge hἑ m–ጓing ण objechs 25 50 75 100 125 150 175 NᤈEℇA ID ℇ18 NᤈEℇB ⤺ ℇ55 NᤈEℇC ⤺ ℇ81 NᤈEℇD ID ℇ133 NAMEℇE ID ℇ169 11 Cओyrighh © 2010 NTT Cऊጉr–hiन Rᄓlic–hiन Rᄓlic–hiन M–ny dish਋bvhed shoਂge syshems vse ch–ନ ਑plଔ–hଉn ho m–ନh–ନ ⥁㸇oਠe਋ng Sheepdog ᐂn vse dଊeᐅ repliᐂhଉn beᐂvse wrଅe ᐉllision ᐂnnoh h–ppen ⴊihe re–d ⴊihe 㜓–r–⸮e⸋e–b⸑) re–d 㜣rom one of hhem) Ch–i⠇਑plicȅion Direch replicȅion g–heⴂy g–hew–y shor–ge ser⨑r shor–ge ser⨑r shor–ge ser⨑r shor–ge ser⨑r shor–ge server shor–ge server 12 Cओyrighh © 2010 NTT Cऊጉr–hiन Clvshᄊ nठᄇm–n–gᄢenh Clvshᄊ nठᄇm–n–gᄢenh Tohem ring pਉhocol D༨–mic memထrsἋጇmȨȚemenh Tअ–l ऊ‑r –nd reliȐle mЮhi-cȖh VirԄ–l ᘏncἊनy M–ᐟନᄇA M–ᐟନᄇB M–ᐟନᄇC MSG1 MSG2 MSG3 B is do⴨ MSG4 MSG5 MSG1 MSG2 MSG3 MSG1 MSG2 MSG3 B is do⴨ MSG4 MSG5 13 Cओyrighh © 2010 NTT Cऊጉr–hiन Clvshᄊ nठᄇm–n–gᄢenh Clvshᄊ nठᄇm–n–gᄢenh ฉrosyncܔlvsherܑngine ⤢plemenԂԋo⠇of ԉԑm-ri⠚ pਉԉcol ⤖ –dopԑd b༇ⴑll-a⠉⴨ ope⠇sovਔe projecԖ (P–cem–aer, GFሜ, eԔ) Sheepdog܄sesܔorosyncܢvlhi〔Ȗh܅o܂voidܢeh–dȅȰ server M–cἋ⠑ A M–cἋ⠑ B M–cἋ⠑ C Loca volvme – ᔉca vo⸄me b ᔉca ⨉⸄me b 㜣–i⸑d) Loca volvme – ᔉca vo⸄me b ᔉca ⨉⸄me b 㜣–i⸑d) Loca volvme – ᔉca vo⸄me b ᔉca ⨉⸄me b 㜣–i⸑d) 14 Cओyrighh © 2010 NTT Cऊጉr–hiन Pᄊform–nᐑ (1 V࠸ Pᄊform–nᐑ (1 V࠸ CPU : Core2 ␄–d 2.4GHe Memory : 1 GB Nehwora : 1 Gbps Dଖa : SATA 7200 rpm M–chନes (Sheepog): 8 D–h– redvnd–ncy (Sheepdog): 1 VM VM FAS 2020 (఑hApp shorȚe VM VM ᔉc–⸇ disa NFS 㜮oc–⸇disa) NFS 㜌ehᤓp Shor–ge) Sheepdog 15 Cओyrighh © 2010 NTT Cऊጉr–hiन Loᐂl disa NFS (loc–l disa) NFS (FAS 2020) Sheepdog (rep=1) Sheepdog (rep=2) Sheepdog (rep=3) Loᐂl disa NFS (FAS 2020) 10 15 20 25 30 35 40 Pᄊform–nᐑ (1 V࠸ Pᄊform–nᐑ (1 V࠸ $ dbench -s -S TἊoКἓЅ (MB/ᘑc) Cढp–r–blᄇho NFS (lऔ–l disa) Physic–l m–chine 16 Cओyrighh © 2010 NTT Cऊጉr–hiन Pᄊform–nce (~ 256 Vࠖ) Pᄊform–nce (~ 256 Vࠖ) CPU : Core2 Qv–d 2.4GHe Mᄢory : 1 GB Neԭora : 1 Gbps Disa : ሙTA 7200 rpm HosԇmȔhinᄖ : 8 ~ 64 VirԄ–l mȔhinᄖ 1 ~ 256 D–h– redvndȨᐏ : 3 FAS 2020 (఑hApp shoਂge OS OS OS OS OS OS OS OS OS OS OS OS Ehheਨeh OS OS OS OS OS OS OS OS OS OS OS OS Ehherneh Shᄑpdog NFS (NᄅApp FAS 2020) 17 Cओyrighh © 2010 NTT Cऊጉr–hiन Pᄊform–nce (~ 256 Vࠖ) Pᄊform–nce (~ 256 Vࠖ) Throvghpvh scȮes Ȕcording ho hhe nvmber o⌇ hosh mȔhines $ dbench -s -S 18 Cओyrighh © 2010 NTT Cऊጉr–hiन TOD㸇ihe∖ TOD㸇ihe∖ Shorh-hermܚo–ls (ନ fᄭܢonhh) Morᄇsc–l–ဋlଅy ሄጓorhܮଐvirh, E⼒ ᤽I Perform–ncᄇଢጊovᄢenh Long-hᄊmܚo–ls (in܉nᄇor܅wo yᄂrs) gv–r–nheᄇrᄮଂဋlihy܂n –v–ମ–ဋlଅy vn‑r he–vy lo–d holᄊ–ncᄇ–g–ନsh nᄅwor؇ጂrhihଉnܷsጮଅ-ည–ନ) lo– b–l–ncନgܔorrᄖጉn​ng hoܩ䄾,܎㴦, mᄢory lo–d 19 Cओyrighh © 2010 NTT Cऊጉr–hiन Cनclvsion Cनclvsion Sheepdog is scȮȐ⸑, mȨȚeȐ⸑, Ȩd reliȐle shorȚe pool for ⤂Ȓ environmenh We ho጑ Sἑepdच will ထcढe hἑ ‑ f–chइ ᘅȨ r ण clऄ ᘅऊ–ge ᘏᘅem Fvrhheਇinformȅion Prृecԇጂge ἅԓ://www.oᘊg.⠑h/ᘟeepdog/ Mȋling lish sheepdog@liᘅᘗwpag.oਚ 20 Cओyrighh © 2010 NTT Cऊጉr–hiन 21 Cओyrighh © 2010 NTT Cऊጉr–hiन Appendix 22 Cओyrighh © 2010 NTT Cऊጉr–hiन Sheepdog clvsher ᤊᐟihᄔhvrᄡ fvll༇s༢mᄅric ᤊᐟihᄔhvrᄡ fvll༇s༢mᄅric 䬑ro configvrȅion Ȑovh c⸄sher members SimilȊ ho ⤖ilon Ȋchihechvre Sheepdog c⸄sher Use sheepdog –s – nehⴉra shor–ge Use sheepdog –s – ⨋rhv–⸇infr–shrvchvre ehherneh swihch VMs Hosh m–chines ehherneh sⴋhch VMs VMs 23 Cओyrighh © 2010 NTT Cऊጉr–hiन Nodᄇmᄢbᄊshiጇhishऊy Nodᄇmᄢbᄊshiጇhishऊy ᤮l nodᄖ ᘅoreᘇhhᄇhiᘅory of nodᄇ membᄊᘟip Objᄔhs Ȋᄇᘅorᄠ wihh hhᄇvᄊᘋon of node mᄢbᄊᘟip ( ᄓoch Time M–chine A M–chine B M–chine C M–chine D M–chine E M–chine C joined M–chine DᬇE joined M–chine ᤛ DᬇE ⸑fh epoch Node membership AᬇB AᬇBᬇC AᬇBᬇCᜇDᬇE BᬇC obj℞ obj℞ obj℞ obj℞ objℜ objℜ objℜ objℜ objℳ objℳ obj℻ obj℻ 24 Cओyrighh © 2010 NTT Cऊጉr–hiन Shrनg cनsishᄨc༇ Shrनg cनsishᄨc༇ Wihhऄh ᄓoch Wihh ᄓऔh obj is vpd–hed ho obj' B –nd C –re f–i⸑d epoch 3 A –nd B shores obj epoch 1 C joined epoch 2 Prevenh from reȠing old objechs obj obj obj obj obj obj obj℞ obj℞ obj℞ obj℞ obj℞ obj℞ obj' obj' obj'ℜ obj'ℜ