The Google File System Sanjay Ghemawat Howard Gobioff and ShunTak Leung Google A - PDF document

Download presentation
The Google File System Sanjay Ghemawat Howard Gobioff and ShunTak Leung Google A
The Google File System Sanjay Ghemawat Howard Gobioff and ShunTak Leung Google A

Embed / Share - The Google File System Sanjay Ghemawat Howard Gobioff and ShunTak Leung Google A


Presentation on theme: "The Google File System Sanjay Ghemawat Howard Gobioff and ShunTak Leung Google A"— Presentation transcript


TheGoogleFileSystemSanjayGhemawat,HowardGobioff,andShun-TakLeungGoogleABSTRACT                                                                   "   (     .                                      5   "       0      #   "   ;  )"1              2.DESIGNOVERVIEW2.1Assumptions         #                  "                                 "              ;01   50 1        =           5 "       (                    41   ;01              6      41 #:                "      $     "    5  8               = " =                     8     $     /               6       5   "             /         0                  $     2.2Interface            6::&#x-480;&#x.300;8   5                 0          9                         .               $                                   )? ))  2.3Architecture6                ;2   @                              !       52         $A?      B     @                  1     #    #       "       "                                             "                          6:          B                :&#x-480;&#x.300;86:     @   C     B  #              C             ,B   -B         @ .#   $    2.4SingleMaster/                 Legend:Data messagesControl messages Linux file systemLinux file system Figure1:GFSArchitecture        /    5           B                           $   @       ;  5        #         "   $         "            "             "    $        "$          $                            $              $"             2.5ChunkSizeB 5     A?01          52    @        @5              3      56   5#       .       $    $      "               $  2                  "1                               "B:       "5   "              DA;8     5  5     6          3 "                  3          $  /    $ (              "        $            $     6               2.6Metadata"  3  (               . 6     .  " ,        -             .         E                  "                  3  2.6.1In-MemoryDataStructures         =          "                               ?) ?? 8                        "     "      A? A?01 0                       $  A?                              !       2.6.2ChunkLocations"                   "                                $      "               3                  6      5        "                        ,     -      2.6.3OperationLog"          C                      , ?F-  $                               8#              "               !           "    !   !         "        "   5         "        5              Write RecordAppend Serial dẽned dẽned success interspersedwith Concurrent consistent inconsistent successes undẽned Failure inconsistent Table1:FileRegionStateAfterMutation      "    1                   "      1        .              "           "                         9       $    8                  6      #        2.7ConsistencyModel                 =     .                     2.7.1GuaranteesbyGFS      ,   - "     (          , ?;-G .         , DA)-"                 " ; 5 6             6                          #   ,     -(        B           (      !     "            6         ,    -(#    # #                "      #       %     6      #6  ,HI-              #. , ))-,  H  I     #      -"#                       "            6$                 ,-         , );- ,-              , ?F-                 "                  "    .               0                         @                           , FD-8          , ?)-6            2        (     2.7.2ImplicationsforApplications           $    (                 :                                            B            9              9         6    =           B                      .                    $9  .       . 9       2            6                ,         -     $              "   78,    -              $     3.SYSTEMINTERACTIONS     5 .                            3.1LeasesandMutationOrder6               2      .          "         "        6       "                 "        5      6     A /       $          "  $                  "       ,            -2                     D      !    ;"                     ,  -D"        ,  - "              PrimaryReplica Secondary Secondary Master ControlData Client2step 145667 Figure2:WriteControlandDataFlow         )"     6     2       @9E#   1  !   !         !            )D?8          $  "$     "                   5         F"  $   2            A"          J"     6                ,          -"  $            8            ,)- ,J-                        "    !            "           #                    "            DJ3.2DataFlow ! !    =     !                          8    5  .                  5     "  5  .                 , -"  .             "           ,     -     H I               ; ?   ;;  D ? ;D   D)? D  8        H I    :     5        "B:  8      :                              B/T           8     ;0 ,-  ; ";01   L 3.3AtomicRecordAppends                # B       5 (                            ,   $ - #.   #  "        E         9           #          B            5                 7   $      #   9          !  );       "        "  $  "             5,A?01-     5               ,9        5          -    5         #               6      #                           "              #                 ##   #                      ,   -        ,   -8           DJD3.4Snapshot"       ,HI-         5        8$    ,     -             @ 6&F'    $         $          "  $   $       "        6                     "          "     B     $     "     B       $        B.     B   B.1                 ,  ;02   -   $        (         B.           3     4.MASTEROPERATION"               (                               4.1NamespaceManagementandLocking0         (                       "                  5 E              C    ,     E  -                        =      2     ,         -   2   $   "     $              C                        "    $         "  $        "    5     !     $      HI      "     =       8                           ($            "     =         "       5               3  5      6   $        (               4.2ReplicaPlacement6                  "         #  B       #       6               0      $         "         (  5        5    5                   5  .         "                M ,       -   =            8  =!     #    4.3Creation,Re-replication,RebalancingB   (                  ,;-           5 8  $ 5  5  ,D-     H I   6           =                        ,)-6     "              "    (                2    5  8                                , ??-     5                 "       H  I            "        ($ 5    5                 "     =     =            6                 $        (               6                  =  "                         $ 5   4.4GarbageCollection6              5                      4.4.1Mechanism            3   /       3          %  .             ,     -E                                  "#                   ,     -                           . "    4.4.2Discussion6                      $        (                  (@      6      H  I "         #                     B            9                .                                         "    50      "          $      "                          #      6                             #        #                            4.5StaleReplicaDetectionB                                           "          "                    "                                "           1#          $    6                               "             5.FAULTTOLERANCEANDDIAGNOSIS8          $     "$  $                (          B                             5.1HighAvailability6                       # (   5.1.1FastRecovery1                        G   3    B            $   ADD   5.1.2ChunkReplication6        #  E  #    #      " "             M       , FD-6               $                       =          5.1.3MasterReplication"                6        !                                                   B        ,  -%C        0HI           "               "                                                   "               $     @       ,  $  -     $                        .    5.2DataIntegrity2                            , J -                0       (                "                 6     A?41  2   )D  @                            $    "                  $        $            6            B     #                                   0           78          78B       5    ,  -     3                             2                                                                         %           "      8             "                   5.3DiagnosticTools2                                                        ,      -  9:B$  "      #   /        "9:B    $         1  $    9:B #            "           "           ,   -   $      "                  6.MEASUREMENTS                              6.1Micro-benchmarks            ;A  ;A  C     "         6      ;?/5: D1  L1F?   ;0   2    /:DFD?6 ;N      ;A    " ;   6.1.1Reads         2     ?01   )D1 " DFA      ;1"     )D1    ; @ #8       ),-       "       ;DF017 ;    ;DF017    ;0      ";017L      3    "  N?017JFO;DF017   ;AA017   "=   LJFO               6.1.2Writes        2  ;1   ;01"        ),-"   AJ017 );A ;DF017   "   A)017   "                   %          6  )F017;A  ,DD017   -   6                     0      ;A ;A  #           3                  #           6.1.3RecordAppends ),-               :                   A017     ?L017;A            #   8                       5  "                           6.2RealWorldClusters             B 6          6           01"1    5    B 1       "    Cluster B Chunkservers 227 Availablediskspace 72TB 180TB Useddiskspace 55TB 155TB NumberofFiles 735k 737k NumberofDead̃les 22k 232k NumberofChunks 992k 1550k Metadataatchunkservers 13GB 21GB Metadataatmaster 48MB 60MB Table2:CharacteristicsoftwoGFSclusters            "1          H I                 6.2.1Storage6          "1        HE I     P     " ;L"1 FD"1   "       1                          6.2.2Metadata"     1     A?41  "          ?F"         01;    "   5 .      .  0          8                  .                  2      F;01 "(        $/    Q  )A Q          6.2.3ReadandWriteRates" )   1           ,"       -"   )017      1      ;017 )017       051015 050100Read rate (MB/s) ,-9 051015 020Write rate (MB/s) ,- 051015 0510Append rate (MB/s) ,-9  Figure3:AggregateThroughputs."         1    "NFO               Cluster B Readrate(lastminute) 583MB/s 380MB/s Readrate(lasthour) 562MB/s 384MB/s Readrate(sincerestart) 589MB/s 49MB/s Writerate(lastminute) 1MB/s 101MB/s Writerate(lasthour) 2MB/s 117MB/s Writerate(sincerestart) 25MB/s 13MB/s Masterops(lastminute) 325Ops/s 533Ops/s Masterops(lasthour) 381Ops/s 518Ops/s Masterops(sincerestart) 202Ops/s 347Ops/s Table3:PerformanceMetricsforTwoGFSClusters"   "      1      6   FL017          JF017 =  B 1    ;)017   3)L0176.2.4MasterLoad" )      DF    "                            $       ,     -          =                              6.2.5RecoveryTime6               "                    1" ;F    A1"                     N;     ,?   -        ADF017,F0 -6   D)D   #  ??017          ;A  AA1"  DAA    "DAA        D   D            6.3WorkloadBreakdown                 ADB &#x-245;&#x.400;      R    6.3.1MethodologyandCaveats"        $!        "    $  $          78          9:B$           9:B                5    2                                    8       5                        !         Operation Read Write RecordAppend Cluster XY 0.42.6 1B..1K 0.14.1 6.64.9 0.29.2 1K..8K 65.238.5 0.41.0 18.915.2 8K..64K 29.945.1 17.843.0 78.02.8 64K..128K 0.10.7 2.31.9 .14.3 128K..256K 0.20.3 31.60.4 .110.6 256K..512K 0.10.1 4.27.7 .131.2 512K..1M 3.96.9 35.528.7 2.225.5 1M..inf 0.11.8 1.512.3 0.72.2 Table4:OperationsBreakdownbySize(%).5        $   #        6.3.2ChunkserverWorkload" ?   595   "  , A?41-               "  ,F;D41-   $     6        R8             $:           8         B �            5    "  ,DFA41-      #  #      5         , A?41-6   R           �    R     " F       5        ,DFA41-        , A?41-           6.3.3AppendsversusWrites9          �  ;(  L(;    R   )J(; DF(;  0         &#xL;-3;†.8;                    #56                  "  Operation Read Write RecordAppend Cluster XY 1B..1K 1K..8K 13.83.9 .10.1 8K..64K 11.49.3 2.45.9 2.30.3 64K..128K 0.30.7 0.30.3 22.71.2 128K..256K 0.80.6 16.50.2 .15.8 256K..512K 1.40.3 3.47.7 .138.4 512K..1M 65.955.1 74.158.0 .146.8 1M..inf 6.430.1 3.328.0 53.97.4 Table5:BytesTransferredBreakdownbyOpera-tionSize(%).5        $" #            Cluster Open 26.116.3 Delete 0.71.5 FindLocation 64.365.8 FindLeaseHolder 7.813.4 FindMatchingFiles 0.62.2 Allothercombined 0.50.8 Table6:MasterRequestsBreakdownbyType(%)             �      )     RFO6               "    $    6.3.4MasterWorkload" A   $ 0$   ,  !-      ,  !!-  B .50; R   #    $ R            #   #  "$            , HI E       -  #     $  H I       E  $             B R                      .50;.                7.EXPERIENCES                            8                    $                   0 $              @  0    @   %2                               ."            "                    2     @ DD       5  5   "                           @ D?6 @                   ,  -     ,  -                 2                                            %       @                        8.RELATEDWORK@     6&F'                       E  6  .     &;' &)'            6          96%&N'                   6   &;D'   55&A'        8                               0  .&;;' :&;   5                5          !     5                                                 ,-       E          "          / &J'                  @&L'               /               :8&#x'-23; .90;    6                         C6%&?' C6%         C6%  E  C6%   5  5     36           $      E  0 . C6%                 "   $          $ 9&D' 9  $    !              "9     $             ;$=  0              9.CONCLUSIONS"     $                  $                                         8   #                 5      ,     -  , $  -           8               B        "$                              6        %2            8                                  0            5   5          "        5                                                                 ACKNOWLEDGMENTS         1 1, -            6  6S#%  %S    B       23     $0            %4         B E/ 5 0  : 9:  %       0            R    REFERENCES&;'" 6  0 %  S C%: %9  9         $ %&'()#*  "*  $   ;B 0  9B % ;NNF&D'9 5/6 %26  C"%2B S 0/  %:  4R  B 789(0     $ %*+,-."$    *  /-"$(*0112  ;6   0;NNN&)'@  B % %2@ (E      78) *  ?,?-(?);NN;&?'6 %C 4  6 S#1 B /#B / 2 9 %9  S T  6#    $ %3( *%$ ! "*    NDQ;) SB  8&F'S /0 450 %C 0   9  0         ()#4) *  A,;-(F;QL;;NLL&A' 055 (77  55 D)&J'1@  3 9: S  @ 0   9   /    *  "*  $   DDAQD)L:B68;NN;&L'@ (77  D)&N'%6: 6  9 /456      ,96%- $ %&133()#*-6#"- )%# %  ;B     ;NLL&;    9 /  :(6         $ % 7*89-:)%   *4   D);QD??0 B  S D&;;' 9 " 09 0"8.4"    $ % %9(*(6  * ))%#**   4 B  : 0    ;NNA&;D'B   6" " 0  24@   (6     $ %&;()#*  "*  $   DD?QD)J 0  8;NNJ

By: luanne-stotts
Views: 260
Type: Public

The Google File System Sanjay Ghemawat Howard Gobioff and ShunTak Leung Google A - Description


It providesfault tolerance while runningoninexpensivecommodityhardware anditdelivers high aggregate performance to a large number of clients While sharing many of the same goals as previous dis tributed 64257le systems our design has been driven by ID: 3719 Download Pdf

Related Documents