258K - views

The Google File System Sanjay Ghemawat Howard Gobioff and ShunTak Leung Google A

It providesfault tolerance while runningoninexpensivecommodityhardware anditdelivers high aggregate performance to a large number of clients While sharing many of the same goals as previous dis tributed 64257le systems our design has been driven by

Embed :
Pdf Download Link

Download Pdf - The PPT/PDF document "The Google File System Sanjay Ghemawat H..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

The Google File System Sanjay Ghemawat Howard Gobioff and ShunTak Leung Google A






Presentation on theme: "The Google File System Sanjay Ghemawat Howard Gobioff and ShunTak Leung Google A"— Presentation transcript:

TheGoogleFileSystemSanjayGhemawat,HowardGobioff,andShun-TakLeungGoogleABSTRACT                                                                   "   (     .                                      5   "       0      #   "   ;  )"1              2.DESIGNOVERVIEW2.1Assumptions         #                  "                                 "              ;01   50 1        =           5 "       (                    41   ;01              6      41 #:                "      $     "    5  8               = " =                     8     $     /               6       5   "             /         0                  $     2.2Interface            6::&#x-480;&#x.300;8   5                 0          9                         .               $                                   )? ))  2.3Architecture6                ;2   @                              !       52         $A?      B     @                  1     #    #       "       "                                             "                          6:          B                :&#x-480;&#x.300;86:     @   C     B  #              C             ,B   -B         @ .#   $    2.4SingleMaster/                 Legend:Data messagesControl messages Linux file systemLinux file system Figure1:GFSArchitecture        /    5           B                           $   @       ;  5        #         "   $         "            "             "    $        "$          $                            $              $"             2.5ChunkSizeB 5     A?01          52    @        @5              3      56   5#       .       $    $      "               $  2                  "1                               "B:       "5   "              DA;8     5  5     6          3 "                  3          $  /    $ (              "        $            $     6               2.6Metadata"  3  (               . 6     .  " ,        -             .         E                  "                  3  2.6.1In-MemoryDataStructures         =          "                               ?) ?? 8                        "     "      A? A?01 0                       $  A?                              !       2.6.2ChunkLocations"                   "                                $      "               3                  6      5        "                        ,     -      2.6.3OperationLog"          C                      , ?F-  $                               8#              "               !           "    !   !         "        "   5         "        5              Write RecordAppend Serial dened dened success interspersedwith Concurrent consistent inconsistent successes undened Failure inconsistent Table1:FileRegionStateAfterMutation      "    1                   "      1        .              "           "                         9       $    8                  6      #        2.7ConsistencyModel                 =     .                     2.7.1GuaranteesbyGFS      ,   - "     (          , ?;-G .         , DA)-"                 " ; 5 6             6                          #   ,     -(        B           (      !     "            6         ,    -(#    # #                "      #       %     6      #6  ,HI-              #. , ))-,  H  I     #      -"#                       "            6$                 ,-         , );- ,-              , ?F-                 "                  "    .               0                         @                           , FD-8          , ?)-6            2        (     2.7.2ImplicationsforApplications           $    (                 :                                            B            9              9         6    =           B                      .                    $9  .       . 9       2            6                ,         -     $              "   78,    -              $     3.SYSTEMINTERACTIONS     5 .                            3.1LeasesandMutationOrder6               2      .          "         "        6       "                 "        5      6     A /       $          "  $                  "       ,            -2                     D      !    ;"                     ,  -D"        ,  - "              PrimaryReplica Secondary Secondary Master ControlData Client2step 145667 Figure2:WriteControlandDataFlow         )"     6     2       @9E#   1  !   !         !            )D?8          $  "$     "                   5         F"  $   2            A"          J"     6                ,          -"  $            8            ,)- ,J-                        "    !            "           #                    "            DJ3.2DataFlow ! !    =     !                          8    5  .                  5     "  5  .                 , -"  .             "           ,     -     H I               ; ?   ;;  D ? ;D   D)? D  8        H I    :     5        "B:  8      :                              B/T           8     ;0 ,-  ; ";01   L 3.3AtomicRecordAppends                # B       5 (                            ,   $ - #.   #  "        E         9           #          B            5                 7   $      #   9          !  );       "        "  $  "             5,A?01-     5               ,9        5          -    5         #               6      #                           "              #                 ##   #                      ,   -        ,   -8           DJD3.4Snapshot"       ,HI-         5        8$    ,     -             @ 6&F'    $         $          "  $   $       "        6                     "          "     B     $     "     B       $        B.     B   B.1                 ,  ;02   -   $        (         B.           3     4.MASTEROPERATION"               (                               4.1NamespaceManagementandLocking0         (                       "                  5 E              C    ,     E  -                        =      2     ,         -   2   $   "     $              C                        "    $         "  $        "    5     !     $      HI      "     =       8                           ($            "     =         "       5               3  5      6   $        (               4.2ReplicaPlacement6                  "         #  B       #       6               0      $         "         (  5        5    5                   5  .         "                M ,       -   =            8  =!     #    4.3Creation,Re-replication,RebalancingB   (                  ,;-           5 8  $ 5  5  ,D-     H I   6           =                        ,)-6     "              "    (                2    5  8                                , ??-     5                 "       H  I            "        ($ 5    5                 "     =     =            6                 $        (               6                  =  "                         $ 5   4.4GarbageCollection6              5                      4.4.1Mechanism            3   /       3          %  .             ,     -E                                  "#                   ,     -                           . "    4.4.2Discussion6                      $        (                  (@      6      H  I "         #                     B            9                .                                         "    50      "          $      "                          #      6                             #        #                            4.5StaleReplicaDetectionB                                           "          "                    "                                "           1#          $    6                               "             5.FAULTTOLERANCEANDDIAGNOSIS8          $     "$  $                (          B                             5.1HighAvailability6                       # (   5.1.1FastRecovery1                        G   3    B            $   ADD   5.1.2ChunkReplication6        #  E  #    #      " "             M       , FD-6               $                       =          5.1.3MasterReplication"                6        !                                                   B        ,  -%C        0HI           "               "                                                   "               $     @       ,  $  -     $                        .    5.2DataIntegrity2                            , J -                0       (                "                 6     A?41  2   )D  @                            $    "                  $        $            6            B     #                                   0           78          78B       5    ,  -     3                             2                                                                         %           "      8             "                   5.3DiagnosticTools2                                                        ,      -  9:B$  "      #   /        "9:B    $         1  $    9:B #            "           "           ,   -   $      "                  6.MEASUREMENTS                              6.1Micro-benchmarks            ;A  ;A  C     "         6      ;?/5: D1  L1F?   ;0   2    /:DFD?6 ;N      ;A    " ;   6.1.1Reads         2     ?01   )D1 " DFA      ;1"     )D1    ; @ #8       ),-       "       ;DF017 ;    ;DF017    ;0      ";017L      3    "  N?017JFO;DF017   ;AA017   "=   LJFO               6.1.2Writes        2  ;1   ;01"        ),-"   AJ017 );A ;DF017   "   A)017   "                   %          6  )F017;A  ,DD017   -   6                     0      ;A ;A  #           3                  #           6.1.3RecordAppends ),-               :                   A017     ?L017;A            #   8                       5  "                           6.2RealWorldClusters             B 6          6           01"1    5    B 1       "    Cluster B Chunkservers 227 Availablediskspace 72TB 180TB Useddiskspace 55TB 155TB NumberofFiles 735k 737k NumberofDeadles 22k 232k NumberofChunks 992k 1550k Metadataatchunkservers 13GB 21GB Metadataatmaster 48MB 60MB Table2:CharacteristicsoftwoGFSclusters            "1          H I                 6.2.1Storage6          "1        HE I     P     " ;L"1 FD"1   "       1                          6.2.2Metadata"     1     A?41  "          ?F"         01;    "   5 .      .  0          8                  .                  2      F;01 "(        $/    Q  )A Q          6.2.3ReadandWriteRates" )   1           ,"       -"   )017      1      ;017 )017       051015 050100Read rate (MB/s) ,-9 051015 020Write rate (MB/s) ,- 051015 0510Append rate (MB/s) ,-9  Figure3:AggregateThroughputs."         1    "NFO               Cluster B Readrate(lastminute) 583MB/s 380MB/s Readrate(lasthour) 562MB/s 384MB/s Readrate(sincerestart) 589MB/s 49MB/s Writerate(lastminute) 1MB/s 101MB/s Writerate(lasthour) 2MB/s 117MB/s Writerate(sincerestart) 25MB/s 13MB/s Masterops(lastminute) 325Ops/s 533Ops/s Masterops(lasthour) 381Ops/s 518Ops/s Masterops(sincerestart) 202Ops/s 347Ops/s Table3:PerformanceMetricsforTwoGFSClusters"   "      1      6   FL017          JF017 =  B 1    ;)017   3)L0176.2.4MasterLoad" )      DF    "                            $       ,     -          =                              6.2.5RecoveryTime6               "                    1" ;F    A1"                     N;     ,?   -        ADF017,F0 -6   D)D   #  ??017          ;A  AA1"  DAA    "DAA        D   D            6.3WorkloadBreakdown                 ADB &#x-245;&#x.400;      R    6.3.1MethodologyandCaveats"        $!        "    $  $          78          9:B$           9:B                5    2                                    8       5                        !         Operation Read Write RecordAppend Cluster XY 0.42.6 1B..1K 0.14.1 6.64.9 0.29.2 1K..8K 65.238.5 0.41.0 18.915.2 8K..64K 29.945.1 17.843.0 78.02.8 64K..128K 0.10.7 2.31.9 .14.3 128K..256K 0.20.3 31.60.4 .110.6 256K..512K 0.10.1 4.27.7 .131.2 512K..1M 3.96.9 35.528.7 2.225.5 1M..inf 0.11.8 1.512.3 0.72.2 Table4:OperationsBreakdownbySize(%).5        $   #        6.3.2ChunkserverWorkload" ?   595   "  , A?41-               "  ,F;D41-   $     6        R8             $:           8         B �            5    "  ,DFA41-      #  #      5         , A?41-6   R           �    R     " F       5        ,DFA41-        , A?41-           6.3.3AppendsversusWrites9          �  ;(  L(;    R   )J(; DF(;  0         &#xL;-3;†.8;                    #56                  "  Operation Read Write RecordAppend Cluster XY 1B..1K 1K..8K 13.83.9 .10.1 8K..64K 11.49.3 2.45.9 2.30.3 64K..128K 0.30.7 0.30.3 22.71.2 128K..256K 0.80.6 16.50.2 .15.8 256K..512K 1.40.3 3.47.7 .138.4 512K..1M 65.955.1 74.158.0 .146.8 1M..inf 6.430.1 3.328.0 53.97.4 Table5:BytesTransferredBreakdownbyOpera-tionSize(%).5        $" #            Cluster Open 26.116.3 Delete 0.71.5 FindLocation 64.365.8 FindLeaseHolder 7.813.4 FindMatchingFiles 0.62.2 Allothercombined 0.50.8 Table6:MasterRequestsBreakdownbyType(%)             �      )     RFO6               "    $    6.3.4MasterWorkload" A   $ 0$   ,  !-      ,  !!-  B .50; R   #    $ R            #   #  "$            , HI E       -  #     $  H I       E  $             B R                      .50;.                7.EXPERIENCES                            8                    $                   0 $              @  0    @   %2                               ."            "                    2     @ DD       5  5   "                           @ D?6 @                   ,  -     ,  -                 2                                            %       @                        8.RELATEDWORK@     6&F'                       E  6  .     &;' &)'            6          96%&N'                   6   &;D'   55&A'        8                               0  .&;;' :&;   5                5          !     5                                                 ,-       E          "          / &J'                  @&L'               /               :8&#x'-23; .90;    6                         C6%&?' C6%         C6%  E  C6%   5  5     36           $      E  0 . C6%                 "   $          $ 9&D' 9  $    !              "9     $             ;$=  0              9.CONCLUSIONS"     $                  $                                         8   #                 5      ,     -  , $  -           8               B        "$                              6        %2            8                                  0            5   5          "        5                                                                 ACKNOWLEDGMENTS         1 1, -            6  6S#%  %S    B       23     $0            %4         B E/ 5 0  : 9:  %       0            R    REFERENCES&;'" 6  0 %  S C%: %9  9         $ %&'()#*  "*  $   ;B 0  9B % ;NNF&D'9 5/6 %26  C"%2B S 0/  %:  4R  B 789(0     $ %*+,-."$    *  /-"$(*0112  ;6   0;NNN&)'@  B % %2@ (E      78) *  ?,?-(?);NN;&?'6 %C 4  6 S#1 B /#B / 2 9 %9  S T  6#    $ %3( *%$ ! "*    NDQ;) SB  8&F'S /0 450 %C 0   9  0         ()#4) *  A,;-(F;QL;;NLL&A' 055 (77  55 D)&J'1@  3 9: S  @ 0   9   /    *  "*  $   DDAQD)L:B68;NN;&L'@ (77  D)&N'%6: 6  9 /456      ,96%- $ %&133()#*-6#"- )%# %  ;B     ;NLL&;    9 /  :(6         $ % 7*89-:)%   *4   D);QD??0 B  S D&;;' 9 " 09 0"8.4"    $ % %9(*(6  * ))%#**   4 B  : 0    ;NNA&;D'B   6" " 0  24@   (6     $ %&;()#*  "*  $   DD?QD)J 0  8;NNJ