Storage Computing in High Energy amp Nuclear Physics CHEP May 812 2023 Andrew Hanushevsky SLAC S3 Gateway Architecture Based on XrdCl xrootd client http plugin Uses Davix an HTTP SDK developed at CERN ID: 1042500
Download Presentation The PPT/PDF document "Xrootd S3 Gateway for WLCG" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Xrootd S3 Gateway for WLCG Storage Computing in High Energy & Nuclear PhysicsCHEP May 8-12, 2023Andrew Hanushevsky, SLAC
2. S3 Gateway ArchitectureBased on XrdCl (xrootd client) http plug-inUses Davix, an HTTP SDK developed at CERNVery reliable and supportedPerforms better than most commercial SDK’sBridges HEP & commercial world securityCloudStorageGcloud borderHMAC keyS3S3 GatewayData SourceOr Targethttp, xrootX509SciTokensCommercial WorldHEP World2
3. S3 Gateway ApplicabilityS3 GatewayhttpxrootAWSCephGCSMinioAny S3 APICommercial or Institutional3
4. S3 Gateway ScalingS3 GatewayhttpxrootTo S3 APIS3 GatewayS3 GatewayUnlimited InstancesS3 GatewayRedirector(s)To S3 APITo S3 APIRedirector selects gatewayeither round robin orbased on actual loadNon-working gateways ignoredPotential to federate Gateway clusters in different regionsPick appropriate regional gatewayS3 Gateway Cluster4
5. S3 Gateway Test SetupCluster of 19 Such NodesXrdEC File system>= 19 GbpsEC Node:12 core CPU24 GB RAM10—12 HD’s1Gpbs NIC8+2 layout1 MB stripesEC Redirector Node S3 GatewayMirrored SSD100 Gbps NICDavixBufferWhat is xrdEC? It’s an XRootD configuration that functions like an erasure encoded parallel file system. Performance is typically stripe size x single disk bandwidth which allows very high throughput.The actual S3 gateway nodeS3Cloud5
6. Ingress PerformanceGCSAWSS3 GatewayInOutCloudSLACdtn’s » 35 minute run time FTS managed 3120 files 1.36TB 50 to 230 concurrent xfers No check summing 100% xfers succeeded In/out non-tracking due to internal Davix buffering6
7. Internal Davix BufferingCurrent Davix fully buffers stream I/OStream I/O creates a physical fileFile is then forwarded to endpointFuture Davix eliminates this bufferingWorking with Davix team to implement thisWill be part of final product7
8. Egress PerformanceS3 GatewayInOutCloudSLACdtn’sGCSAWS » 10 minute run time FTS managed 312files 136GB 50 to 230 concurrent xfers Local check summing 100% xfers succeededUnexplained 400 MB/shard limit (hardware?)8
9. S3 Gateway CKS PerformanceS3 GatewayInOutCloudSLACdtn’sGCSAWS » 40 minute run time FTS managed Ingress 3120 files 1.36TB 50 to 130 concurrent xfers Check summing Out transfer ½ of in because data read back w/ egress charge to compute checksum9
10. Avoiding CKS Egress Charge IAWS & GCS provide server-less computingAWS via lambdaPython, Java, Google Go and C#GCS via Google Cloud FunctionsPython, Java, Google Go, .NET, Ruby, and PHPLeverage these to compute checksumS3 Gateway triggers server-less cks programChecksum computed in the cloud (no egress)Result transmitted back to S3 gateway10
11. S3 Gateway for multiple API’sThe S3 Gateway is universalWork with all S3 storage flavors we testedwork with both s3v4 and older s3v2S3 credentials: different names, but same thingAWS: ACCESS_KEY_ID & SECRET_ACCESS_KEYCeph: HMAC key pairGCS: HMAC key pairMinIO: username & passwordS3 Gateway/Davix uses AWS naming convention11
12. ConclusionS3 Gateway is extremely economicalAvoids most cloud chargesExcept egress when fetching data from the cloudBuilt-in authorization can restrict who can do thisS3 Gateway provides uniform accessRegardless of S3 provider access is the sameAutomatic conversion of HEP auth to S3 authProven scalability and performanceDoc on Xrootd-HowTo 12