Ryan Hennig. Hadoop Platform Team. ABOUT ME. RYAN HENNIG. Born and raised in Seattle, WA. Studied Computer Science at University of Washington in Seattle. Worked on Microsoft SQL Server 2006 – 2012. ID: 565760
DownloadNote - The PPT/PDF document "Hadoop @ eBay: Past, Present and Future" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Hadoop @ eBay: Past, Present and Future
Ryan HennigHadoop Platform Team
Slide2ABOUT ME
Slide3RYAN HENNIG
Born and raised in Seattle, WAStudied Computer Science at University of Washington in SeattleWorked on Microsoft SQL Server 2006 – 2012Shipped SQL Server 2008, 2008 R2, 2012Joined eBay Hadoop team in early 2012 - Based in Bellevue, suburb of Seattle
COMPUTE AND DATA INFRASTRUCTURE
3
Slide4AGENDA
Past
: Growth of Hadoop at eBay
Present
: Hadoop Use Cases, Operations Tools
Future
: Hadoop 2.0
Slide5HADOOP AT EBAY: PAST
Growth of Hadoop at eBay
Adventures in Forking
Partnership with
Hortonworks
Slide6HADOOP EVOLUTION @ eBay
HADOOP AT EBAY: PAST
6
2007
Single digit nodes
2010
Shared cluster
100s nodes
1000s + core
PBCDH2
2011
Shared clusters1000s node10,000+ core10s PBWilma (0.20)
2012
Shared clusters1000s node10,000+ core10s PBArgon (0.22)
2013
Shared clusters 4k+ node40,000+ core50s PBHDP 1.x
2009
Search
10s
- nodes
Slide7ADVENTURES IN FORKING
2007-2010: eBay runs shared clusters on Cloudera Distribution of Hadoop2010-2012: eBay runs shared clusters on custom Hadoop versions2010: Wilma (based on 0.20)2011: Argon (based on 0.22)2012: Custom branch abandonedLessons LearnedForking a fast-changing open source project is difficult and riskyBalancing Development and operations needsDevelopment team sizeFacebook had 100eBay had 15Coordination with open source community = lots of overheadDivergence from open source: Push changes early and often
HADOOP AT EBAY: PAST
7
Slide8HADOOP AT EBAY: PAST
8
Slide9EBAY AND HORTONWORKS
2012: eBay enters partnership with HortonWorksGoalsFocus on eBay-specific development internallyLeverage HortonWorks expertise for general Hadoop DevelopmentAvoid source code divergence by making open source contribution a priorityBenefits to HortonWorksCredibility enhanced by having a well-known customerAbility to test at large scale
HADOOP AT EBAY: PAST
9
Slide10HADOOP AT EBAY:PRESENT
Shared and Dedicated Clusters
Job Distribution
Use Case Examples
eBay Data Platform Overview
Slide11SHARED AND DEDICATED CLUSTERS
Shared clusters10s of PB and 10s of thousands of slots per clusterUsed primarily for analytics of user behavior and inventoryMix of production and ad-hoc jobsMix of MR, Hive, PIG, Cascading etc.Hadoop and HBase security enabledDedicated clustersVery specific use cases like Index BuildingTight SLAs for jobs (in order of minutes)Immediate revenue impactUsually smaller than our shared clusters, but still big (100s of nodes…)
HADOOP AT EBAY: PRESENT
11
Slide12JOB DISTRIBUTION BY TYPE
HADOOP AT EBAY: PRESENT
12
Slide13USE CASE EXAMPLES
Cassini, eBay’s new search engine:Use MR to build full and incremental near-real-time indexesRaw Data is stored in HBase for efficient updates and random readStrong SLAs: < 10 minutesRun on dedicated clustersRelated and similar Items recommendations:Use transactional data, click stream data, search index, etc.Production MR jobs on a shared clusterAnalytics dashboard:Run Mobius MR jobs to join click stream data and transactional data Store summary data in HBaseWeb application to query HBase
HADOOP AT EBAY: PRESENT
13
Slide14HADOOP OPERATIONS
LDAP Integration - All users stored in Active Directory, accessed via LDAP - Access to MapReduce Queues granted via MapReduce queues - Batch users: shared by a group of usersSecurity - Kerberos as implemented by Microsoft Active Directory - One domain for users, another for service/server principals - Batch users authenticated via keytabs, not passwordsMisc - 10’s of slave nodes are broken at any given time - Often need to add several racks of machines at a time
HADOOP AT EBAY: PRESENT
14
Slide15HADOOP OPERATIONS
Team has Development and Operations Responsibilities2 Huge shared clusters1800+ users, exponential growthAbout 10 Hadoop developersRecently: operations work moved to dedicated teamDeveloped several tools to manage operationsHadoop Management Console: user-facing web appldap-admin: swiss-army knife style tool for hadoop adminsPuppet: for adding machines to the clusters, many racks at a timeDecom/Recom scripts: automatic detection, repair, decommission, and recommission of slave nodes
HADOOP AT EBAY: PRESENT
15
Slide16HADOOP MANAGEMENT CONSOLE
Custom Web application built on Ruby on RailsSelf-service tools are continually added to reduce support loadUser ManagementAccess RequestsGroup MembershipBatch User ManagementNew RequestsSudoer managementDataset ManagementExplore DatasetsRequest New dataset transfer between Teradata and HadoopMetadata toolsEach dataset is stored in custom XML formatCode Generation: Hive Tables, Java POJOs
HADOOP AT EBAY: PRESENT
16
Slide17HADOOP AT EBAY: PRESENT
17
Slide18HADOOP AT EBAY: PRESENT
18
Slide19HADOOP AT EBAY: PRESENT
19
Slide20HADOOP AT EBAY: PRESENT
20
Slide21HADOOP AT EBAY: PRESENT
21
Slide22HADOOP AT EBAY: PRESENT
22
Slide23HADOOP AT EBAY: PRESENT
23
Slide24ldap-admin
Command-line tool written in RubySwiss-army knife tool, features added on demand for support issuesOften used features:Add a user to a groupView key details for LDAP users and groupsList all users, batch users, hadoop groupsReset batch user passwords and keytabsShow/add/remove sudoers for a batch accountRun user diagnostics: check permissions, keytabs, etc
HADOOP AT EBAY: PRESENT
24
Slide25HADOOP AT EBAY:FUTURE
HDFS Federation
YARN
New Scenarios
Storage and Operational Efficiency
Slide26HDFS HA and Federation
HDFS High-Availability for ReliabilityNameNode in Hadoop 1.0 is a Single Point of FailureAutomated failover to hot standby Depends on ZooKeeperHDFS Federation for Scalability and IsolationHadoop 1.0: Single NameNode service“Secondary NameNode” is not for failoverStorage scales horizontally, but Namespace scales verticallyNo isolation for different tenants or applicationsHadoop 2.0: HDFS FederationPartition the HDFS Namespace Many independent NameNodesAllows direct access to Block Storage w/o going through HDFS interface
HADOOP AT EBAY: FUTURE
26
Slide27HDFS HA
HADOOP AT EBAY: FUTURE
27
Slide28HDFS HA
HADOOP AT EBAY: FUTURE
28
Slide29HDFS HA
HADOOP AT EBAY: FUTURE
29
Slide30HDFS Federation
HADOOP AT EBAY: FUTURE
30
Horizontal Scalability of HDFS Namespace
Multiple independent
NameNodes
serving a
subtree
of the
NameSpace
Example: NN1 provides /users, NN2 provides /reports
Slide31YARN
HADOOP AT EBAY: FUTURE
31
Hadoop 1.0:
MapReduce
JobTracker
and
TaskTracker
services
Handles Resource Management, Job Execution
Hadoop 2.0: YARN
Refactoring
Responsiblities
of
JobTracker
and
TaskTracker
into more general platform
Global
ResourceManager
Cluster-wide resource managements
Per-application
ApplicationMaster
Application-specific job control
Slide32YARN
HADOOP AT EBAY: FUTURE
32
Slide33YARN
HADOOP AT EBAY: FUTURE
33
Slide34YARN
HADOOP AT EBAY: FUTURE
34
Slide35YARN
HADOOP AT EBAY: FUTURE
35
Slide36New Scenarios
Iterative QueryStinger (Hive), Impala, etcRapid Data exploration and analysisGraph DatabasesTitanDB, Giraph Billions of vertices and edgesComplex Graph TraversalsApplications: PayPal fraud detection, Social Graph AnalysisReal-Time ProcessingStorm (Twitter), Apache S4Reinforcement Learning, Monitoring
HADOOP AT EBAY: FUTURE
36
Slide37Efficiency and Reliability
Storage EfficiencyHDFS introduces a 3x storage cost for its replicasHDFS-RAID: more reliability for 1.5x storage costReed-SolomonLocally Repairable Codes (Project Xorbas)Tradeoff: the cost of repairing lost data is much higherOperational EfficiencyMore automationMore self-service toolsBetter Monitoring
HADOOP AT EBAY: FUTURE
37
Slide38Open Source
HMC MetadataLong term goal: standardize on open source technologies (HCatalog)Short term: explore what should be open sourcedHadoop Management ConsoleHadoop Access Request AutomationBatch user creation and managementMetadata managementCode generation of dataset to Hive tables and Java POJOsldap_admin toolsVery useful but tightly coupled to eBay’s LDAP configurationWilling to open source if there is interest
HADOOP AT EBAY: FUTURE
38
Slide39THANK YOU
Questions?
Next Slides
Past, present and future of a retail concept: the hypermarke
The past, present and future of traditional knowledge
Past, present and future
Chaos in the past, present and future
Climate in spain: past, present and future
Past, present and future
Walks past present and future
Moses, past, present, future
Coal: past, present and future