Download Presentation - The PPT/PDF document "Hadoop @ eBay: Past, Present and Future" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation on theme: "Hadoop @ eBay: Past, Present and Future"— Presentation transcript:
Slide1
Hadoop @ eBay: Past, Present and Future
Ryan HennigHadoop Platform Team
Slide2
ABOUT ME
Slide3
RYAN HENNIG
Born and raised in Seattle, WAStudied Computer Science at University of Washington in SeattleWorked on Microsoft SQL Server 2006 – 2012Shipped SQL Server 2008, 2008 R2, 2012Joined eBay Hadoop team in early 2012 - Based in Bellevue, suburb of Seattle
2007-2010: eBay runs shared clusters on Cloudera Distribution of Hadoop2010-2012: eBay runs shared clusters on custom Hadoop versions2010: Wilma (based on 0.20)2011: Argon (based on 0.22)2012: Custom branch abandonedLessons LearnedForking a fast-changing open source project is difficult and riskyBalancing Development and operations needsDevelopment team sizeFacebook had 100eBay had 15Coordination with open source community = lots of overheadDivergence from open source: Push changes early and often
HADOOP AT EBAY: PAST
7
Slide8
HADOOP AT EBAY: PAST
8
Slide9
EBAY AND HORTONWORKS
2012: eBay enters partnership with HortonWorksGoalsFocus on eBay-specific development internallyLeverage HortonWorks expertise for general Hadoop DevelopmentAvoid source code divergence by making open source contribution a priorityBenefits to HortonWorksCredibility enhanced by having a well-known customerAbility to test at large scale
HADOOP AT EBAY: PAST
9
Slide10
HADOOP AT EBAY:PRESENT
Shared and Dedicated Clusters
Job Distribution
Use Case Examples
eBay Data Platform Overview
Slide11
SHARED AND DEDICATED CLUSTERS
Shared clusters10s of PB and 10s of thousands of slots per clusterUsed primarily for analytics of user behavior and inventoryMix of production and ad-hoc jobsMix of MR, Hive, PIG, Cascading etc.Hadoop and HBase security enabledDedicated clustersVery specific use cases like Index BuildingTight SLAs for jobs (in order of minutes)Immediate revenue impactUsually smaller than our shared clusters, but still big (100s of nodes…)
HADOOP AT EBAY: PRESENT
11
Slide12
JOB DISTRIBUTION BY TYPE
HADOOP AT EBAY: PRESENT
12
Slide13
USE CASE EXAMPLES
Cassini, eBay’s new search engine:Use MR to build full and incremental near-real-time indexesRaw Data is stored in HBase for efficient updates and random readStrong SLAs: < 10 minutesRun on dedicated clustersRelated and similar Items recommendations:Use transactional data, click stream data, search index, etc.Production MR jobs on a shared clusterAnalytics dashboard:Run Mobius MR jobs to join click stream data and transactional data Store summary data in HBaseWeb application to query HBase
HADOOP AT EBAY: PRESENT
13
Slide14
HADOOP OPERATIONS
LDAP Integration - All users stored in Active Directory, accessed via LDAP - Access to MapReduce Queues granted via MapReduce queues - Batch users: shared by a group of usersSecurity - Kerberos as implemented by Microsoft Active Directory - One domain for users, another for service/server principals - Batch users authenticated via keytabs, not passwordsMisc - 10’s of slave nodes are broken at any given time - Often need to add several racks of machines at a time
HADOOP AT EBAY: PRESENT
14
Slide15
HADOOP OPERATIONS
Team has Development and Operations Responsibilities2 Huge shared clusters1800+ users, exponential growthAbout 10 Hadoop developersRecently: operations work moved to dedicated teamDeveloped several tools to manage operationsHadoop Management Console: user-facing web appldap-admin: swiss-army knife style tool for hadoop adminsPuppet: for adding machines to the clusters, many racks at a timeDecom/Recom scripts: automatic detection, repair, decommission, and recommission of slave nodes
HADOOP AT EBAY: PRESENT
15
Slide16
HADOOP MANAGEMENT CONSOLE
Custom Web application built on Ruby on RailsSelf-service tools are continually added to reduce support loadUser ManagementAccess RequestsGroup MembershipBatch User ManagementNew RequestsSudoer managementDataset ManagementExplore DatasetsRequest New dataset transfer between Teradata and HadoopMetadata toolsEach dataset is stored in custom XML formatCode Generation: Hive Tables, Java POJOs
HADOOP AT EBAY: PRESENT
16
Slide17
HADOOP AT EBAY: PRESENT
17
Slide18
HADOOP AT EBAY: PRESENT
18
Slide19
HADOOP AT EBAY: PRESENT
19
Slide20
HADOOP AT EBAY: PRESENT
20
Slide21
HADOOP AT EBAY: PRESENT
21
Slide22
HADOOP AT EBAY: PRESENT
22
Slide23
HADOOP AT EBAY: PRESENT
23
Slide24
ldap-admin
Command-line tool written in RubySwiss-army knife tool, features added on demand for support issuesOften used features:Add a user to a groupView key details for LDAP users and groupsList all users, batch users, hadoop groupsReset batch user passwords and keytabsShow/add/remove sudoers for a batch accountRun user diagnostics: check permissions, keytabs, etc
HADOOP AT EBAY: PRESENT
24
Slide25
HADOOP AT EBAY:FUTURE
HDFS Federation
YARN
New Scenarios
Storage and Operational Efficiency
Slide26
HDFS HA and Federation
HDFS High-Availability for ReliabilityNameNode in Hadoop 1.0 is a Single Point of FailureAutomated failover to hot standby Depends on ZooKeeperHDFS Federation for Scalability and IsolationHadoop 1.0: Single NameNode service“Secondary NameNode” is not for failoverStorage scales horizontally, but Namespace scales verticallyNo isolation for different tenants or applicationsHadoop 2.0: HDFS FederationPartition the HDFS Namespace Many independent NameNodesAllows direct access to Block Storage w/o going through HDFS interface
Iterative QueryStinger (Hive), Impala, etcRapid Data exploration and analysisGraph DatabasesTitanDB, Giraph Billions of vertices and edgesComplex Graph TraversalsApplications: PayPal fraud detection, Social Graph AnalysisReal-Time ProcessingStorm (Twitter), Apache S4Reinforcement Learning, Monitoring
HADOOP AT EBAY: FUTURE
36
Slide37
Efficiency and Reliability
Storage EfficiencyHDFS introduces a 3x storage cost for its replicasHDFS-RAID: more reliability for 1.5x storage costReed-SolomonLocally Repairable Codes (Project Xorbas)Tradeoff: the cost of repairing lost data is much higherOperational EfficiencyMore automationMore self-service toolsBetter Monitoring
HADOOP AT EBAY: FUTURE
37
Slide38
Open Source
HMC MetadataLong term goal: standardize on open source technologies (HCatalog)Short term: explore what should be open sourcedHadoop Management ConsoleHadoop Access Request AutomationBatch user creation and managementMetadata managementCode generation of dataset to Hive tables and Java POJOsldap_admin toolsVery useful but tightly coupled to eBay’s LDAP configurationWilling to open source if there is interest
