Hadoop @ eBay: Past, Present and Future
43K - views

Hadoop @ eBay: Past, Present and Future

Ryan Hennig. Hadoop Platform Team. ABOUT ME. RYAN HENNIG. Born and raised in Seattle, WA. Studied Computer Science at University of Washington in Seattle. Worked on Microsoft SQL Server 2006 – 2012.

Download Presentation

Hadoop @ eBay: Past, Present and Future




Download Presentation - The PPT/PDF document "Hadoop @ eBay: Past, Present and Future" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.



Presentation on theme: "Hadoop @ eBay: Past, Present and Future"— Presentation transcript:

Slide1

Hadoop @ eBay: Past, Present and Future

Ryan HennigHadoop Platform Team

Slide2

ABOUT ME

Slide3

RYAN HENNIG

Born and raised in Seattle, WAStudied Computer Science at University of Washington in SeattleWorked on Microsoft SQL Server 2006 – 2012Shipped SQL Server 2008, 2008 R2, 2012Joined eBay Hadoop team in early 2012 - Based in Bellevue, suburb of Seattle

COMPUTE AND DATA INFRASTRUCTURE

3

Slide4

AGENDA

Past

: Growth of Hadoop at eBay

Present

: Hadoop Use Cases, Operations Tools

Future

: Hadoop 2.0

Slide5

HADOOP AT EBAY: PAST

Growth of Hadoop at eBay

Adventures in Forking

Partnership with

Hortonworks

Slide6

HADOOP EVOLUTION @ eBay

HADOOP AT EBAY: PAST

6

2007

Single digit nodes

2010

Shared cluster

100s nodes

1000s + core

PBCDH2

2011

Shared clusters1000s node10,000+ core10s PBWilma (0.20)

2012

Shared clusters1000s node10,000+ core10s PBArgon (0.22)

2013

Shared clusters 4k+ node40,000+ core50s PBHDP 1.x

2009

Search

10s

- nodes

Slide7

ADVENTURES IN FORKING

2007-2010: eBay runs shared clusters on Cloudera Distribution of Hadoop2010-2012: eBay runs shared clusters on custom Hadoop versions2010: Wilma (based on 0.20)2011: Argon (based on 0.22)2012: Custom branch abandonedLessons LearnedForking a fast-changing open source project is difficult and riskyBalancing Development and operations needsDevelopment team sizeFacebook had 100eBay had 15Coordination with open source community = lots of overheadDivergence from open source: Push changes early and often

HADOOP AT EBAY: PAST

7

Slide8

HADOOP AT EBAY: PAST

8

Slide9

EBAY AND HORTONWORKS

2012: eBay enters partnership with HortonWorksGoalsFocus on eBay-specific development internallyLeverage HortonWorks expertise for general Hadoop DevelopmentAvoid source code divergence by making open source contribution a priorityBenefits to HortonWorksCredibility enhanced by having a well-known customerAbility to test at large scale

HADOOP AT EBAY: PAST

9

Slide10

HADOOP AT EBAY:PRESENT

Shared and Dedicated Clusters

Job Distribution

Use Case Examples

eBay Data Platform Overview

Slide11

SHARED AND DEDICATED CLUSTERS

Shared clusters10s of PB and 10s of thousands of slots per clusterUsed primarily for analytics of user behavior and inventoryMix of production and ad-hoc jobsMix of MR, Hive, PIG, Cascading etc.Hadoop and HBase security enabledDedicated clustersVery specific use cases like Index BuildingTight SLAs for jobs (in order of minutes)Immediate revenue impactUsually smaller than our shared clusters, but still big (100s of nodes…)

HADOOP AT EBAY: PRESENT

11

Slide12

JOB DISTRIBUTION BY TYPE

HADOOP AT EBAY: PRESENT

12

Slide13

USE CASE EXAMPLES

Cassini, eBay’s new search engine:Use MR to build full and incremental near-real-time indexesRaw Data is stored in HBase for efficient updates and random readStrong SLAs: < 10 minutesRun on dedicated clustersRelated and similar Items recommendations:Use transactional data, click stream data, search index, etc.Production MR jobs on a shared clusterAnalytics dashboard:Run Mobius MR jobs to join click stream data and transactional data Store summary data in HBaseWeb application to query HBase

HADOOP AT EBAY: PRESENT

13

Slide14

HADOOP OPERATIONS

LDAP Integration - All users stored in Active Directory, accessed via LDAP - Access to MapReduce Queues granted via MapReduce queues - Batch users: shared by a group of usersSecurity - Kerberos as implemented by Microsoft Active Directory - One domain for users, another for service/server principals - Batch users authenticated via keytabs, not passwordsMisc - 10’s of slave nodes are broken at any given time - Often need to add several racks of machines at a time

HADOOP AT EBAY: PRESENT

14

Slide15

HADOOP OPERATIONS

Team has Development and Operations Responsibilities2 Huge shared clusters1800+ users, exponential growthAbout 10 Hadoop developersRecently: operations work moved to dedicated teamDeveloped several tools to manage operationsHadoop Management Console: user-facing web appldap-admin: swiss-army knife style tool for hadoop adminsPuppet: for adding machines to the clusters, many racks at a timeDecom/Recom scripts: automatic detection, repair, decommission, and recommission of slave nodes

HADOOP AT EBAY: PRESENT

15

Slide16

HADOOP MANAGEMENT CONSOLE

Custom Web application built on Ruby on RailsSelf-service tools are continually added to reduce support loadUser ManagementAccess RequestsGroup MembershipBatch User ManagementNew RequestsSudoer managementDataset ManagementExplore DatasetsRequest New dataset transfer between Teradata and HadoopMetadata toolsEach dataset is stored in custom XML formatCode Generation: Hive Tables, Java POJOs

HADOOP AT EBAY: PRESENT

16

Slide17

HADOOP AT EBAY: PRESENT

17

Slide18

HADOOP AT EBAY: PRESENT

18

Slide19

HADOOP AT EBAY: PRESENT

19

Slide20

HADOOP AT EBAY: PRESENT

20

Slide21

HADOOP AT EBAY: PRESENT

21

Slide22

HADOOP AT EBAY: PRESENT

22

Slide23

HADOOP AT EBAY: PRESENT

23

Slide24

ldap-admin

Command-line tool written in RubySwiss-army knife tool, features added on demand for support issuesOften used features:Add a user to a groupView key details for LDAP users and groupsList all users, batch users, hadoop groupsReset batch user passwords and keytabsShow/add/remove sudoers for a batch accountRun user diagnostics: check permissions, keytabs, etc

HADOOP AT EBAY: PRESENT

24

Slide25

HADOOP AT EBAY:FUTURE

HDFS Federation

YARN

New Scenarios

Storage and Operational Efficiency

Slide26

HDFS HA and Federation

HDFS High-Availability for ReliabilityNameNode in Hadoop 1.0 is a Single Point of FailureAutomated failover to hot standby Depends on ZooKeeperHDFS Federation for Scalability and IsolationHadoop 1.0: Single NameNode service“Secondary NameNode” is not for failoverStorage scales horizontally, but Namespace scales verticallyNo isolation for different tenants or applicationsHadoop 2.0: HDFS FederationPartition the HDFS Namespace Many independent NameNodesAllows direct access to Block Storage w/o going through HDFS interface

HADOOP AT EBAY: FUTURE

26

Slide27

HDFS HA

HADOOP AT EBAY: FUTURE

27

Slide28

HDFS HA

HADOOP AT EBAY: FUTURE

28

Slide29

HDFS HA

HADOOP AT EBAY: FUTURE

29

Slide30

HDFS Federation

HADOOP AT EBAY: FUTURE

30

Horizontal Scalability of HDFS Namespace

Multiple independent

NameNodes

serving a

subtree

of the

NameSpace

Example: NN1 provides /users, NN2 provides /reports

Slide31

YARN

HADOOP AT EBAY: FUTURE

31

Hadoop 1.0:

MapReduce

JobTracker

and

TaskTracker

services

Handles Resource Management, Job Execution

Hadoop 2.0: YARN

Refactoring

Responsiblities

of

JobTracker

and

TaskTracker

into more general platform

Global

ResourceManager

Cluster-wide resource managements

Per-application

ApplicationMaster

Application-specific job control

Slide32

YARN

HADOOP AT EBAY: FUTURE

32

Slide33

YARN

HADOOP AT EBAY: FUTURE

33

Slide34

YARN

HADOOP AT EBAY: FUTURE

34

Slide35

YARN

HADOOP AT EBAY: FUTURE

35

Slide36

New Scenarios

Iterative QueryStinger (Hive), Impala, etcRapid Data exploration and analysisGraph DatabasesTitanDB, Giraph Billions of vertices and edgesComplex Graph TraversalsApplications: PayPal fraud detection, Social Graph AnalysisReal-Time ProcessingStorm (Twitter), Apache S4Reinforcement Learning, Monitoring

HADOOP AT EBAY: FUTURE

36

Slide37

Efficiency and Reliability

Storage EfficiencyHDFS introduces a 3x storage cost for its replicasHDFS-RAID: more reliability for 1.5x storage costReed-SolomonLocally Repairable Codes (Project Xorbas)Tradeoff: the cost of repairing lost data is much higherOperational EfficiencyMore automationMore self-service toolsBetter Monitoring

HADOOP AT EBAY: FUTURE

37

Slide38

Open Source

HMC MetadataLong term goal: standardize on open source technologies (HCatalog)Short term: explore what should be open sourcedHadoop Management ConsoleHadoop Access Request AutomationBatch user creation and managementMetadata managementCode generation of dataset to Hive tables and Java POJOsldap_admin toolsVery useful but tightly coupled to eBay’s LDAP configurationWilling to open source if there is interest

HADOOP AT EBAY: FUTURE

38

Slide39

THANK YOU

Questions?