Architecture Oct 27 2015 Anand Palanisamy Sr Manager Cloud Platform Engineering About PayPal 2 Founded in 1998 Gobal Company and head quartered in San Jose CA USA Public ID: 667254
Download Presentation The PPT/PDF document "PayPal Cloud Journey &" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
PayPal Cloud Journey & Architecture
Oct 27 2015
Anand Palanisamy
Sr. Manager, Cloud Platform EngineeringSlide2
About PayPal2
Founded
in 1998
Gobal
Company
and head
quartered
in San
Jose
, CA, USA
Public
company
, trading in Nasdaq (PYPL)
169
million
active
accounts
Operating
in 203
markets
with
100+
currencies
Processed
4
Billion
payments
in 2014
$235
Billion
(
Payment
Volume
)
More
info
at:
https://www.paypal.com/aboutSlide3
Design Goals3
A
platform
to
colloborate
internally
and
externally
Agility
for
both
Cloud
Users
and Cloud
Builder
Well-defined
Cloud
APIsSlide4
About our PayPal Cloud4
Background
Started in July 2012 with 1 engineer and 16 decommissioned servers
Today, one of the world’s
Largest
OpenStack
Private
Cloud
Number
of Physical Servers:
8064
Total
Cores
:
~400,000+
Number
of
VMs
: 82,000+
Block Storage:
2 peta bytes
Availability
Zones
: 10+
Largest
AZ
with
2,500
+
hypervisors
Business
Enablement
H
osting ~
100% of PayPal’s
production traffic (except Databases and Messaging)
Powers 100%
of
PaaS
,
Dev
/
QA and M&As
First production workload on SDN in 2013Slide5
Infrastructure-as-a- Service (
IaaS
)
Monitoring Alerting
Remediation
Compliance
Patching
Upgrades
CMS (Configuration Management System)
The Vision
5
Provide a platform that enables agility, availability and innovation.
Risk
Payments
Wallet
Mobile
Ads
DEV
QA
PROD
M&A’s
PAAS
L&P
Homegrown
CMS (CMDB)
Open Source
Puppet
Salt
Ansible
Homegrown
Reparo
Blink
StackWatch
StackMetrics
CloudInfo
FlyWay
Open Source
ELK (Elastic Search, Log Stash,
Kibana
)Zabbix
Nova, Neutron, Cinder, Swift, KeyStone, Glance
PayPal Compute SKUs, ToRs/Routers, LBs, Firewalls, Storage
CI, ALM, Monitoring, Release/Change
Mgmt
(Home Grown, Jenkins,
Docker,Mesos
..)
Platform Frameworks (Java, C++,
node.js
, Python,
Scala
..)Slide6
The Cloud Lab + First Seed6Slide7
The Journey so far…7Slide8
Cloud Deployment Architecture8
10,000
foot
level
Regions
Availability Zones- AZs
Region 2
AZ
1
AZ
2
AZ
n
Region 1
AZ
1
AZ
2
AZ
n
Region
m
AZ
1
AZ
2
AZ
n
AZ = fault-domain, single
OpenStack
control-plane instance
AZ maps to one or more physical network
bubbles
Multiple nova cells in
AZSlide9
Inside AZ9
Internet
Racks
Access
Aggregation
Core
SDN Gateways
LBs
Firewalls
Availability Zone
Cloud (
IaaS
+
PaaS
) Controllers, VMs, SDN controllersSlide10
Multi-tenancy model10
Virtual private cloud (VPC)
VPC is a collection of
OpenStack
tenants
It is a security zone
Keystone changes
Ideally single large (virtual) router or VRF per VPC
VPC model introduces scale issues
VPC Isolation evolved over period of time based on the network gearsSlide11
Challenges in managing 10+ AZs11
1000s of services across multiple data centers
GBs of logs every hour to dig for identifying potential issue
Many moving parts (Backbone network, LDAP Infra, DNS Infra, CORP LDAP & Network)
Network Infrastructure differences
Overall failure
Vs
1 or 2 failures
Global Keystone
Sync issue between AZ and Cells
Firewall between Control Plane and Hypervisors
Generations of Hardware, Firmware Versions, BIOS difference
Config
drift management
Capacity Management and Cloud
BackofficeSlide12
Challenges with DEV/QA VPC12
Unpredictable API Usage pattern
Single point of failure for VMs
Variety of workloads
Adhoc
Connectivity requirements
Patching VMs
Enforcing disciplineSlide13
13
Questions?
W
e
are hiring
https://
jobsearch.paypal-corp.com/jobs?keywords=cloud&page=1