Norgan Systems Engineer BeardyMcBeards in openvswitch Managing Open vSwitch Across a Large Heterogeneous Fleet 2 wwwrackspacecom About Rackspace 9 Worldwide Data Centers 5000 ID: 600806
Download Presentation The PPT/PDF document "Chad" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Chad NorganSystems EngineerBeardyMcBeards in #openvswitch
Managing Open vSwitch Across a Large Heterogeneous
FleetSlide2
2
www.rackspace.com
About Rackspace
9
Worldwide
Data Centers
5,000+ Rackers
300,000+ Customers≅70 PB Stored
Global FootprintCustomers in 120+ Countries
Portfolio of
Hosted Solutions
Dedicated - Cloud - Hybrid
Annualized Revenue
Over
$1B
60%
100
OF
THE
We
Serve
FORTUNE
®
OVERSlide3
Rackspace’s Public CloudLarge Fleet
Heterogenous
Several different hardware manufacturersSeveral XenServer major versions (sometimes on varying kernels)
Five
networking configurationsSix production public cloudsSix internal private clouds
Various non-production environments
Tens of thousands of hypervisorsHundreds of thousands of virtual
machines1 - 3+ Interfaces Slide4
Networks Available to CustomersSlide5
Rackspace has used Open vSwitch since the 0.9 versionBehind most of First Generation Cloud Servers (Slicehost)
Powers 100% of Next Generation Cloud
ServersUpgraded OVS nine times since the launch of Next Gen Public Cloud in August
2012
Our History With OVSSlide6
Service provider features:Overlay NetworksQoSVLAN TaggingPort SecurityLACPSoftware = FlexibleUpgrades are easier than hardwareWhy We Use OVSSlide7
Our Favorite ImprovementsSlide8
Example: Busy HV With Syslog Collector Slide9
Mission Accomplished!We moved the bottleneck!New bottlenecks:Guest OS kernel configurationXen Netback/Netfront
DriverSlide10
Matching the OVS kernel module to both the running and staged kernelHypervisor updates often come with a newer kernelWe often don’t immediately rebootRunning kernel != Kernel at next rebootDetect both kernels and install both sets of OVS kernel modulesHeterogeneous ScaleChallenges of Upgrading OVSSlide11
Playbook-style upgradesAsynchronous plays with parallel limitsExtensibleEasy to build validations and pre-checks to prevent unwanted impact
OVS
Upgrade SolutionSlide12
Architectural Basics
Patch Port
Tunnel Encapsulation
VIF
PIF
VIF
VIF
PIF
Integration BridgeInterface BridgeSlide13
Ansible + OVS = Flexible Network Rewiring
Tunnel
Encap
VIF
PIF
VIF
VIF
Integration Bridge
Interface BridgePatch PortPIFSlide14
Ansible + OVS = Flexible Network Rewiring
Tunnel
Encap
VIF
PIF
VIF
VIF
Integration Bridge
Interface BridgePatch PortPIF
Public NetBridgePatch PortSlide15
Ansible + OVS = Flexible Network Rewiring
Tunnel
Encap
VIF
PIF
VIF
VIF
Integration Bridge
Interface BridgePatch Port
PIFPublic NetBridge
Patch PortSlide16
Ansible + OVS = Flexible Network Rewiring
Patch Port
Tunnel
Encap
VIF
PIF
VIF
VIF
Integration BridgePublic NetBridgeService Net Bridge
Interface BridgePatch PortSlide17
Ansible + OVS = Flexible Network Rewiring
Patch Port
Tunnel
Encap
VIF
PIF
VIF
VIF
Integration BridgePublic NetBridgeService Net Bridge
Cloud Net BridgeInterface Bridge
Patch Port
Patch PortSlide18Slide19
Ansible + OVS = Flexible Network Rewiring
Patch Port
PIF
VIF
Public Net
Bridge
Public Net Bridge
Interface Bridge
Patch PortPublic Net Bridge_oldSlide20
Measuring OVS – PavlOVS.pyPublishes metrics to StatsD/GraphitePer
bridge
byte, packet, open flow countDatapath hit, missed,
lost, flow counts
Open vSwitch CPU utilizationInstance countTunnels configured and in fault stateSlide21
2000 Eviction Threshold
Datapath
Flow CountSlide22
Datapath Flow CountSlide23
Hit, Miss, Lost
Hit, Miss, LostSlide24
OVS CPU
OVS CPU By CellSlide25
The OVS Of Our DreamsConnection TrackingMore (efficient) performance
JSON Output
from ovs-*ctl commandsSlide26Slide27
QUESTIONS?