Very Fast and Flexible Cloud/NFV - PowerPoint Presentation

1. Very Fast and Flexible Cloud/NFVSolution Stacks with FD.ioJerome Tollet, Frank Brockners (Cisco)1

2. Solution Stacks – A User Perspective:Above and below “The Line”2Network Controller;IO Abstraction & Feature Path Virtual Machine/ContainerLife Cycle Manager Service/WF Life Cycle ManagerVM Policy, Network PolicyService Provisioning, Service ConfigurationService Chaining, Service MonitoringAuto Recovery, Elastic Scaling, Workload Placement, Service AssuranceHypervisor/Host//ContainerComputeNetworkStoragePhys./virtual Network ControlGroup Policy, ChainingHigh-PerformanceFlexible Feature PathsVM/Container PolicyService MonitoringAuto HealingElastic ScalingService ProvisioningWorkload PlacementService ConfigurationService ModelApp IntentWorkFlow Topology

3. “The 20th century was about invention, the 21st is about mashups and integration”Toby FordFD.io Mini-Summit Sept, 20163

4. OpenSource Building Blocks4PaaSCloud Infra & ToolingInfrastructureHardwareNetwork ControlOperating SystemsVIM Management SystemOrchestrationApplication Layer / App ServerIO Abstraction & Feature Path Network Data AnalyticsAdditionalPaaS platforms

5. Composing the NO-STACK-WORLD5HardwareNetwork ControlOperating SystemsVIM Management SystemOrchestrationIO Abstraction & Feature Path Network Data AnalyticsApplication Layer / App ServerThe “No-Stack-Developer”Evolve/Integrate/Install/TestOPNFVComposeDeployTestEvolveIterate

6. Building Cloud/NFV Solution StacksOPNFV performs System Integration as an open community effort:Create/Evolve Components (in lock-step with Upstream Communities)Compose / Deploy / TestIterate (in a distributed, multi-vendor CI/CD system)Let’s add “fast and flexible networking” as another focus…6Network ControllerForwarder – Switch/RouterVirtual Machine/ContainerLife Cycle Manager Service/WF Life Cycle ManagerServiceModelApp IntentWorkFlow Topology

7. Foundational Assets For NFV Infrastructure:A stack is only as good as its foundationForwarderFeature rich, high performance, highly scalable virtual switch-routerLeverages hardware acceleratorsRuns in user spaceModular and easy extensibleForwarder Diversity: Hardware and SoftwareVirtual Domains link and interact with physical domainsDomains and PolicyConnectivity should reflect business logic instead of physical L2/L3 constructs7Network ControllerForwarder – Switch/RouterVirtual Machine/ContainerLife Cycle Manager Service/WF Life Cycle ManagerServiceModelApp IntentWorkFlow Topology

8. Evolving The OPNFV Solution Set:OPNFV FastDataStacks ProjectOPNFV develops, integrates, and continuously tests NFV solution stacks: Historically OPNFV solution stacks only used OVS as virtual forwarderObjective: Create a new stacks which significantly evolve networking for NFVCurrent scenariosOpenStack – OpenDaylight (Layer2) – VPPOpenStack – OpenDaylight (Layer3) – VPPOpenStack – VPP...Diverse set of contributors: https://wiki.opnfv.org/display/fds + VPPInstall ToolsVM ControlNetwork ControlApex, Compass,Fuel, JujuOpenStackOpenDaylight,ONOS, OpenContrailHypervisorKVM, KVM4NFVForwarderOVS, OVS-DPDKComponents in OPNFVCategory

9. The Foundation is the Fast Forwarder:FD.io/VPP – The Universal Dataplane9

10. FD.io is…Project at Linux FoundationMulti-partyMulti-projectSoftware DataplaneHigh throughputLow LatencyFeature RichResource EfficientBare Metal/VM/ContainerMultiplatform 10Bare Metal/VM/ContainerFd.io Scope:Network IO - NIC/vNIC <-> cores/threadsPacket Processing – Classify/Transform/Prioritize/Forward/Terminate Dataplane Management Agents - ControlPlaneDataplane Management AgentPacket ProcessingNetwork IO

11. Multiparty: Contributor/Committer Diversity

12. Multiproject: FD.io Projects12Honeycombhc2vppDataplane Management AgentCSITpuppet-fdiotrexTesting/SupportNSH_SFCONETLDKodp4vppCICNVPP SandboxVPPPacket Processingdeb_dpdkrpm_dpdkNetwork IOgovpp

13. Vector Packet Processor - VPPPacket Processing Platform:High performanceLinux User spaceRun’s on commodity CPUs: / /Shipping at volume in server & embedded products since 2004.13Bare Metal/VM/ContainerDataplane Management AgentPacket ProcessingNetwork IO

14. VPP Universal Fast Dataplane: Performance at Scale [1/2]Per CPU core throughput with linear multi-thread(-core) scalingHardware:Cisco UCS C240 M4Intel® C610 series chipset2 x Intel® Xeon® Processor E5-2698 v3 (16 cores, 2.3GHz, 40MB Cache)2133 MHz, 256 GB Total6 x 2p40GE Intel XL710=12x40GETopology:Phy-VS-PhySoftwareLinux: Ubuntu 16.04.1 LTSKernel: ver. 4.4.0-45-genericFD.io VPP: VPP v17.01-5~ge234726 (DPDK 16.11)Resources1 physical CPU core per 40GE portOther CPU cores available for other services and other work20 physical CPU cores available in 12x40GE seuptLots of Headroom for much more throughput and featuresIPv4 RoutingIPv6 Routing

15. Hardware:Cisco UCS C240 M4Intel® C610 series chipset2 x Intel® Xeon® Processor E5-2698 v3 (16 cores, 2.3GHz, 40MB Cache)2133 MHz, 256 GB Total6 x 2p40GE Intel XL710=12x40GETopology:Phy-VS-PhySoftwareLinux: Ubuntu 16.04.1 LTSKernel: ver. 4.4.0-45-genericFD.io VPP: VPP v17.01-5~ge234726 (DPDK 16.11)Resources1 physical CPU core per 40GE portOther CPU cores available for other services and other work20 physical CPU cores available in 12x40GE seuptLots of Headroom for much more throughput and featuresVPP Universal Fast Dataplane: Performance at Scale [2/2]Per CPU core throughput with linear multi-thread(-core) scalingL2 SwitchingL2 Switching with VXLAN Tunneling

16. VPP: How does it work?* approx. 173 nodes in default deploymentethernet-inputdpdk-inputaf-packet-inputvhost-user-inputmpls-inputlldp-input...-no-checksumip4-inputip6-inputarp-inputcdp-inputl2-inputip4-lookupip4-lookup-mulitcastip4-rewrite-transitip4-load-balanceip4-midchainmpls-policy-encapinterface-outputPacket 0Packet 1Packet 2Packet 3Packet 4Packet 5Packet 6Packet 7Packet 8Packet 9Packet 1012Packet processing is decomposed into a directed graph node …… packets moved through graph nodes in vector …Instruction CacheData CacheMicroprocessor… graph nodes are optimized to fit inside the instruction cache …34… packets are pre-fetched, into the data cache …

17. dispatch fn()Get pointer to vectorPREFETCH #3 and #4PROCESS #1 and #2ASSUME next_node same as last packetUpdate counters, advance buffersEnqueue the packet to next_node<as above but single packet>while packets in vectorwhile 4 or more packetswhile any packetsMicroprocessorethernet-inputPacket 1Packet 2… packets are processed in groups of four, any remaining packets are processed on by one … 4… instruction cache is warm with the instructions from a single graph node … 5… data cache is warm with a small number of packets .. 6VPP: How does it work?

18. dispatch fn()Get pointer to vectorPREFETCH #1 and #2PROCESS #1 and #2ASSUME next_node same as last packetUpdate counters, advance buffersEnqueue the packet to next_node<as above but single packet>while packets in vectorwhile 4 or more packetswhile any packetsMicroprocessorethernet-inputPacket 1Packet 2… prefetch packets #1 and #2 …7VPP: How does it work?

19. dispatch fn()Get pointer to vectorPREFETCH #3 and #4PROCESS #1 and #2ASSUME next_node same as last packetUpdate counters, advance buffersEnqueue the packet to next_node<as above but single packet>while packets in vectorwhile 4 or more packetswhile any packetsMicroprocessorethernet-inputPacket 1Packet 2Packet 3Packet 4… process packet #3 and #4 …… update counters, enqueue packets to the next node …VPP: How does it work?8

20. VPP Features as of 17.01 Release20Tunnels/EncapsGRE/VXLAN/VXLAN-GPE/LISP-GPE/NSHIPSEC Including HW offload when availableInterfacesDPDK/Netmap/AF_Packet/TunTapVhost-user - multi-queue, reconnect,Jumbo Frame SupportMPLSMPLS over Ethernet/GREDeep label stacks supportedSegment RoutingSR MPLS/IPv6 Including MulticastInband iOAMTelemetry export infra (raw IPFIX)iOAM for VXLAN-GPE (NGENA)SRv6 and iOAM co-existenceiOAM proxy mode / cachingiOAM probe and responder LISPLISP xTR/RTR L2 Overlays over LISP and GRE encaps Multitenancy Multihome Map/Resolver Failover Source/Dest control plane support Map-Register/Map-Notify/RLOC-probingLanguage BindingsC/Java/Python/LuaHardware PlatformsPure Userspace - X86,ARM 32/64,PowerRaspberry PiRoutingIPv4/IPv614+ MPPS, single coreHierarchical FIBsMultimillion FIB entriesSource RPFThousands of VRFs Controlled cross-VRF lookupsMultipath – ECMP and Unequal CostNetwork ServicesDHCPv4 client/proxyDHCPv6 ProxyMAP/LW46 – IPv4aasMagLev-like LoadIdentifier Locator AddressingNSH SFC SFF’s & NSH ProxyLLDPBFDPolicerMultiple million Classifiers – Arbitrary N-tupleSwitchingVLAN Support Single/ Double tag L2 forwd w/EFP/BridgeDomain conceptsVTR – push/pop/Translate (1:1,1:2, 2:1,2:2)Mac Learning – default limit of 50k addrBridging Split-horizon group support/EFP FilteringProxy ArpArp terminationIRB - BVI Support with RouterMac assigmtFloodingInput ACLsInterface cross-connectL2 GRE over IPSec tunnelsMonitoringSimple Port Analyzer (SPAN)IP Flow Export (IPFIX)Counters for everythingLawful InterceptSecurityMandatory Input Checks: TTL expiration header checksum L2 length < IP length ARP resolution/snooping ARP proxySNATIngress Port Range FilteringPer interface whitelistsPolicy/Security Groups/GBP (Classifier)

21. 21Rapid Release Cadence – ~3 months16-02Fd.io launch16-06Release- VPP16-09Release:VPP, Honeycomb, NSH_SFC, ONE17-01Release:VPP, Honeycomb, NSH_SFC, ONE16-06 New FeaturesEnhanced Switching & Routing IPv6 SR multicast support LISP xTR support VXLAN over IPv6 underlay per interface whitelists shared adjacencies in FIBImproves interface support vhost-user – jumbo frames Netmap interface support AF_Packet interface supportImproved programmability Python API bindings Enhanced JVPP Java API bindings Enhanced debugging cliHardware and Software Support Support for ARM 32 targets Support for Raspberry Pi Support for DPDK 16.0416-09 New FeaturesEnhanced LISP support for L2 overlays Multitenancy Multihoming Re-encapsulating Tunnel Routers (RTR) support Map-Resolver failover algorithmNew plugins for SNAT MagLev-like Load Identifier Locator Addressing NSH SFC SFF’s & NSH ProxyPort range ingress filteringDynamically ordered subgraphs17-01 New FeaturesHierarchical FIBPerformance Improvements DPDK input and output nodes L2 Path IPv4 lookup nodeIPSEC Softwand HWCrypto SupportHQoS supportSimple Port Analyzer (SPAN)BFDIPFIX ImprovementsL2 GRE over IPSec tunnelsLLDPLISP Enhancements Source/Dest control plane L2 over LISP and GRE Map-Register/Map-Notify RLOC-probingACLFlow Per PacketSNAT – Multithread, Flow ExportLUA API Bindings

22. 22New in 17.04 ReleaseVPP Userspace Host StackTCP stackDHCPv4 relay multi-destinationDHCPv4 option 82DHCPv6 relay multi-destinationDHPCv6 relay remote-idND ProxySNATCGN: Configurable port allocationCGN: Configurable Address poolingCPE: External interface DHCP supportNAT64, LW46Security GroupsRouted interface supportL4 filters with IPv6 Extension HeadersAPIMove to CFFI for Python bindingPython Packaging improvementsCLI over APIImproved C/C++ language bindingSegment Routing v6SR policies with weighted SID listsBinding SIDSR steering policiesSR LocalSIDsFramework to expand local SIDs w/pluginsiOAMUDP Pinger w/path fault isolationIOAM as type 2 metadata in NSHIOAM raw IPFIX collector and analyzerAnycast active server selectionIPFIXCollect IPv6 informationPer flow state

23. Key TakeawaysFeature Rich High Performance480 Gbps/200mpp at scale (8 million routes)Many Use Cases:Bare Metal/Embedded/Cloud/Containers/NFVi/VNFs 23

24. Solutions Stacks:Fast and extensible networking natively integrated into OpenStack24

25. What is “networking-vpp”FD.io / VPP is a fast software dataplane that can be used to speed up communications for any kind of VM or VNF.VPP can speed-up both East-West and North-South communicationsNetworking-vpp is a project aiming at providing a simple, robust, production grade integration of VPP in OpenStack using ml2 interfaceGoal is to make VPP a first class citizen component in OpenStack for NFV and Cloud applications25

26. Networking-vpp: Design PrinciplesMain design goals are: simplicity, robustness, scalabilityEfficient management communicationsAll communication is asynchronousAll communication is REST basedRobustnessBuilt for failure – if a cloud runs long enough, everything will happen eventuallyAll modules are unit and system testedCode is small and easy to understand (no spaghetti/legacy code)26

27. Networking-vpp: current feature setNetwork typesVLAN: supported since version 16.09VXLAN-GPE: supported since version 17.04Port typesVM connectivity done using fast vhostuser interfacesTAP interfaces for services such as DHCPSecuritySecurity-groups based on VPP stateful ACLsPort Security can be disabled for true fastpathRole Based Access Control and secure TLS connections for etcdLayer 3 NetworkingNorth-South Floating IPNorth-South SNATEast-West Internal GatewayRobustnessIf Neutron commits to it, it will happenComponent state resync in case of failure: recovers from restart of Neutron, the agent and VPP27

28. Networking-vpp, what is your problem?You have a controller and you tell it to do somethingIt talks to a device to get the job doneDid the message even get sent, or did the controller crash first? Does the controller believe it sent the message when it restarts?Did the message get sent and the controller crash before it got a reply?Did it get a reply but crash before it processed it?If the message was sent and you got a reply, did the device get programmed?If the message was sent and you didn’t get a reply, did the device get programmed?28

29. Networking-vpp, what is your problem?If you give a device a list of jobs to do, it’s really hard to make sure it gets them all and acts on themIf you give a device a description of the state you want it to get to, the task can be made much easier29

30. Networking-vpp: overall architectureCompute NodeVPPML2 AgentVMVMVMvhostuserCompute NodeVPPML2 AgentVMVMVMvhostuserNeutron ServerML2 VPPMechanism DriverjournalingHTTP/jsondpdkdpdkvlan / flat network30

31. Networking-vpp: Resync mechanismThe agent marks everything it puts in VPPIf the agent restarts, it comes up with no knowledge of what’s in VPP, so it reads those marks backWhile it’s been gone, things may have happened and etcd will have changedFor each item, it can see if it’s correct or if it’s out of date (for instance, deleted), and it can also spot new ports it’s been asked to makeWith that information, it does the minimum necessary to fix VPP’s stateThis means that while the agent’s gone, traffic keeps movingNeutron is abiding by its promises (‘I will do this thing for you at the first possible moment’)31Compute NodeVPPML2 AgentVMvhostuserdpdk

32. Networking-vpp: HA etcd deploymentetcd is a strictly consistent store that can run with multiple processesThey talk to each other as they get the job doneA client can talk to any of themVarious deployment models – the client can use a proxy or talk directly to the etcd processes32The serviceThe processesClient

33. Networking-vpp: Role Based Access ControlSecurity Hardening TLS communication between nodes and ETCD : Confidentiality and integrity of the messages in transit and authentication of ETCD server.ETCD RBAC :Limit the impact of a compromised Compute Node to the node itself33Compute Node 1Neutron ServerHTTPS/jsonCompute Node 2

34. Networking-vpp: RoadmapNext version will be networking-vpp 17.07Security Groupssupport for remote-group-IDSupport for additional protocol fieldsVXLAN-GPEsupport for ARP handling in VPPResync states in case of agent restartImproved layer 3 support:support for HA (VRRP based)Resync for layer ports34

35. Solution Stacks for enhanced Network Control:OpenStack – OpenDaylight – FD.io/VPP35

36. Towards a Stack with enhanced Network ControlFD.io/VPPHighly scalable, highperformance, extensible virtualforwarderOpenDaylight Network ControllerExtensible controller platformDecouple business logic from network constructs: Group Based Policy as mediator between business logic and network constructsSupport for a diverse set of network devicesClustering for HA36

37. Solution Stack Ingredients and their EvolutionOpenDaylightGroup Based Policy (GBP) Neutron MapperGBP Renderer Manager enhancementsVPP RendererVirtual Bridge Domain Mgr / Topology ManagerFD.ioHoneyComb – EnhancementsVPP – Enhancements CSIT – VPP component testsOPNFVOverall System Composition – Integration into CI/CDInstaller: Integration of VPP into APEXSystem Test: FuncTest and Yardstick system test application to FDS37See also:FDS Architecture: https://wiki.opnfv.org/display/fds/OpenStack-ODL-VPP+integration+design+and+architecture NeutronNeutron NorthBoundGBP Neutron MapperTopology Mgr - VBDHoneycomb (Dataplane Agent)VPPRESTNetconf/YANG...VPP rendererDPDKGBP Renderer ManagerSystem Install(APEX)System Test(FuncTest, Yardstick)

38. HoneycombVPP 2Example: Creating a Neutron vhostuser port on VPP38Topology Manager (vBD)HoneycombVPP 1VPP RendererNeutronNeutron NorthBoundGBP Neutron MapperGBP Renderer ManagerNetconf/YANGPOST PORT(id=<uuid>, host_id=<vpp>, vif_type=vhostuser) Update PortMap Port to GBP EndpointUpdate/Create Policy involving GBP EndpointResolve PolicyApply policy, update nodesconfigure interfaces over NetconfNetconf/YANGBridge domain and tunnel configvhostuserVMVXLAN TunnelConfigure bridge domain on nodes over NetConf

39. Computenode-0Computenode-1HoneyCombHoneyCombHoneyCombqrouter (NAT)Tenant network i/fTenant network i/fTenant network i/fBridgeDomainVM 2vhost-userControlnode-0VXLANVPPExternal network i/fDHCPtapOVS (br-ex)tapVXLANVXLANFastDataStacks: OS – ODL(L2) – FD.io Example: 3 node setup: 1 x Controller, 2 x Compute39VPPVPPOpenStack ServicesNetwork ControlVM 1BridgeDomainBridgeDomainvhost-userInternet

40. Computenode-0Computenode-1Tenant network i/fTenant network i/fTenant network i/fBridgeDomainVM 2vhost-userControlnode-0VXLANVPPExternal network i/fInternetDHCPtapVXLANVXLANFastDataStacks: OS – ODL(L3) – FD.io Example: 3 node setup: 1 x Controller, 2 x Compute40VPPVPPOpenStack ServicesNetwork ControlVM 1BridgeDomainBridgeDomainvhost-userHoneyCombHoneyCombHoneyComb

41. FastDataStacks: Status41Danube 1.0 (March 2017)Enhanced O/S-ODL(L3)-VPP stack (Infra complete: Neutron / GBP Mapper / GBP Renderer / VBD / Honeycomb / VPP)L2 and L3 networking using ODL (incl. east-west security groups)Colorado 1.0 (September 2016)Base O/S-ODL(L2)-VPP stack (Infra: Neutron / GBP Mapper / GBP Renderer / VBD / Honeycomb / VPP)Automatic InstallBasic system-level testingL2 networking using ODL (no east-west security groups), L3 networking uses qrouter/OVSOverlays: VXLAN, VLANColorado 3.0 (December 2016)Enhanced O/S-ODL(L2)-VPP stack (Infra complete: Neutron / GBP Mapper / GBP Renderer / VBD / Honeycomb / VPP)Enhanced system-level testingL2 networking using ODL (incl. east-west security groups), L3 networking uses qrouter/OVSO/S-VPP (Infra: Neutron ML2-VPP / Networking-vpp-agent / VPP)Automatic Install, Overlays: VLANDanube 2.0 (May 2017)Enhanced O/S-ODL(L3/L2)-VPP stack: HA for OpenStack and ODL (clustering)

42. FastDataStacks – Next StepsSimple and efficient forwarding modelClean separation of “forwarding” and “policy”:Pure Layer 3 with distributed routing: Every VPP node serves as a router, “no bridging anywhere”Contracts/Isolation managed via Group Based PolicyFlexile Topology Services: LISP integration, complementing VBDAnalytics integration into the solution stacksIntegration of OPNFV projects:Bamboo (PNDA.io for OPNFV)Virtual Infrastructure Networking Assurance (VINA)NFVbench (Full Stack NFVI one-shot benchmarking)Container Stack using FD.io/VPPIntegrating Docker, K8s, Contiv, FD.io/VPP container networking, Spinnaker 42

43. 43An NFV Solution Stack is only as good as its foundation

44. Thank you44