Test Achievements 2 Sept 2013 June 2015 Beryllium Release First Release Performance report Helium Release First Performance test OpenFlow Cbench Sept 2014 Integration group ID: 1040145
Download Presentation The PPT/PDF document "Test and Performance Integration Group" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Test and PerformanceIntegration Group
2. Test Achievements2Sept2013June2015Beryllium ReleaseFirst Release Performance reportHelium ReleaseFirst Performance test: OpenFlow (Cbench)Sept2014Integration groupNew project scope: distribution and CI system testFeb2014Hydrogen ReleaseFirst system test: OpenFlow, OVSDB and VTN ManagerFeb2016Lithium ReleaseFirst Cluster testFirst SSP suite for OpenFlow
3. Build test: Junit, integration, SingleFeatureSystem test: feature, multi-featureSSP tests: Stability, Scalability, PerformanceCluster tests: Data consistency, HAContinuous Integration Tests3
4. CI Test Strategy4Code MergeAuto-ReleaseMerge buildDistributionfeature testfeature test cluster testFor all projectsCode SubmitVerify buildBuild testcluster testFor all projects in downstreamfeature testFor all projects in downstreamBuild testDistribution-check testAuto-Release testcluster testDaily triggerSSP tests Multi-feature testFor all projectsMulti-feature test
5. Build test: https://jenkins.opendaylight.org/releng/view/Merge-Jobs/System test: https://jenkins.opendaylight.org/releng/view/CSIT-1node/SSP tests: https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Performance_Test:ResultsCluster tests: https://jenkins.opendaylight.org/releng/view/CSIT-3node/CI Tests Results5
6. Real network test scenarios.Uniform and dedicated test environment.More than one environment to compare results.Tests documented and step-by-step reproducible.Other SDN controllers results added for reference.Recommendations for ODL best performance.Beryllium Performance Report6
7. OpenFlowEnd-to-End performanceSouthbound performanceNetconfNorthbound performanceSouthbound scaleSouthbound performanceEnd-to-End performanceOVSDBSouthbound scaleBGPData Ingest performanceData Advertise performancePCEPLSP Read performanceLSP Update performanceProtocol Summary7
8. Setup 1Test Environment8Hardware:CPU: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz, 40 coresTotal RAM: 128GBOS: Ubuntu Linux 4.2.0-30-generic x86_64Network:Every test component is run directly on the machineall communication happens over the 127.0.0.1/8 loopback network.JVM: Oracle 1.8.0_738G of java heapDefault garbage collector
9. Setup 2Test Environment9Hardware:CPU: 2 Intel Xeon E5-2699 v3sz @ 2.30 GHz, 72 coresTotal RAM: 64GBOS: Fedora release 23 Linux 4.2.3-300.fc23.x86_64 x86_64Network:Test component is run directly on the machineall communication happens over 127.0.0.1/8 loopback networkException: OVSDB Scale Tests which use multiple other identical systems to scale test toolsJVM:OpenJDK 64-Bit 1.8.0_728G of java heapDefault garbage collector
10. Controller SoftwareOpenDaylight Beryllium release.OpenDaylight Beryllium with OF Boron plugin.ONOS Falcon pre-release (RC3).FloodLight 1.2.NOTE: Data for other SDN controllers is only for general comparative reference. They should not be considered as definitive maximum performance values for those platforms.Test Environment10
11. Test procedure:Configure 100k flows and wait until we get confirmation of flows being programmed.Remove flows and wait for confirmation.Step-by-step guide: https://wiki.opendaylight.org/view/Openflow:TestingOpenFlow E2E Performance11
12. Beryllium plugin:Flow Add Controller Rate= 2k flows/s100k flows Switch program Time= 53 s100k flows Confirmation time= 23-143 sBoron plugin:Flow Add flow Controller Rate= 2k flows/s100k flows Switch program Time= 53 s100k flows Confirmation time= 7-10 sONOS:Flow Add Controller Rate= 2.4k flows/s100k flows Switch program Time= 43-61 s100k flows Confirmation time= 32-404 sFloodlight:Flow Add Controller Rate= 2k flows/s100k flows Switch program Time= 54 sOpenFlow E2E Performance12
13. Beryllium plugin:Single flow per REST call:Flow Add Controller Rate= 2k flows/s100k flows Switch program Time= 53 s100k flows Confirmation time= 23-143 s200 flows per REST call:Flow Add Controller Rate= 10k flows/s100k flows Switch program Time= 11 s100k flows Confirmation time= 23-143 sBoron plugin:Single flow per REST call:Flow Add Controller Rate= 2k flows/s100k flows Switch program Time= 53 s100k flows Confirmation time= 7-10 s200 flows per REST call:Flow Add Controller Rate= 10k flows/s100k flows Switch program Time= 11 s100k flows Confirmation time= 7-10 sOpenFlow E2E Performance13
14. Test procedure:Configure 100k flows in the OpenFlow network.Restart network (mininet).Wait until we get confirmation of flows being re-programmed.Step-by-step guide: https://wiki.opendaylight.org/view/Openflow:TestingOpenFlow SB Performance14
15. Beryllium plugin:Flow Add Switch Rate= 10-20k flows/s100k flows Confirmation time= 160-204 sBoron plugin:Flow Add Switch Rate= 10k flows/s100k flows Confirmation time= 7-10 sONOS:Flow Add Switch Rate= 1-14k flows/s (flows were missing)100k flows Confirmation time= 60-64sFloodlight:Flow Add Switch Rate= 10-20k flows/s100k flows Confirmation time= No API found for thisOpenFlow SB Performance15
16. Test procedure:ODL starts with simple l2fib models and NETCONF northbound enabled.External client writes lots of l2fib entries in ODL Datastore using NETCONF northbound.The client measures time since 1st request is sent until last response is received.Step-by-step guide: https://wiki.opendaylight.org/view/NETCONF:TestingNetconf NB Performance16
17. Beryllium Release:Single l2fib entry per Netconf request:l2fib Add Rate with TCP client= 7k l2fib/sl2fib Add Rate with SSH client= 6.6k l2fib/s100 l2fib entries per Netconf request:l2fib Add Rate with TCP client= 42k l2fib/sl2fib Add Rate with SSH client= 38.5k l2fib/sNetconf NB Performance17
18. Test procedure:Start netconf-testtool with desired amount of netconf servers.Testtool generates initial configuration for OpenDaylight.Check OpenDaylight connects to all of the simulated devices (TCP or SSH).Step-by-step guide: https://wiki.opendaylight.org/view/NETCONF:TestingNetconf SB Scale18
19. Beryllium release:Devices connected via TCP= 28,000Device connected via SSH= 14,000Netconf SB Scale19
20. Test procedure:OpenDaylight mounts a simulated NETCONF device.Small ODL application triggers a NETCONF notification stream for new mount point.Simulated device starts sending routes with a certain number of prefixes into ODL.Application waits until all notifications have been processed.Step-by-step guide: https://wiki.opendaylight.org/view/NETCONF:TestingNetconf SB Performance20
21. Beryllium Release:Single prefix per Netconf notification:Prefix Learn Rate with TCP= 10.7k prefixes/sPrefix Learn Rate with SSH= 9.8k prefixes/s10 prefixes per Netconf notification:Prefix Learn Rate with TCP= 20k prefixes/sPrefix Learn Rate with SSH= 16.4k prefixes/sNetconf SB Performance21
22. Test procedure:OpenDaylight mounts a simulated NETCONF device.External REST client starts sending prefixes via RESTCONF.ODL application transforms the requests into device specific models and writes to the device.The REST client waits until all of its requests are handled in RESTCONF.Step-by-step guide: https://wiki.opendaylight.org/view/NETCONF:TestingNetconf E2E Performance22
23. Beryllium Release:Single prefix per Netconf notification:Prefix Learn Rate with TCP= 3.2k prefixes/sPrefix Learn Rate with SSH= 2.3k prefixes/s50 prefixes per Netconf notification:Prefix Learn Rate with TCP= 85.3k prefixes/sPrefix Learn Rate with SSH= 61k prefixes/sNetconf E2E Performance23
24. Test procedure:Start multiple OVS systems and configure OpenDaylight as the OVS manager.Run mininet in the OVS system to generate many OVS bridges.Check all OVS systems connect to OpenDaylight. Step-by-step guide: https://wiki.opendaylight.org/view/OVSDB:S3POVSDB Scale24ControllerOVSOVSOVSOVSRESTOVSDB
25. Beryllium release:OVS bridges connected= 1,800OVSDB SB Scale25
26. Test procedure:OpenDaylight connects to the BGP tool (play.py).BGP tool sends 1 million updates (single prefix).Wait until 1 million prefixes are present in ODL.Disconnect BGP tool and wait until all prefixes are removed from ODL. Step-by-step guide:https://wiki.opendaylight.org/view/BGP_LS_PCEP:Specific_Performance_TestingBGP Data Ingest Perf26ControllerBGP ToolRESTBGP (1M prefixes)
27. Beryllium release:Prefix Add rate= 16k prefixes/sPrefix Delete rate= 23k prefixes/sBGP Data Ingest Perf27
28. Test procedure:OpenDaylight connects to the BGP tool (play.py).Use REST and a Java application in controller to populate the ODL RIB.Restart the BGP connection and wait until all prefixes are learned by the BGP tool. Delete all prefixes in ODL RIB and wait until they clear from BGP tool.Step-by-step guide:https://wiki.opendaylight.org/view/BGP_LS_PCEP:Specific_Performance_TestingBGP Data Advertise Perf28ControllerBGP ToolRESTBGP (1M prefixes)
29. Beryllium Release:REST to populate ODL RIB:Single prefix/request rate= 200 prefixes/s100k prefix/request rate= 2k prefixes/sJava app to populate ODL RIB:Single prefix/request rate= 6k prefixes/s100k prefix/request rate= 6k prefixes/sAfter reconnect BGP tool:Prefix add rate= 20k prefixes/sAfter remove all prefixes:Prefix delete rate= 15k prefixes/sBGP Data Advertise Perf29
30. Test procedure:PCC tool (pcc-mock) connects to OpenDaylight (PCE).PCC tool sends information for 65535 tunnels.Wait until 65536 tunnels are present in ODL.Step-by-step guide:https://wiki.opendaylight.org/view/BGP_LS_PCEP:Specific_Performance_TestingPCEP LSP Read Performance30ControllerPCC ToolRESTPCEP (65535 tunnels)
31. Beryllium release:Read rate (1 PCC)= 4.6k tunnels/sRead rate (512 PCC)= 4k tunnels/sPCEP LSP Read Performance31
32. Test procedure:PCC tool (pcc-mock) connects to OpenDaylight (PCE).REST tool (updater.py) is used to send tunnel updates.Wait until 65536 tunnels are updated in PCC tool.Step-by-step guide:https://wiki.opendaylight.org/view/BGP_LS_PCEP:Specific_Performance_TestingPCEP LSP Update Performance32ControllerPCC ToolRESTPCEP (65535 tunnels)REST Tool
33. Beryllium release:Update rate (1 PCC)= 1.8k tunnels/sUpdate rate (512 PCC)= 1.9k tunnels/sPCEP LSP Update Performance33
34. “Batching” multiple flows/prefixes into a single protocol message improves overall performance:OpenFlow NB (REST) performance is 5x faster with “batch” requests.Netconf SB performance is x20 faster than OpenFlow 1.3 as the last does not support “batching” requests.By default OpenDaylight persists configuration data (other tested controllers do not persist), this has negative impact on performance and depending on the Use Case you may want to:Use fast disk (e.g. SSD)Disable persistence if you do not need.Performance Recommendations34