Common NFVI Telco Taskforce - PowerPoint Presentation

1. Common NFVI Telco Taskforce Paris Face-To-Face SessionsFu Qiao, China MobileBin Hu, AT&TReference Implementation OVP & CVC ecosystemJuly 2019

2. Intent & OutputIntent: Where, and how, to augment OPNFV’s delivery stream including the associated OVP Test Ecosystem and CVC Governing board to perform Compliance, Verification, and Certification, of VNFs on CNTT Reference Architecture.Output:Community members Provide Feedback to harden CNTT Certification Methodologies, including reaching consensus to leverage & update current OPNFV/ONAP test processes & harnesses.Assess Delivery Partnerships, Roles, and Responsibilities, (e.g. CVC, TSC, OPNFV, OVP) to ensure successful certifications of VNFs with Identified Reference Architectures, managed via a CVC Governing Process.

3. Discussion OutlineCurrent OPNFV/OVP Certification ProcessProcessTerminologyProjects & Release StatusProposed CNTT Certification MethodologyObjectiveCertification Strategy & VehicleCNTT Transition into OPNFV/OVPDiscussion & Next StepsPriority TopicsOffline Recommendation AppendixTest Categories (Potential)OPNFV/OVP Entrance & Exit Criteria (Example)Chapter 10 – Proposed Outline [Compliance, Verification, & Certification]References

4. Current Process7/19/2019The Linux Foundation Internal Use Only4

5. Current OPNFV/OVP Certification Process CVC Compliance, Verification, and Certification governing framework: Compliance testing: compares the system under test against the specifications / standardsValidation testing: ensures the system under test is operating according to its intended / required purposePerformance testing: measures how well the system under test performs its specific purpose(s)OVP certifications are accomplished as a two-part process: OPNFV provides Test Tools and Test Cases to OVP.OVP provides vendor NFVI products a “badge” claiming “OPNFV-certified” once OVP testing passes successfully (via Dovetail).Test frameworks and supported test cases for OVP Certifications include (OPNFV and ONAP):1. OPNFV – NFVI Testing and CertificationCVC Category: Validation, CompliancePurpose: NFVI platform is checked against the Open Stack requirementsTest framework, test result database and Web UIDovetail projectTest tools, test cases and test executionAPI testing (FuncTest)Performance and HA (Yardstick)Load testing (Bottleneck)2. ONAP – VNF Testing and Certification CVC Category: Compliance, Performance Purpose: VNF template is checked against the ONAP Requirements VNF Validation/Packaging Compliance (HEAT and TOSCA/CSAR/VFD)Web front-end integrated with OPNFV Dovetail Web UIFuture plan: testing of VNF lifecycle, VNF functions, VNF performanceVNF Validation Testing - CVC and ONAP teams are working to develop the initial life-cycle tests for VNF devices.  The first release will run on the "basic open stack" meeting ONAP requirements. Future versions would require this to be the reference NFVI defined by CNTT

6. OPNFV Release StatusCurrent release status for OPNFV: Installer oriented release Multiple installers existMultiple scenarios exist, with one scenario represents a certain integration group and a certain configurationAll releases must pass CI（Continuous Integration） testOVP includes test cases that at lease one OPNFV release scenario can pass

7. OPNFV Feature ProjectsFeature projects in OPNFVDefine a certain feature for the OPNFV release, e.g. high availabilityWork with upstream projects to develop featuresWork with testing projects to define and co-develop feature testsWork with installer projects to develop certain scenarios including the features

8. OPNFV projects and releasesFeature projectsTesting projectsInstaller projectsTest casesScenariosReleaseCIScenariosDovetailTest cases that can pass in one of the release scenarios of OPNFVOVP verified Commercial ProductEUAGTest WGInfra WGOVPCommercial productsTSCCVC

9. Relation between scenarios and OVPMostly, test scenarios are derived from Projects coming into OPNFV for release scoping, resulting in test cases used by OVP for verification and certification. Only test cases that can let one of the scenarios pass can be included in OVP. This is a procedure to make sure OVP test cases are written in a right wayIn order to work with OVP, we should also consider the procedure of verifying OVP. Which is the installer projects and the test scenarios, like “deploy_scenarios”In the past, there are some test cases excluded from OVP since no scenario can pass these cases. Need to ensure CNTT test cases should be involved in OVP. Two possible ways:Define CNTT ref arch as scenarios, and have all test cases for the ref arch be involved in OVP (could also be addressed in OVP as CNTT test)Add test cases from operators, which operators already tested in their environment

10. Proposed Process7/19/2019The Linux Foundation Internal Use Only10

11. Proposed CNTT Certification Methodology Methodology: Perform VNF certifications using CNTT reference architecture, leveraging existing OPNFV Intake Process. Upstream projects will define features/capabilities, test scenarios, and test cases to augment existing OVP test harnesses to be executed via the OVP Ecosystem.OVP/CVC Certification Strategy & Vehicle:NFVI Certification (Compliance): NFVI is the SUT, ensuring NFVI is compliant with specs of Ref Model and Ref ArchitectureEmpirical Validation with Reference VNF (Validation): NFVI is the SUT, ensuring NFVI runs with Golden VNFsCandidate VNF Certification (Validation & Performance): VNF is the SUT, ensuring VNFs operate with Ref Model and Ref ArchSecurity: Ensures VNF is free from known security vulnerabilities, utilizing industry standard cyber security frameworksBest Practices: Standardized test methodology / flow, Test Plan, and Test Case SuitesIntegration with Dovetail and OVP flow (code, docs, cert criteria, etc.)Leveraging ONAP Network and Service Models, with identified VNF-specific parametersStandardized certification criteriaRefNFVICertified VNFVNF SuppliersOVP Testing EcosystemCandidate VNF

12. CNTT transition-a few suggestions The best way to transition CNTT to OPNFV is to make use of the current status of OPNFV rather than re-define a new release modelAdoption of CNTT Ref Arch NFVI it OPNFV starts with identification of Feature projects for CNTT.OPNFVs process would consist of:Working with CNTT, and other initiatives, to bring in release scope, and test scenarios OPNFV will then host the CNTT models, and ref architecturesOPNFV will work with installer & tester projects to define CNTT scenarios and test casesRecommendation: Setup more than one installer/test project, since it is not healthy to put too much work into single project

13. CNTT transition-a few suggestions Set up new committee focusing on driving CNTT effort in OPNFVCVC is the committee for certification in LFN. A new committee focusing on driving common NFVI should be created, to drive the whole process, including feature projects, installers projects, and OVPWe can open to the community to hear more suggestionsSet up new WG focusing on driving CNTT effort in OPNFVA WG has specific deliverables, and scope can include management of projects, installers, test certification, and security. A WG can drive cross Test, cross Infra, discussions for overall certificationsWe can open to the community to hear more suggestions

14. CNTT transition-a few suggestions Key things for smooth transition to OPNFVSolid feature projects, that can address exact reference implementation details. We need to work out the detailed project series.Enough installers involve, otherwise we can’t get the ‘reference implementation’ into solid codes and releasesActive test projects. CNTT needs lots of new test cases, the current test projects probably can’t cover all of them. We need active contributors in the current test projects, and also figure out gaps and probably create new test projects.Recommendation: As a next step, OPNFV, in partnership with Telcos, will perform a gap assessment of existing test harnesses (e.g. Dovetail, ONAP) to ensure certification scenarios and test cases are robust, and complete.

15. Discussion & Next Steps7/19/2019The Linux Foundation Internal Use Only15

16. Things we can work out during F2FPriority Discussion TopicsWhat exactly are we going to create in OPNFV? (Series of projects?) Do we create a ‘committee’ or WG within OPNFV to drive the whole thing? Can our VNF certification goal be technically supported by OPNFV’s current test framework and test cases that are targeted for NFVI?Is there alignment with CNTT’s Three-Part Certification Methodology (NFVI, Empirical, VFN – validations)? Secondary, or Offline discussion:Do we begin with small set of goals, e.g. create feature projects focusing on first ref Arch?After OPNFV test framework (cases, tools, etc) evaluation, if current test tools in OPNFV don’t fit, what toolset do we prefer?What will the RACI chart look like with regards to CNTT, CVC, OVP, and OPNFV? What will the relationship and communication mediums be between these teams?How will issue resolution be handled for certifications, knowing issues could surface in VNFs, Reference Architectures, or the SUT including Dovetail interfaces?Who will delivery, host, and support the various environments (i.e. SUT) needed for NFVI/VNF certifications?What procedure should be used to ensure OVP test cases are more objectively, comprehensively and accurately to test SUT?

17. Next Steps Incorporate Paris feedback into CNTT Certification Methodology materials LeadDate Perform gap assessment of OPNFV/ONAP test harnesses (i.e. Dovetail, ONAP, and Test Project Feeds) to accommodate VFN/NFVI certifications LeadDateParticipantsEstablish a WG & Identify participants for working certification sessions* (e.g. ratify Ch. 10, CVC Process Changes, etc) LeadParticipant NamesTarget Date for Tech Track kickoffMeeting cadenceTimebox delivery milestones* Implies the CNTT will establish a tech track for Compliance discussions and deliveries.

18. Appendix7/19/2019The Linux Foundation Internal Use Only18

19. Test CategoriesPotential Areas of TestingOn-Boarding (ONAP MANO, NFVI Agnostic)Instantiation, Deletion, Recent Change (ONAP MANO, NFVI Agnostic)VNF Functional TestingSecurity TestingCharging / Revenue Assurance VerificationMicroServices SupportClosed Loop TestingVNF Coexistence (ETSI NFV-TST001 “Noisy Neighbor”)VNF Coexistence (ETSI NFV-TST001 “Noisy Neighbor”)VNF Interactions with Extended NFVI TopologyVNF Interactions with Complex NFVI (Akraino)Scalability TestingHA TestingFault Recovery TestingPM/KPI/Service Assurance Testing

20. 20 EXIT CRITERIAENTRANCE CRITERIAOPNFV entrance criteria must be satisfied for VNF certification.Demonstration of Reference Architecture ImplementationDesign & RequirementsDesign, Configuration, Features, SLAs, and Capability documentation completeUsers stories / Adherence to CNTT Model principles and guidelinesChosen RA Matches a RA from the product catalogEnvironmentLab / Flavor, component s/w rev levels specified, with confirmation of compatibility with external systemsTenant needs identifiedAll connectivity, network, image, VMs, delivered with successful pairwise tests Lab instrumented for proper monitoringPlanning & DeliveryKickoff / Acceptance Criteria reviews performed Delivery commitments, timelines, and cadence acceptedConfirm backward compat. across software/flavor rev levelsData / VNFs / SecurityImages, Heat Templates, Preload Sheets availableImages uploaded to tenant spaceExternal system test data needs identifiedOwners (NFVI, VNF, PTL, etc) documentedSecurity Compliance Satisfied (see Ch. 8 – scans, vulnerabilities)Test Case ContributionsVNF Developer/Supplier validations to be performed documented and suppliedNFVI validations to be performed supplied (e.g. echo, manifest)Test to ensure users are added and have correct privileges for the tenantTest to ensure quota against submitted request for the respective tenantsTest to ensure custom flavors against submitted request for respective tenantsCERTIFICATION TENANTS (via OVP Ecosystem)All Test Cases have a status of “Passed”, “Failed”, “N/A” or “Out-scoped”.All Severity 1 and Severity 2 issues are resolved.All Issues have been Resolved or the Project/Component Team has voted unanimously for a Conditional Certification.Known defects, or issues, are clearly documented and furnished to Telco providers with certification notes.Certification Notification(s) issued.OPERATIONAL CONSIDERATIONS (TELCO PERSPECTIVE)Orchestration capabilities verified to be working as expectedFabric setup/configuration validations successfully passedOpenstack API endpoint is reachable and working for that zoneCompute zones and cinder types verifiedStandard images verified to exist (and usable)Network object created (and working, as in IPs are bindable and usable)Resolver overlay/DNS traffic/port 53 overlay on gateway is working properlyDesignate is working, domain preferably created, and maybe test A record created/verified to be resolvableStandard NTP servers are working and verified (using tenant's CIDR source IP)NFVI/VNF is tested at steady state and high loadContinuously monitored to ensure SLAs are met and used as feedback to load/perf testsEND USER CONSIDERATIONS (TELCO PERSPECTIVE)Component redundancy to ensure graceful updates without disruption of servicesThin provisioning storage should handle actual full quota usage casesLoad balancing should support elasticitySRIOV Network configuration via SDN must be aware of all VMs on a host (and their network config)Auto-healing databases (any component related db) when out of syncObvious, but, supports all required network functionality (all protocols, service chaining, VLAN trunking, QoS marking, probe/mirror, etc)Supports NFV migrationSupports snapshots and backups of large volumesPre-check or audit failures during NFV deployment should allow follow-up mitigation, when possible, rather than killing deployment and rolling backChapter 10 – Compliance, Verification, & CertificationOPNFV/OVP: Example Entrance/Exit Criteria

21. Chapter 10 – Compliance, Verification, & CertificationProposed Chapter OutlineSynopsis - Overview, problem statement, scopeTerms and Definitions – Glossary and foundation of terms and concepts used in chapterPrinciples and Guidelines – Goals, objectives, details which drive decisions, test certification strategyLifecycle and Process Flow – Project management guidelines, onboarding, SLAs and Issue ResolutionCurrent OVP/CVC Process – Existing process, test frameworks, tools, test cases/scenarios and test certification guidelinesCNTT/NFVI Certification Approach – Augmented OPNFV and OVP certification process using NFVI Certification, Empirical Validation, and VNF CertificationQuality Assurance – Dependencies, Recommendations, Assumptions, System Under Test (SUT) pre-reqs for certification, Entrance/Exit Criteria, Test Frameworks, Categories, Harness(es), and ToolsAutomation Considerations – Leverage automation capabilities within Opensource communitiesResults Reporting – Metrics, Measurements, and Respective Certifications and Badges – e.g. pass/fail, measure only, etcFuture Planning – Outline evolution timeframe and feature capabilities with NFVI/VNF Test Certification

22. ReferencesCVC - https://wiki.lfnetworking.org/display/LN/Compliance+and+Verification+Committee Conducting OVP Testing with Dovetail https://docs.opnfv.org/en/stable-danube/submodules/dovetail/docs/testing/user/userguide/testing_guide.html DovetailFramework https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Case+Requirements Test Plan: https://wiki.opnfv.org/display/dovetail/Dovetail+%28Danube%29+Documentation+for+Review?preview=/11698759/11698757/User%20Guide.pdf TCs: https://wiki.opnfv.org/display/dovetail/Dovetail+%28Danube%29+Documentation+for+Review Called by functest (repo): https://github.com/opnfv/dovetail/tree/master/etc/testcase Per OVP release in the release notes:https://docs.opnfv.org/en/stable-fraser/submodules/dovetail/docs/release/release-notes/index.htmlhttps://docs.opnfv.org/en/stable-danube/submodules/dovetail/docs/release/release-notes/index.htmlOverall documentation is on docs.opnfv.org for the corresponding Fraser and Danube releaseshttps://docs.opnfv.org/en/stable-fraser/testing/testing-user.html (Fraser)https://docs.opnfv.org/en/stable-fraser/testing/testing-dev.html (Fraser)