ISVV Effectiveness Measurement in ESA Space
Description: ISVV Effectiveness Measurement in ESA Space Projects Pedro A. Barrios, Maria Hernek, Marek Prochazka European Space Agency NASA IVV Workshop 11-13 September 2012 Objective Outline Objective Present the results of an ESA study to assess
Related Topics
Download Presentation
"ISVV Effectiveness Measurement in ESA Space" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. ISVV Effectiveness Measurement in ESA Space Projects Pedro A. Barrios, Maria Hernek, Marek Prochazka European Space Agency
NASA IV&V Workshop 11-13 September 2012<br>
slide2. Objective / Outline Objective
Present the results of an ESA study to assess the effectiveness of the ISVV process carried out in the scope of ESA missions
Assessment of past ISVV projects, with the following final objectives:
Identify what is useful in ISVV process (i.e. what brings results)
Identify what needs to be improved (i.e. added/removed/clarified/...)
Make unified metrics collection an integrated part of the process
Outline
ESA ISVV process: a quick overview
ISVV metrics definition
ISVV metrics collection & analysis
Conclusions and future work<br>
slide3. Independent Software Verification & Validation (ISVV) by ESA ISVV is required for Mission and Safety Critical software, (ECSS-E-40/ECSS-Q-80)
ISVV tasks are additional and complementary to the nominal SW supplier’s verification and validations tasks
ISVV tasks cover verification and validation of software requirements, design, code and tests (typically starting at SW-SRR and finishing before the SW-QR)
ISVV supplier is required to be an organization independent of the software supplier as well as the prime/system integrator (full technical, managerial, and financial independence)
Most ESA projects implement the ISVV process as an industrial contract placed by the Prime contractor<br>
slide4. ESA ISVV Process overview 6 activities/STAGES: Management (MAN), Verification (IVE) and Validation (IVA)
Activities are composed of TASKS, and these are further split into SUBTASKS
Management (MAN.PM and MAN.VV) is concerned with issues such as ISVV objectives and scope, planning, roles, responsibilities, budget, communication, competence, confidentiality, schedule and ISVV level definition (to limit the scope of ISVV)
Technical Specification Analysis (IVE.TA) is verification of the software requirements
Design Analysis (IVE.DA) is verification of the SW Architectural Design and the Software Detailed Design
Code Analysis (IVE.CA) is verification of the SW source code
Validation (IVA) is testing of the SW to demonstrate that the implementation meets the technical specification<br>
slide5. ESA ISVV Process overview Example of a Task/Subtask description
Activity: Technical Specification Analysis
Task: SW Requirements Verification
Subtasks: T1.S1, T1.S2 … T1.S11
Start/End Events
Inputs/Outputs
Methods are identified for each subtask
Some numbers:
IVE.TA 1 task 11 subtasks
IVE.DA 3 tasks 15/12/5 subtasks
IVE.CA 3 tasks 10/5/3 subtasks
IVA 3 tasks 3/3/3 subtasks<br>
slide6. ESA ISVV Process overview IVE: Technical Specification Analysis Subtasks: To verify
Software Requirements external consistency with the system requirements
Interface Requirements external consistency with the system requirements
software requirements correctness
consistent documentation of the software requirements
software requirements completeness
dependability and safety requirements
readability of the software requirements
timing and sizing budgets of the software requirements
Identify test areas and test cases for Independent Validation
that software requirements are testable
software requirements conformance with applicable standards TA.T1: Software Requirements Verification<br>
slide7. ESA ISVV Process overview IVE: Design Analysis Subtasks: To verify
SW architectural design external consistency with Technical Specification
SW architectural design external consistency with Interface Control Documents
interfaces consistency between different SW components
architectural design correctness
architectural design completeness
dependability & safety of the design
readability of the architectural design
timing and sizing budgets of the software
Identify test areas and test cases for independent Validation
architectural design conformance with applicable standards
if models are produced by the SW suppliers:
Verify test performed on high level model
Verify development and verification and testing methods and environment
then construct model test cases & model test procedures
then execution of model test procedures DA.T1: Architectural Design Verification<br>
slide8. ESA ISVV Process overview IVE: Design Analysis Subtasks: To verify
detailed design external consistency with Technical Specification
detailed design external consistency with Interface Control Documents
detailed design external consistency with Architectural Design
interfaces consistency between different SW components
detailed design correctness
detailed design completeness
dependability & safety of design
readability of detailed design
timing and sizing budgets of software
accuracy of the model (in case models are produced by the SW suppliers)
Identify test areas and test cases for independent Validation
Verify detailed design conformance with applicable standards Subtasks: To verify
timing and sizing budgets of software
that dependability & safety aspects on product are specified in the SUM
readability of User Manual
completeness of User Manual
correctness of User Manual DA.T2: Detailed Design Verification DA.T3: Software User Manual Verification<br>
slide9. ESA ISVV Process overview IVE: Code Analysis CA.T1: Source Code Verification Subtasks: To verify
source code external consistency with Technical Specification
source code external consistency with Interface Control Documents
source code external consistency with Architectural Design and Detailed Design
interfaces consistency between different SW units
source code correctness with respect to technical specification, architectural design and detailed design
source code readability, maintainability and conformance with the applicable standards
dependability & safety of source code
Source code accuracy
Identify test areas and test cases for independent Validation
timing and sizing budgets of the software<br>
slide10. ESA ISVV Process overview IVE: Code Analysis CA.T2: Integration Test Specification and Test Data Verification
Subtasks: To verify
consistency with Technical Specification
consistency with Software Architectural Design
integration test procedures correctness and completeness
If models are produced by the SW suppliers, then evaluate model verification and validation test results
integration test reports
CA.T3: Unit Test Procedure and Test Data Verification
Subtasks: To verify
consistency with Software Detailed Design
unit test procedures correctness and completeness
unit test reports<br>
slide11. ISVV effectiveness metrics Key goal of activity is to estimate effectiveness of the ISVV process carried out in scope of ESA projects
Major objective is to provide measurements and conclusions to support identification and prioritization of ISVV activities based on their ‘efficiency’
Improve ISVV process is an additional objective
ISVV effectiveness to be calculated based on number of findings and their acceptance and impact
Based on number of findings, the following metrics are computed: findings per ISVV stage / task / subtask; finding per severity; findings per type and effective findings.<br>
slide12. Measurement Process 3 steps activity: ISVV metrication definition / ISVV metrics collection / ISVV metrics assessment
Industrial context:
Measurement needs and processes started by ESA
Provision of metrics performed through different small contracts granted by ESA to different ESA ISVV suppliers
Data analysis, collection and metrics analysis and calculation performed by an ESA contractor to this activity<br>
slide13. Measurement Process Data gathering, with following contents:
SW product metrics (size in kLOC, number of requirements, criticality)
ISVV project metrics (ISVV level, ISVV scope and stages, documentation quality at reviews)
Findings (task, subtask, which document, type, severity, use of tools, acceptance, impact measured in number of changes)
Note: excel tool was used<br>
slide14. Measurement Process 15 products from 5 projects
4 different ISVV suppliers
The IVE effectiveness metrics are assessed:
per product
per SW products of similar size
In total, i.e. in all projects and SW products considered
Analysis is performed:
Per all stages
Per ISVV project stage
Per ISVV task /subtask
Findings per stage/task/sub-task, per severity, per type, Effective Findings & Tools usage Note: Only one product classified as small<br>
slide15. ISVV metrics collection & analysis (1/10) Total Findings
Total number of IVE findings for 15 products within this analysis is 2492
No clear relationship between findings & product size red=big products ; blue=medium; green=small Mean & Standard deviation<br>
slide16. ISVV metrics collection & analysis (2/10) Findings per stage TA: Technical Specification Analysis
CA: Code Analysis
DA: Design Analysis Although there is some variability per product, number of findings are roughly 1/3 for three stages
The majority of findings are at the TA stage for big type products, and CA stage for small<br>
slide17. ISVV metrics collection & analysis (3/10) Findings per task (TA: Technical Specification Analysis) Majority of findings of TA stage are at TA.T2 task (Software Requirements Verification) for all products, all projects, all product sizes with only one exception.
As the size of products decreases, more findings are discovered at TA.T2 task<br>
slide18. ISVV metrics collection & analysis (4/10) Findings per task (DA: Design Analysis) Total share of findings for DA stage Majority of findings of the DA stage are either at DA.T2 (Architectural Design Verification) task or at DA.T4 task (Detailed Design Verification) depending of the different products. Findings per product for DA tasks Findings per size for DA tasks<br>
slide19. ISVV metrics collection & analysis (5/10) Findings per task (CA: Code Analysis) Total share of findings for CA stage Share of findings per product for CA tasks Findings per size for CA tasks Majority of findings of CA stage are at the CA.T2 task (Source Code Verification) in totals, then it varies product by product. CA.T3 (IT tests Verification) also represents a big share<br>
slide20. ISVV metrics collection & analysis (6/10) Findings per sub-task (e.g. TA subtasks) Exact numbers are available for all the subtasks
There are subtasks producing a reduced number of findings. Three possible cases: subtask not performed within the ISVV project, subtask not producing findings or data not available for the subtask T2: Software Requirements Verification T1: Requirements Traceability Verification<br>
slide21. ISVV metrics collection & analysis (7/10) Findings per severity Most of findings are minor. Major findings account for 36%.
Proportions found across the three stages (TA, DA, CA) are similar to these numbers<br>
slide22. ISVV metrics collection & analysis (8/10) Findings per type Most of findings are of type correctness, followed by findings of type completeness<br>
slide23. ISVV metrics collection & analysis (9/10) Tools usage Majority of findings were discovered manually (97% of the total findings) and only very few of them using tools (either to automatically discover the finding or the so-called ‘semi-automated’, using tools to further evaluate to discover any finding)<br>
slide24. ISVV metrics collection & analysis (10/10) Effective findings (ISVV findings that implied a change, improvement, correction to the software product) Majority of findings are effective no matter product & size, except for small product for which majority of findings are not effective
Majority of findings per stage are effective (72% TA & DA stages; 61% at CA stage)
Majority of findings are effective for all severities (70% for major, 69% for minor)<br>
slide25. Conclusions (1/2) Total number of findings
Measurements based on number of findings
Focus on IVE metrics
No correlation found between number of findings & product size
Total number of findings per ISVV stage / task /subtask
Stage: Roughly even distribution (39% TA, 28% DA, 33% CA)
Task/Subtasks: Identified the tasks producing most of the findings for TA, DA, CA (e.g. ‘Correctness/Completeness’ subtasks are producing many findings; ‘consistency’ subtasks produce some)
Type of findings: The majority of findings are of type correctness (36%) & completeness (28%)
Effective findings: The majority of findings (69%) are effective (i.e. implying changes/corrections to the software product)<br>
slide26. Conclusions (2/2) Severity: Most findings are minor at all stages, with 58% minor, 36% major and remaining 6% for other severity (comment, very low)
Tools: The majority of the findings were discovered manually (97%) and only very few of them using tools. Tools were used only for 3% of findings (especially at the CA stage)
Example: if we started today an ISVV contract on a project, we could expect, on average: 166 findings, from which 115 would be effective findings; out of those, 41 would be major findings, and those would be spread on the different stages as: IVE: 16 TA, 11 DA, 14 CA<br>
slide27. Future work Collect metrics for the upcoming ISVV projects.
Analyze tasks/sub-tasks not producing many findings (they might need better explanations within the ISVV guide, review the methods and tools proposed to be used when performing them, …)
Analyze Independent Validation:
Define useful metrics for IVA and asses IVA effectiveness
Extend the scope of IVA, to cover Qualification & Acceptance of the SW and the Operational scenarios (i.e. having the operational view to create SW validation campaigns)
Modeling:
Some model related sub-tasks have not been ‘profiled’
Understand how models produced during SW development could be used during ISVV activities (e.g. Model-Based Testing techniques to produce validation campaigns)
ISVV effectiveness Metrics:
Some other way how to measure effectiveness?
Cost figures?<br>
slide28. Thanks for your attention !!!!
For more information, please contact:
Pedro A. Barrios, European Space Agency
Pedro.Barrios@esa.int<br>
slide29. Back-up Slides<br>
slide30. Findings per sub-task (1/3) IVE: Technical Specification Analysis
TA.T1: Software Requirements Verification
(===) IVE.TA.T1.S1: Verify Software Requirements external consistency with the system requirements
(===) IVE.TA.T1.S2: Verify Interface Requirements external consistency with the system requirements
(+++) IVE.TA.T1.S3: Verify software requirements correctness
(===) IVE.TA.T1.S4: Verify the consistent documentation of the software requirements
(+++) IVE.TA.T1.S5: Verify software requirements completeness
(+++) IVE.TA.T1.S6: Verify the dependability and safety requirements
(+++) IVE.TA.T1.S7: Verify the readability of the software requirements
(---) IVE.TA.T1.S8: Verify the timing and sizing budgets of the software requirements
(---) IVE.TA.T1.S9: Identify test areas and test cases for Independent Validation
(---) IVE.TA.T1.S10: Verify that the software requirements are testable
(---) IVE.TA.T1.S11: Verify software requirements conformance with applicable standards High level view of number of findings per sub-task
Legend:
(+++): Subtask producing a considerable number of findings
(===): Subtask producing some findings
(---): Subtask producing a reduced number of findings
(xxx): Metrics not available for that subtask<br>
slide31. Findings per sub-task (2/3) IVE: Design Analysis
DA.T1: Architectural Design Verification
(===) IVE.DA.T1.S1: Verify the SW architectural design external consistency with the Technical Specification
(---) IVE.DA.T1.S2: Verify the SW architectural design external consistency with the Interface Control Documents
(===) IVE.DA.T1.S3: Verify interfaces consistency between different SW components
(===) IVE.DA.T1.S4: Verify architectural design correctness
(===) IVE.DA.T1.S5: Verify architectural design completeness
(===) IVE.DA.T1.S6: Verify the dependability & safety of the design
(+++) IVE.DA.T1.S7: Verify the readability of the architectural design
(===) IVE.DA.T1.S8: Verify the timing and sizing budgets of the software
(---) IVE.DA.T1.S9: Identify test areas and test cases for independent Validation
(---) IVE.DA.T1.S10: Verify architectural design conformance with applicable standards
(xxx) IVE.DA.T1.S11: Verify the test performed on the high level model
(xxx) IVE.DA.T1.S12: Verify the development and verification and testing methods and environment
(xxx) IVE.DA.T1.S13: then construct model test cases
(xxx) IVE.DA.T1.S14: then construct model test procedures
(xxx) IVE.DA.T1.S15: then execution of model test procedures
DA.T2: Detailed Design Verification
(---) IVE.DA.T2.S1: Verify the detailed design external consistency with the Technical Specification
(---) IVE.DA.T2.S2: Verify the detailed design external consistency with the Interface Control Documents
(---) IVE.DA.T2.S3: Verify the detailed design external consistency with the Architectural Design
(+++) IVE.DA.T2.S4: Verify interfaces consistency between different SW components
(===) IVE.DA.T2.S5: Verify detailed design correctness
(===) IVE.DA.T2.S6: Verify detailed design completeness
(+++) IVE.DA.T2.S7: Verify the dependability & safety of the design
(---) IVE.DA.T2.S8: Verify the readability of the detailed design
(===) IVE.DA.T2.S9: Verify the timing and sizing budgets of the software
(xxx) IVE.DA.T2.S10: Verify the accuracy of the model (in case models are produced by the SW suppliers)
(---) IVE.DA.T2.S11: Identify test areas and test cases for independent Validation
(---) IVE.DA.T2.S12: Verify detailed design conformance with applicable standards<br>
slide32. Findings per sub-task (3/3) DA.T3: Software User Manual Verification
(---) IVE.DA.T3.S1: Verify the timing and sizing budgets of the software
(---) IVE.DA.T3.S2: Verify the dependability & safety aspects on the product are specified in the SUM
(---) IVE.DA.T3.S3; Verify the readability of the User Manual
(---) IVE.DA.T3.S4; Verify the completeness of the User Manual
(---) IVE.DA.T3.S5: Verify the correctness of the User Manual
IVE: Code Analysis
CA.T1: Source Code Verification
(---) IVE.CA.T1.S1: Verify source code external consistency with Technical Specification
(---) IVE.CA.T1.S2: Verify source code external consistency with Interface Control Documents
(---) IVE.CA.T1.S3: Verify source code external consistency with Architectural Design and Detailed Design
(---) IVE.CA.T1.S4: Verify interfaces consistency between different SW units
(+++) IVE.CA.T1.S5: Verify source code correctness with respect to technical specification, architectural design & detailed design
(+++) IVE.CA.T1.S6: Verify the source code readability, maintainability and conformance with the applicable standards
(+++) IVE.CA.T1.S7: Verify the dependability & safety of the source code
(---) IVE.CA.T1.S8: Verify the accuracy of the source code
(---) IVE.CA.T1.S9: Identify test areas and test cases for independent Validation
(===) IVE.CA.T1.S10: Verify the timing and sizing budgets of the software
CA.T2: Integration Test Specification and Test Data Verification
(===) IVE.CA.T2.S1: Verify consistency with Technical Specification
(---) IVE.CA.T2.S2: Verify consistency with Software Architectural Design
(+++) IVE.CA.T2.S3: Verify integration test procedures correctness and completeness
(xxx) IVE.CA.T2.S4: If models are produced by the SW suppliers, then evaluate model verification and validation test results
(xxx) IVE.CA.T2.S5: Verify integration test reports
CA.T3: Unit Test Procedure and Test Data Verification
(---) IVE.CA.T3.S1: Verify consistency with Software Detailed Design
(===) IVE.CA.T3.S2: Verify unit test procedures correctness and completeness
(xxx) IVE.CA.T3.S3: Verify unit test reports<br>
slide33. IVA: Independent Validation IVA.T1: Identification of Test Cases
IVA.T1.S1: Evaluate Task Input Inspection
IVA.T1.S2: Perform Analysis
IVA.T1.S3: Writing Independent Validation Test Plan
IVA.T2: Construction of Test Procedures
IVA.T2.S1: Achieve knowledge about the SVF
IVA.T2.S2: Implement Test Cases into Test Procedures
IVA.T2.S3: Updating the Independent Validation Test Plan
IVA.T3: Execution of Test Procedures
IVA.T3.S1: Execute the Test Procedures
IVA.T3.S2: Investigation of failed tests
IVA.T3.S3: Produce Test Report<br>
NASA IV&V Workshop 11-13 September 2012<br>
slide2. Objective / Outline Objective
Present the results of an ESA study to assess the effectiveness of the ISVV process carried out in the scope of ESA missions
Assessment of past ISVV projects, with the following final objectives:
Identify what is useful in ISVV process (i.e. what brings results)
Identify what needs to be improved (i.e. added/removed/clarified/...)
Make unified metrics collection an integrated part of the process
Outline
ESA ISVV process: a quick overview
ISVV metrics definition
ISVV metrics collection & analysis
Conclusions and future work<br>
slide3. Independent Software Verification & Validation (ISVV) by ESA ISVV is required for Mission and Safety Critical software, (ECSS-E-40/ECSS-Q-80)
ISVV tasks are additional and complementary to the nominal SW supplier’s verification and validations tasks
ISVV tasks cover verification and validation of software requirements, design, code and tests (typically starting at SW-SRR and finishing before the SW-QR)
ISVV supplier is required to be an organization independent of the software supplier as well as the prime/system integrator (full technical, managerial, and financial independence)
Most ESA projects implement the ISVV process as an industrial contract placed by the Prime contractor<br>
slide4. ESA ISVV Process overview 6 activities/STAGES: Management (MAN), Verification (IVE) and Validation (IVA)
Activities are composed of TASKS, and these are further split into SUBTASKS
Management (MAN.PM and MAN.VV) is concerned with issues such as ISVV objectives and scope, planning, roles, responsibilities, budget, communication, competence, confidentiality, schedule and ISVV level definition (to limit the scope of ISVV)
Technical Specification Analysis (IVE.TA) is verification of the software requirements
Design Analysis (IVE.DA) is verification of the SW Architectural Design and the Software Detailed Design
Code Analysis (IVE.CA) is verification of the SW source code
Validation (IVA) is testing of the SW to demonstrate that the implementation meets the technical specification<br>
slide5. ESA ISVV Process overview Example of a Task/Subtask description
Activity: Technical Specification Analysis
Task: SW Requirements Verification
Subtasks: T1.S1, T1.S2 … T1.S11
Start/End Events
Inputs/Outputs
Methods are identified for each subtask
Some numbers:
IVE.TA 1 task 11 subtasks
IVE.DA 3 tasks 15/12/5 subtasks
IVE.CA 3 tasks 10/5/3 subtasks
IVA 3 tasks 3/3/3 subtasks<br>
slide6. ESA ISVV Process overview IVE: Technical Specification Analysis Subtasks: To verify
Software Requirements external consistency with the system requirements
Interface Requirements external consistency with the system requirements
software requirements correctness
consistent documentation of the software requirements
software requirements completeness
dependability and safety requirements
readability of the software requirements
timing and sizing budgets of the software requirements
Identify test areas and test cases for Independent Validation
that software requirements are testable
software requirements conformance with applicable standards TA.T1: Software Requirements Verification<br>
slide7. ESA ISVV Process overview IVE: Design Analysis Subtasks: To verify
SW architectural design external consistency with Technical Specification
SW architectural design external consistency with Interface Control Documents
interfaces consistency between different SW components
architectural design correctness
architectural design completeness
dependability & safety of the design
readability of the architectural design
timing and sizing budgets of the software
Identify test areas and test cases for independent Validation
architectural design conformance with applicable standards
if models are produced by the SW suppliers:
Verify test performed on high level model
Verify development and verification and testing methods and environment
then construct model test cases & model test procedures
then execution of model test procedures DA.T1: Architectural Design Verification<br>
slide8. ESA ISVV Process overview IVE: Design Analysis Subtasks: To verify
detailed design external consistency with Technical Specification
detailed design external consistency with Interface Control Documents
detailed design external consistency with Architectural Design
interfaces consistency between different SW components
detailed design correctness
detailed design completeness
dependability & safety of design
readability of detailed design
timing and sizing budgets of software
accuracy of the model (in case models are produced by the SW suppliers)
Identify test areas and test cases for independent Validation
Verify detailed design conformance with applicable standards Subtasks: To verify
timing and sizing budgets of software
that dependability & safety aspects on product are specified in the SUM
readability of User Manual
completeness of User Manual
correctness of User Manual DA.T2: Detailed Design Verification DA.T3: Software User Manual Verification<br>
slide9. ESA ISVV Process overview IVE: Code Analysis CA.T1: Source Code Verification Subtasks: To verify
source code external consistency with Technical Specification
source code external consistency with Interface Control Documents
source code external consistency with Architectural Design and Detailed Design
interfaces consistency between different SW units
source code correctness with respect to technical specification, architectural design and detailed design
source code readability, maintainability and conformance with the applicable standards
dependability & safety of source code
Source code accuracy
Identify test areas and test cases for independent Validation
timing and sizing budgets of the software<br>
slide10. ESA ISVV Process overview IVE: Code Analysis CA.T2: Integration Test Specification and Test Data Verification
Subtasks: To verify
consistency with Technical Specification
consistency with Software Architectural Design
integration test procedures correctness and completeness
If models are produced by the SW suppliers, then evaluate model verification and validation test results
integration test reports
CA.T3: Unit Test Procedure and Test Data Verification
Subtasks: To verify
consistency with Software Detailed Design
unit test procedures correctness and completeness
unit test reports<br>
slide11. ISVV effectiveness metrics Key goal of activity is to estimate effectiveness of the ISVV process carried out in scope of ESA projects
Major objective is to provide measurements and conclusions to support identification and prioritization of ISVV activities based on their ‘efficiency’
Improve ISVV process is an additional objective
ISVV effectiveness to be calculated based on number of findings and their acceptance and impact
Based on number of findings, the following metrics are computed: findings per ISVV stage / task / subtask; finding per severity; findings per type and effective findings.<br>
slide12. Measurement Process 3 steps activity: ISVV metrication definition / ISVV metrics collection / ISVV metrics assessment
Industrial context:
Measurement needs and processes started by ESA
Provision of metrics performed through different small contracts granted by ESA to different ESA ISVV suppliers
Data analysis, collection and metrics analysis and calculation performed by an ESA contractor to this activity<br>
slide13. Measurement Process Data gathering, with following contents:
SW product metrics (size in kLOC, number of requirements, criticality)
ISVV project metrics (ISVV level, ISVV scope and stages, documentation quality at reviews)
Findings (task, subtask, which document, type, severity, use of tools, acceptance, impact measured in number of changes)
Note: excel tool was used<br>
slide14. Measurement Process 15 products from 5 projects
4 different ISVV suppliers
The IVE effectiveness metrics are assessed:
per product
per SW products of similar size
In total, i.e. in all projects and SW products considered
Analysis is performed:
Per all stages
Per ISVV project stage
Per ISVV task /subtask
Findings per stage/task/sub-task, per severity, per type, Effective Findings & Tools usage Note: Only one product classified as small<br>
slide15. ISVV metrics collection & analysis (1/10) Total Findings
Total number of IVE findings for 15 products within this analysis is 2492
No clear relationship between findings & product size red=big products ; blue=medium; green=small Mean & Standard deviation<br>
slide16. ISVV metrics collection & analysis (2/10) Findings per stage TA: Technical Specification Analysis
CA: Code Analysis
DA: Design Analysis Although there is some variability per product, number of findings are roughly 1/3 for three stages
The majority of findings are at the TA stage for big type products, and CA stage for small<br>
slide17. ISVV metrics collection & analysis (3/10) Findings per task (TA: Technical Specification Analysis) Majority of findings of TA stage are at TA.T2 task (Software Requirements Verification) for all products, all projects, all product sizes with only one exception.
As the size of products decreases, more findings are discovered at TA.T2 task<br>
slide18. ISVV metrics collection & analysis (4/10) Findings per task (DA: Design Analysis) Total share of findings for DA stage Majority of findings of the DA stage are either at DA.T2 (Architectural Design Verification) task or at DA.T4 task (Detailed Design Verification) depending of the different products. Findings per product for DA tasks Findings per size for DA tasks<br>
slide19. ISVV metrics collection & analysis (5/10) Findings per task (CA: Code Analysis) Total share of findings for CA stage Share of findings per product for CA tasks Findings per size for CA tasks Majority of findings of CA stage are at the CA.T2 task (Source Code Verification) in totals, then it varies product by product. CA.T3 (IT tests Verification) also represents a big share<br>
slide20. ISVV metrics collection & analysis (6/10) Findings per sub-task (e.g. TA subtasks) Exact numbers are available for all the subtasks
There are subtasks producing a reduced number of findings. Three possible cases: subtask not performed within the ISVV project, subtask not producing findings or data not available for the subtask T2: Software Requirements Verification T1: Requirements Traceability Verification<br>
slide21. ISVV metrics collection & analysis (7/10) Findings per severity Most of findings are minor. Major findings account for 36%.
Proportions found across the three stages (TA, DA, CA) are similar to these numbers<br>
slide22. ISVV metrics collection & analysis (8/10) Findings per type Most of findings are of type correctness, followed by findings of type completeness<br>
slide23. ISVV metrics collection & analysis (9/10) Tools usage Majority of findings were discovered manually (97% of the total findings) and only very few of them using tools (either to automatically discover the finding or the so-called ‘semi-automated’, using tools to further evaluate to discover any finding)<br>
slide24. ISVV metrics collection & analysis (10/10) Effective findings (ISVV findings that implied a change, improvement, correction to the software product) Majority of findings are effective no matter product & size, except for small product for which majority of findings are not effective
Majority of findings per stage are effective (72% TA & DA stages; 61% at CA stage)
Majority of findings are effective for all severities (70% for major, 69% for minor)<br>
slide25. Conclusions (1/2) Total number of findings
Measurements based on number of findings
Focus on IVE metrics
No correlation found between number of findings & product size
Total number of findings per ISVV stage / task /subtask
Stage: Roughly even distribution (39% TA, 28% DA, 33% CA)
Task/Subtasks: Identified the tasks producing most of the findings for TA, DA, CA (e.g. ‘Correctness/Completeness’ subtasks are producing many findings; ‘consistency’ subtasks produce some)
Type of findings: The majority of findings are of type correctness (36%) & completeness (28%)
Effective findings: The majority of findings (69%) are effective (i.e. implying changes/corrections to the software product)<br>
slide26. Conclusions (2/2) Severity: Most findings are minor at all stages, with 58% minor, 36% major and remaining 6% for other severity (comment, very low)
Tools: The majority of the findings were discovered manually (97%) and only very few of them using tools. Tools were used only for 3% of findings (especially at the CA stage)
Example: if we started today an ISVV contract on a project, we could expect, on average: 166 findings, from which 115 would be effective findings; out of those, 41 would be major findings, and those would be spread on the different stages as: IVE: 16 TA, 11 DA, 14 CA<br>
slide27. Future work Collect metrics for the upcoming ISVV projects.
Analyze tasks/sub-tasks not producing many findings (they might need better explanations within the ISVV guide, review the methods and tools proposed to be used when performing them, …)
Analyze Independent Validation:
Define useful metrics for IVA and asses IVA effectiveness
Extend the scope of IVA, to cover Qualification & Acceptance of the SW and the Operational scenarios (i.e. having the operational view to create SW validation campaigns)
Modeling:
Some model related sub-tasks have not been ‘profiled’
Understand how models produced during SW development could be used during ISVV activities (e.g. Model-Based Testing techniques to produce validation campaigns)
ISVV effectiveness Metrics:
Some other way how to measure effectiveness?
Cost figures?<br>
slide28. Thanks for your attention !!!!
For more information, please contact:
Pedro A. Barrios, European Space Agency
Pedro.Barrios@esa.int<br>
slide29. Back-up Slides<br>
slide30. Findings per sub-task (1/3) IVE: Technical Specification Analysis
TA.T1: Software Requirements Verification
(===) IVE.TA.T1.S1: Verify Software Requirements external consistency with the system requirements
(===) IVE.TA.T1.S2: Verify Interface Requirements external consistency with the system requirements
(+++) IVE.TA.T1.S3: Verify software requirements correctness
(===) IVE.TA.T1.S4: Verify the consistent documentation of the software requirements
(+++) IVE.TA.T1.S5: Verify software requirements completeness
(+++) IVE.TA.T1.S6: Verify the dependability and safety requirements
(+++) IVE.TA.T1.S7: Verify the readability of the software requirements
(---) IVE.TA.T1.S8: Verify the timing and sizing budgets of the software requirements
(---) IVE.TA.T1.S9: Identify test areas and test cases for Independent Validation
(---) IVE.TA.T1.S10: Verify that the software requirements are testable
(---) IVE.TA.T1.S11: Verify software requirements conformance with applicable standards High level view of number of findings per sub-task
Legend:
(+++): Subtask producing a considerable number of findings
(===): Subtask producing some findings
(---): Subtask producing a reduced number of findings
(xxx): Metrics not available for that subtask<br>
slide31. Findings per sub-task (2/3) IVE: Design Analysis
DA.T1: Architectural Design Verification
(===) IVE.DA.T1.S1: Verify the SW architectural design external consistency with the Technical Specification
(---) IVE.DA.T1.S2: Verify the SW architectural design external consistency with the Interface Control Documents
(===) IVE.DA.T1.S3: Verify interfaces consistency between different SW components
(===) IVE.DA.T1.S4: Verify architectural design correctness
(===) IVE.DA.T1.S5: Verify architectural design completeness
(===) IVE.DA.T1.S6: Verify the dependability & safety of the design
(+++) IVE.DA.T1.S7: Verify the readability of the architectural design
(===) IVE.DA.T1.S8: Verify the timing and sizing budgets of the software
(---) IVE.DA.T1.S9: Identify test areas and test cases for independent Validation
(---) IVE.DA.T1.S10: Verify architectural design conformance with applicable standards
(xxx) IVE.DA.T1.S11: Verify the test performed on the high level model
(xxx) IVE.DA.T1.S12: Verify the development and verification and testing methods and environment
(xxx) IVE.DA.T1.S13: then construct model test cases
(xxx) IVE.DA.T1.S14: then construct model test procedures
(xxx) IVE.DA.T1.S15: then execution of model test procedures
DA.T2: Detailed Design Verification
(---) IVE.DA.T2.S1: Verify the detailed design external consistency with the Technical Specification
(---) IVE.DA.T2.S2: Verify the detailed design external consistency with the Interface Control Documents
(---) IVE.DA.T2.S3: Verify the detailed design external consistency with the Architectural Design
(+++) IVE.DA.T2.S4: Verify interfaces consistency between different SW components
(===) IVE.DA.T2.S5: Verify detailed design correctness
(===) IVE.DA.T2.S6: Verify detailed design completeness
(+++) IVE.DA.T2.S7: Verify the dependability & safety of the design
(---) IVE.DA.T2.S8: Verify the readability of the detailed design
(===) IVE.DA.T2.S9: Verify the timing and sizing budgets of the software
(xxx) IVE.DA.T2.S10: Verify the accuracy of the model (in case models are produced by the SW suppliers)
(---) IVE.DA.T2.S11: Identify test areas and test cases for independent Validation
(---) IVE.DA.T2.S12: Verify detailed design conformance with applicable standards<br>
slide32. Findings per sub-task (3/3) DA.T3: Software User Manual Verification
(---) IVE.DA.T3.S1: Verify the timing and sizing budgets of the software
(---) IVE.DA.T3.S2: Verify the dependability & safety aspects on the product are specified in the SUM
(---) IVE.DA.T3.S3; Verify the readability of the User Manual
(---) IVE.DA.T3.S4; Verify the completeness of the User Manual
(---) IVE.DA.T3.S5: Verify the correctness of the User Manual
IVE: Code Analysis
CA.T1: Source Code Verification
(---) IVE.CA.T1.S1: Verify source code external consistency with Technical Specification
(---) IVE.CA.T1.S2: Verify source code external consistency with Interface Control Documents
(---) IVE.CA.T1.S3: Verify source code external consistency with Architectural Design and Detailed Design
(---) IVE.CA.T1.S4: Verify interfaces consistency between different SW units
(+++) IVE.CA.T1.S5: Verify source code correctness with respect to technical specification, architectural design & detailed design
(+++) IVE.CA.T1.S6: Verify the source code readability, maintainability and conformance with the applicable standards
(+++) IVE.CA.T1.S7: Verify the dependability & safety of the source code
(---) IVE.CA.T1.S8: Verify the accuracy of the source code
(---) IVE.CA.T1.S9: Identify test areas and test cases for independent Validation
(===) IVE.CA.T1.S10: Verify the timing and sizing budgets of the software
CA.T2: Integration Test Specification and Test Data Verification
(===) IVE.CA.T2.S1: Verify consistency with Technical Specification
(---) IVE.CA.T2.S2: Verify consistency with Software Architectural Design
(+++) IVE.CA.T2.S3: Verify integration test procedures correctness and completeness
(xxx) IVE.CA.T2.S4: If models are produced by the SW suppliers, then evaluate model verification and validation test results
(xxx) IVE.CA.T2.S5: Verify integration test reports
CA.T3: Unit Test Procedure and Test Data Verification
(---) IVE.CA.T3.S1: Verify consistency with Software Detailed Design
(===) IVE.CA.T3.S2: Verify unit test procedures correctness and completeness
(xxx) IVE.CA.T3.S3: Verify unit test reports<br>
slide33. IVA: Independent Validation IVA.T1: Identification of Test Cases
IVA.T1.S1: Evaluate Task Input Inspection
IVA.T1.S2: Perform Analysis
IVA.T1.S3: Writing Independent Validation Test Plan
IVA.T2: Construction of Test Procedures
IVA.T2.S1: Achieve knowledge about the SVF
IVA.T2.S2: Implement Test Cases into Test Procedures
IVA.T2.S3: Updating the Independent Validation Test Plan
IVA.T3: Execution of Test Procedures
IVA.T3.S1: Execute the Test Procedures
IVA.T3.S2: Investigation of failed tests
IVA.T3.S3: Produce Test Report<br>