Assurance Cases for Software Releases in ISS
Description: Assurance Cases for Software Releases in ISS Sustaining Phase of Development Sarma Susarla IVV Team Lead Sarma.V.Susarlanasa.gov 9102013 Introduction IVV has been performing ISS IVV for the past 18 years Software architecture and
Related Topics
Download Presentation
"Assurance Cases for Software Releases in ISS" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Assurance Cases for Software Releases in ISS Sustaining Phase of Development Sarma Susarla
IV&V Team Lead
Sarma.V.Susarla@nasa.gov
9/10/2013<br>
slide2. Introduction IV&V has been performing ISS IV&V for the past 18 years
Software architecture and design is mature
However new releases are being developed to add new functionality and product improvements in the existing architecture.
IV&V had been generating technical reports at the end of each milestone event such as :
Technical Design Review (TDR)
Test Readiness Review (TRR)
Software Transition Readiness Review (STRR)
These reports document software assurance stating that the CSCI is ready for next stage in life cycle development.
This presentation provides a framework for an integrated software assurance approach:
To assert that CSCI is ready for on-orbit deployment
Using existing IV&V processes and analyses. Sarma Susarla IV&V Workshop 9/10/2013 2<br>
slide3. Evidence Based Assurance (EBA) Implementation Approach Use ISS life cycle development model
Use life cycle development milestones as individual claims into EBA network model
Requirements development completion
Code development completion etc.
Use Biwar report as a mechanism to report evidence as it becomes available during life cycle development support activity
Maximize usage of existing evidence data from technical reports
Minimize new documentation work
Existing analyses remain same but may become deeper and more rigorous to support each claim Sarma Susarla IV&V Workshop 9/10/2013 3<br>
slide4. ISS CSCI Life Cycle In normal waterfall mode, the life cycle Sequence is SRR, PDR, CDR, TRR, and FQT.
In ISS above model is used for CSCI first release.
Subsequent CSCI releases every 15 months to
Enhance functionality of existing functions
Add new mission requirements such as Visiting Vehicle support
Fix existing critical software problems
Implement code changes to avoid operator workarounds (SPNs) for code problems
In sustaining phase , each release is an incremental change to the previous release
Only changes to the review artifacts are subject to formal reviews
To cut down development time, parallel development activity for:
Requirements development
Design/Code development
FQT test development Sarma Susarla IV&V Workshop 9/10/2013 4<br>
slide5. Life cycle Timeline RCS Review Process Test design development Design Test Review Process Test Implementation- Test Script Development Implementation Test Review Process Requirements Development 6 months 6 months 2 months Sarma Susarla IV&V Workshop 9/10/2013 Stage
TRRs FQT end Stage End STRR Legend
TDR: Technical Design Review
TRR: Test Readiness Review
FQT : Formal Qualification Testing
STRR: Software Transition Readiness Review
RCS: Requirement Change Sheet 4 months 5<br>
slide6. CSCI Life Cycle Reviews Sustaining engineering CSCI development life cycle
Content review
SCR (Software Change Request) database contains desired new functionality, existing problems, and product improvement suggestions
SCR Content is selected based on community high priority selections
IV&V selects open SCRs from the database
To meet mission requirements
To fix critical problems
Requirement Change Reviews
On going, held after requirement changes for each SCR are worked
IV&V evaluates changes for Q1, Q2, Q3 and provides issue feedback
Technical Design Review (TDR)
Held after all requirement changes and new designs are complete
Combines SRR, PDR, and CDR in waterfall model
IV&V evaluates design changes for Q1, Q2, Q3 and provides issue feedback
Code Reviews
On going, held after code developed for each SCR triggered change
IV&V evaluates changes for Q1, Q2, Q3 and provides issue feedback
Test Design Reviews
On going, held after test case designs to verify new requirements are developed
IV&V evaluates design to ensure that the tests verify the requirements correctly and completely.
Q2 and Q3 considered as applicable Sarma Susarla IV&V Workshop 9/10/2013 6<br>
slide7. CSCI Life Cycle Reviews Test Implementation Reviews
On going, review held after a test script for an FQT test is developed and dryrun on FQT platform
IV&V verifies from test log and script that the test is implemented as per test design and test cases ran in the expected sequence and order
IV&V also verifies that test failures are captured in SCRs.
Test Readiness Review (TRR)
Held after all tests completed test implementation reviews, review issues were incorporated, and tests were dryrun on flight release.
IV&V verifies that there are no open issues on any test cleared for FQT run
Formal Qualification Test Results Review
IV&V reviews test logs to verify that tests ran as expected and test failures are properly described in referenced SCRs.
Stage Test Reviews
IV&V reviews integrated tests that verify the CSCI performance in the on-orbit CSCIs integrated environment
IV&V ensures that test procedures match test objectives and test failures are correctly reported in SCRs
Software Transition Readiness Review
IV&V evaluates open SCRs/Issues to ensure that the corresponding problem does not impact on-orbit CSCI transition to new release or immediate operation with new release
If such SCRs/Issues are found, IV&V would articulate to the Program to address the problem before the transition. Sarma Susarla IV&V Workshop 9/10/2013 7<br>
slide8. Proposed CSCI Assurance Model Components
A Generic assurance claim network
The Generic model is instantiated for each CSCI to suit the CSCI’s specific life cycle specifics.
Sub-claims may be added to suite
Listed evidence in the model may be substituted with different evidence applicable evidence
TBDs in the model will be replaced with actual numbers
A Generic evidence reporting mechanism
Analysis Drivers
Technical Reference for Adverse Conditions
Technical Reference for undocumented CSCI behaviors as needed
PAL assets as extensions to Catalog of Methods for ISS specific artifacts analysis for:
Requirements
Software design
Flight code
Test design and implementation Sarma Susarla IV&V Workshop 9/10/2013 8<br>
slide9. Assurance Model Concept Inverted Tree structure of various claims each claim representing completion of a life cycle development event
Top most claim represents final assurance claim for Software on-orbit deployment
For each Sub-Claim
Claim Statement
Second level claims representing each major milestone in software development at Process level E.g.
Requirements development complete
Code development complete etc.
Third level claim represents sub-claims if second level claim is made up of several distinguishable life cycle events
The last level claim represents analysis activity requiring certain rigor such as IV&V 3 Questions
Evidence
Most of the data is what is generated for existing technical reports
TDR Report
TRR report
STRR report
List of developer products such as SCRs, RCSs and Tests reviewed and IV&V issues
Life cycle analysis activity and results reported in Biwar
For activity which is outside the scope of IV&V (such as simulation, unit tests etc), use developer’s status reports as supporting evidence
Argument
Description of life cycle analysis activity pertinent to realizing the claim
Contains reference to PAL assets and technical reference as appropriate Sarma Susarla IV&V Workshop 9/10/2013 9<br>
slide10. Assurance Case Claim Network Most Claims supported by IV&V Life cycle review activity Sarma Susarla IV&V Workshop 9/10/2013 10 Entire claim network is a word document containing for each claim: Claim statement; Evidence, Argument , Caveats , Supplemental info<br>
slide11. Analysis Rigor Methods
Analysis methods expanded from COM methods to detailed ISS specific guidelines.
Separate PAL asset each for Requirements, Design, Code and Test.
Adverse conditions
Full list of adverse conditions captured in TR folder in ECM, next slide shows a shortened list
These conditions are evaluated as applicable to Requirements, Design, Code, and Test reviews with respect to Q2, Q3.
If an adverse condition is integrated into a requirement, then further life cycle analysis for that will be under Q1.
There will be several adverse conditions in code that will not be captured in requirements. Code will be reviewed against such adverse conditions
Very few adverse conditions for test analysis because FQT will not test behaviors not captured in requirements
Adverse conditions not captured in requirements will be topics for Independent Analysis(IA). Sarma Susarla IV&V Workshop 9/10/2013 11<br>
slide12. ISS Adverse Conditions (Partial list) Sarma Susarla IV&V Workshop 9/10/2013 12<br>
slide13. Evidence Reporting Next set of slides contain reporting samples for some assurance claims in the claim network
Reported as part of Biwar as claims are realized during the life cycle development
At the end of the life cycle, a technical report is prepared containing:
claim network
IV&V activity and evidence that supports each claim Sarma Susarla IV&V Workshop 9/10/2013 13<br>
slide14. Requirements Claim (1.2) Claim
Requirements are developed as per CSCI SCR content and completed development prior to design and code development
Evidence
List of RCSs developed, and reviewed by IV&V and others
Biwar report confirming that Review issues were incorporated into the requirements and requirements development is completed
IV&V RCS review status reported in Biwar
RCS status reported at TDR
(Sub-Claim) Requirement evaluation per IV&V 3 Questions
Argument
Each RCS is reviewed against the corresponding SCR to ensure that the requirement adequately captures the SCR intent
Review issues were submitted and dispositioned with reviewer’s concurrence
Requirements development is completed by TDR except for post-TDR SCR content changes where the requirements development is completed before their design/code implementation. Sarma Susarla IV&V Workshop 9/10/2013 14<br>
slide15. Requirements Analysis Sub-claim (1.2.1) Claim
The requirements adequately capture what the system is supposed to do, what it is not supposed to do and how the system should behave under adverse conditions for defined conditionsÂ
Evidence
List of RCSs and the SCRs driving changes to requirements
List of accepted issues submitted by IV&V
Usage of PAL asset “ ISS Requirements Review and Analysis Guidelines” for analyzing the requirementsÂ
Argument
IV&V analyst is instructed to use the guidelines in the PAL asset for requirements analysis.
Each requirement is analyzed to match the concept in the SCR that drives the new requirement or change to the existing requirement.
Issues were submitted to the developer where the requirement is deficient from IV&V three questions perspective.
The analysis will cover all aspects for IV&V Q1 and practical situations for IV&V Q2 and Q3.Â
Supplemental information
Adverse conditions to be evaluated for are listed in ISS Technical Reference folder. The PAL asset contains specific guidelines designed for ISS development context. These guidelines provide instructions on how to verify for IV&V Q1, Q2, and Q3.Â
Caveats
For IV&V Q2, it is not possible to specify what the software is not supposed to do (negative specifications) for all cases and hence this specification is limited. During requirements analysis, IV&V will ensure that negative specifications are provided where practical.
For IV&V Q3, it is not possible to specify software actions for all possible adverse conditions However, certain adverse conditions (such as communication error, stale data, data exceeding a threshold, software timeout etc. can be integrated into requirement specifications . Sarma Susarla IV&V Workshop 9/10/2013 15<br>
slide16. Requirements Claim Reporting Reported after SRS incorporated all requirement changes as specified in content SCRs, usually after TDR (combines 1.2 and 1.2.1)
CSCI requirements development is completed and the new requirements are captured in SRS version <tbd>, in time for design and coding to proceed. Requirements were developed through <tbd> RCS reviews. IV&V evaluated the requirements to ensure that they capture what the software is supposed to do as indicated in the respective SCRs. IV&V also evaluated the requirements to ensure that as specified, the software does not do what it is not supposed to do and behaves adequately under applicable adverse conditions as indicated in ISS Technical reference “adverse conditions”. All accepted review issues were incorporated correctly into the SRS.
Evidence attachments to Technical report
Table showing SCR #, RCS number, review date
List of IV&V’s accepted issues
Table showing for each SRS function, # of new/changed requirements Sarma Susarla IV&V Workshop 9/10/2013 16<br>
slide17. Code Development Claim (1.4) Claim
Code development required by the SCR content is completed
Evidence
Status report in Biwar of IV&V Code reviews, either in developer’s peer reviews or separately and incorporation of review issues.
List of accepted IV&V issues/ or SCRs generated during code reviews
(Sub-claim) Code evaluation per IV&V 3 questions
Argument
IV&V reviewed code in code peer reviews and provided issue feedback.
If peer reviews were not held or missed, IV&V reviewed code changes after the new release code is available
Peer review issues were submitted to the developer during peer reviews and the accepted issues were incorporated into the final code.
If code review is done after the code is formally released, any issues found during the review are reported as SCRs and the SCR process ensures that the issue is appropriately addressed via seal-break process if the problem warrants such action; otherwise the issue will be fixed in a future release.
Supplemental information
Tools used to perform code analysis included Understand Ada, Understand C, and Textpad. Sarma Susarla IV&V Workshop 9/10/2013 17<br>
slide18. Code Evaluation Sub-claim(1.4.1) Claim
Code implemented correctly to produce behavior desired by the requirements and/or SCR on what it is supposed to do, what it is not supposed to do and perform adequately under applicable adverse conditions.
Evidence
Biwar report indicating code review completion by IV&V for all code changing SCRs in the release
Usage of PAL asset “ISS Code Analysis Guidelines” for code review
Argument
IV&V analyst is instructed to use the guidelines in the PAL asset for code analysis.
The code is evaluated to ensure that it implements the design documented in the design document and/or documented in requirements
For those code change SCRs that do not have a corresponding requirement change, the code is evaluated to ensure that it implements the behavior or code correction as specified in the SCR analysis recommendation.
Issues were submitted to the developer where the design does not match functionality desired from the requirement.
The code analysis will cover all aspects for IV&V Q1, Q2, Q3 where formal requirements exist. For those cases where formal requirements do not exist, IV&V will evaluate the code for all practical and credible aspects of Q2 and Q3 code scenarios that can take place during code execution; these practical scenarios are specified in the code analysis guidelines.
Supplemental information
Understand ADA/C, Textpad tools are used to navigate through the code and perform code analysis Sarma Susarla IV&V Workshop 9/10/2013 18<br>
slide19. Code Development Claim Reporting Reported after new flight code is released and IV&V has completed code review (combines 1.4 and 1.4.1)
IV&V reviewed code changes made for this release and confirmed that the code implements what is required as per the SCRs, respective requirements and the design documents TLDD, DBDD, and SUM. IV&V also evaluated the code changes to ensure that the code performs adequately under applicable adverse conditions as listed in ISS technical reference “ISS adverse conditions” and does not cause any unwanted behavior. All accepted code issues were implemented in the code.
Evidence attachments to technical report
List of code change SCRs, corresponding review date
List of accepted IV&V issues Sarma Susarla IV&V Workshop 9/10/2013 19<br>
slide20. FQT Completion Claim (1.6.6) Claim
Required CSCI behavior is verified though formal tests and test anomalies were reported correctly in SCRs
Evidence
List of FQT tests, pass/fail status, corresponding SCR for failed tests
Biwar report indicating IV&V evaluation of FQT results
Argument
IV&V evaluated test logs from formal testing to ensure all defined FQT tests were executed
IV&V examined the test failures and evaluated the corresponding SCR that reported the failure to make sure that the SCR correctly describes the failure.
Supplemental information
Program board, SCSRP, evaluates the impact of each SCR on ISS on-orbit operation and authorizes software changes via PPLs or patches to supplement the flight software for on-orbit usage.
IV&V provides feedback to the Program in the SCR dispositioning process Sarma Susarla IV&V Workshop 9/10/2013 20<br>
slide21. FQT completion reporting Reported after IV&V completed evaluation of all FQT test results (for 1.6.6)
CSCI Formal Qualification Testing completed. From the test report IV&V verified, that all planned FQT tests were run. <tbd1> of a total of <tbd2>, tests passed confirming that the software is performing as per the mission requirements except for the requirements listed below. For the failed tests IV&V verified that the test failure is correctly reported in the referenced SCR. The CSCI life cycle development is completed and the CSCI is verified to perform as required for the mission, with the exception of the following requirements. These failures will be assessed by the Program through the SCSRP process and appropriate changes will be made to the CSCI via PPL/Patch as needed for on-orbit operation
Evidence attachments to technical report
List of FQT tests with pass/fail status and associated SCR info for failed tests Sarma Susarla IV&V Workshop 9/10/2013 21<br>
slide22. CSCI Transition Readiness Claim (1.8) Claim
All software issues that affect on-orbit transition of the new release have been fixed
Evidence
List of undispositioned SCRs on the CSCI along with IV&V assessment as reported in Biwar
List of dispositioned SCRs that are targeted for fix in a future release along with IV&V assessment as reported in Biwar
IV&V’s assessment of work status of all review issues and SCRs targeted for fix in the current release as reported in Biwar
IV&V’s assessment of completion of SPNs targeted for this release as reported in Biwar
Argument
IV&V evaluates all undispositoned SCRs on this release to determine their impact on software behavior for software transition or immediately following transition
IV&V evaluates all SCRs that are targeted for fix in a future release to determine their impact on software behavior for software transition or immediately following transition
If any of those SCRs affect safety critical on-orbit operation, IV&V will request the SCSRP board to evaluate those SCRs prior to on-orbit software transition to determine if any patches or PPLs need to be developed.
IV&V evaluates any open issues from the CSCI reviews or SCRs targeted for fix in the current release to determine their impact on on-orbit transition and immediate software.
IV&V reviews SPNs developed for the release to make sure that operational workaround is documented Sarma Susarla IV&V Workshop 9/10/2013 22<br>
slide23. Transition readiness reporting Report this after the CSCI readiness is evaluated for on-orbit transition (1.8)
IV&V completed on-orbit transition readiness review and no exceptions have been uncovered. IV&V evaluated <tbd1> undispositoned SCRs on the current and previous releases and <tbd2> dispositioned but targeted for fix in a future CSCI release. IV&V analyzed these SCRs and determined that they do not impact the software operation for successful on-orbit transition from the previous release and immediate on-orbit operation with the new release. All work targeted for fix on this release from SCRs and milestone review issues is completed. IV&V verified that all SPNs, numbering <tbd>, triggered by the new release have been developed. IV&V’s analysis concludes that the CSCI is ready for on-orbit transition.
Evidence attachments to technical report
List of undispositioned SCRs showing SCR #, Title, CSCI Status, IV&V evaluation comment
List of SCRs approved for future work showing SCR #, Title, CSCI. Status, IV&V evaluation comment Sarma Susarla IV&V Workshop 9/10/2013 23<br>
slide24. CSCI Technical report Final Technical Report is prepared and will be delivered at Transition Readiness Review
Report will focus on the EBA network model and realization of claims
For data, report relies heavily on using what is reported in biwar and internal worksheets currently generated during IV&V life cycle work.
Report Format
Overall claim tree chart Plus for each major claim in the overall claim chart, the following:
Claim statement, Evidence, Argument, Caveats, supplemental information (as given in previous slides)
IV&V statement on claim realization as reported in Biwar (as given in previous slides)
Evidence data attachments Sarma Susarla IV&V Workshop 9/10/2013 24<br>
slide25. Assurance cases for software releases in ISS sustaining phase of development Questions/Comments Sarma Susarla IV&V Workshop 9/10/2013 25<br>
IV&V Team Lead
Sarma.V.Susarla@nasa.gov
9/10/2013<br>
slide2. Introduction IV&V has been performing ISS IV&V for the past 18 years
Software architecture and design is mature
However new releases are being developed to add new functionality and product improvements in the existing architecture.
IV&V had been generating technical reports at the end of each milestone event such as :
Technical Design Review (TDR)
Test Readiness Review (TRR)
Software Transition Readiness Review (STRR)
These reports document software assurance stating that the CSCI is ready for next stage in life cycle development.
This presentation provides a framework for an integrated software assurance approach:
To assert that CSCI is ready for on-orbit deployment
Using existing IV&V processes and analyses. Sarma Susarla IV&V Workshop 9/10/2013 2<br>
slide3. Evidence Based Assurance (EBA) Implementation Approach Use ISS life cycle development model
Use life cycle development milestones as individual claims into EBA network model
Requirements development completion
Code development completion etc.
Use Biwar report as a mechanism to report evidence as it becomes available during life cycle development support activity
Maximize usage of existing evidence data from technical reports
Minimize new documentation work
Existing analyses remain same but may become deeper and more rigorous to support each claim Sarma Susarla IV&V Workshop 9/10/2013 3<br>
slide4. ISS CSCI Life Cycle In normal waterfall mode, the life cycle Sequence is SRR, PDR, CDR, TRR, and FQT.
In ISS above model is used for CSCI first release.
Subsequent CSCI releases every 15 months to
Enhance functionality of existing functions
Add new mission requirements such as Visiting Vehicle support
Fix existing critical software problems
Implement code changes to avoid operator workarounds (SPNs) for code problems
In sustaining phase , each release is an incremental change to the previous release
Only changes to the review artifacts are subject to formal reviews
To cut down development time, parallel development activity for:
Requirements development
Design/Code development
FQT test development Sarma Susarla IV&V Workshop 9/10/2013 4<br>
slide5. Life cycle Timeline RCS Review Process Test design development Design Test Review Process Test Implementation- Test Script Development Implementation Test Review Process Requirements Development 6 months 6 months 2 months Sarma Susarla IV&V Workshop 9/10/2013 Stage
TRRs FQT end Stage End STRR Legend
TDR: Technical Design Review
TRR: Test Readiness Review
FQT : Formal Qualification Testing
STRR: Software Transition Readiness Review
RCS: Requirement Change Sheet 4 months 5<br>
slide6. CSCI Life Cycle Reviews Sustaining engineering CSCI development life cycle
Content review
SCR (Software Change Request) database contains desired new functionality, existing problems, and product improvement suggestions
SCR Content is selected based on community high priority selections
IV&V selects open SCRs from the database
To meet mission requirements
To fix critical problems
Requirement Change Reviews
On going, held after requirement changes for each SCR are worked
IV&V evaluates changes for Q1, Q2, Q3 and provides issue feedback
Technical Design Review (TDR)
Held after all requirement changes and new designs are complete
Combines SRR, PDR, and CDR in waterfall model
IV&V evaluates design changes for Q1, Q2, Q3 and provides issue feedback
Code Reviews
On going, held after code developed for each SCR triggered change
IV&V evaluates changes for Q1, Q2, Q3 and provides issue feedback
Test Design Reviews
On going, held after test case designs to verify new requirements are developed
IV&V evaluates design to ensure that the tests verify the requirements correctly and completely.
Q2 and Q3 considered as applicable Sarma Susarla IV&V Workshop 9/10/2013 6<br>
slide7. CSCI Life Cycle Reviews Test Implementation Reviews
On going, review held after a test script for an FQT test is developed and dryrun on FQT platform
IV&V verifies from test log and script that the test is implemented as per test design and test cases ran in the expected sequence and order
IV&V also verifies that test failures are captured in SCRs.
Test Readiness Review (TRR)
Held after all tests completed test implementation reviews, review issues were incorporated, and tests were dryrun on flight release.
IV&V verifies that there are no open issues on any test cleared for FQT run
Formal Qualification Test Results Review
IV&V reviews test logs to verify that tests ran as expected and test failures are properly described in referenced SCRs.
Stage Test Reviews
IV&V reviews integrated tests that verify the CSCI performance in the on-orbit CSCIs integrated environment
IV&V ensures that test procedures match test objectives and test failures are correctly reported in SCRs
Software Transition Readiness Review
IV&V evaluates open SCRs/Issues to ensure that the corresponding problem does not impact on-orbit CSCI transition to new release or immediate operation with new release
If such SCRs/Issues are found, IV&V would articulate to the Program to address the problem before the transition. Sarma Susarla IV&V Workshop 9/10/2013 7<br>
slide8. Proposed CSCI Assurance Model Components
A Generic assurance claim network
The Generic model is instantiated for each CSCI to suit the CSCI’s specific life cycle specifics.
Sub-claims may be added to suite
Listed evidence in the model may be substituted with different evidence applicable evidence
TBDs in the model will be replaced with actual numbers
A Generic evidence reporting mechanism
Analysis Drivers
Technical Reference for Adverse Conditions
Technical Reference for undocumented CSCI behaviors as needed
PAL assets as extensions to Catalog of Methods for ISS specific artifacts analysis for:
Requirements
Software design
Flight code
Test design and implementation Sarma Susarla IV&V Workshop 9/10/2013 8<br>
slide9. Assurance Model Concept Inverted Tree structure of various claims each claim representing completion of a life cycle development event
Top most claim represents final assurance claim for Software on-orbit deployment
For each Sub-Claim
Claim Statement
Second level claims representing each major milestone in software development at Process level E.g.
Requirements development complete
Code development complete etc.
Third level claim represents sub-claims if second level claim is made up of several distinguishable life cycle events
The last level claim represents analysis activity requiring certain rigor such as IV&V 3 Questions
Evidence
Most of the data is what is generated for existing technical reports
TDR Report
TRR report
STRR report
List of developer products such as SCRs, RCSs and Tests reviewed and IV&V issues
Life cycle analysis activity and results reported in Biwar
For activity which is outside the scope of IV&V (such as simulation, unit tests etc), use developer’s status reports as supporting evidence
Argument
Description of life cycle analysis activity pertinent to realizing the claim
Contains reference to PAL assets and technical reference as appropriate Sarma Susarla IV&V Workshop 9/10/2013 9<br>
slide10. Assurance Case Claim Network Most Claims supported by IV&V Life cycle review activity Sarma Susarla IV&V Workshop 9/10/2013 10 Entire claim network is a word document containing for each claim: Claim statement; Evidence, Argument , Caveats , Supplemental info<br>
slide11. Analysis Rigor Methods
Analysis methods expanded from COM methods to detailed ISS specific guidelines.
Separate PAL asset each for Requirements, Design, Code and Test.
Adverse conditions
Full list of adverse conditions captured in TR folder in ECM, next slide shows a shortened list
These conditions are evaluated as applicable to Requirements, Design, Code, and Test reviews with respect to Q2, Q3.
If an adverse condition is integrated into a requirement, then further life cycle analysis for that will be under Q1.
There will be several adverse conditions in code that will not be captured in requirements. Code will be reviewed against such adverse conditions
Very few adverse conditions for test analysis because FQT will not test behaviors not captured in requirements
Adverse conditions not captured in requirements will be topics for Independent Analysis(IA). Sarma Susarla IV&V Workshop 9/10/2013 11<br>
slide12. ISS Adverse Conditions (Partial list) Sarma Susarla IV&V Workshop 9/10/2013 12<br>
slide13. Evidence Reporting Next set of slides contain reporting samples for some assurance claims in the claim network
Reported as part of Biwar as claims are realized during the life cycle development
At the end of the life cycle, a technical report is prepared containing:
claim network
IV&V activity and evidence that supports each claim Sarma Susarla IV&V Workshop 9/10/2013 13<br>
slide14. Requirements Claim (1.2) Claim
Requirements are developed as per CSCI SCR content and completed development prior to design and code development
Evidence
List of RCSs developed, and reviewed by IV&V and others
Biwar report confirming that Review issues were incorporated into the requirements and requirements development is completed
IV&V RCS review status reported in Biwar
RCS status reported at TDR
(Sub-Claim) Requirement evaluation per IV&V 3 Questions
Argument
Each RCS is reviewed against the corresponding SCR to ensure that the requirement adequately captures the SCR intent
Review issues were submitted and dispositioned with reviewer’s concurrence
Requirements development is completed by TDR except for post-TDR SCR content changes where the requirements development is completed before their design/code implementation. Sarma Susarla IV&V Workshop 9/10/2013 14<br>
slide15. Requirements Analysis Sub-claim (1.2.1) Claim
The requirements adequately capture what the system is supposed to do, what it is not supposed to do and how the system should behave under adverse conditions for defined conditionsÂ
Evidence
List of RCSs and the SCRs driving changes to requirements
List of accepted issues submitted by IV&V
Usage of PAL asset “ ISS Requirements Review and Analysis Guidelines” for analyzing the requirementsÂ
Argument
IV&V analyst is instructed to use the guidelines in the PAL asset for requirements analysis.
Each requirement is analyzed to match the concept in the SCR that drives the new requirement or change to the existing requirement.
Issues were submitted to the developer where the requirement is deficient from IV&V three questions perspective.
The analysis will cover all aspects for IV&V Q1 and practical situations for IV&V Q2 and Q3.Â
Supplemental information
Adverse conditions to be evaluated for are listed in ISS Technical Reference folder. The PAL asset contains specific guidelines designed for ISS development context. These guidelines provide instructions on how to verify for IV&V Q1, Q2, and Q3.Â
Caveats
For IV&V Q2, it is not possible to specify what the software is not supposed to do (negative specifications) for all cases and hence this specification is limited. During requirements analysis, IV&V will ensure that negative specifications are provided where practical.
For IV&V Q3, it is not possible to specify software actions for all possible adverse conditions However, certain adverse conditions (such as communication error, stale data, data exceeding a threshold, software timeout etc. can be integrated into requirement specifications . Sarma Susarla IV&V Workshop 9/10/2013 15<br>
slide16. Requirements Claim Reporting Reported after SRS incorporated all requirement changes as specified in content SCRs, usually after TDR (combines 1.2 and 1.2.1)
CSCI requirements development is completed and the new requirements are captured in SRS version <tbd>, in time for design and coding to proceed. Requirements were developed through <tbd> RCS reviews. IV&V evaluated the requirements to ensure that they capture what the software is supposed to do as indicated in the respective SCRs. IV&V also evaluated the requirements to ensure that as specified, the software does not do what it is not supposed to do and behaves adequately under applicable adverse conditions as indicated in ISS Technical reference “adverse conditions”. All accepted review issues were incorporated correctly into the SRS.
Evidence attachments to Technical report
Table showing SCR #, RCS number, review date
List of IV&V’s accepted issues
Table showing for each SRS function, # of new/changed requirements Sarma Susarla IV&V Workshop 9/10/2013 16<br>
slide17. Code Development Claim (1.4) Claim
Code development required by the SCR content is completed
Evidence
Status report in Biwar of IV&V Code reviews, either in developer’s peer reviews or separately and incorporation of review issues.
List of accepted IV&V issues/ or SCRs generated during code reviews
(Sub-claim) Code evaluation per IV&V 3 questions
Argument
IV&V reviewed code in code peer reviews and provided issue feedback.
If peer reviews were not held or missed, IV&V reviewed code changes after the new release code is available
Peer review issues were submitted to the developer during peer reviews and the accepted issues were incorporated into the final code.
If code review is done after the code is formally released, any issues found during the review are reported as SCRs and the SCR process ensures that the issue is appropriately addressed via seal-break process if the problem warrants such action; otherwise the issue will be fixed in a future release.
Supplemental information
Tools used to perform code analysis included Understand Ada, Understand C, and Textpad. Sarma Susarla IV&V Workshop 9/10/2013 17<br>
slide18. Code Evaluation Sub-claim(1.4.1) Claim
Code implemented correctly to produce behavior desired by the requirements and/or SCR on what it is supposed to do, what it is not supposed to do and perform adequately under applicable adverse conditions.
Evidence
Biwar report indicating code review completion by IV&V for all code changing SCRs in the release
Usage of PAL asset “ISS Code Analysis Guidelines” for code review
Argument
IV&V analyst is instructed to use the guidelines in the PAL asset for code analysis.
The code is evaluated to ensure that it implements the design documented in the design document and/or documented in requirements
For those code change SCRs that do not have a corresponding requirement change, the code is evaluated to ensure that it implements the behavior or code correction as specified in the SCR analysis recommendation.
Issues were submitted to the developer where the design does not match functionality desired from the requirement.
The code analysis will cover all aspects for IV&V Q1, Q2, Q3 where formal requirements exist. For those cases where formal requirements do not exist, IV&V will evaluate the code for all practical and credible aspects of Q2 and Q3 code scenarios that can take place during code execution; these practical scenarios are specified in the code analysis guidelines.
Supplemental information
Understand ADA/C, Textpad tools are used to navigate through the code and perform code analysis Sarma Susarla IV&V Workshop 9/10/2013 18<br>
slide19. Code Development Claim Reporting Reported after new flight code is released and IV&V has completed code review (combines 1.4 and 1.4.1)
IV&V reviewed code changes made for this release and confirmed that the code implements what is required as per the SCRs, respective requirements and the design documents TLDD, DBDD, and SUM. IV&V also evaluated the code changes to ensure that the code performs adequately under applicable adverse conditions as listed in ISS technical reference “ISS adverse conditions” and does not cause any unwanted behavior. All accepted code issues were implemented in the code.
Evidence attachments to technical report
List of code change SCRs, corresponding review date
List of accepted IV&V issues Sarma Susarla IV&V Workshop 9/10/2013 19<br>
slide20. FQT Completion Claim (1.6.6) Claim
Required CSCI behavior is verified though formal tests and test anomalies were reported correctly in SCRs
Evidence
List of FQT tests, pass/fail status, corresponding SCR for failed tests
Biwar report indicating IV&V evaluation of FQT results
Argument
IV&V evaluated test logs from formal testing to ensure all defined FQT tests were executed
IV&V examined the test failures and evaluated the corresponding SCR that reported the failure to make sure that the SCR correctly describes the failure.
Supplemental information
Program board, SCSRP, evaluates the impact of each SCR on ISS on-orbit operation and authorizes software changes via PPLs or patches to supplement the flight software for on-orbit usage.
IV&V provides feedback to the Program in the SCR dispositioning process Sarma Susarla IV&V Workshop 9/10/2013 20<br>
slide21. FQT completion reporting Reported after IV&V completed evaluation of all FQT test results (for 1.6.6)
CSCI Formal Qualification Testing completed. From the test report IV&V verified, that all planned FQT tests were run. <tbd1> of a total of <tbd2>, tests passed confirming that the software is performing as per the mission requirements except for the requirements listed below. For the failed tests IV&V verified that the test failure is correctly reported in the referenced SCR. The CSCI life cycle development is completed and the CSCI is verified to perform as required for the mission, with the exception of the following requirements. These failures will be assessed by the Program through the SCSRP process and appropriate changes will be made to the CSCI via PPL/Patch as needed for on-orbit operation
Evidence attachments to technical report
List of FQT tests with pass/fail status and associated SCR info for failed tests Sarma Susarla IV&V Workshop 9/10/2013 21<br>
slide22. CSCI Transition Readiness Claim (1.8) Claim
All software issues that affect on-orbit transition of the new release have been fixed
Evidence
List of undispositioned SCRs on the CSCI along with IV&V assessment as reported in Biwar
List of dispositioned SCRs that are targeted for fix in a future release along with IV&V assessment as reported in Biwar
IV&V’s assessment of work status of all review issues and SCRs targeted for fix in the current release as reported in Biwar
IV&V’s assessment of completion of SPNs targeted for this release as reported in Biwar
Argument
IV&V evaluates all undispositoned SCRs on this release to determine their impact on software behavior for software transition or immediately following transition
IV&V evaluates all SCRs that are targeted for fix in a future release to determine their impact on software behavior for software transition or immediately following transition
If any of those SCRs affect safety critical on-orbit operation, IV&V will request the SCSRP board to evaluate those SCRs prior to on-orbit software transition to determine if any patches or PPLs need to be developed.
IV&V evaluates any open issues from the CSCI reviews or SCRs targeted for fix in the current release to determine their impact on on-orbit transition and immediate software.
IV&V reviews SPNs developed for the release to make sure that operational workaround is documented Sarma Susarla IV&V Workshop 9/10/2013 22<br>
slide23. Transition readiness reporting Report this after the CSCI readiness is evaluated for on-orbit transition (1.8)
IV&V completed on-orbit transition readiness review and no exceptions have been uncovered. IV&V evaluated <tbd1> undispositoned SCRs on the current and previous releases and <tbd2> dispositioned but targeted for fix in a future CSCI release. IV&V analyzed these SCRs and determined that they do not impact the software operation for successful on-orbit transition from the previous release and immediate on-orbit operation with the new release. All work targeted for fix on this release from SCRs and milestone review issues is completed. IV&V verified that all SPNs, numbering <tbd>, triggered by the new release have been developed. IV&V’s analysis concludes that the CSCI is ready for on-orbit transition.
Evidence attachments to technical report
List of undispositioned SCRs showing SCR #, Title, CSCI Status, IV&V evaluation comment
List of SCRs approved for future work showing SCR #, Title, CSCI. Status, IV&V evaluation comment Sarma Susarla IV&V Workshop 9/10/2013 23<br>
slide24. CSCI Technical report Final Technical Report is prepared and will be delivered at Transition Readiness Review
Report will focus on the EBA network model and realization of claims
For data, report relies heavily on using what is reported in biwar and internal worksheets currently generated during IV&V life cycle work.
Report Format
Overall claim tree chart Plus for each major claim in the overall claim chart, the following:
Claim statement, Evidence, Argument, Caveats, supplemental information (as given in previous slides)
IV&V statement on claim realization as reported in Biwar (as given in previous slides)
Evidence data attachments Sarma Susarla IV&V Workshop 9/10/2013 24<br>
slide25. Assurance cases for software releases in ISS sustaining phase of development Questions/Comments Sarma Susarla IV&V Workshop 9/10/2013 25<br>