/
Assurance  Cases  for  Software Assurance  Cases  for  Software

Assurance Cases for Software - PowerPoint Presentation

luanne-stotts
luanne-stotts . @luanne-stotts
Follow
370 views
Uploaded On 2018-11-09

Assurance Cases for Software - PPT Presentation

R eleases in ISS Sustaining P hase of Development Sarma Susarla IVampV Team L ead SarmaVSusarlanasagov 9102013 Introduction IVampV has been performing ISS IVampV for the past 18 years ID: 725676

code amp requirements test amp code test requirements software review csci claim development scrs scr sarma susarla design workshop issues evidence 2013

Share:

Link:

Embed:

Download Presentation from below link

Download Presentation The PPT/PDF document "Assurance Cases for Software" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Slide1

Assurance Cases for Software Releases in ISS Sustaining Phase of Development

Sarma Susarla

IV&V Team

L

ead

Sarma.V.Susarla@nasa.gov

9/10/2013Slide2

IntroductionIV&V has been performing ISS IV&V for the past 18 yearsSoftware architecture and design is matureHowever new releases are being developed to add new functionality and product improvements in the existing architecture.IV&V had been generating technical reports at the end of each milestone event such as :

Technical Design Review (TDR)

Test Readiness Review (TRR)

Software Transition Readiness Review (STRR)These reports document software assurance stating that the CSCI is ready for next stage in life cycle development.This presentation provides a framework for an integrated software assurance approach:To assert that CSCI is ready for on-orbit deploymentUsing existing IV&V processes and analyses.

Sarma Susarla IV&V Workshop 9/10/2013

2Slide3

Evidence Based Assurance (EBA) Implementation ApproachUse ISS life cycle development model Use life cycle development

milestones

as

individual claims into EBA network modelRequirements development completionCode development completion etc. Use Biwar report as a mechanism to report evidence as it becomes available during life cycle development support activity

Maximize usage of existing evidence data from technical reportsMinimize new documentation work

Existing analyses remain same but may become deeper

and more rigorous to support each claim

Sarma Susarla IV&V Workshop 9/10/2013

3Slide4

ISS CSCI Life Cycle In normal waterfall mode, the life cycle Sequence is SRR, PDR, CDR, TRR, and FQT. In ISS above model is used for CSCI first release.Subsequent CSCI releases every 15 months to

Enhance functionality of existing functions

Add new mission requirements such as Visiting Vehicle support

Fix existing critical software problems Implement code changes to avoid operator workarounds (SPNs) for code problemsIn sustaining phase , each release is an incremental change to the previous release

Only changes to the review artifacts are subject to formal reviewsTo cut down development time, parallel development activity for:Requirements development

Design/Code development

FQT test development

Sarma Susarla IV&V Workshop 9/10/2013

4Slide5

Life cycle TimelineRCS Review Process

Test design development

Design Test Review Process

Test Implementation- Test Script Development

Implementation Test Review Process

Software design/code development

Peer Review Process

Requirements Development

6 months

6 months

Content Review

TDR

TRR

2 months

Sarma Susarla IV&V Workshop 9/10/2013

Stage

TRRs

FQT end

Stage End

STRR

CSCI

ready

for on-orbit transition

Legend

TDR: Technical Design Review

TRR: Test Readiness Review

FQT : Formal Qualification Testing

STRR: Software Transition Readiness Review

RCS: Requirement Change Sheet

4 months

5Slide6

CSCI Life Cycle ReviewsSustaining engineering CSCI development life cycleContent

review

SCR

(Software Change Request) database contains desired new functionality, existing problems, and product improvement suggestionsSCR Content is selected based on community high priority selectionsIV&V selects open SCRs from the databaseTo meet mission requirementsTo fix critical problems Requirement Change Reviews

On going, held after requirement changes for each SCR are workedIV&V evaluates changes for Q1, Q2, Q3 and provides issue feedback

Technical Design Review (TDR

)

Held after all requirement changes and new designs are complete

Combines SRR, PDR, and CDR in waterfall model

IV&V evaluates design changes for Q1, Q2, Q3 and provides issue feedback

Code Reviews

On going, held after code developed for each SCR triggered change

IV&V evaluates changes for Q1, Q2, Q3 and provides issue feedback

Test Design Reviews

On going, held after test case designs to verify new requirements are developed

IV&V evaluates design to ensure that the tests verify the requirements correctly and completely.

Q2 and Q3 considered as applicable

Sarma Susarla IV&V Workshop 9/10/2013

6Slide7

CSCI Life Cycle ReviewsTest Implementation ReviewsOn going, review held after a test script for an FQT test is developed and

dryrun

on FQT platform

IV&V verifies from test log and script that the test is implemented as per test design and test cases ran in the expected sequence and orderIV&V also verifies that test failures are captured in SCRs.Test Readiness Review (TRR)Held after all tests completed test implementation reviews, review issues were incorporated, and tests were dryrun

on flight release.IV&V verifies that there are no open issues on any test cleared for FQT runFormal Qualification Test Results

R

eview

IV&V reviews test logs to verify that tests ran as expected and test failures are properly described in referenced SCRs.

Stage Test Reviews

IV&V reviews integrated tests that verify the CSCI performance in the on-orbit CSCIs integrated environment

IV&V ensures that test procedures match test objectives and test failures are correctly reported in SCRs

Software Transition Readiness Review

IV&V evaluates open SCRs/Issues to ensure that the corresponding problem does not impact on-orbit CSCI transition to new release or immediate operation with new release

If such SCRs/Issues are found, IV&V would articulate to the Program to address the problem before the transition.

Sarma Susarla IV&V Workshop 9/10/2013

7Slide8

Proposed CSCI Assurance ModelComponentsA Generic assurance claim networkThe Generic model is instantiated for each CSCI to suit the CSCI’s specific life cycle specifics.

Sub-claims may be added to suite

Listed evidence in the model may be substituted with different evidence applicable evidence

TBDs in the model will be replaced with actual numbersA Generic evidence reporting mechanismAnalysis DriversTechnical Reference for Adverse ConditionsTechnical Reference for undocumented CSCI behaviors as needed

PAL assets as extensions to Catalog of Methods for ISS specific artifacts analysis for:RequirementsSoftware design

Flight code

Test design and implementation

Sarma Susarla IV&V Workshop 9/10/2013

8Slide9

Assurance Model ConceptInverted Tree structure of various claims each claim representing completion of a life cycle development eventTop most claim represents final assurance claim for Software on-orbit deploymen

t

For each Sub-Claim

Claim StatementSecond level claims representing each major milestone in software development at Process level E.g.Requirements development completeCode development complete etc.Third level claim represents sub-claims if second level claim is made up of several distinguishable life cycle events

The last level claim represents analysis activity requiring certain rigor such as IV&V 3 Questions Evidence

Most of the data is what is generated for existing technical reports

TDR Report

TRR report

STRR reportList of developer products such as SCRs, RCSs and Tests reviewed and IV&V issues Life

cycle analysis activity and results

reported

in

B

iwar

For activity which is outside the scope of IV&V (such as simulation, unit tests etc), use developer’s status reports as supporting evidence

Argument

Description of life cycle analysis activity pertinent to realizing the claim

Contains reference to PAL assets and technical reference as appropriate

Sarma Susarla IV&V Workshop 9/10/2013

9Slide10

Assurance Case Claim NetworkMost Claims supported by IV&V Life cycle review activity

Code behavior verified by unit and integration tests

All CSCI issues that impact on-orbit transition have been addressed

FQT tests Verified CSCI behavior

CSCI behavior verified in integrated on-orbit configuration

1.5

1.7

1.8

All tests completed dry runs using flight software on FQT platform

TRR established CSCI, SIM and test rig readiness for formal testing

CSCI behavior verified through formal testing

1.6.4

1.6.5

1.6.6

Test designs are adequate to verify requirement changes

Test scripts and procedures implemented the designs accurately

Regression tests were adequate

Test designs verified as per IV&V 3 questions

1.6.1.1

1.0

SCR content represents mission concept

Design adequately represents requirements functionality

Code developed to match requirements and SCRs

Requirements developed per SCR content

1.1

1.2

1.4

Requirements evaluated per IV&V 3 Questions

Design evaluated per IV&V 3 Questions

Code evaluated per IV&V 3 Questions

1.4.1

1.2.1

1.3.1

Final claim at STRR

1.3

1.6

1.6.1

1..

6.3

1.6.2

Each Box is

a claim

CSCI ready for on-orbit transition

Sarma Susarla IV&V Workshop 9/10/2013

10

Entire claim network is a word document containing for each claim: Claim statement; Evidence, Argument , Caveats , Supplemental infoSlide11

Analysis RigorMethodsAnalysis methods expanded from COM methods to detailed ISS specific guidelines.Separate PAL asset each for Requirements, Design, Code and Test.

Adverse conditions

Full list of adverse conditions captured in TR folder in ECM, next slide shows a shortened list

These conditions are evaluated as applicable to Requirements, Design, Code, and Test reviews with respect to Q2, Q3.If an adverse condition is integrated into a requirement, then further life cycle analysis for that will be under Q1.There will be several adverse conditions in code that will not be captured in requirements. Code will be reviewed against such adverse conditionsVery few adverse conditions for test analysis because FQT will not test behaviors not captured in requirementsAdverse conditions not captured in requirements will be topics for Independent Analysis(IA).

Sarma Susarla IV&V Workshop 9/10/2013

11Slide12

ISS Adverse Conditions (Partial list)Sarma Susarla IV&V Workshop 9/10/2013

Adverse condition

Comment

Req

Design/Code

Test

Recoverable 1553 error (including channel switch)

Potential problem in meeting

real time

response requirements due to additional recovery time

X

X

 

Un recoverable 1553 com error (RT fail)

Potential functional problem to high level software which is communicating to RT and not monitoring for this error

X

X

 

LOC with ground

This can happen

up to

15

minutes

every orbit; Software should be analyzed for

ground

controlled hazards for consequences when LOC could happen in the middle of manual control. Also

ground

initiated command sequences like arm and fire. If fire is missed the arm should be cleared.

xx

MDM in BC role going to diagnostics

Autonomous software responses interrupted due to this

condition must be recoverable by the backup MDM and still meet time to effect requirements for potential hazardsXX

 

MDM in RT role going to diagnotics

A BC MDM sending autonomous responses to this MDM or through this MDM must be able to detect this condition, re-attempt the response after MDM recovery or generate C&W

xx

 

Hardware errorswhen software is acquiring data through an interfaces, the interface may provide some error information along with data. The software must check the error information and accept data only under no error condition 

No heartbeat from RT

The data acquired from RT would be stale. Software that uses this data must have smarts to detect that the data is stalex

x

 Transient

sensor data changes due to hardware problemsP

otential to trigger un-intended software functionality. To prevent this, data persistency checks must be made

x

xx

Out of range sensor dataPotential that software can crash without any proper diagnostic informationx

xxMultiple adverse condition occurrenceIt is possible that occurrence of multiple adverse conditions in a narrow time window could result in unacceptable system

failure like losing a catastrophic hazard control. System should be analyzed for at least

2 fault occurrence.x

xTask overruns

A task overrun can impact the execution time line of other tasks and cause function failures in other tasks

x

xMemory corruptionDefective software corrupting memory used by another software functionx

x

xOperator errorThis can be sending wrong command or wrong command parameter. The software should reject the command and provide such indication

xx 

12Slide13

Evidence ReportingNext set of slides contain reporting samples for some assurance claims in the claim networkReported as part of Biwar as claims are realized during the life cycle developmentAt the end of the life cycle, a technical report is prepared containing:

claim network

IV&V activity and evidence that supports each claim

Sarma Susarla IV&V Workshop 9/10/201313Slide14

Requirements Claim (1.2)ClaimRequirements are developed as per CSCI SCR content and completed development prior to design and code development

Evidence

List of RCSs developed, and reviewed by IV&V and others

Biwar report confirming that Review issues were incorporated into the requirements and requirements development is completedIV&V RCS review status reported in BiwarRCS status reported at TDR

(Sub-Claim) Requirement evaluation per IV&V 3 QuestionsArgument

Each RCS is reviewed against the corresponding SCR to ensure that the requirement adequately captures the SCR intent

Review issues were submitted and

dispositioned

with reviewer’s concurrenceRequirements development is completed by TDR except for post-TDR SCR content changes where the requirements development is completed before their design/code implementation.

 

Sarma Susarla IV&V Workshop 9/10/2013

14Slide15

Requirements Analysis Sub-claim (1.2.1)Claim

The requirements adequately capture what the system is supposed to do, what it is not supposed to do and how the system should behave under adverse conditions for defined conditions 

Evidence

List of RCSs and the SCRs driving changes to requirementsList of accepted issues submitted by IV&VUsage of PAL asset “ ISS Requirements Review and Analysis G

uidelines” for analyzing the requirements Argument

IV&V analyst is instructed to use the guidelines in the PAL asset for requirements analysis.

Each requirement is analyzed to match the concept in the SCR that drives the new requirement or change to the existing requirement.

Issues were submitted to the developer where the requirement is deficient from IV&V three questions perspective.

The analysis will cover all aspects for IV&V Q1 and practical situations for IV&V Q2 and Q3. 

Supplemental information

Adverse conditions to be evaluated for are listed in ISS Technical Reference folder. The PAL asset contains specific guidelines designed for ISS development context. These guidelines provide instructions on how to verify for IV&V Q1, Q2, and Q3. 

Caveats

For IV&V Q2, it is not possible to specify what the software is not supposed to do (negative specifications) for all cases and hence this specification is limited. During requirements analysis, IV&V will ensure that negative specifications are provided where practical.

For IV&V Q3, it is not possible to specify software actions for all possible adverse conditions However, certain adverse conditions (such as communication error, stale data, data exceeding a threshold, software timeout etc. can be integrated into requirement specifications .

Sarma Susarla IV&V Workshop 9/10/2013

15Slide16

Requirements Claim ReportingReported after SRS incorporated all requirement changes as specified in content SCRs, usually after TDR (combines 1.2 and 1.2.1)CSCI requirements development is completed and the new requirements are captured in SRS version

<

tbd

>, in time for design and coding to proceed. Requirements were developed through <tbd

> RCS reviews. IV&V evaluated the requirements to ensure that they capture what the software is supposed to do as indicated in the respective SCRs. IV&V also evaluated the requirements to ensure that as specified, the software does not do what it is not supposed to do and behaves adequately under applicable adverse conditions as indicated in ISS Technical reference “adverse conditions”.

All accepted review issues were incorporated correctly into the SRS.

Evidence attachments to Technical report

Table showing SCR #, RCS number, review date

List of IV&V’s accepted issues

Table showing for each SRS function, # of new/changed requirements

Sarma Susarla IV&V Workshop 9/10/2013

16Slide17

Code Development Claim (1.4)ClaimCode development required by the SCR content is completed

Evidence

Status report in

Biwar of IV&V Code reviews, either in developer’s peer reviews or separately and incorporation of review issues. List of accepted IV&V issues/ or SCRs generated during code reviews (Sub-claim) Code evaluation per IV&V 3 questions

Argument

IV&V reviewed code in code peer reviews and provided issue feedback.

If peer reviews were not held or missed, IV&V reviewed code changes after the new release code is available

Peer review issues were submitted to the developer during peer reviews and

the accepted issues were incorporated into the final code.If code review is done after the code is formally released, any issues found during the review are reported as SCRs and the SCR process ensures that the issue is appropriately addressed via seal-break process if the problem warrants such action;

otherwise

the issue will be fixed in a future release.

Supplemental information

Tools used to perform code analysis included Understand Ada, Understand C, and

Textpad

.

Sarma Susarla IV&V Workshop 9/10/2013

17Slide18

Code Evaluation Sub-claim(1.4.1)ClaimCode implemented correctly to produce behavior desired by the requirements and/or SCR on what it is supposed to do, what it is not supposed to do and perform adequately under applicable adverse conditions.

Evidence

Biwar

report indicating code review completion by IV&V for all code changing SCRs in the releaseUsage of PAL asset “ISS Code Analysis Guidelines” for code review

ArgumentIV&V analyst is instructed to use the guidelines in the PAL asset for code analysis.

The code is evaluated to ensure that it implements the design documented in the design document and/or documented in requirements

For those code change SCRs that do not have a corresponding requirement change, the code is evaluated to

ensure that it implements

the behavior or code correction as specified in the SCR analysis recommendation.Issues were submitted to the developer where the design does not match functionality desired from the requirement.

The code analysis will cover all aspects for IV&V Q1, Q2, Q3 where formal requirements exist. For those cases where formal requirements do not exist, IV&V will evaluate the code for all practical and credible aspects of Q2 and Q3 code scenarios that can take place during code execution; these practical scenarios are specified in the code analysis guidelines.

Supplemental information

Understand

ADA/C,

Textpad

tools are used to navigate through the code and perform code analysis

Sarma Susarla IV&V Workshop 9/10/2013

18Slide19

Code Development Claim ReportingReported after new flight code is released and IV&V has completed code review (combines 1.4 and 1.4.1)

IV&V reviewed code changes made for this release and confirmed that

the code implements what is required as per the SCRs,

respective requirements and the design documents TLDD, DBDD, and SUM. IV&V also evaluated the code changes to ensure that the code performs adequately under applicable adverse conditions as listed in ISS technical reference “ISS adverse conditions” and does not cause any unwanted behavior. All accepted code issues were implemented in the code.Evidence attachments to technical reportList of code change SCRs, corresponding review date

List of accepted IV&V issues

Sarma Susarla IV&V Workshop 9/10/2013

19Slide20

FQT Completion Claim (1.6.6)ClaimRequired CSCI

behavior is verified though formal tests and test anomalies were reported correctly in SCRs

Evidence

List of FQT tests, pass/fail status, corresponding SCR for failed testsBiwar report indicating IV&V evaluation of FQT resultsArgument

IV&V evaluated test logs from formal testing to ensure all defined FQT tests were executedIV&V examined the test failures and evaluated the corresponding SCR that reported the failure to make sure that the SCR correctly describes the failure.

Supplemental information

Program board, SCSRP, evaluates the impact of each SCR on ISS on-orbit operation and authorizes software changes via PPLs or patches to supplement the flight software for on-orbit usage.

IV&V provides feedback to the Program in the SCR

dispositioning

process

Sarma Susarla IV&V Workshop 9/10/2013

20Slide21

FQT completion reportingReported after IV&V completed evaluation of all FQT test results (for 1.6.6)CSCI Formal Qualification Testing completed. From the test report IV&V verified, that all planned FQT tests were run.

<tbd1>

of a total of <tbd2>, tests passed confirming that the software is performing as per the mission requirements except for the requirements listed below. For the failed tests IV&V verified that the test failure is correctly reported in the referenced SCR. The

CSCI life cycle development is completed and the CSCI is verified to perform as required for the mission, with the exception of the following requirements. These failures will be assessed by the Program through the SCSRP process and appropriate changes will be made to the CSCI via PPL/Patch as needed for on-orbit

operation

Evidence attachments to technical report

List of FQT tests with pass/fail status and associated SCR info for failed tests

Sarma Susarla IV&V Workshop 9/10/2013

Failed SRS

Requirement

SCR

Describing

the

Problem

Req#, req text

SCR #, SCR title

Req#, req text

SCR #, SCR title

Req#, req text

SCR #, SCR title

21Slide22

CSCI Transition Readiness Claim (1.8)ClaimAll software issues that affect on-orbit transition of the new release have been fixed

Evidence

List of

undispositioned SCRs on the CSCI along with IV&V assessment as reported in BiwarList of dispositioned SCRs that are targeted for fix in a future release along with IV&V assessment as reported in B

iwarIV&V’s assessment of work status of all review issues and SCRs targeted for fix in the current release as reported in B

iwar

IV&V’s assessment of completion of SPNs targeted for this release as reported in

B

iwar Argument

IV&V evaluates all

undispositoned

SCRs on this release to determine their impact on software behavior for software transition or immediately following transition

IV&V evaluates all SCRs that are targeted for fix in a future release to determine their impact on software behavior for software transition or immediately following transition

If any of those SCRs affect safety critical on-orbit operation, IV&V will request the SCSRP board to evaluate those SCRs prior to on-orbit software transition to determine if any patches or PPLs need to be developed.

IV&V evaluates any open issues from the CSCI

reviews or SCRs targeted for fix in the current release

to determine their impact on

on

-orbit

transition and immediate software.

IV&V reviews SPNs developed for the release to make sure that operational workaround is documented

Sarma Susarla IV&V Workshop 9/10/2013

22Slide23

Transition readiness reporting Report this after the CSCI readiness is evaluated for on-orbit transition (1.8)IV&V completed on-orbit transition readiness review and no exceptions have been uncovered. IV&V evaluated

<tbd1>

undispositoned SCRs on the current and previous releases and <tbd2> dispositioned but targeted for fix in a future CSCI release. IV&V analyzed these SCRs and determined that they do not impact the software operation for successful on-orbit transition from the previous release and immediate on-orbit operation with the new release. All work targeted for fix on this release from SCRs and milestone review issues is completed.

IV&V verified that all SPNs, numbering <

tbd

>,

triggered by the new release have been developed. IV&V’s analysis concludes that the CSCI is ready for on-orbit transition.

Evidence attachments to technical reportList of

undispositioned

SCRs showing SCR #, Title, CSCI Status, IV&V evaluation comment

List of

SCRs approved for future work showing SCR

#, Title, CSCI. Status, IV&V evaluation comment

Sarma Susarla IV&V Workshop 9/10/2013

23Slide24

CSCI Technical reportFinal Technical Report is prepared and will be delivered at Transition Readiness ReviewReport will focus on the EBA network model and realization of claimsFor data, report relies heavily on using what is reported in biwar

and internal worksheets currently generated during IV&V life cycle work.

Report Format

Overall claim tree chart Plus for each major claim in the overall claim chart, the following:Claim statement, Evidence, Argument, Caveats, supplemental information (as given in previous slides)IV&V statement on claim realization as reported in Biwar (as given in previous slides)Evidence data attachments

Sarma Susarla IV&V Workshop 9/10/2013

24Slide25

Assurance cases for software releases in ISS sustaining phase of developmentQuestions/Comments

Sarma Susarla IV&V Workshop 9/10/2013

25