A Methodology for Using Measures to Assess Software Safety ...

Fraunhofer USACenter for ExperimentalSoftware EngineeringA Methodology for Using Measures to Assess Software Safety Risk from an Independent Testing PerspectiveVictor R. Basili, Kathleen Dangle, Linda EskerFraunhofer Center for Experimental Software Engineering, MarylandITEA Symposium© 2008 Fraunhofer Center MarylandFraunhofer USACenter for ExperimentalSoftware EngineeringOutline• Problem• Safety Context and Visibility• Approach Overview• Approach Details• Steps and Examples• Benefits and Future Work2© 2008 Fraunhofer Center MarylandFraunhofer USACenter for ExperimentalSoftware EngineeringThe Problem•Independent evaluation of the safety of a system is traditionally done at the end of the system’s development life cycle, i.e., during independent test, ( e.g., DT) – Late visibility into problems– Limited time to do analysis and test •Resources during independent software test for safety are limited– Time and effort are limited resources – Need to be used effectively•There is a need to improve the safety analysis during independent software test to gain more confidence in the safety of a system•There is a need to maximize the opportunity of identifying potential safety risks that may not be exposed during operation3© 2008 Fraunhofer Center MarylandFraunhofer USACenter for ExperimentalSoftware EngineeringThe Problem: More specifically •There is insufficient start up information to assess the cost and schedule for independent testing: – How do we plan effective use of resources?•Developer processes are insufficient or lack safety deliverables– What kind of useful information can we gather?– How do we do it contractually?•There is a need to focus resources by understanding where the higher risks are– How do we take advantage of this information in a cost effective way?•There is a need for assessment of independent software safety test– How do we focus and evaluate our activities?4© 2008 Fraunhofer Center MarylandFraunhofer USACenter for ExperimentalContextSoftware EngineeringSystem Development PhaseDuring development, measures are SARneeded to monitor and track safety activities from a program management perspectiveSAR = Safety Assessment ReportIndependent System TestIndependent Software TestWhile in development, planning for independent software test beginsField TestA SAR isn’t done until the end, don’t even know what is fragile until the end of the development phase5© 2008 Fraunhofer Center MarylandFraunhofer USACenter for ExperimentalSoftware EngineeringContext•From a safety point of view, in independent software test– Testing and analysis are both required• Analysis involves assessing the hazards, the causes, the controls, and the verification for completeness and correctness,and testing involves checking that verifications on controls arecomplete, regression testing of those verifications as new changes are made, etc.– Testing and analysis are complex• Emphasis on “rainy day” testing vs. “sunny day”• Software by its nature introduces more off-nominal and out-of-bounds cases into the system– It is the last milestone focused on assuring software safety6© 2008 Fraunhofer Center MarylandFraunhofer USACenter for ExperimentalSoftware EngineeringVisibility Into System Safety Risks•What happened before? What is independent software test receiving?– What kind of information can be gathered from development that will provide the testers insights into the focus, amount, and types of analysis and testing needed? •How can we leverage prior safety activities performed, so that independent software testing can be tailored to the system it is receiving? – What functionality of the system is ready for independent software testing?– What are the high risk safety issues for this system? •How can we measure our progress during independent test and prioritize our activities according to highest safety risk issues?7© 2008 Fraunhofer Center MarylandFraunhofer USACenter for ExperimentalSoftware EngineeringApproach• Goal is to develop and implement a set of metrics that provide management visibility into system (and software) safety• For the purpose of asking the right questions, identifying safety risks and monitoring the quality of the safety process•Measure process OUTPUTS, intermediate products generated during development and test•Is sufficient material there? Where are the potential risks based upon missing information?– This is a syntactic, quantitative analysis. – Can be measured directly; can be automated•Is the right material there?– This is a semantic analysis – Can generate statistical samples, based upon the lack of sufficient materials, that can be manually inspected for quality attributes, e.g., correctness8© 2008 Fraunhofer Center MarylandFraunhofer USACenter for ExperimentalSoftware EngineeringApproach•Apply a set of metrics to objectively assist in identifying areas where safety may not have been properly addressed •Use development knowledge to focus analysis and test– Understand what data is available and how we might reinterpret that data from an independent test viewpoint– Develop an independent test plan that focuses on high risk areas– Whenever possible, use existing data (i.e., do not impose additional costs, time burden) •For example– During development perform this syntactic and semantic analysis– Make data available to independent software safety tester for planning– During independent test, perform this syntactic and semantic analysis to provide insight into safety concerns9© 2008 Fraunhofer Center MarylandFraunhofer USACenter for ExperimentalSoftware EngineeringDefining Measures to Provide Insights into Software Safety 1. Articulate the purpose of the safety related activity and Identify potential insight areas that sufficiently cover the important aspects of the software safety process for the specific environment2. State the goals associated with each insight area3. Develop a set of Readiness Assessment questions that– Provide initial insight into the areas of interest– Allow a quick and easy status report of the area– Identify whether it is possible to go deeper into the area4. Define Software Safety Visibility goals and questions to expose risks associated with outputs of the safety analysis process5. Develop/enumerate measures and models to define what will be measured and how it will be interpreted6. Identify responses to potential risks indicated by measures outside the model thresholds and further actions to be taken7. Apply the measures and interpret the results 10© 2008 Fraunhofer Center Maryland