The software tool certification is an integral part of the avionics industry that guarantees that the avionics software satisfies safety, reliability, and regulatory standards. Avionics systems are designed, developed, verified, validated, and maintained using software tools. To make sure that they do not introduce any risk into the software that may impact flight safety, these tools need to be qualified.
In the past, the process was manual. Today, a lot of automated tools have been developed for doing the job, which are potentially more reliable than any human effort and are able to generate code, test cases, and test procedures. Some of the tools are developed by a third party. This saves the company time, and the cost of generating the in-house tools. As more activities are performed by these tools, the DO178C needs to be updated to provide suitable evidence that these tools are available and are being used to take the credit through the certification process.
Tool qualification typically involves activities such as documenting the tool’s purpose and requirements, conducting tool verification and validation, performing tool and host environment integration testing, and generating the necessary documentation to support the qualification process. Based on complexity of the projects, the organization capability and tools used, there are many factors which influences the automation and process optimization ratio in development and verification under the DO178B/C; a few insights are shown below:
The DO-330 ”Software Tool Qualification Considerations”, a new “domain independent, external document”, was developed to provide guidance for an acceptable tool qualification process.
As per the RTCA, the DO-330 a supplement definition of tool qualification is needed when the process of DO-178C objectives is eliminated, reduced, or automated by the use of the software tool without its outputs being verified.
Development Automation: Organizations can use tools to automate tasks such as code generation, unit testing, regression testing, and documentation generation. The extent of automation can vary based on the organization’s capabilities and the availability of appropriate tools.
Verification Automation: Tools for an automated test case generation, test execution, and coverage analysis can be employed to optimize the verification process.
Process Optimization: Process optimization involves improving the efficiency and effectiveness of the software development and verification processes. This can be achieved through best practices, standardization, and continuous improvement initiatives. Organizations can adopt process optimization methodologies such as Lean or Six Sigma to identify areas of improvement and implement strategies to eliminate waste and optimize resource utilization.
In most cases, tool qualification is recommended if the output is not verified, but there are few scenarios where tool qualification may not be required, these are listed below:
- Documentation tools used for development/verification do not directly impact the safety critical functions of airborne systems.
- In-house tools specially designed for intranet use should ensure that these tools are reliable and safe to use and that the tool output is not embedded in airborne software
- The user need not perform tool qualification if the tool vendor has provided evidence of its qualification for any commercial off the shelf (COTS) tool and the output is verified using a peer review process.
DO178C/ED-12 standard describes requirements for the tool qualification*. If the output of any tool is fed as input to any of the development stages of safety critical SW without any added facts of reliable verification, then section-12.2 of DO178C and section-11.4 of DO-254 demands that the tool needs qualification.
The utilization of a tool in achieving a specific safety objective within the design and implementation of a software application is referred to as tool qualification. Tool Qualification constitutes of activities like:
- Recognizing whether they are development or audit type of tools.
- Figuring out the level of the tool
- Making sure the tool’s specification meets the standards as per section 12.2.1 DO178C.
- Analyzing the performance for the tool, managing the defects by analysis and correction.
The DO-178C primarily concentrates on offering guidance for certifying airborne software. However, it does not explicitly cover the qualification of tools utilized in the software life cycle process. Whereas the DO-330 fills this gap by providing a framework for the qualification of software tools
The DO-330, called the ‘Software Tool Qualification Considerations’ standard published by the Radio Technical Commission for Aeronautics (RTCA), provides guidance on the qualification of tools. This is used in safety critical airborne systems both from a tool developer’s and a tool user’s perspective. Based on this the software tools are categorized under two categories:
- Tools which can automate or aid any software process or activity.
- Tools which help programmers to develop software lifecycle relics like design documents, requirement specific object code etc.
1. Software Tools can support the tasks which can be:
- Recurring tasks
- Complicated tasks that need interpretation
- Tasks liable to human error
- Tasks/process that are flexible and
- Re-engineering.
It takes less time to do tasks, and we can focus more on more important tasks.
Tools are sometimes counterproductive as they can add more delay to the process than making it more efficient. For example, traceability and dynamic test coverage metrices will be one of the processes which are suited for tools.
2. Classification of Tools as per DO178B:
- Development Tools which can induce errors and whose output can be part of airborne software. Eg:- Any compiler when compiled in order to generate executable code can potentially add errors to the executable code.
- Verification Tools cannot introduce any errors but instead fail to detect them. Eg: It will not modify either the source code or the requirements but also fails to detect the error.
As per the above description, it is inevitable that software tools influence the final output of the software product and as per the D0178B guidelines, states that any tool needs to be qualified if it is going to satisfy any of the above points mentioned in Section I and if the output of the tool is the only means of delivering proof of evidence to the regulatory authorities(FAA/EASA).
3. The main characteristics of tool qualification are as follows:
- The tool should offer certainty which is equivalent to tasks/process mentioned in section I.
- The tool should be qualified to be implemented on specific configuration. Eg :- The tool used for level D software cannot be re-used for level B. A thorough assessment is required to determine if the tool is qualified or not.
- Only those utilities that are used to remove, ease, or automate SW lifecycle process activities and whose output is not verified need tool qualification.
It is important to note that a verification tool does not necessarily replace verification tests, for example we have a verification tool that instruments the code and afterwards removes the instrumentation code. In this case, this is a verification process, and the tool used helps with it. However, this tool changes the source code, so if there is a mistake in that tool it would introduce errors in the final product which makes it a development tool. This is really confusing hence, what we can newly see in the DO178C Tool Qualification in terms of development and verification tools are no longer used, instead the Certification authorities came up with tool qualification levels. In addition to the DO178C/ED-12C, RTCA Inc./EUROCAE has created one among the five sub-supplements on tools (DO330/ED-215) which describes objectives specific to tool qualifications.
Brochure: Aerospace Engineering Services
The motive behind developing the DO330/ED-215 is to provide tool specific guidance for tool users and tool developers. The objectives for a tool-user focus on selecting the tool to be employed and evaluating its influence on software processes. The objectives of a tool developer on the other hand, outline the development process, conduct verification and integration procedures, and ensure that it complies with the user expectations published in Tool Operational Requirements (TOR).
4. DO178B/ED-12B Vs DO178C/ED-12C Tool Qualification Criteria:
The phrases “development tool” and “verification tool” in the DO178B are substituted by the following tool qualification criteria in the DO178C:
| Criteria 1: | A tool whose output is part of the airborne software and thus could insert an error |
| Criteria 2: | A tool that automates verification processes and thus could fail to detect an error, and whose output is used to justify the elimination or reduction of : 1. Verification processes other than that automated by the tool, or 2. Development processes that could have an impact on the airborne software. |
| Criteria 3: | A tool that, within the scope of its intended use, could fail to detect an error |
Depending on what the Assurance level (DAL A, B, C, D) is guaranteed and based on what criteria are applied the following tool qualification levels are defined in the DO178C/ED-12C sub supplements.
Figure 2:
5. Qualified Tools Life Cycle:
Just like the DO178C defines software lifecycle process, similarly even the DO-330/ED-215 implements lifecycle process for tools that are qualified, to achieve the quality for a product.
The tool qualification lifecycle is classified under: –
- Tool Planning
- Tool Development
- Tool Verification
Tool Planning: During this phase, the project defines which tools are being used, assigning Tool qualification level (TQL) based on the safety impact (DAL); it outlines the qualification strategy/plans along with verification and development environments like plans, checklists, process.
Tool Development: This phase describes how the tool is implemented to meet its qualification objectives which involves verifying tool’s functional and safety requirements, implement the tool in accordance with the design, coding, and integration activities.
Tool Verification: Ensures the verification activities demonstrate the correctness and reliability of the tool, make sure there is traceability between the tool’s requirements and its design and implementation.
All the above three need to be implemented in a sequential order making sure Integral process which constitutes of Tool Configuration Management, Tool Quality Assurance, and Tool Qualification Liaison are parallelly implemented in the background.
The DO-330 is often used in conjunction with other aerospace standards such as DO-178C, DO-278A, DO-254, DO-200A, DO-331, DO-332, DO-333.
Some tools are difficult to qualify based on the DO-178C/DO-330, but in such cases a workaround has been created. The DO-330 also mentions the feasibility of usage of alternative methods to qualify a tool. One such method is a formal method which is an alternative for qualifying a tool.
E.g., Model Checking as a formal method is a mathematical approach that describes all possible states of the tool to verify that it satisfies formal specifications.
6. Objectives of Tool Qualification:
There may be multiple stakeholders involved in the tool qualification process; hence all their duties and functions need to be documented in the Plan for Software Aspects of Certification (PSAC) and Tool Qualification Plan (TQP). Finally, the applicant is responsible for meeting all the objectives of Tool Qualification.
A summary is mentioned below for the tool qualification objectives at each phase. In addition to the 10 listed tables in the DO178C/ED-12C, one more table has been introduced (Table T0) in the DO-330/ED-215.
| Summary of Table T-1 to T-10 | |
| Table T-1 Tool Planning Process | Defines tool development and integration environment, process, standards, transition criteria. |
| Table T-2 Tool Development Processes | Define the tool’s derived requirements, the tool’s LLR, its derived LLR, and the source code, which is developed, and the executable code is produced. |
| Table T-3 Verification of Outputs of Tool Requirements Processes | Describes the tool’s verification process like tool requirements; complies and is traceable to the tool’s operational requirements. Tool requirements are verifiable, consistent and conform to the tool requirement standards, etc. |
| Table T-4 Verification of Outputs of Tool Design Process | Describes the tool verification process; whether the tool’s LLR complies and is traceable to tool requirements; the LLR is consistent and conforms to tool design standards etc. |
| Table T-5 Verification of Outputs of Tool Coding & Integration Process | Describes the tool verification process; whether the source code complies and is traceable to tools LLR; source code is consistent and conforms to tool’s LLR etc. |
| Table T-6 Testing of Outputs of Integration Process | Tool executable object code complies and is robust with tool requirements and the tool’s LLR. |
| Table T-7 Verification of Outputs of Tool Testing | Test procedures and test results are correct. Test coverage of tool requirement and tool LLR are achieved; structural coverage to the level of MCDC, decision coverage, statement coverage and data and control coupling is achieved. |
| Table T-8 Tool Configuration Management Process | Configuration items are identified, baselined and traceable; problem reporting, change control and change review, archival retrieval and release are established. |
| Table T-9 Tool Quality Assurance Process | Assurance is obtained that the tool plans and standards are reviewed for consistency and that the transition criteria is satisfied. |
| Table T-10 Tool Qualification Liaison Process | The means of compliance is proposed, and an agreement is obtained between the applicant and the certifying authority. |
7. DO-330/ED-215 Table T-0 Tool Operational Processes
The table below illustrates all the objectives focusing the uses of the tool in the software life cycle process.
Figure 3:
8. Artifacts Required for Tool Certification
- Plan for Software Aspects of Certification (PSAC): PSAC includes tool-specific information like the type of tool (COTS or in-house), its intended use, the tool’s effect on software life cycle, and the reference to the tool’s operational, integration and verification and validation process/environments.
- Tool Qualification Plan (TQP): The TQP includes the tool qualification process for a Tool to be classified under TQL-1,2,3 and 4 levels. Description of the tool’s overview, architecture, interfaces, tool qualification activities and the data to be produced.
- Tool Development Plan (TDP): The TDP includes tool’s requirement, design and development and coding standards along with the tool’s lifecycle.
- Tool Verification Plan (TVP): The TVP includes tool verification activities, transition criteria, independence and reverification which satisfies the tool’s verification objectives.
- Tool Configuration Management Plan (TCMP): The TCMP describes activities like identification of configuration item, baseline, traceability, Problem reporting and change control which satisfies the objectives of TCM throughout the tool life cycle.
- Tool Quality Assurance Plan (TQAP): Describes the approach to meet the goals/objectives of TQA process which includes revie
*‘Tool qualification’ means ‘certification.’
Conclusion: It is important to make a note that the choice of tool qualification methos depends on aspects such as the criticality of the software being implemented and the impact of the tool on the safety standards. It is the organizations who carefully choose the suitable methods for tool qualification in compliance with regulatory authorities – Federal Aviation Administration and European Aviation Safety Agency (FAA/EASA).
Know More: Avionics Engineering Services
Bibliography:
- RTCA, DO-178B Software Considerations in Airborne Systems and Equipment Certification, Radio Technical Commission for Aeronautics, 1992.
- RTCA, DO-178C Software Considerations in Airborne Systems and Equipment Certification, Radio Technical Commission for Aeronautics, 2011.
- RTCA: DO-330 Software Tool Qualification and Considerations. Radio Technical Commission for Aeronautics, 2011.
