According to the industrial needs and to the recommendations of international standards, there is the need to automate some stages of the life cycle of embed-ded safety-critical systems. The original contribution of this thesis is in the def-inition of a model-driven framework, able to enable the automatic generation of proper formal models, starting from an high-level specification of a system behaviour. In particular, formal models described in this work are involved in the context of the V&V activities of a embedded safety-critical system life cycle and in particular in the automatic generation of test cases. To realize such frame-work, model-driven principles have been exploited. More in detail, in order to capture the needs of the application domain, and enable the automatic gener-ation of test case, proper domain specific modelling languages (DSMLs) have been defined. In particular topics related to the definition of formal syntax and semantics of DSMLs have been address. Another relevant activity described in this thesis is in the definition of a structured methodology to address mapping issues related to the translation from the high-level languages defined and the languages adopted to describe the generated formal models. A concrete frame-work, based on model transformations realizing the defined mapping, has been realized during the thesis work. The validation of the developed framework has been performed by applying it within the context of the mass-transit trans-portation and in particular within the railway domain. Even if the focus is on automatic generation of test cases, the proposed Model-Driven methodology is useful also for the complete design (based on centralized information), hence it is possible to extend it for automatic verification of several non-functional prop-erties of embedded critical systems as the survivability and the vulnerability.
Chapter 2
State of the art in automatic test case generation
The testing activities affect up to the 50% of the total costs and time of a criti-cal system development cycle. Thus, in order to reduce this effort, academician and industrial interests have been focused on the concept of testing automation.
Literature is rich of definitions addressing the this concept:
"The management and performance of test activities, to include the development and execution of test script so as to verify test requirement, using an automated test tool "
(Dustin et al.[29])
"A system that includes technologies that support automatic generation and/or execution of tests for unit, integration, functional, performance, and load testing"
(Huzinga et al.[51]) Testing automation means that tests are run and checked without manual intervention
(...) you make use of a test automation framework to write and run
Sommerville [86]
This chapter is focused on well-know techniques and research trends adopted for testing automation and in particular for the automatic generation of test case.
The chapter will provide the needed background to understand research activi-ties at the basis of the present thesis. First a brief overview of the a general testing automation process is provided in order to understand which are its the key fac-tors. Then, with the respect of the state of the art, Model-based techniques for automation of test cases generation are discussed.
2.1 The process of automation
As emerging from previous definitions, the process of testing automation, is composed by several stages taking into account different aspects of a testing process. Despite the absence of reference standards, several methodologies have been proposed in the year as an attempt to provide a systematic schema to per-form testing automation. Among such proposed methodologies, one of the most famous it the Automated Test Lifecycle Methdology (ATLM) proposed by [29]. The ATLM, depicted in Figure 2.1, shows that, the automation process is anything but easy, due to the presence of several stages to address:
FIGURE2.1: The ATLM process ([29])
1. Decision to automate: the first step might seem obvious but in some cases it may not be necessary to introduce testing automation. This step, in fact, should help the team to outlines the potential benefits of automation and the effort to spent in an automated test development; There are several possible reasons that motivate the introduction of testing automation:
• Manual testing is a laborious task, that can lead to human errors;
• Automation allows to better verify that the SUT meets its require-ments;
• Automation reduces the presence of defects and failures of the devel-oped system;
2. Test tool acquisition: during this step, the software team, can decide to ac-quire an existing automation tool suite, or develop it from scratch. The outcomes of the step varying on the basis of the choice. In the first case, a set of criteria to drive the choice have to be defined, and the tool suite vendor has to be contacted. In the second case, this step can be assimilated to the design phase of a software development cycle.
3. Introduction of Automated testing: this phase is focused on the process that allows to integrate the automation strategy within the considered software system. This phase can be divided in two sub-steps:
• Test Process Analysis, which ensures that the overall test strategy is in place and adapted to the considered system under development;
• Test Tool Consideration, that focuses on the benefits that can be achieved from the introduction of the tool suite for automation, during the sys-tem development.
4. Test planning, design and development: this and the next one are the most important steps as they include the identification of the test procedures, defines the tests cases and allows for their generation. The test plan iden-tifies among others test items, the features to be tested, the testing tasks, the test environment, the test design techniques and a preliminary testing schedule. Then, according to the plan, test cases are designed in terms of the steps that have to be followed in order to verify a given system fea-ture. Finally the test development allows to obtain test script that can be executed on the system.
5. Execution and management. In this step, the previously obtained scripts are executed on the system and the results are collected, in order to perform the assessment.
6. Test assessment: this step has a twofold objective: first it allows to evaluate the automated testing process, in order to exploit eventual weakness and improve them at an early stage of development. The second, is the result analysis to find out the defects distribution of the considered system.
The first three steps of such general methodology are more comparable to management process that has to carry out the advantages of an automation cess and the needed effort to introduce it within an existing development pro-cess. Instead the process that lead to the effective automation of testing, is com-posed by three defined phases:
1. Test Generation, in which high-level description of test cases is generated from test specification. The outcomes of this step are usually not exe-cutable;
2. Test Development, that allows to translate the generated test cases in codes or scripts written in machine-readable language and therefore executable;
3. Test Execution, in which, scripts are executed on the SUT;
4. Test Reporting, in which outcomes from tests executions are analysed and compared with predicted outcomes.
With the respect to the context of the testing automation, this work is focused on the Test Generation step. Next sections provide the state of the art related to approaches for automatic test case generation from an high-level specification of the SUT.