of the software/system.
For these reasons, system level testing has been traditionally accomplished by means of black-box testing. As previously stated, this approach expresses its limitations in case of safety-critical systems. First limiting to the observa-tion of the interfaces can prevent the testing engineer to observe the possible passage through some hazardous states. Second, as stated in Section1.3.1 inter-national safety and quality standards prescribe methodologies and techniques for system level testing with a limited or full knowledge of the system. Fur-thermore complex systems, can have internal emergent dynamics that must be known and controlled during the testing phase and system testing is the first moment when these dynamics start to appear. Such problem is worsening in case of critical systems where emergent behaviours can bring the system into unexpected hazardous states: thus black-box testing is considered as not suffi-cient to eliminate all the possible system hazards and motivate the adoption of gray-box approaches. In order to perform gray box (system-level) testing and having “limited knowledge” about the SUT, a model of the internal system be-haviour is needed. Indeed, gray box testing approaches are quite always model based-testing approaches, too. Many approaches that exploit models for gray-box testing are present in the scientific literature, for example in [61] where UML activity diagrams are used to generate test cases, [77] where FMS are used to un-derstand about the internal behaviour of the system or the work in [28] where the starting point of the generation is represented by a set of architectural models of the system specified with the AADL language. In the context of the gray-box ap-proaches, next section exploits model-based ones for test case generation and in particular poses the focus on approaches based on the modelling of the internal structure of the system.
its meaning. The expression "Model-based" applied to generation of test cases, can be applied in different forms. According to this [93], identifies the following meanings:
• Generation of test input data from informations about the domains of the input values (domain model). This reduces hand-made work but does not provide any information to know whether a test has passed or failed;
• Generation of test cases from an environment model. A model represents the environment of SUT. This approach does not provide information on testing outcome, too;
• Generation of tests cases from a behavioural model. A model describes the expected behaviour of the SUT. This approach needs oracle information to compare outcomes;
• Generation of test scripts from abstract representation of tests. Models describe test cases. From them proper transformations generate machine-readable tests.
The [93] taxonomy, was extended by [94] in order to address the embedded criti-cal systems related to criticriti-cal sectors as the railway, the healthcare or the avionic.
A summary of [94] taxonomy is depicted in Figure2.2
[94] taxonomy is very helpful, because not only surveys main approaches found in literature, but also underlines the key factors of a MBT process. In fact, there are some relevant factors, identified by [94] with the respect to the model-based generation of test cases:
• the selection criteria, that drives the generation of test cases. Among the criteria collected by [94] the approach described in this work is focused on requirement coverage, that means that we are interested to trace, on the model, SUT requirements and verify that generated test cases cover them;
• the choice of the technology to perform the model based generation. There are several manual and automatic approach. Among them, the one ex-ploited in this work, is the one based on Model Checking.
• the result of generation, that allows to classify the approaches on the basis of the nature of the generation results. In fact, generation results, can be, in turn, model, or executable test scripts/code.
FIGURE2.2: MBT taxonomy for Embedded Systems [94]
2.3.1 Automatic generation through Model Checking
A well-know technique to automatically generate test cases is the adoption of model checking techniques.
Model checking is a formal technique, used to analyse a finite-state represen-tation of a system for property violations. The core of the technique is the model checker which is able to analyse all reachable states of a system model, in order to verify a property. There are two main reasons that lead to the adoption of model checking for testing generation automation ([40]):
1. The model checker can be used as a oracle for test outcomes evaluation;
2. If a violation is detected, the model checker produces a counterexample, that is a sequence of analysed states whose crossing lead to the violation.
Several approaches that apply model checking techniques to testing automation have been surveyed in the years ([27, 36]). Black et al [12] combine mutation analysis with model checking in order to automatically generate complete test suites from formal specifications by defining new specification mutation oper-ators Cimatti et al.[Cimatti:2013] describe a formal approach for the validation of functional requirements of hybrid systems. In their work, for instance, coun-terexamples generation is adopted to verify, in a formal way, the consistency of requirements, during early stages of system development.
The idea emerging from the surveyed works is that a test can be modelled as a property that the system must satisfy. In fact, to verify a given property P, is nec-essary to to translate it in a Linear Temporary Logic (LTL) or in a Computational Tree Logic (CTL) and to give it to a model checker. However, with the respect to the generation of test cases, the negation of P is given in input to the model checker that, in order to demonstrate the violation, produces a counterexample which is a trace of steps that represents a single test case of a test suite (Figure 2.3).
As the same surveys state, there are some open issues with the respect to the automatic generation with a model checker. In particular the problem of the state explosion when dealing with complex and large models, that may produce a state-space which makes intractable the specification by a model checkers soft-ware. The second relevant problem is related to how to force the model checker to generate a complete test suite, since it usually return only one counterexample for each found violation.
FIGURE2.3: Testing with Model Checking process
2.3.2 Model-based vs Model-driven generation
Despite Model based (MB) and model driven (MD) use models as primary arte-fact in system development and encourage both the adoption of a domain-specific modelling language, sometimes there is a confusion in understanding the differ-ences between these two concepts. Several differdiffer-ences exist that definitely sep-arate MB and MD. First the concept of MB was introduced in the 1970 as an attempt to overcome the concept of computer-aided design (CAD) and to sup-port the system development process during its whole life cycle. Model driven, instead, was introduced later, to use abstract representation of a system to cretely generate system code from it. From this point of view, MD can be con-sidered as a subset of MB. Moreover, there is an important difference also in the objectives of these two paradigms. Models produced with a MB approach not necessarily are used to produce concrete system artefacts. In fact, models in MB approach are used to discover, at early stage of development and before the im-plementation, lacks in requirements specification or in system design. Models in a MD approaches instead, thanks to the support of model transformations, are usually used as inputs to produce new models, that allows to analyse the system from different point views or to obtain concrete realizations of the same system (as for example system code). A relevant example is the the research trend that encourages the adoption of formal methods in industrial settings . In fact, with the introduction of MD, is possible to model the SUT specification with a simple
and user-friendly formalism and deriving a complex formal model, in automatic way without any effort spent by the modeller.