Validation Methods and Techniques derived by lessons learnt in EMMA2

Abstract

Beyond the scientific results the European Project EMMA2 was able to learn lessons how to validate in the future. The strengths of the E-OCVM – the basis of validation – are obvious: its six sub-steps facilitate planning of validation studies for both laymen and experts. In the same way E-OCVM’s levels of maturity are helpful in determining the status of a new concept. However good advice about how to find the appropriate methods and techniques for each maturity level is lacking. The sketchiness and novelty of concepts with levels V1/V2 should lead to measure “Operational Feasibility” only but not “Operational Improvements” in terms of Efficiency, Capacity, Safety, Human Factors issues etc. It is unlikely to expect meaningful or even statistically significant effects e.g. for increased capacity or enhanced situation awareness, if the system is too unclear or maybe even too complex or simply too novel for the user. Operational Feasibility consists of acceptance and usability of the new system by tomorrow’s user. It should be measured for example with a tailor-made questionnaire consisting of items directly derived from the operational concept. Hence the number of baseline scenarios may be reduced as far as possible because scenarios with today’s concepts won’t shed too much light in the acceptance of new systems. Given a transformation of a new concept to more mature levels like V3/V4, additional measurements are recommended to be applied to test finally Operational Improvements. Such kind of tests should only be conducted in shape of real-time simulations: Effects of increased efficiency or decreased workload can only be measured if an accurate designing of test scenarios took place in sub-step 2.5. It has to be guaranteed that baseline scenarios make the results of a new applied system comparable. Here the use of standard questionnaires like NASA-TLX or simple scales like ISA are suitable to measure operators’ reaction to new systems. For experiments in a complex and increasingly cooperative ATM the need for distributed- or coupled simulation setups is getting more and more relevant. The efficient use of such an experimental environment for validation requires the provision of standardised ways to connect simulators and exchange simulation data. To improve interoperability between validation platforms the EUROCAE WG81 is therefore developing the required standards and recommendation.