A survey on test and evaluation methodologies of pedestrian localization systems

Abstract

The first formal attempt to standardize the test and evaluation of localization systems was the ISO/IEC 18305 standard. As reviewed in this paper, the ISO/IEC 18305 provides guidelines to evaluate an indoor localization system as a black box. Such evaluation is useful for end users, but not fully useful for either developers or researchers. As a result, both developers and researchers use their own evaluation methodologies, which have a set of weaknesses. Our objective is to compare the standard to the evaluation methodologies implemented by developers. We focus on pedestrian localization systems because they are the most numerous. In this work, we survey both competitions and individual works, between 2008-2018, with the ISO/IEC 18305 standard as a guideline. That is, we have reviewed the evaluation methodologies regarding the different aspects defined in the standard. We finalize with a summary of the shortcomings of the evaluation methodologies implemented by the developers, as well as the shortcomings of the standard. Our article ends with a summary of open issues that need to be addressed in order to foster the development of evaluation methodologies of localization systems. The main conclusion of our work is that the developers of localization systems are in need of guidelines to design evaluation methodologies, especially for the evaluation of system components. The standard fails to satisfy such requirement by proposing an evaluation strategy that focuses more on the end user than the developer.