Reproducibility in Modeling: Methods implemented in the ABM Simulation Framework FAME

Kurzfassung

Reproducibility of modelling results is one of the most important pillars of the scientific process. Still, reproducibility is not an easy achievement. Scientists need to consider several aspects during model design, versioning, and execution. In the design phase, non-deterministic behaviour must be ruled out by, e.g., avoiding race conditions, and controlling pseudo-random number generation. The latter is of special relevance if multiple random number generators are employed in parallel processes. Different versions of model components need to be identifiable using, e.g., semantic versioning, and findable, e.g. on repositories or package indices. Furthermore, model users need to store the model, linked libraries, input data, and corresponding metadata along with the model results to guarantee their reproducibility. The open framework FAME helps to create agent-based simulations. Agent logic is written in Java by extending framework classes from FAME-Core. Configurations are provided in YAML and CSV files and converted to a compressed binary format using the Python tool FAME-Io. A schema file describing the Java model allows FAME-Io to check input file consistency ahead of simulations. These can be executed on one or multiple processes using MPI. FAME employs multiple techniques to achieve result reproducibility. A central service provides reproducible random numbers based on configurable seeds and independent of the number of computing processes involved. The execution order of methods is implicitly derived from the configuration which helps to avoid race conditions. Strict memory separation of model components avoids accidental interference between agents. All FAME components are versioned with semantic versioning and frequently released to package indices. FAME models are typically packaged as a single executable Java archive, including all framework code, model code, and their dependencies in a single file. Model input, model output, and accompanying metadata are also stored in a single file. Thus, scientists need to store exactly two files to achieve full reproducibility -- regardless of the model complexity.