FoxBench - Benchmark Array File Formats

Abstract

For effective data exchange and transfer, choosing the proper file format is crucial. Different domains have specific standards for file formats. While CSV files are commonly used, they lack reusability. Data files are well-suited for computing clusters as they enable efficient data processing and storage across multiple interconnected systems. Data analytics pipelines can be time-consuming due to handling large volumes of data. Timely data access is crucial for efficient processing and analysis. Earth system science (ESS) data commonly manifests as dense or sparse n-dimensional data. Dense n-dimensional data is conventionally stored in arrays, while sparse n-dimensional data is typically housed in data frames. In the realm of ESS, an array of file formats is leveraged for the storage of dense n-dimensional data, including NetCDF4, TileDB, and Zarr. The paper at hand aims to evaluate data file formats for retrieving multidimensional data, specifically focusing on tools within the ESS domain. The insights from this exploration will be applicable to other data analytics projects.