DataFinder: A Python Application for Scientific Data Management

Kurzfassung

The DataFinder is a data management client developed in Python that primarily targets the management of scientific technical data. Data can be attached with individual meta data that is based on a free-definable data model to achieve data structuring and ordering. Moreover, the system is able to handle large amounts of data and can be easily integrated in existing working environments. The system is based upon a client-server-architecture and uses open, stable standards. One of its main features is extensibility using Python scripts. The server side consists of the meta data server that stores the meta data as well as the system configuration. The server is accessed with the standardised protocol WebDAV by the client side. Data can be stored on the same server as the meta data or separated from it. The concept of separated meta data and data storage allows the flexible usage of heterogeneous storage resources (for example, FTP, GridFTP, operating system data interfaces, or WebDAV). On the client side, graphical user interfaces for the administration of the whole system (e.g., for configuration of storage resources or meta data modelling) and for the usage of the provided data management functionalities (e.g., storing and retrieving data, searching for data, or executing scripts on data) exist. The user interfaces are implemented in Python and the Qt GUI toolkit (using PyQt). Furthermore, the basic functionality is accessible through a Python-API so that these functionalities can be easily extended or used within other software systems. For example, typical usage scenarios include the automatic migration of existing data archives into the DataFinder-based data storage by provision of specific Python scripts. The talk is to gives an overview about the basic concepts of the DataFinder and demonstration of typical usage scenarios. Additionally, we want to present our experiences developing such a system using the Python programming language as well as on overview over the used Python modules, for example the Open Source Python WebDAV library developed at DLR. It also presents advantages for using Python in science and engineering, both in research and industry.