Deep integration of big data technologies with the ODS standard
The amount of data that is collected during tests and simulations has increased significantly in recent years and there is no end in sight. The current practice of copying the data to a workstation, where it is analyzed, reaches its limit. A single workstation is not capable of dealing with the load. In addition, transferring and copying the data itself becomes a cost and time factor.
The automotive industry is not the only one that struggles with enormous amounts of data and therefore new techniques were developed to offer a solution, which is generally summarised with the term “Big Data”. Instead of analyzing all data on a single computer, the workload is distributed among several computers and analyzed in parallel. Besides, data is kept close to the computer where it is going to be analyzed which reduces the need to copy and transfer data.
Unfortunately, ODS, the standard for managing test measurement data, has evolved in parallel and independently of Big Data techniques. Only recently a few cautious steps to integrate ODS and Big Data were taken. But in our opinion, the available (and proposed) solutions are insufficient to really bring all benefits of Big Data technologies to bear.
We are working on a platform that deeply integrates Big Data technologies with the ODS-standard. The following image shows an overview of the general concept:
A cloud-ready ODS server
The central piece of our platform is a server that fulfills the ODS-standard. It is extended by KPI-functionality which enables the user to perform calculations on mass-data and mix it with standard ODS-functionality. As an example a user may be interested to find all tests, which were done after 1/1/2020 (meta-data) and where the maximum temperature exceeds 95° C for at least 10 seconds (calculation on mass-data). More complex queries would be possible.
The ODS-server with extended KPI-functionality would use Hive and Spark to interact with the data. The actual meta-data is stored in a standard relational database while the mass-data is stored in a Hadoop File System. Hive, Spark, and Hadoop are standard Big Data technologies well equipped to handle huge amounts of data. The server itself uses exclusive technology that is Cloud Native compliant.
This solution offers a true and deep integration of the ODS-standard with Big Data technologies. All data is stored only once and does not need to be copied or transferred. By storing the data in a Hadoop File System, we can take full advantage of all innovations that Big Data technologies bring to the table.