WP5

Integration of Raman Spectra into Interoperable Harmonised Databases

Objective

The Raman Spectra integration calls for an IoT data architecture able to capture data streams generated by heterogeneous devices and sensors. It should be able to apply the spectra harmonisation workflow developed by WP4. The raw data as well as the processed data shall be stored with sufficient provenance metadata and accessible both programmatically and by graphical interfaces, customised for different user scenarios (experts, public, etc).

Methodology

The data integration will adopt modern big data streaming architecture, with distinct intake, processing and (multiple) service layers, using open standards and domain specific ontologies. We will review the relevant IoT standards and API and existing resources for representation of Raman spectra (formats, databases, ontologies). Next step is the design and implementation of a high-performance intake layer, responsible for accumulating the data from instruments or sensors and ensuring the data is accompanied by proper provenance metadata. It is important that the non-processed data is retained, as it will allow reapplying new methods. The harmonisation workflow developed in CHARISMA will be integrated within the processing layer and apply to sensor data streams handled by intake layer. The harmonised data stored in agreed formats and available via API will be the basis of building user friendly applications for selected scenarios (search and accessing standardized spectra, integration of Raman spectra in databases, curation of existing spectra from e.g. literature, supporting industry use cases and modelling). We will mine the data from diverse low-, medium-, and high-resolution spectrometers available in our consortium to minimise discrepancies and enable data exchange. We will collaborate with WP8 on evolving Data Management Plan according to the FAIR and OPEN principles.

Work Package Leader

Nina Jeliazkova, IdeaConsult Ltd.