Raw data queries during data-intensive parallel workflow execution - INRA - Institut national de la recherche agronomique Accéder directement au contenu
Article Dans Une Revue Future Generation Computer Systems Année : 2017

Raw data queries during data-intensive parallel workflow execution

Résumé

Computer simulations consume and produce huge amounts of raw data files presented in different formats, e.g., HDF5 in computational fluid dynamics simulations. Users often need to analyze domain-specific data based on related data elements from multiple files during the execution of computer simulations. In a raw data analysis, one should identify regions of interest in the data space and retrieve the content of specific related raw data files. Existing solutions, such as FastBit and RAW, are limited to a single raw data file analysis and can only be used after the execution of computer simulations. Scientific Workflow Management Systems (SWMS) can manage the dataflow of computer simulations and register related raw data files at a provenance database. This paper aims to combine the advantages of a dataflow-aware SWMS and the raw data file analysis techniques to allow for queries on raw data file elements that are related, but reside in separate files. We propose a component-based architecture, named as ARMFUL (Analysis of Raw data from Multiple Files) with raw data extraction and indexing techniques, which allows for a direct access to specific elements or regions of raw data space. ARMFUL innovates by using a SWMS provenance database to add a dataflow access path to raw data files. ARMFUL facilitates the invocation of ad-hoc programs and third party tools (e.g., FastBit tool) for raw data analyses. In our experiments, a real parallel computational fluid dynamics is executed, exploring different alternatives of raw data extraction, indexing and analysis.
Fichier principal
Vignette du fichier
paper-FGCS-2016-author version.pdf (1.76 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

lirmm-01445219 , version 1 (24-01-2017)

Identifiants

Citer

Vítor J Silva, José J Leite, José J Camata, Daniel de Oliveira, Alvaro L G A Coutinho, et al.. Raw data queries during data-intensive parallel workflow execution. Future Generation Computer Systems, 2017, 75, pp.402-422. ⟨10.1016/j.future.2017.01.016⟩. ⟨lirmm-01445219⟩
446 Consultations
544 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More