Corral Framework: Trustworthy and Fully Functional Data Intensive Parallel Astronomical Pipelines

CABRAL, JUAN; SÁNCHEZ, BRUNO; BEROIZ, MARTÍN; DOMINGUEZ, MARIANO; LARES, MARCELO; GUROVICH, SEBASTIÁN; GRANITO, PABLO

Astronomy and Computing

Elsevier

Lugar: Amsterdam; Año: 2017

Data processing pipelines are one of most common astronomical software. This kind of programs are chains of processes that transform raw data into valuable information. In this work a Python framework for astronomical pipeline generation is presented. It features a design pattern (Model-View-Controller) on top of a SQL Relational Database capable of handling custom data models, processing stages, and result communication alerts, as well as producing automatic quality and structural measurements. This pattern provides separation of concerns between the user logic and data models and the processing flow inside the pipeline, delivering for free multi processing and distributed computing capabilities. For the astronomical community this means an improvement on previous data processing pipelines, by avoiding the programmer deal with the processing flow, and Parallelization issues, and by making him focusing just in the algorithms involved in the successive data transformations. This software as well as working examples of pipelines are available to the community at https://github.com/toros-astro.