Corral Framework: Trustworthy and Fully Functional Data Intensive Parallel Astronomical Pipelines

JUAN B CABRAL; BRUNO SANCHEZ; MARTÍN BEROIZ; MARIANO DOMINGUEZ; MARCELO LARES; PABLO GRANITTO

Astronomy and Computing

Elsevier

Año: 2017

2213-1337

Data processing pipelines represent an important slice of the astronomical software library that include chains of processes thattransform raw data into valuable information via data reduction and analysis. In this work we present Corral, a Python frameworkfor astronomical pipeline generation. Corral features a Model-View-Controller design pattern on top of an SQL Relational Databasecapable of handling: custom data models; processing stages; and communication alerts, and also provides automatic quality andstructural metrics based on unit testing. The Model-View-Controller provides concept separation between the user logic and thedata models, delivering at the same time multi-processing and distributed computing capabilities. Corral represents an improve-ment over commonly found data processing pipelines in Astronomy since the design pattern eases the programmer from dealingwith processing flow and parallelization issues, allowing them to focus on the specific algorithms needed for the successive datatransformations and at the same time provides a broad measure of quality over the created pipeline. Corral and working examplesof pipelines that use it are available to the community at https://github.com/toros-astro.