2018
Publication Year
2020-10-14T14:37:39Z
Acceptance in OA@INAF
GIANO-B online data reduction software at the TNG
Title
HARUTYUNYAN, AVET; RAINER, Monica; Hernandez, Nauzet; OLIVA, Ernesto;
GUERRA, JOSE; et al.
Authors
10.1117/12.2312690
DOI
http://hdl.handle.net/20.500.12386/27816
Handle
PROCEEDINGS OF SPIE
Series
10706
Number
PROCEEDINGS OF SPIE
SPIEDigitalLibrary.org/conference-proceedings-of-spie
GIANO-B online data reduction
software at the TNG
Harutyunyan, Avet, Rainer, Monica, Hernandez, Nauzet,
Oliva, Ernesto, Guerra, Jose, et al.
Avet Harutyunyan, Monica Rainer, Nauzet Hernandez, Ernesto Oliva, Jose Guerra,
Marcello Lodi, Jose San Juan, Andrea Bignamini, Adriano Ghedina, Francesca
Ghinassi, Emilio Molinari, Serena Benatti, Ilaria Carleo, Riccardo Claudi, Giuseppina
Micela, Andrea Tozzi, Carlo Baffa, Andrea Baruffolo, Valdemaro Biliotti, Nicolas
Buchschacher, Massimo Cecconi, Rosario Cosentino, Gilberto Falcini, Daniela
Fantinel, Luca Fini, Alberto Galli, Elisabetta Giani, Carlos Gonzalez, Esther
Gonzalez-Alvarez, Manuel Gonzalez, Raffaele Gratton, Marcos Hernandez Diaz,
Marcella Iuzzolino, Luca Malavolta, Jesus Maldonado, Livia Origlia, Ennio Poretti,
Hector Perez Ventura, Alfio Puglisi, Carlos Riverol, Luis Riverol, Nicoletta Sanna,
Salvatore Scuderi, Ulf Seeman, Alessandro Sozzetti, Mauro Sozzi, "GIANO-B online
data reduction software at the TNG," Proc. SPIE 10706, Advances in Optical and
Mechanical Technologies for Telescopes and Instrumentation III, 1070642 (10 July
2018); doi: 10.1117/12.2312690
Event: SPIE Astronomical Telescopes + Instrumentation, 2018, Austin, Texas, United
States
GIANO-B Online Data Reduction Software at the TNG
Avet Harutyunyan
a, Monica Rainer
b,c, Nauzet Hernandez
a, Ernesto Oliva
b, Jose Guerra
a,
Marcello Lodi
a, Jose San Juan
a, Andrea Bignamini
d, Adriano Ghedina
a, Francesca Ghinassi
a,
Emilio Molinari
e,a, Serena Benatti
f, Ilaria Carleo
g,f, Riccardo Claudi
f, Giuseppina Micela
h,
Andrea Tozzi
b, Carlo Baffa
b, Andrea Baruffolo
f, Valdemaro Biliotti
b, Nicolas Buchschacher
i,
Massimo Cecconi
a, Rosario Cosentino
a, Gilberto Falcini
b, Daniela Fantinel
f, Luca Fini
b,
Alberto Galli
a, Elisabetta Giani
b, Carlos Gonzalez
a, Esther Gonzalez-Alvarez
h, Manuel
Gonzalez
a, Raffaele Gratton
f, Marcos Hernandez Diaz
a, Marcella Iuzzolino
j, Luca Malavolta
g,
Jesus Maldonado
h, Livia Origlia
k, Ennio Poretti
a, Hector Perez Ventura
a, Alfio Puglisi
b,
Carlos Riverol
a, Luis Riverol
a, Nicoletta Sanna
b, Salvatore Scuderi
l, Ulf Seeman
m, Alessandro
Sozzetti
n, and Mauro Sozzi
ba
Fundac´ıon Galileo Galilei INAF, Rambla Jos´
e Ana Fern´
andez P´
erez, 7, 38712 Bre˜
na Baja TF,
Spain
b
INAF Osservatorio Astrofisico di Arcetri, Largo Enrico Fermi 5, I-50125 Firenze, Italy
cINAF Osservatorio Astronomico di Brera, via Emilio Bianchi 46, I-23807 Merate LC, Italy
d
INAF Osservatorio Astronomico di Trieste, Via G.B. Tiepolo 11, I-34143 Trieste, Italy
eINAF Osservatorio Astronomico di Cagliari, Via della Scienza 5, I-09047 Selargius CA, Italy
f
INAF Osservatorio Astronomico di Padova, Vicolo Osservatorio 5, I-35122 Padova, Italy
gUniversit`
a di Padova, Via 8 Febbraio 2, I-35122 Padova, Italy
h
INAF Osservatorio Astronomico di Palermo, Piazza del Parlamento 1, I-90134 Palermo, Italy
iUniversit´
e de Gen`
eve, 24 rue du G´
en´
eral-Dufour, 1211 Gen`
eve 4, Switzerland
j
Officina Stellare, Via Della Tecnica, 87/89, I-36030 Sarcedo VI, Italy
k
INAF Osservatorio Astronomico di Bologna, via Gobetti 93/3, I-40129 Bologna, Italy
lINAF Osservatorio Astrofisico di Catania, via S.Sofia 78, I-95123 Catania, Italy
mInstitut f¨
ur Astrophysik - Georg-August Universit¨
at G¨
ottingen, Friedrich-Hund-Platz 1,
D-37077 G¨
ottingen, Germany
n
INAF Osservatorio Astrofisico di Torino, via Osservatorio 20, I-10025 Pino Torinese TO, Italy
ABSTRACT
GIANO-B is the high resolution near-infrared (NIR) spectrograph of the Telescopio Nazionale Galileo (TNG), which started its regular operations in October 2017. Here we present GIANO-B Online Data Reduction Software (DRS) operating at the Telescope.
GIANO-B Online DRS is a complete end-to-end solution for the spectrograph real-time data handling. The Online DRS provides management, processing and archival of GIANO-B scientific and calibration data. Once the instrument control software acquires the exposure ramp segments from the detector, the DRS ensures the complete data flow until the final data products are ingested into the science archive. A part of the Online DRS is GOFIO software, which performs the reduction process from ramp-processed 2D spectra to extracted and calibrated 1D spectra.
A User Interface (UI) developed as a part of the Online DRS provides basic information on the final reduced data, thus allowing the observer to take decisions in real-time during the night and adjust the observational strategy as needed.
Keywords: NIR high resolution spectroscopy, data reduction software, data processing, python
Send correspondence to Avet Harutyunyan, E-mail: [email protected]
Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation III, edited by Ramón Navarro, Roland Geyl, Proc. of SPIE Vol. 10706, 1070642 · © 2018 SPIE
CCC code: 0277-786X/18/$18 · doi: 10.1117/12.2312690 Proc. of SPIE Vol. 10706 1070642-1
1. INTRODUCTION
GIANO-B is a near-infrared (NIR) high resolution spectrograph1,2mounted at the Telescopio Nazionale Galileo
(TNG), which provides cross-dispersed echelle spectroscopy at a resolution of ∼50000 in the 0.9-2.45µm spectral range in a single exposure.
With the aim of providing a complete solution to the real-time handling of GIANO-B observational and calibra-tion data at the Telescope, we developed the GIANO-B Online Data Reduccalibra-tion Software (DRS). The Online DRS kicks in as soon as exposure ramp segments are acquired from the detector by the Instrument Control System (ICS), and manages the data processing and the storage of data products. The data are first stored locally at
the telescope and then sent to the TNG data archive (Sec.7), from where they are available for download shortly
after they were acquired and processed.
2. ONLINE DRS OVERVIEW
The Online DRS dataflow is shown in Fig.1. The Online DRS code is mainly written in Python∗, with exception
of a fortran procedure (see Sec.3). For the user interface some additional web programming technologies were
used (Sec.5).
Ramp Segments GIANO-B Instrument control software
Segments -> Ramp-reduced 2D spectra GIANO-B RAMP PROCESSOR
2D spectra -> 1D spectra GOFIO
User feedback GIANO-B WEBUI BACKEND
DB DB
GAINO-B ONLINE DRS WEB UI
GIANO-B ARCHIVER
TNG Archive
TNG FITS FILE SERVICE
Trieste Archive
Message Broker
ActiveMQ
NFS Disk
Figure 1. The Online DRS dataflow.
One of the basic ideas behind the Online DRS design is to decouple as much as possible the data handling system from that of the observation management. With this approach, the potential issues of data handling
∗
and processing would not interfere with the more critical instrument control or detector acquisition operations. Thus, the Online DRS and the instrument control and acquisition systems are set to operate on separate nodes. In such a context, a message-oriented middleware is particularly well-suited for providing the interaction of the two components. GIANO-B Online DRS exploits this infrastructure through a message broker, to receive information about newly acquired data from the instrument acquisition system.
The communication between the GIANO-B data acquisition system and the Online DRS is done through STOMP†
protocol. These two applications use an Apache ActiveMQ‡message broker already implemented for the telescope
control system at the TNG.
For reading and writing the files, GIANO-B Online DRS accesses a dedicated Network File System (NFS)
storage space, created to hold all GIANO-B data. The Online DRS also makes use of SQLite§ databases for its
operations.
In the following sections of the text the different parts of the flowchart in Fig.1will be discussed more in detail.
3. RAMP PROCESSING AND 2D SPECTRA CREATION
During the exposure the GIANO-B data acquisition system carries out multiple non-destructive readouts of the
detector data. The multiple read-outs are performed following Fowler sampling,3 as this reduces the read-out
noise. The readouts are performed every 10 seconds and for each of these exposure segments a FITS file is written to the disk. More information on the GIANO-B detector and its acquisition system can be found in
Ref.4.
In order to create a single file containing the entire exposure data, the ramp segments data are analysed and combined into a FITS file. This is done by the Online DRS in real-time once a complete set of ramp exposure segments is obtained. The core ramp analysis and processing is done using a fortran code especially developed
for this aim, which follows the procedure described in Ref. 4.
The 2D spectra images created by the ramp processor are written into FITS files and stored to the NFS disk. Additionally, these files are ingested into the TNG data archive through a FITS file archive service running at
the Telescope (Sec.7). The information both on the processed images and ramp exposure segments are added to
the database (Fig.1). These database entries are further used for 1D spectra extraction, as well as for providing
the corresponding feedback through the Online DRS user interface (Sec.5).
Ramp processor does not require any start up and permanently watches for new ramp segments. The process is always kept alive via cron job.
4. 1D SPECTRA EXTRACTION
The reduction of the ramp processed raw files and subsequent extraction of 1D spectra is done by GOFIO,5
which uses the 2D spectra generated by the ramp processor. The real-time information on the new 2D spectra available for extraction is obtained from the database entries added by the ramp processor every time an exposure
sequence data is written into a FITS file (Sec.3).
As with ramp processing, at this step too the resulting 1D spectra are stored to the NFS disk, while the
information on these spectra together with processing success/failure status is added to the database (see Fig.1).
The 1D spectra extraction has to be started-up and shut-down, which can be done in particular through the
user interface (Sec.5).
5. USER INTERFACE
We opted for a Web-based User Interface (Web UI) for GIANO-B Online DRS to benefit from the access flexi-bility it provides. Also, Web UI enables the separation of the user interface of the data processing from the data themselves, avoiding direct data access potential drawbacks. The interface is available throughout the local net-work at the telescope and benefits from the gigabit-fast connection between the web server and client browsers.
† https://stomp.github.io/ ‡ http://activemq.apache.org/ § https://www.sqlite.org
Proc. of SPIE Vol. 10706 1070642-3
®GIANO -B GIANO -B Online Data Reduction Software (DRS)
GIANO -B 2D Spectra Ramp Processing
20 Spectra Ramp Processing t D Spectra Extrac[bn
Ramp -processed 20 spectra Download log S Ramp segments
Image nerve Image type Object Slllpoa Elptlma Groups AB Pairs NN Status GIANO -8. 2018 -02- 23T01 -25 -05.000.119 SCIENCE AT04 A 200.0 1/1 4/27 S 01 -32 -42.000 SCIENCE 2323 Processing GIANO -8. 2018 -02- 23101 -20- 42.000119 SCIENCE ÁT04 B 200.0 1/1 3/27 A 01 -32 -32.000 SCIENCE 22/23 Processing GIANO -8. 2018- 02- 23T01 -16- 12.00011s SCIENCE ÁT04 A 200.0 1/1 3/27 S 01 -32 -22.000 SCIENCE 21/23 Processing GIANO -8. 2018- 02- 23T01 -11- 49.000.119 SCIENCE AT04 B 200.0 1/1 2/27 A 01-32-12.000 SCIENCE 20/23 Processing
GIANO -8. 2018 -D2- 23101 -07 - 21.000119 SCIENCE AT04 A 200.0 1/1 2/27 S 01 -32 -03.000 SCIENCE 19/23 Processing
GIANO -8. 2018 -02 -23101 -03- 04.000119 SCIENCE AT04 B 200.0 1/1 1/27 S
0131 -63.000 SCIENCE 1323 Processing GIANO -8. 2018- 02- 231C0 -58- 42.000119 SCIENCE AT04 A 203.0 1/1 1/27 S
013143.000 SCIENCE 17/23 Processing GIANO- 8.2018- 02- 22T18 -08- 30.000.11s FLAT FLAT A 100.0 1, 1/1 S
013134.000 0131 -24.000 SCIENCE 15,23 SCIENCE 15/23 Processing Processing GIANO- 8.2018 -02- 22T18 -07- 23.000119 FLAT FLAT A 103.0 6/7 1/1 S 0131 -14.000 SCIENCE 14/23Processing GIANO- 8.2018 -02 -22T18-05- 18.000119 FLAT FLAT A 103.0 Y/ 1/1 S 0131 -04.000 SCIENCE 13/23 Processing GIANO- 8.2018 -02- 22T18-03- 09.000119 FLAT FLAT A 103.0 4/7 1/1 S 01 -30 -55000 SCIENCE 12/23 Processing GIANO- 8.2018 -02 -22T18 -01-02.000119 FLAT FLAT A 103.0 317 1/1 S 01-3045000 SCIENCE 11/23 Processing GIANO- 8.2018- D2- 22T17 -58- 55.000119 FLAT FLAT A 103.0 2/7 1/1 S 01 -30 -35000 SCIENCE 10/23 Processing GIANO -8.2018 -02- 22T17á8- 48.000.119 FLAT FLAT A 100.0 1/7 1/1 S 0130-26.000 SCIENCE 9/22 Processing
GIANO -8.2018- 02- 22T17á2á0.000115 DARK DARK A 100.0 7/7 1/1 S 0130.16000 SCIENCE 9/22 Processing
GIANO -8.2018 -02- 22T11á2á0.000- 010.15 DARK DARK A 10.0 1/1 1/1 S
0130.00.000 SCIENCE 7/23 Processing
GIANO -8.2018 -02- 22T17á2á0.000 03D.10S DARK DARK A 30.0 in 1/1 S
01- 29-56.000 0149-47000 SCIENCE 522 SCIENCE 5/23 Processing Processing
GIANO -8.2018 -D2- 22T17á2á0.000 C80.RS DARK DARK A 80.0 in 1/1 S 014931000 SCIENCE 4/23 Processing GIANO- 8.2018- D2- 22T17á0-43.0W11s DARK DARK A 100.0 5//7 1/1 S 01 -29 -21000 SCIENCE 3/23 Processing
GIANO- 8.2018 -D2- 22T17á0-03.000- 010.k5 DARK DARK A 10.0 5//7 1/1 S 0149 -11000 SCIENCE 2/23 Processing
GIANO- 8.2018 -D2- 22T17á043.000 030Ás DARK DARK A 30.0 6/7 1/1 S 014619000 SCIENCE 23/23 Pmcess4
GIANO- 8.2018 -D2- 22T17á043.000 080.äs DARK DARK A 80.0 67 1/1 S 01 -28-09.000 SCIENCE 22/23 Pmcess4 01 -28-00000 SCIENCE 21/23 Pmcess4
The Web UI is delivered by nginx¶ HTTP server, while both the UI back- and front-end are developed using
DjangokPython Web framework. nginx requests are forwarded to the Framework via WSGI∗∗, which is provided
by uWSGI††. Additionally, the usual HTML/CSS/JavaScript combination is used for the front end construction.
Figure 2. The 2D Spectra Ramp Processing tab Web UI.
The Web UI consists of two main sections accessed through tabs. “2D Spectra Ramp Processing” (Fig.2) shows
the progress of the 2D spectra creation through real-time ramp processing as described in Sec.3. No action has
to be carried out by the user in order turn on the ramp processing, this process is always active watching for new ramp segments.
If necessary (i.e. for inspection), the ramp-processed spectra files can be individually downloaded. Also, the contents of the ramp-processed 2D spectra table can be downloaded as a text file.
“1D Spectra Extraction” (Fig.3) displays the state of the calibration and scientific spectra processing. Unlike the
ramp processing, the spectra extraction has to be explicitly turned on. A node.js‡‡ application was developed
to enable the spectra extraction process start-up and shut-down from the Web UI.
Beside showing basic information on the extracted spectra (e.g. S/N in different spectral regions), the Web UI
allows individual download of the extracted 1D spectra. Also, the spectra can be plotted on an overlay (Fig.4).
The plots are done on client side and are developed using plotly.js§§.
¶ https://nginx.org/ k https://www.djangoproject.com/ ∗∗ https://wsgi.readthedocs.io/ †† https://uwsgi-docs.readthedocs.io/ ‡‡ https://nodejs.org/ §§ https://plot.ly/javascript/
)GIANO -B GIANO -B Online Data Reduction Software (DRS)
GIANO -B 'ID Spectra Extraction
Start / Stop processing
®.. Status: Running
FxtntReO spectra
Ramp -process. RD Spectre Extracted- cellbratee ID spectre
Calibrations
calibration type
20 Spectra Ramp Processing 10 Spectra Extraction
Plumber Smem FIATO OARKs 7 28 7 OK 28 OK Obten SM YJXK GIANO -8.2018 -02 -23101- 33-50.0059s GIANO -8.2018 -02 - 23101- 38- 13.0OS.9s Processing :' GIANO-0.2 01 8-02-23101-2 5-05.00J.fte GIANO-0.2018-02-23101-29-27.009.fte GIANO-8.2018-02-23T01-25-05000_A0_m9l9fha GIANO-8.2018-02-23T01-25-05000_40 s1999 GIANO-8.2018-02-23T01-25-05000_A_mO1e.111s GIANO-8.2018-02-23T01-25-05000_A_O1e.flte GIANO-8.2018-02-23T01-29-27 000_B_msl9fltas GIAN0-8.2018-02-23T01-29-27.00010 s16fRO AT04 AB 29.56 39.18 35.46 26.41 ATOO AB AT04 A 21.35 28.47 24.71 17.95 ATOO A AT04 B 19.13 28.89 25.35 18.98 ATOO 8 GIANO-0.2018-02-23T01-18-12.000.95 GIANO-0.2018-02-23T01-2032.000.95 CIANO-8.2018-02-23101-16-12.000 AB ms1ófha GIANO-8.2018-02-23T01-16-12000_413 s1999 GIANO-8.2018-02-23T01-16-12000_A_mO1e.11ts GIANO-8.2018-02-23T01-16-12000_A_O1e.flte GIANO-8.2018-02-23101-20-42.000_0 msttl.fltO AT04 AB 29.7 39.49 35.5 28.67 ATOO AB AT04 A 23.9 30.92 25.26 18.79 ATOO A AT04 13 16.89 24.45 24.87 18.59 L J. L L s L L Extracted CIANO -0.201 CIANO -0.201 CIANO -B.201 GIANO-B.201 400 00 60 55 m 5 a 40
Spectrum Flux and S/N plot - trace 16 / order 47
615 1620 1615 1630 1635 16a0 1615
615 1620 1625 1630 1635 1640 1645
Wavelength (nm)
trace lo, order 47. rage 16126 -16466 1
CIANO- 0.2018 -02
CIANO- 0.2018 -02- 23101 -20-
42.000.-SIANG
CIANO -8 .2018 -02- 23T01 -16 -12900 A msl0.fts AT04 A 23 .9 30 92 25.
Figure 3. 1D Spectra Extraction Tab of the Web UI showing spectra processing running during a typical observing night.
Figure 4. The plot overlay window of the Web UI.
6. DATA ARCHIVING
After the GIANO-B data are acquired and processed, they are first archived locally at the TNG. Then, the data
are sent to the TNG public data archive at the Italian center for Astronomical Archives¶¶, from where they are
available to the community for downloading. The sync times between the two nodes are typically around 20
¶¶
http://archives.ia2.inaf.it/tng/
minutes.
Proc. of SPIE Vol. 10706 1070642-5
At the TNG side, a common HTTP service of the FITS data archive ingestion (FITS File Service - FFS) is available, which has been used for archiving of the data of all the TNG instruments. The GIANO-B Online DRS archives 2D and 1D spectra FITS data using the FFS. The access to the FFS is accomplished through a specifically developed Python client to the service. The FITS files with 2D data are archived directly by the
ramp processor as soon as they are created (Sec.3). The 1D spectra are archived shortly after their creation by
a dedicated process watching for newly extracted spectra.
7. FUTURE WORK
GIANO-B Online DRS has been operating at the TNG from March 2018, without any stability issues or failures so far. That said, fast development timescales were a priority and for some of the parts of the system the imple-mented solutions were not the most optimal. In fact, the system can probably benefit from certain more efficient solutions, and the future work we envisage will possibly go towards the implementation of those improvements. Even if the Online DRS works within the local network and virtually has no limitation in moving large amounts of data on frequent requests, we will change from polling to event-driven approach. Beside of the evident efficiency gains, this will also simplify the overall system architecture. In particular, technologies like Server-Sent Events
(SSE)∗∗∗ will be considered for this.
Other updates may include the fortran code (Sec.3) translation to Python, while still ensuring fast array
ma-nipulations (e.g. via numpy†††). Also, overall code translation from Python 2.x to Python 3.x is considered.
REFERENCES
[1] Oliva, E., Origlia, L., Maiolino, R., Baffa, C., Biliotti, V., Bruno, P., Falcini, G., Gavriousev, V., Ghinassi, F., Giani, E., Gonzalez, M., Leone, F., Lodi, M., Massi, F., Mochi, I., Montegriffo, P., Pedani, M., Rossetti, E., Scuderi, S., Sozzi, M., and Tozzi, A., “The GIANO spectrometer: towards its first light at the TNG,” in [Ground-based and Airborne Instrumentation for Astronomy IV ], Proc. SPIE 8446, 84463T (2012).
[2] Origlia, L., Oliva, E., Baffa, C., Falcini, G., Giani, E., Massi, F., Montegriffo, P., Sanna, N., Scuderi, S., Sozzi, M., Tozzi, A., Carleo, I., Gratton, R., Ghinassi, F., and Lodi, M., “High resolution near IR spectroscopy with GIANO-TNG,” in [Ground-based and Airborne Instrumentation for Astronomy V ], Proc. SPIE 9147, 91471E (2014).
[3] Fowler, A. M. and Gatley, I., “Demonstration of an algorithm for read-noise reduction in infrared arrays,” ApJ 353, L33 (1990).
[4] Oliva, E., Biliotti, V., Baffa, C., Giani, E., Gonzalez, M., Sozzi, M., Tozzi, A., and Origlia, L., “Performances and results of the detector acquisition system of the GIANO spectrometer,” in [High Energy, Optical, and Infrared Detectors for Astronomy V ], Proc. SPIE 8453, 84532T (2012).
[5] Rainer, M., Harutyunyan, A., Carleo, I., Oliva, E., Benatti, S., Bignamini, A., Claudi, R., Gonzalez-Alvarez, E., Sanna, N., Ghedina, A., Micela, G., Molinari, E., Tozzi, A., Baffa, C., Baruffolo, A., Biliotti, V., Buch-schacher, N., Cecconi, M., Cosentino, R., Falcini, G., Fantinel, D., Fini, L., Galli, A., Ghinassi, F., Giani, E., Gonzalez, C., Gonzalez, M., Gratton, R., Guerra, J., Hernandez Diaz, M., Hernandez, N., Iuzzolino, M., Lodi, M., Malavolta, L., Maldonado, J., Origlia, L., Perez Ventura, H., Puglisi, A., Riverol, C., Riverol, L., San Juan, J., Scuderi, S., Seemann, U., Sozzetti, A., and Sozzi, M., “Introducing GOFIO: a DRS for the GIANO-B near-infrared spectrograph,” in [Ground-based and Airborne Instrumentation for Astronomy VII ], Proc. SPIE TBD, TBD (2018).
∗∗∗
http://www.w3.org/TR/eventsource/
†††
