FITS keyword mapping

Rationale

Ingest the metadata present in FITS keywords into existing data models.

Test case

A few datasets were chosen to serve as test case. The examples provided in this document apply to UVES data. The metadata will be ingested into ESO's Science Archive Facility, which will become the backend repository for ESO's registry.

The mappings

This section contains the several types of mapping rules necessary to ingest the metadata from the test datasets. Some rules are not necessary to ingest the test data, so examples are artificial.

The syntax itself is provided as illustration; it will be formally defined at a later time, and is subject to change. One of the goals is that it should be easily readable.

Keyword/value mapping

What: The value is the content of a know FITS keyword.
How: Specify FITS keyword
Example: TotalExposureTime = EXPTIME

Constant

What: A value is fixed across all the data to ingest
How: Specify constant value
Example: TypeOfObservation = "SPECTROSCOPY"

Conversion

What: A value is present, but in different unit/formatting than the one expected
How: Add unit/formatting info along with the value; a list of supported conversions must be defined (see below)
Example 1 (expected unit: degrees): SpatialInfo.ERR_SLIT = ERR_SLIT,arcsec
Example 2 (expected format: ISO8601): StartDateTime = {MJD-OBS,mjd}

Arithmetic expressions, string concatenation

What: A value must be calculated
How: Evaluate expressions with simple arithmetic operators (+. -. /, *) and string concatenation (&)
Example 1: Wavelength Min = CRVAL1 - CRPIX1 * CDELT1
Example 2: ProcessingType = "ESO UVES pipeline" & HIERARCH ESO PRO REC1 PIPE ID

Choice

What: A value exists in one of several keywords
How: A list of candidate keywords is defined using a regex; the first one to be found is used.
Example: OpticalElements.FILTER = FILTER[23] (either FILTER2 or FILTER3 keyword exist on a file)

Indirection

What: A keyword contains a keyword name to lookup
Example: AverageResolution = >RESOLAVG

Addressing other FITS extensions

What: A keyword resides on an extension other that the primary
How: Specify the extension along with the keyword. Extension 0 will be the primary FITS unit; 1 will be the first extension.
Example: Instrument = 1:INSTRUME

Addressing other FITS files

What: A keyword resides on an external FITS file
How: Specify the file name along with the keyword
Example: AverageResolution = {QC-FILE}:0:RESOLAVG

Precomputed table

What: Some values are extraordinarily hard to get, or even missing. An example from UVES data: lookup the keyword name which has a value of 'LINE_TABLE_*2' (where * = BLUE, REDL or REDU), replace the last 4 characters 'CATG' with 'NAME', get the value of that keyword, use the value as a filename to retrieve from the ESO archive, and lookup in that file the value of the keyword HIERARCH ESO QC RESOLAVG
How: Such complicated mapping will not be automated, as they are not likely to occur more than once. It will involve user intervention, in the form of an external file, in tsv format, provided with the values for each file (this effectively corresponds to hard-coding the values in this file). This scheme also allows for inclusion of metadata that is not present in any form on the FITS files.
Example (first line contain the concepts to map to):

filename                                            AverageResolution          other....
r.UVES.2002-02-03T04:20:57.989_0011.fits            50465.089
r.UVES.2002-02-03T04:20:57.989_0012.fits            79456.827
r.UVES.2002-02-03T04:20:57.989_0013.fits            67034.8444
.....

Sample

• Sample FITS file (headers only)
• Sample mappings
• The database model:
  

More on requirements

'Standard' unit conversion

Unit conversion is a broad issue that should be addressed in a standard and general way.

We will use a standard unit conversion library (CDS's devcorner?) in order to cover all necessary unit conversions.

References

The SAADA tool (as of version 1.1.1) was previously evaluated to determine whether it was an applicable solution to the problem. A full report was presented at DS5PlanningStage02

The requirements presented are based on the work of Jörg Retzlaff and Remco Slijkhuis, at ESO's Advanced Data Products group.

Presented at: DS5PlanningStage03

-- Main.BrunoRino - 07 Mar 2006