-
Notifications
You must be signed in to change notification settings - Fork 13
Adding New Image Sources
In this section, I discuss how images go from an observatory to being available on Helioviewer. From this information, I hope that the following steps for adding new data will make sense, and will give you some context about the updates that need to be made.
During an observatory's regular operation, the measurements made are typically processed into a Flexible Image Transport System (FITS) file. There are various kinds of FITS files (typically called level X) which have gone through various levels of processing. There are FITS files that consist of raw measurement data, these usually aren't suitable for visualization purposes. Then there are FITS files that have gone through some processing to make them presentable. These are the types of FITS files we prefer for Helioviewer.
Helioviewer makes use of the JPEG2000 (*.jp2) file format.
This type of image format boasts excellent compression ratios,
allows embedding arbitrary metadata in the files,
allows reading partial regions of an image,
and allows multiple images to be merged into a single file.
In order for data to be used in Helioviewer, the FITS images have to be converted into a grayscale JPEG2000 files, with the FITS header containing World Coordinate System (WCS) information embedded as XML within the jpeg2000 file.
Creation of FITS and JPEG2000 files are considered upstream processes. The Helioviewer Project doesn't manage creation of jpeg2000 files, we retrieve them from data providers.
When Helioviewer gets a new source of jpeg2000 files, a patch has to be applied to Helioviewer's source code to be able to serve the images. The patch has to do the following:
- Define where to get the jp2 image files.
- Add new database identifiers for the new data source.
- Define a selection hierarchy for the Helioviewer UI (i.e. user will select SOHO > LASCO > C2, or SDO > AIA > 304.
- Define how to read the jp2 files if non-standard fields are used.
- Define image processing steps such as adding color or making portions of the image transparent.
- Define watermark information, the text that appears for the data source Helioviewer movies & screenshots.
- Add attribution to the helioviewer.org About Dialog.
After these updates are made, Helioviewer will be able to read a list of jpeg2000 files, add them to its internal database of images, and enable loading the images on Helioviewer.org and JHelioviewer.
In general, you will need to know the following things about your new datasource:
- Where to get jp2 images. This can be a remote web directory with jp2 files, or a local folder with jp2 files.
- All the different instruments and measurements that are provided by your new datasource, and an idea of the selection hierarchy
- Machine readable information available from your jp2 files. This comes from FITS metadata embedded as XML in the jp2 file.
This is the name that the data ingestion scripts will see when importing the jp2 image. To get the exact name, download a jp2 for each new measurement, and execute <api>/management/data/get_jp2_info.py <jp2 file>
NOTE: This script requires a connection to the Helioviewer database, so it should be executed within a Helioviewer server environment.
If you're using the docker compose environment, then execute this within the helioviewer-cli container. Inside the
container, source the python virtual environment via source ~/venv/bin/activate. This script lives in ~/api/management/data.
This will print the information found in the FITS header which is needed for data ingestion. For example, for EUI HRI 1216, it prints the following
observatory: Solar_Orbiter
instrument: EUI
detector: HRI_LYA
measurement: 1216
These are the values that must be set in the datasource_property table for data ingestion to work.
The default observatory, instrument, detector, measurement covers the hierarchy for many data sources,
but it doesn't cover all of them. For example, the RHESSI instrument uses observatory, energy band, reconstruction method instead.
To enable custom fields, you must enable the pipeline to extract your desired labels from the jp2 metadata by updating jp2parser.py:JP2parser:getData. For reference, RHESSI is handled by adding extra processing. pseudocode below:
def getData():
...
if image["observatory"] == "RHESSI":
execute rhessi specific processing
...
def _process_rhessi_extras(...):
add energy band to the data dictionary.
add reconstruction method to the data dictionary.
Then in jp2.py:insert_images, update the processing to read your labels.
This function reads the fields returned by jp2parser's getData().
if img["observatory"] == "RHESSI":
leafs = ["observatory", "energy_band", "reconstruction_method"]
The first step is to create new ids for each measurement. Update <api>/install/helioviewer/db.py:create_datasource_table with the new data source.
If you already have a database instance running, then also insert that new data source into the existing datasources table. Create a sql file with the insert statement in <api>/install/database to help mirrors easily update their database.
TODO: Still need to understand layeringOrder, sourceIdGroup, and displayOrder. For now just set:
- layeringOrder to 1
- sourceIdGroup to ''
- displayOrder to 0
- Set enabled to 0
Each new data source created in step 2 need fields added to the datasource_property table. This is the text that will show up in the UI. Typically this will use uiOrder 1, 2, 3, 4 in the form Observatory, Instrument, Detector, Measurement. All of this information can be gathered with information in section 1.a. If you're using custom fields, the name field can be used to set the label, and the
fitsName field should match the value read from a jp2 file for your custom label.
Add these property insertions to the sql file created in section 2.
name Is the display name that will be shown in the web application
fitsName Is the value in the image metadata used for data ingestion
description should be the expanded name of the acronym.
uiOrder is the order the dropdowns should appear.
The server description points Helioviewer to the URL used for downloading images for the new datasource. It lives in <api>/helioviewer/hvpull/servers. Refer to other server definitions in that folder for implementation details.
After defining the server definition, you must update daemon.py with the new server in the get_servers() function. This simply returns a dictionary in the form {filename: server_class}, add your new filename, classname pair to the dictionary.
The filename mentioned above is the name that will be used for the -d server parameter in downloader.py.
The API backend needs to know how to handle the new image type. Therefore a new class must be defined in <api>/src/Image/ImageType. The new image should have a class name of Image_ImageType_xxxImage. Use EUIVImage.php as your starting point as its the simplest form of image source.
In the constructor you must set a color table (see 5.b) and you must define the getWaterMarkName() function.
In this context, $uiLabels is an array of Observatory, Instrument, Detector, Measurement in the form:
{
{
'label': 'Observatory',
'name': Your Observatory name
},
{
'label': 'Instrument',
'name': Your Instrument Name
},
{
'label': 'Detector',
'name': Your Detector Name
},
{
'label': 'Measurement',
'name': Your measurement
}
}
These were defined in section 3.
You will also need to create color tables for the image. For this you will need to work with Jack and/or Bogden to gather information about the desired colors. There may be some manual effort here. For EUI I was given a text file describing the RGB colors that I needed to convert into the png usable for helioviewer. If you have a text file with the RGB values available, you may use <api>/scripts/gen_color_tables/gen_color_table.php to create the png.
When you have the appropriate color pngs in <api>/docroot/resources/images/color-tables, then program your ***Image.php class to return the appropriate color table.
That's all that needs to be done. Test downloader.py and make sure that the downloader is able to parse files from the new server. Then check Helioviewer to verify how the images are displayed.
This section details problems that have been observed when trying to add new image sources.
Seen when attempting to add Solar Orbiter data to Helioviewer.
ValueError: time data 'solo_L3_eui-fsi174-i' does not match format '%Y_%m_%d__%H_%M_%S'
At the time of writing daemon.py assumed the first 20 characters of the filename contain the date. That is not the case for Solar Orbiter images. To work around this, the server definitions have been updated with a function get_datetime_from_file which can be used to override this behavior. If this function is not defined, then the old behavior will be used. Otherwise, you must implement your own time parser in the server definition by implementing a function named get_datetime_from_file for this. See solar_orbiter.py for details.
If you get a KeyError in jp2.py:insert_images it is most likely because some field is missing from the database.
The error looks like this:
source = source[str(img[leaf])]
KeyError: '304'
If you see this, check the file that it's failing on, and refer to section 1.a. for details on what to add to the database.
For the best compatibility and user experience, the FITS header should be transformed into XML and embedded as an XMLBox in the JPEG2000 header. It looks like this (as an example):
Sample Header
XML Box (xml ) @ (6500, 6423)
<meta>
<fits>
<SIMPLE>1</SIMPLE>
<BITPIX>16</BITPIX>
<NAXIS>2</NAXIS>
<NAXIS1>4096</NAXIS1>
<NAXIS2>4096</NAXIS2>
<EXTEND>1</EXTEND>
<ORIGIN>SDO</ORIGIN>
<DATE>2021-06-01T00:09:33.000</DATE>
<TELESCOP>SDO</TELESCOP>
<INSTRUME>AIA_4</INSTRUME>
<DATE-OBS>2021-06-01T00:01:29.132</DATE-OBS>
<T_OBS>2021-06-01T00:01:30.583</T_OBS>
<TOBSSTEP>1.00000</TOBSSTEP>
<TOBSEPOC>1977.01.01_00:00:00.000_TAI</TOBSEPOC>
<CAMERA>4</CAMERA>
<IMG_TYPE>LIGHT</IMG_TYPE>
<EXPTIME>2.9020560</EXPTIME>
<EXPSDEV>0.00020240554</EXPSDEV>
<INT_TIME>3.1484375</INT_TIME>
<WAVELNTH>304</WAVELNTH>
<WAVEUNIT>angstrom</WAVEUNIT>
<WAVE_STR>304_THIN</WAVE_STR>
<FSN>235893077</FSN>
<FID>0</FID>
<LVL_NUM>1.50000</LVL_NUM>
<QUALLEV0>0</QUALLEV0>
<QUALITY>1073741824</QUALITY>
<TOTVALS>16777216</TOTVALS>
<DATAVALS>16777216</DATAVALS>
<MISSVALS>0</MISSVALS>
<PERCENTD>100.000</PERCENTD>
<DATAMIN>-7</DATAMIN>
<DATAMAX>988</DATAMAX>
<DATAMEDN>3</DATAMEDN>
<DATAMEAN>4.7359357</DATAMEAN>
<DATARMS>6.4962573</DATARMS>
<DATASKEW>7.4229717</DATASKEW>
<DATAKURT>284.68384</DATAKURT>
<OSCNMEAN>0</OSCNMEAN>
<OSCNRMS>0</OSCNRMS>
<FLAT_REC>aia.flatfield[:#655]</FLAT_REC>
<CTYPE1>HPLN-TAN</CTYPE1>
<CUNIT1>arcsec</CUNIT1>
<CRVAL1>0.00000</CRVAL1>
<CDELT1>0.60000002</CDELT1>
<CRPIX1>2048.50</CRPIX1>
<CTYPE2>HPLT-TAN</CTYPE2>
<CUNIT2>arcsec</CUNIT2>
<CRVAL2>0.00000</CRVAL2>
<CDELT2>0.60000002</CDELT2>
<CRPIX2>2048.50</CRPIX2>
<CROTA2>0.00000</CROTA2>
<R_SUN>1577.4635</R_SUN>
<MPO_REC>sdo.master_pointing[:#1458]</MPO_REC>
<INST_ROT>-0.13163900</INST_ROT>
<IMSCL_MP>0.60016501</IMSCL_MP>
<X0_MP>2068.8899</X0_MP>
<Y0_MP>2009.7500</Y0_MP>
<RSUN_LF>0</RSUN_LF>
<X0_LF>0</X0_LF>
<Y0_LF>0</Y0_LF>
<ASD_REC>sdo.lev0_asd_0004[:#89405241]</ASD_REC>
<SAT_Y0>-5.8003411</SAT_Y0>
<SAT_Z0>9.1231413</SAT_Z0>
<SAT_ROT>3.1858184e-05</SAT_ROT>
<ACS_MODE>SCIENCE</ACS_MODE>
<ACS_ECLP>NO</ACS_ECLP>
<ACS_SUNP>YES</ACS_SUNP>
<ACS_SAFE>NO</ACS_SAFE>
<ACS_CGT>GT3</ACS_CGT>
<ORB_REC>sdo.fds_orbit_vectors[2021.06.01_00:01:00_UTC]</ORB_REC>
<DSUN_REF>1.4959787e+11</DSUN_REF>
<DSUN_OBS>1.5167896e+11</DSUN_OBS>
<RSUN_REF>6.9600000e+08</RSUN_REF>
<RSUN_OBS>946.47807</RSUN_OBS>
<GCIEC_X>0</GCIEC_X>
<GCIEC_Y>0</GCIEC_Y>
<GCIEC_Z>0</GCIEC_Z>
<HCIEC_X>0</HCIEC_X>
<HCIEC_Y>0</HCIEC_Y>
<HCIEC_Z>0</HCIEC_Z>
<OBS_VR>2327.1037</OBS_VR>
<OBS_VW>28223.582</OBS_VW>
<OBS_VN>5788.5049</OBS_VN>
<CRLN_OBS>88.404388</CRLN_OBS>
<CRLT_OBS>-0.66814470</CRLT_OBS>
<CAR_ROT>2244</CAR_ROT>
<ROI_NWIN>268435456</ROI_NWIN>
<ROI_SUM>0</ROI_SUM>
<ROI_NAX1>0</ROI_NAX1>
<ROI_NAY1>0</ROI_NAY1>
<ROI_LLX1>0</ROI_LLX1>
<ROI_LLY1>0</ROI_LLY1>
<ROI_NAX2>0</ROI_NAX2>
<ROI_NAY2>0</ROI_NAY2>
<ROI_LLX2>0</ROI_LLX2>
<ROI_LLY2>0</ROI_LLY2>
<ISPSNAME>aia.lev0_isp_0011</ISPSNAME>
<ISPPKTIM>2021-06-01T00:01:27.507</ISPPKTIM>
<ISPPKTVN>001.197</ISPPKTVN>
<AIVNMST>453</AIVNMST>
<AIMGOTS>2001196927</AIMGOTS>
<ASQHDR>3457118549</ASQHDR>
<ASQTNUM>3</ASQTNUM>
<ASQFSN>235893077</ASQFSN>
<AIAHFSN>235893069</AIAHFSN>
<AECDELAY>1534</AECDELAY>
<AIAECTI>0</AIAECTI>
<AIASEN>0</AIASEN>
<AIFDBID>241</AIFDBID>
<AIMGOTSS>13697</AIMGOTSS>
<AIFCPS>4</AIFCPS>
<AIFTSWTH>0</AIFTSWTH>
<AIFRMLID>3337</AIFRMLID>
<AIFTSID>40960</AIFTSID>
<AIHISMXB>7</AIHISMXB>
<AIHIS192>0</AIHIS192>
<AIHIS348>8252211</AIHIS348>
<AIHIS604>8388011</AIHIS604>
<AIHIS860>8388608</AIHIS860>
<AIFWEN>204</AIFWEN>
<AIMGSHCE>2900</AIMGSHCE>
<AECTYPE>0</AECTYPE>
<AECMODE>ON</AECMODE>
<AISTATE>CLOSED</AISTATE>
<AIAECENF>1</AIAECENF>
<AIFILTYP>0</AIFILTYP>
<AIMSHOBC>53.891998</AIMSHOBC>
<AIMSHOBE>67.351997</AIMSHOBE>
<AIMSHOTC>39.855999</AIMSHOTC>
<AIMSHOTE>24.927999</AIMSHOTE>
<AIMSHCBC>2955.7241</AIMSHCBC>
<AIMSHCBE>2969.4719</AIMSHCBE>
<AIMSHCTC>2941.7720</AIMSHCTC>
<AIMSHCTE>2927.2839</AIMSHCTE>
<AICFGDL1>0</AICFGDL1>
<AICFGDL2>25</AICFGDL2>
<AICFGDL3>89</AICFGDL3>
<AICFGDL4>236</AICFGDL4>
<AIFOENFL>1</AIFOENFL>
<AIMGFSN>3</AIMGFSN>
<AIMGTYP>0</AIMGTYP>
<AIAWVLEN>8</AIAWVLEN>
<AIAGP1>0</AIAGP1>
<AIAGP2>0</AIAGP2>
<AIAGP3>0</AIAGP3>
<AIAGP4>0</AIAGP4>
<AIAGP5>0</AIAGP5>
<AIAGP6>0</AIAGP6>
<AIAGP7>0</AIAGP7>
<AIAGP8>281</AIAGP8>
<AIAGP9>345</AIAGP9>
<AIAGP10>748</AIAGP10>
<AGT1SVY>7</AGT1SVY>
<AGT1SVZ>-20</AGT1SVZ>
<AGT2SVY>-4</AGT2SVY>
<AGT2SVZ>-16</AGT2SVZ>
<AGT3SVY>-2</AGT3SVY>
<AGT3SVZ>0</AGT3SVZ>
<AGT4SVY>57</AGT4SVY>
<AGT4SVZ>123</AGT4SVZ>
<AIMGSHEN>12</AIMGSHEN>
<RECNUM>230361239</RECNUM>
<DRMS_ID>aia_test.lev1p5:230361239:image_lev1p5</DRMS_ID>
<PRIMARYK>T_OBS, WAVELNTH</PRIMARYK>
<HEADSUM>HfdQKZaNHdaNHZaN</HEADSUM>
<LONGSTRN>OGIP 1.0</LONGSTRN>
<BLANK>-32768</BLANK>
<CHECKSUM>ZMjebJhdZJhdbJhd</CHECKSUM>
<DATASUM>3404142404</DATASUM>
<DATE_OBS>2021-06-01T00:01:29.132</DATE_OBS>
<XCEN>0.0000000</XCEN>
<YCEN>0.0000000</YCEN>
<history>
FITSHEAD2STRUCT run at: Tue Jun 1 14:07:45 2021
</history>
<comment>
FITS (Flexible Image Transport System) format is defined in 'Astronomy
and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
This FITS file may contain long string keyword values that are
continued over multiple keywords. The HEASARC convention uses the &
character at the end of each substring which is then continued
on the next keyword which has the name CONTINUE.
FITSHEAD2STRUCT
</comment>
</fits>
<helioviewer>
<HV_ROTATION>0.00000</HV_ROTATION>
<HV_JP2GEN_VERSION>0.8</HV_JP2GEN_VERSION>
<HV_JP2GEN_BRANCH_REVISION>No valid revision number found. Bazaar not installed? Using HV_WRITTENBY manually included revision number: 84 [2011/01/10, https://launchpad.net/jp2gen] : % SPAWN: Error managing child process.: No such file or directory</HV_JP2GEN_BRANCH_REVISION>
<HV_HVS_DETAILS_FILENAME>hvs_version5.pro</HV_HVS_DETAILS_FILENAME>
<HV_HVS_DETAILS_FILENAME_VERSION>5.0</HV_HVS_DETAILS_FILENAME_VERSION>
<HV_COMMENT>JP2 file created locally at Lockheed LMSAL using hv_aia_list2jp2_gs2 at Tue Jun 1 14:07:45 2021.
Contact Helioviewer LMSAL Franchise ([email protected]) for more details/questions/comments regarding this JP2 file.
HVS (Helioviewer setup) file used to create this JP2 file: hvs_version5.pro (version 5.0).
FITS to JP2 source code provided by ESA/NASA Helioviewer Project [contact the Helioviewer Project at [email protected]][NASA-GSFC] and is available for download at https://launchpad.net/jp2gen.
Please contact the source code providers if you suspect an error in the source code.
Full source code for the entire Helioviewer Project can be found at https://launchpad.net/helioviewer.</HV_COMMENT>
<HV_SUPPORTED>TRUE</HV_SUPPORTED>
</helioviewer>
</meta>
Helioviewer depends on information in the header for position images. Helioviewer.org also allows viewing the full FITS metadata in case users are interested in more specific details. Having the full header means if there is an update to take advantage of other features present in the header, the information will already be present in the header.
In case the header is missing some fields required by Helioviewer, this is a list of required values:
| FITS Key | Description |
|---|---|
| DSUN_OBS or DSUN | Distance between the observer and the sun, in meters |
| SOLAR_R or RADIUS | Width of the sun's radius in the image, in pixels. |
| NAXIS1 | Image width in pixels |
| NAXIS2 | Image height in pixels |
| CRPIX1 / CRPIX2 | x,y pixel coordinate of the reference pixel measured from the bottom left of the image. Can be outside of the image |
| CRVAL1 / CRVAL2 | HPC coordinate of the reference pixel in arcseconds |
| CDELT1 / CDELT2 | Area represented by the pixel in arcseconds per pixel. Often called Image Scale. Helioviewer requires CDELT1 == CDELT2 |
| TELESCOP | Name of the observatory i.e. SDO, SOHO, Hinode |
| DETECTOR | Detector used to create the image i.e C2/C3 for LASCO or COR1, COR2 for STEREO. |
| INSTRUME | Instrument of the observatory used to create the image i.e. EIT, MDI for SOHO or AIA, HMI for SDO. |
| DATE-OBS | Date and time the image was taken |