Failure and database corruption when commit after requesting ambient gas broadening parameters that are valid, but have no entries in HITRAN database

[jackdobinson]

Description

Parameters that are valid, but have no entries in HITRAN (e.g. "gamma_CO2" for any isotope of H2O) has the following problems:

1. The lack of data is not reported (at least a logging.warning should be issued). A quick fix would be fetch and fetch_by_ids could return a boolean flag indicating if all retrieved fields have data. Then the user can make an informed decision about how to handle that case.

2. The on-disk database becomes corrupted and will not re-initialise (after close and re-open of python interpreter) after hapi.db_commit() is called after fetching valid parameter that has no entries in HITRAN database. Seems to be due to the header['position'] dictionary not containing the offsets of the new data columns.

Recreation of Error

The following code snippet recreates the error:

# HITRAN API testing
import os
import hapi

# Make folder for database
db_dir = 'local_test_database'
os.makedirs(db_dir, exist_ok=True)

# Initialise database and download problematic data
hapi.db_begin(db_dir)
hapi.fetch('H2O_TEST', 1, 1, 2750, 3250, Parameters=('nu','gamma_CO2')) 

# no indication that `gamma_CO2` is not present in HITRAN database

# Commit database transaction to save to disk
hapi.db_commit()

# Re-open database, fails with KeyError(...)
hapi.db_begin(db_dir)

Possible fix for (2)

            # pre-defined positions are needed to skip the existing parameters in headers (new feature)
            if 'position' in header:
                # FIX STARTS HERE
                start = header['position'].get(qnt, None) 
                if start is None:
                    start = end
                    header['position'][qnt] = start
                # FIX ENDS HERE
            else:
                start = end