README

This project is intended to illustrate the hacking of "I/O API metadata" (aka IOAPI metadata) in netCDF files. (For more info on those standards, see

http://www.baronams.com/products/ioapi/
http://www.unidata.ucar.edu/software/netcdf/

) The top-level drivers are two bash scripts:

1 computeCropSum.sh: this takes an input of valid IOAPI (but which is semantically crippled), and

* removes all but the data variable of interest
* fixes the problem with its data
* assimilates some BELD data to a separate layer
* integrates the data over each gridcell
* "windows" the data to a subdomain
* plots the data, for each layer, recording metadata

2 mergeWithCmaqEmissions.sh: this takes as input

* the output of computeCropSum.sh (aka "EPIC emissions")
* emissions files from specified CMAQ-5 evaluation dates (aka "CMAQ emissions")
* MCIP output, containing map scale factors (MSFs), from one of those dates (since the MSFs are presumed time-invariant)

  and

* windows each of the CMAQ emissions to match the EPIC emissions' domain
* adds to each CMAQ emissions file a specie=N2O for contain the integration of the EPIC emissions
* converts the EPIC emissions' units to match those of the CMAQ emissions

while ensuring IOAPI-validity is conserved (e.g., the output can be

* loaded by VERDI, which is notoriously "hard to please"
* consumed by CCTM (aka CMAQ)

). The top-level drivers mostly drive two sets of netCDF-aware tools:

1 NCO

http://nco.sourceforge.net/

2 R, specifically packages=ncdf4, M3

http://cirrus.ucsd.edu/~pierce/ncdf/
http://cran.r-project.org/web/packages/M3/

  Note that package=M3 requires (among others) package=ncdf4, which requires netCDF 4.

For IOAPI testing, this project also uses

3 m3tools, specifically m3stat and m3wndw

www.baronams.com/products/ioapi/M3STAT.html
www.baronams.com/products/ioapi/M3WNDW.html

For display, this project also uses

4 R package=fields

http://www.image.ucar.edu/Software/Fields/

5 a commandline PDF viewer of your choice (edit the script, or comment out)

Since the code requires bash and netCDF 4, you may have difficulty running on windows: cygwin may be helpful. (If you're only interested in reading the most-IOAPI-aware code, see ioapi.r.) To run the example,

0 Ensure you have proper dependencies. Check your

* NCO: should be uplevel (I have version=4.0.5, latest=4.1.0) and in the path
* R: it should be uplevel (I have version=2.14.0, latest=2.15.0), in the path, and include the above packages
* bash: should be uplevel (I have version=3.2.25, latest=4.2) and your shell (always :-)
* m3tools: should be uplevel and in path
* PDF viewer: edit the driver script to call yours (or comment it out)

1 Fork the git repository, which will create a folder for following activity.
  E.g. (in bash),

$ rm -fr /tmp/ioapi-hack-R/
$ pushd /tmp
/tmp $ env GIT_SSL_NO_VERIFY=true git clone https://TomRoche@github.com/TomRoche/ioapi-hack-R.git
> Cloning into 'ioapi-hack-R'...
> remote: Counting objects: 29, done.
> remote: Compressing objects: 100% (22/22), done.
> remote: Total 29 (delta 9), reused 27 (delta 7)
> Unpacking objects: 100% (29/29), done.
/tmp $ popd ; pushd /tmp/ioapi-hack-R/

  The folder contents should resemble

> drwxr-xr-x  you  yours  4.0K  date  time  .
> -rwxr-xr-x  you  yours  6.6K  date  time  beldRead.r
> -rwxr-xr-x  you  yours   963  date  time  beldToRDS.r
> -rw-r--r--  you  yours   15K  date  time  computeCropSum.r
> -rwxr-xr-x  you  yours   13K  date  time  computeCropSum.sh
> -rw-r--r--  you  yours  6.2K  date  time  demonotonicizeVar.r
> drwxr-xr-x  you  yours  4.0K  date  time  .git
> -rw-r--r--  you  yours   120  date  time  .gitignore
> -rw-r--r--  you  yours   16K  date  time  ioapi.r
> -rwxr-xr-x  you  yours  3.8K  date  time  plotLayersForTimestep.r
> -rw-r--r--  you  yours  6.5K  date  time  processLayers.r
> -rw-r--r--  you  yours  6.1K  date  time  processVars.r
> -rw-r--r--  you  yours  1.7K  date  time  README
> -rw-r--r--  you  yours  1.8K  date  time  tlrRutilities.r
> -rw-r--r--  you  yours  8.4K  date  time  windowEmissions.r
> -rw-r--r--  you  yours  8.1K  date  time  writeBELDlayer.r

2 Download the input data from

https://github.com/TomRoche/ioapi-hack-R/downloads

  E.g. (in bash),

URI_ROOT="https://github.com/downloads/TomRoche/ioapi-hack-R"
DIR="/tmp/ioapi-hack-R"
for FN in \
  "map.CMAQkm.world.dat" \
  "epic_site_crops_0529_USA_2Ellen.csv" \
  "epic_site_crops_0529_USA_2Ellen.rds" \
  "5yravg.test.nc" \
; do
  URI="${URI_ROOT}/${FN}"
  FP="${DIR}/${FN}" # file path
  # curl can't handle the redirects, so use wget
#  ARGS="-c -O ${FP}"
  # for broken-cert environments (like EPA terrae) use '--no-check-certificate'
  ARGS="--no-check-certificate -c -O ${FP}"
  for CMD in \
    "wget ${ARGS} ${URI}" \
  ; do
    echo -e "$ ${CMD}"
    eval "${CMD}"
  done
done

  The folder contents should resemble

-rwxr-xr-x  you  yours  6.6K  date  time  beldRead.r
-rwxr-xr-x  you  yours   963  date  time  beldToRDS.r
-rw-r--r--  you  yours   15K  date  time  computeCropSum.r
-rwxr-xr-x  you  yours   13K  date  time  computeCropSum.sh
-rw-r--r--  you  yours  6.2K  date  time  demonotonicizeVar.r
drwxr-xr-x  you  yours  4.0K  date  time  .git
-rw-r--r--  you  yours   120  date  time  .gitignore
-rw-r--r--  you  yours   16K  date  time  ioapi.r
-rwxr-xr-x  you  yours  3.8K  date  time  plotLayersForTimestep.r
-rw-r--r--  you  yours  6.5K  date  time  processLayers.r
-rw-r--r--  you  yours  6.1K  date  time  processVars.r
-rw-r--r--  you  yours  1.7K  date  time  README
-rw-r--r--  you  yours  1.8K  date  time  tlrRutilities.r
-rw-r--r--  you  yours  8.4K  date  time  windowEmissions.r
-rw-r--r--  you  yours  8.1K  date  time  writeBELDlayer.r
-rw-r--r--  you  yours  1.1M  date  time  map.CMAQkm.world.dat
-rw-r--r--  you  yours  1.5M  date  time  epic_site_crops_0529_USA_2Ellen.rds
-rw-r--r--  you  yours  5.5M  date  time  epic_site_crops_0529_USA_2Ellen.csv
-rw-r--r--  you  yours  638M  date  time  5yravg.test.nc

3 Run it: e.g.,

/tmp/ioapi-hack-R $ ./computeCropSum.sh

  It should produce

3.1 a ginormous console trace, at times paused

3.2 several new netCDF files, e.g.,

-rw-r--r--  you  yours  400K  date  time  5yravg.DN2windowed.nc
-rw-r--r--  you  yours   24M  date  time  5yravg.DN2summed.nc
-rw-r--r--  you  yours   24M  date  time  5yravg.DN2belded.nc
-rw-r--r--  you  yours   24M  date  time  5yravg.DN2layers_fixed.nc
-rw-r--r--  you  yours   24M  date  time  5yravg.DN2layered.nc
-rw-r--r--  you  yours  1.1M  date  time  temp.extend.nc
-rw-r--r--  you  yours   24M  date  time  temp.full.nc
-rw-r--r--  you  yours   22M  date  time  5yravg.DN2demonotonicized.nc
-rw-r--r--  you  yours   22M  date  time  5yravg.DN2vars_fixed.nc
-rw-r--r--  you  yours   22M  date  time  5yravg.DN2stripped.nc

    The main output is '5yravg.DN2windowed.nc', which you should be able to open with VERDI (the point of the IOAPI part of this exercise :-) Compared to the source NetCDF (5yravg.test.nc), that output

*   covers a smaller horizontal extent

*   has many fewer datavars: in fact, only one (DN2) other than the IOAPI "metadatavar" TFLAG.

3.3 a plot file containing plots of each data layer in the window, plus

*   a plot of the sum of the BELD proportions in each gridcell (penultimate layer)

*   a plot of the integrated DN2 emissions over each gridcell (final layer)

After you run it, improve it! The code is currently quite crude, so feel free to fix/extend--and then send me pull request!

HTH, Tom Roche <Tom_Roche@pobox.com>