Add process for splitting PRISM climos

actions/split-prism-climos/1971.yaml

@@ -0,0 +1,3 @@
+global:
+    climo_start_time: "1971-01-01T00:00:00Z"
+    climo_end_time: "2000-12-31T00:00:00Z"

actions/split-prism-climos/1981.yaml

@@ -0,0 +1,3 @@
+global:
+    climo_start_time: "1981-01-01T00:00:00Z"
+    climo_end_time: "2010-12-31T00:00:00Z"

actions/split-prism-climos/DESCRIPTION.md

@@ -0,0 +1,41 @@
+# Split Aggregated PRISM Climologies
+
+## Purpose
+
+The [PRISM data portal](https://data.pacificclimate.org/portal/bc_prism/map/) currently features climatologies that contain both monthly and yearly data. Each file contains data for the twelve monthly means, followed by the yearly mean. The yearly mean is assigned to a Mid-Year (July) date, so the timestamps in the file are not in order.
+
+We want to upgrade to a snazzy new version of ncWMS, but the new version requires monotonic timestamps. So each file has to be split into separate monthly and yearly datasets, which can be viewed or downloaded separately by users.
+
+## Splitting the files
+
+The files were split with CDO rather than `split_merged_climos`, as `split_merged_climos` isn't set up to handle monthly+yearly aggregations and there was some urgency. Commands are in `cdo.txt`. 
+
+## Updating metadata
+
+The original files predated PCIC's metadata standards and contained none of the global metadata required. 
+
+* `prism.yaml` - metadata needed by all these files
+* `1971.yaml` and `1981.yaml` - `climo_start_time` and `climo_end_time` attributes for the two climatologies 
+* `monthly.yaml` and `yearly.yaml` - `frequency` attributes for the two resolutions
+
+## Dimensionizing CRS
+
+CDO automatically corrects the CRS variable from a variable with a
+time dimension to a variable with no dimensions.
+CDO's corrections are 100% correct - the CF Standards specify that
+CRSs should be dimensionless variables - but the PDP cannot at this
+time support downloading a dimensionless variable, so after the CDO
+operators, we have the run the `dimensionize_crs.py` script to restore
+the dimensional variable.
+
+## Files Processed
+* pr_monClim_PRISM_historical_run1_197101-200012.nc
+* pr_monClim_PRISM_historical_run1_198101-201012.nc
+* tmax_monClim_PRISM_historical_run1_197101-200012.nc
+* tmax_monClim_PRISM_historical_run1_198101-201012.nc
+* tmin_monClim_PRISM_historical_run1_197101-200012.nc
+* tmin_monClim_PRISM_historical_run1_198101-201012.nc
+
+## Next Steps
+
+These files are still missing some required metadata; we're waiting to hear back from scientists, but this process filled in enough metadata to make it possible to add them to the database and map servers.

actions/split-prism-climos/PRISM.yaml

@@ -0,0 +1,12 @@
+global:
+    project_id: other
+    institution: Pacific Climate Impacts Consortium (PCIC), Victoria, BC, www.pacificclimate.org
+    institute_id: PCIC
+    contact: Faron Anslow ([email protected])
+    model_id: PRISM
+    product: gridded_observations
+    domain: bc
+    table_id: Table Amon (10 Jun 2010)
+    experiment_id: historical
+    run: run1
+    modeling_realm: atmos

actions/split-prism-climos/cdo.txt

@@ -0,0 +1,33 @@
+$ cdo seltimestep,1/12 input/pr_monClim_PRISM_historical_run1_197101-200012.nc output/pr_mClimMean_PRISM_historical_19710101-20001231.nc
+
+$ cdo seltimestep,13 input/pr_monClim_PRISM_historical_run1_197101-200012.nc output/pr_aClimMean_PRISM_historical_19710101-20001231.nc
+
+
+$ cdo seltimestep,1/12 input/pr_monClim_PRISM_historical_run1_198101-201012.nc output/pr_mClimMean_PRISM_historical_19810101-20101231.nc
+
+$ cdo seltimestep,13 input/pr_monClim_PRISM_historical_run1_198101-201012.nc output/pr_aClimMean_PRISM_historical_19810101-20101231.nc
+
+
+
+
+$ cdo seltimestep,1/12 input/tmax_monClim_PRISM_historical_run1_197101-200012.nc output/tasmax_mClimMean_PRISM_historical_19710101-20001231.nc
+
+$ cdo seltimestep,13 input/tmax_monClim_PRISM_historical_run1_197101-200012.nc output/tasmax_aClimMean_PRISM_historical_19710101-20001231.nc
+
+
+$ cdo seltimestep,1/12 input/tmax_monClim_PRISM_historical_run1_198101-201012.nc output/tasmax_mClimMean_PRISM_historical_19810101-20101231.nc
+
+$ cdo seltimestep,13 input/tmax_monClim_PRISM_historical_run1_198101-201012.nc output/tasmax_aClimMean_PRISM_historical_19810101-20101231.nc
+
+
+
+
+
+$ cdo seltimestep,1/12 input/tmin_monClim_PRISM_historical_run1_197101-200012.nc output/tasmin_mClimMean_PRISM_historical_19710101-20001231.nc
+
+$ cdo seltimestep,13 input/tmin_monClim_PRISM_historical_run1_197101-200012.nc output/tasmin_aClimMean_PRISM_historical_19710101-20001231.nc
+
+
+$ cdo seltimestep,1/12 input/tmin_monClim_PRISM_historical_run1_198101-201012.nc output/tasmin_mClimMean_PRISM_historical_19810101-20101231.nc
+
+cdo seltimestep,13 input/tmin_monClim_PRISM_historical_run1_198101-201012.nc output/tasmin_aClimMean_PRISM_historical_19810101-20101231.nc

actions/split-prism-climos/dimensionize_crs.py

@@ -0,0 +1,79 @@
+'''The CF Standards support using data variables with no associated dimensions
+to store coordinate reference information. The variable is used to tightly group
+attributes that specify various values for the CRS. Here's an example:
+
+int crs ;
+        crs:grid_mapping_name = "latitude_longitude" ;
+        crs:long_name = "CRS definition" ;
+        crs:longitude_of_prime_meridian = 0. ;
+        crs:semi_major_axis = 6378137. ;
+        crs:inverse_flattening = 298.257222101 ;
+        crs:GeoTransform = "-140.0041666666665 0.008333333332999999 0 61.9958333333325 0 -0.008333333332999999 " ;
+        crs:crs_wkt = "GEOGCS[\"NAD83\",DATUM[\"North_American_Datum_1983\",SPHEROID[\"GRS 1980\",6378137,298.257222101,AUTHORITY[\"EPSG\",\"7019\"]],TOWGS84[0,0,0,0,0,0,0],AUTHORITY[\"EPSG\",\"6269\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4269\"]]" ;
+
+Unfortunately, the PDP cannot actually serve a dimensionless variable. 
+
+This script is a quick hack. It takes a file with a dimensionless crs variable and
+outputs a file identical except for the fact that the crs variable has a time
+dimension and a single value (0). All other variables and attributes are simply copied.
+'''
+
+import argparse
+from netCDF4 import Dataset
+import time
+
+parser = argparse.ArgumentParser('convert a dimensionless CRS variable to time-based')
+parser.add_argument('infile', metavar='infile', help='input netCDF file')
+parser.add_argument('outfile', metavar='outfile', help='location to write output file to')
+args = parser.parse_args()
+
+infile = Dataset(args.infile, "r", format="NETCDF4")
+
+print("Processing {} to {}".format(args.infile, args.outfile))
+# check to see if there's a dimensionless CRS variable
+if not "crs" in infile.variables:
+    print("Error: No CRS variable in {}".format(args.infile))
+    infile.close()
+    sys.exit()
+elif infile.variables["crs"].dimensions is None:
+    print("Error: CRS variable already has a dimension".format(args.infile))
+    infile.close()
+    sys.exit()
+
+outfile = Dataset(args.outfile, "w", format="NETCDF4")
+
+#Copy over everything else.
+
+# Dimensions
+for dim in infile.dimensions:
+    size = None if infile.dimensions[dim].isunlimited() else len(infile.dimensions[dim])
+    outfile.createDimension(dim, size)
+
+# Non-CRS variables
+for var in infile.variables:
+    if var != "crs":
+        print("    Copying variable {}".format(var))
+        outfile.createVariable(var, infile.variables[var].dtype, infile.variables[var].dimensions)
+        outfile.variables[var].setncatts(infile.variables[var].__dict__)
+        outfile.variables[var][:] = infile.variables[var][:]
+
+# Global attributes
+print("    Copying global attributes")
+outfile.setncatts(infile.__dict__)
+
+# Update the CRS
+print("    Updating CRS")
+outfile.createVariable("crs", "i", ("time"))
+outfile.variables["crs"].setncatts(infile.variables["crs"].__dict__)
+outfile.variables["crs"][:] = 0        
+
+# Update the history attribute
+print("    Updating history global attribute")
+entry = "{}: dimensionize_crs {} {}".format(time.ctime(time.time()), args.infile, args.outfile)
+outfile.history = "{} {}".format(entry, outfile.history if "history" in outfile.ncattrs() else "")
+
+
+# clean up
+infile.close()
+outfile.close()
+print("Successfully updated CRS")

actions/split-prism-climos/monthly.yaml

@@ -0,0 +1,2 @@
+global:
+    frequency: mClim

actions/split-prism-climos/yearly.yaml

@@ -0,0 +1,2 @@
+global:
+    frequency: aClim