Add support for fmu-datio output

.gitignore

@@ -60,4 +60,3 @@ target/
 
 # pyenv python configuration file
 .python-version
-docs/test_images/

README.rst

@@ -35,3 +35,7 @@ Features
 * Outputs PNG figures and Irap (RMS) binary maps
 * Simple configuration through a YAML file
 * Configuration from YAML can be overriden by command line options
+* From version 1.3, output via `fmu-dataio`_ (to `sumo`_) is supported.
+
+.. _fmu-dataio: https://github.com/equinor/fmu-dataio/
+.. _sumo: https://github.com/equinor/fmu-sumo

docs/config-hc.rst

@@ -98,9 +98,9 @@ to make HC thickness maps from geomodels directly.
      stoiip: {STOIIP: tests/data/reek/reek_geo_stooip.roff}
      dates: [19900101]
 
-The "STOIIP" in the example above is the name of the property within the ROFF file.
-If this name is noit known, try 'unknown' as name instead. The date is purely
-informative.
+The "STOIIP" in the example above is the name of the property within the ROFF file. If this name is
+not known, try 'null' as name instead (also 'unknown' is supported for backward compatibility). The
+date is purely informative.
 
 
 Input Eclipse with dates from external config
@@ -141,7 +141,7 @@ properties, either continuous of discrete. For example:
 The ``zonation`` section
 ------------------------
 
-This section is optopnal (more text to come)
+This section is optional (more text to come)
 
 
 The ``mapsettings`` section
@@ -159,4 +159,9 @@ Text coming...
 The ``output`` section
 ----------------------
 
-Text coming...
+A missing output will (from version 1.3):
+
+* Output maps via `fmu-dataio <https://github.com/equinor/fmu-dataio/>`_.
+* Output plots to ``/tmp`` folder
+
+For other settings, see :ref:`allexamples`

docs/examples.rst

@@ -1,3 +1,5 @@
+.. _allexamples:
+
 ============
 Examples
 ============
@@ -29,13 +31,13 @@ The zonation file:
 
 Plots (examples):
 
-Zone 1+3 oilthickness in 19991201:
+All zones oilthickness in 19991201:
 
-.. image:: test_images/z1+3--hc1b_oilthickness--19991201.png
+.. image:: test_to_docs/all--hc1b_oilthickness--19991201.png
 
-Difference oil column in Zone3:
+Difference oil column:
 
-.. image:: test_images/z3--hc1b_oilthickness--20010101_19991201.png
+.. image:: test_to_docs/all--hc1b_oilthickness--20010101_19991201.png
 
 HC thickness 1c
 """""""""""""""
@@ -150,6 +152,33 @@ a cumulative facies thickness when filter is activated, as in this example.
 .. literalinclude:: ../tests/yaml/hc_thickness4a.yml
    :language: yaml
 
+HC thickness hcdataio1a (dataio)
+""""""""""""""""""""""""""""""""
+
+Thickness maps (oil) using PORV from Eclipse files. Here fmu-dataio is the backend, for upload to
+SUMO. See ``mapfolder`` is set to `magic` ``fmu-dataio``!
+
+.. literalinclude:: ../docs/test_to_docs/hcdataio1a.yml
+   :language: yaml
+
+HC thickness hcdataio1b (dataio)
+""""""""""""""""""""""""""""""""
+
+Thickness maps for both gas and oil using PORV from Eclipse files and grid from roff. See
+``mapfolder`` is set to `magic` ``fmu-dataio``! Alternatively ``mapfolder`` could just be missing.
+
+.. literalinclude:: ../docs/test_to_docs/hcdataio1b.yml
+   :language: yaml
+
+HC thickness hcdataio1c (dataio)
+""""""""""""""""""""""""""""""""
+
+Thickness maps using STOIIP from roff files. Here fmu-dataio is the backend, for upload to
+SUMO. This is default behaviour (from version 1.3) when ``mapfolder`` is missing
+
+.. literalinclude:: ../docs/test_to_docs/hcdataio1c.yml
+   :language: yaml
+
 
 ---------------------------------------
 Examples of YAML setup for Average maps
@@ -173,9 +202,9 @@ AVG example 1b
 .. literalinclude:: ../tests/yaml/avg1b.yml
    :language: yaml
 
-.. image:: test_images/z1--avg1b_average_por.png
+.. image:: test_to_docs/z1--avg1b_average_por.png
 
-.. image:: test_images/z3--avg1b_average_por.png
+.. image:: test_to_docs/z3--avg1b_average_por.png
 
 
 AVG example 1c
@@ -231,3 +260,12 @@ or whatever. Both Eclipse and ROFF input should be possible.
 
 .. literalinclude:: ../tests/yaml/avg2b.yml
    :language: yaml
+
+AVG example dataio 1a
+"""""""""""""""""""""
+
+Using FMU dataio as backend. Otherwise similar to previous case 2c. Note that a metadata section is
+required for average data, and properties must occur under ``properties:``
+
+.. literalinclude:: ../docs/test_to_docs/avgdataio1a.yml
+   :language: yaml

docs/test_images/.gitignore → docs/test_to_docs/.gitignore

@@ -1 +1,2 @@
 *.png
+*.yml

docs/usage.rst

@@ -84,3 +84,42 @@ file, e.g. as:
      diffdates: !include_from  ../../share/fmuconfig/output/global_variables.yml::global.DIFFDATES
 
 Note that dates and diffdates are two separate lists
+
+
+--------------------------------
+Using fmu-dataio for SUMO output
+--------------------------------
+
+From version 1.3, a ``fmu-dataio`` pipeline is supported and encouraged. This will make results
+compatible with upload to SUMO. This involves some changes:
+
+  * The environment variable ``FMU_GLOBAL_CONFIG`` shall be set in advance and this shall
+    point to a fmu-dataio compatible global variables file (on YAML format), for example:
+
+  .. code-block:: bash
+
+      export FMU_GLOBAL_CONFIG=fmuconfig/output/global_variables.yml
+
+  * The output.mapfolder must either:
+
+    * be missing, as default now is fmu-dataio
+
+    * be set to the magical string ``fmu-dataio``
+
+  * For average maps, some additional metadata are needed, which shall be given in the
+    ``metadata`` block under ``input.properties`` section, e.g.
+
+  .. code-block:: yaml
+
+      input:
+        eclroot: tests/data/reek/REEK
+        grid: $eclroot.EGRID
+
+        properties:
+          - name: SWAT
+            source: $eclroot.UNRST
+            dates: !include_from tests/yaml/global_config3a.yml::global.DATES
+            diffdates: !include_from tests/yaml/global_config3a.yml::global.DIFFDATES
+            metadata:
+              content: saturation
+              unit: fraction

scripts/setup_functions.py

@@ -1,5 +1,6 @@
 """Setup helpers for setup.py."""
 import fnmatch
+import glob
 import os
 from distutils.command.clean import clean as _clean
 from os.path import exists
@@ -46,6 +47,7 @@ class CleanUp(_clean):
 
     CLEANFOLDERSRECURSIVE = ["__pycache__", "_tmp_*", "*.egg-info"]
     CLEANFILESRECURSIVE = ["*.pyc", "*.pyo"]
+    CLEANSPECIAL = ["docs/test_to_docs/*.png", "docs/test_to_docs/*.yml"]
 
     @staticmethod
     def ffind(pattern, path):
@@ -91,6 +93,11 @@ def run(self):
                 print(f"Remove file {pfil}")
                 os.unlink(pfil)
 
+        for fil_ in CleanUp.CLEANSPECIAL:
+            for pfil in glob.glob(fil_):
+                print(f"Remove (auto doc) file {pfil}")
+                os.unlink(pfil)
+
 
 # ======================================================================================
 # Sphinx

src/xtgeoapp_grd3dmaps/avghc/_compute_avg.py

@@ -92,10 +92,8 @@ def get_avg(config, specd, propd, dates, zonation, zoned, filterarray):
                 coarsen=mycoarsen,
             )
 
-            use_fmu_dataio = False
-            if config["output"]["mapfolder"] == "fmu-dataio":
-                use_fmu_dataio = True
-            else:
+            filename = None
+            if config["output"]["mapfolder"] != "fmu-dataio":
                 filename = _avg_filesettings(config, zname, propname, mode="map")
 
             usename = (zname, propname)
@@ -104,7 +102,7 @@ def get_avg(config, specd, propd, dates, zonation, zoned, filterarray):
                 xmap.values = ma.masked_inside(xmap.values, -1e-30, 1e-30)
 
             avgd[usename] = xmap.copy()
-            if use_fmu_dataio:
+            if filename is None:
                 export_avg_map_dataio(avgd[usename], usename, config)
             else:
                 xtg.say("Map file to {}".format(filename))

src/xtgeoapp_grd3dmaps/avghc/_compute_hcpfz.py

@@ -1,5 +1,4 @@
 import numpy.ma as ma
-
 from xtgeo.common import XTGeoDialog
 
 xtg = XTGeoDialog()
@@ -25,7 +24,7 @@ def get_hcpfz(config, initd, restartd, dates, hcmode, filterarray):
     hcpfzd = dict()
 
     # use the given date from config if stoiip, giip, etc as info
-    gdate = config["input"]["dates"][0]  # will give 'unknowndate' if unset
+    gdate = str(config["input"]["dates"][0])  # will give 'unknowndate' if unset
 
     if "rock" in hcmode:
         hcpfzd[gdate] = initd["dz"] * filterarray
@@ -54,17 +53,14 @@ def _get_hcpfz_ecl(config, initd, restartd, dates, hcmode, filterarray):
         raise RuntimeError("Dates er missing. Bug?")
 
     for date in dates:
-
         if hcmode in ("oil", "gas"):
             usehc = restartd["s" + hcmode + "_" + str(date)]
         elif hcmode == "comb":
             usehc1 = restartd["s" + "oil" + "_" + str(date)]
             usehc2 = restartd["s" + "gas" + "_" + str(date)]
             usehc = usehc1 + usehc2
         else:
-            raise ValueError(
-                'Invalid mode "{}" in "computesettings: method"'.format(hcmode)
-            )
+            raise ValueError(f"Invalid mode '{hcmode}'' in 'computesettings: method'")
 
         if hcmethod == "use_poro":
             usehc[usehc < shcintv[0]] = 0.0
@@ -92,8 +88,7 @@ def _get_hcpfz_ecl(config, initd, restartd, dates, hcmode, filterarray):
 
         else:
             raise ValueError(
-                'Unsupported method "{}" in "computesettings:'
-                ' method"'.format(hcmethod)
+                f"Unsupported method '{hcmethod}' in 'computesettings' method"
             )
 
     for key, val in hcpfzd.items():
@@ -104,23 +99,21 @@ def _get_hcpfz_ecl(config, initd, restartd, dates, hcmode, filterarray):
     # config and select the right one, and delete those that are not
     # relevant; e.g. one may ask for 20050816--19930101 but not for
     # 20050816; in that case the difference must be computed but
-    # after that the 20050816 entry will be removed fomr the list
+    # after that the 20050816 entry will be removed from the list
 
     cdates = config["input"]["dates"]
 
     for cdate in cdates:
         cdate = str(cdate)
-        logger.debug("cdate is: %s", cdate)
         if "-" in cdate:
-            d1 = str(cdate.split("-")[0])
-            d2 = str(cdate.split("-")[1])
-
-            if d1 in hcpfzd and d2 in hcpfzd:
-                hcpfzd[cdate] = hcpfzd[d1] - hcpfzd[d2]
+            dt1 = str(cdate.split("-")[0])
+            dt2 = str(cdate.split("-")[1])
+            if dt1 in hcpfzd and dt2 in hcpfzd:
+                hcpfzd[cdate] = hcpfzd[dt1] - hcpfzd[dt2]
             else:
                 xtg.warn(
-                    "Cannot retrieve data for date {} and/or {}. "
-                    "Some TSTEPs failed?".format(d1, d2)
+                    f"Cannot retrieve data for date {dt1} and/or {dt2}. "
+                    "Some TSTEPs failed?"
                 )
 
     alldates = hcpfzd.keys()