v0.3 dev

README.md

@@ -30,6 +30,8 @@
     </tr>
 </table>
 
+<!-- TODO: README needs updating for v1-dev! -->
+
 <!-- Insert catchy summary -->
 
 VirtualShipParcels is a command line simulator allowing students to plan and conduct a virtual research expedition, receiving measurements as if they were coming from actual oceanographic instruments including:

docs/user-guide/documentation/copernicus_products.md

@@ -2,7 +2,7 @@
 
 VirtualShip supports running experiments anywhere in the global ocean from 1993 through to the present day (and approximately two weeks into the future), using the suite of products available from the [Copernicus Marine Data Store](https://data.marine.copernicus.eu/products).
 
-The data sourcing task is handled by the `virtualship fetch` command. The three products relied on by `fetch` to source data for all [VirtualShip instruments](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) (both physical and biogeochemical) are:
+The data sourcing task is handled by the `virtualship run` command, which in turn relies on the [copernicusmarine toolbox](https://github.com/mercator-ocean/copernicus-marine-toolbox?tab=readme-ov-file) for 'streaming' data from the Copernicus Marine Data Store. The three products relied on in `run` to source data for all [VirtualShip instruments](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) (both physical and biogeochemical) are:
 
 1. **Reanalysis** (or "hindcast" for biogeochemistry).
 2. **Renalysis interim** (or "hindcast interim" for biogeochemistry).
@@ -15,7 +15,7 @@ The Copernicus Marine Service describe the differences between the three product
 As a general rule of thumb the three different products span different periods across the historical period to present and are intended to allow for continuity across the previous ~ 30 years.
 
 ```{note}
-The ethos for automated dataset selection in `virtualship fetch` is to prioritise the Reanalysis/Hindcast products where possible (the 'work horse'), then _interim products where possible for continuity, and finally filling the very near-present (and near-future) temporal range with the Analysis & Forecast products.
+The ethos for automated dataset selection in `virtualship run` is to prioritise the Reanalysis/Hindcast products where possible (the 'work horse'), then _interim products where possible for continuity, and finally filling the very near-present (and near-future) temporal range with the Analysis & Forecast products.
 ```
 
 ```{warning}
@@ -24,13 +24,13 @@ In the rare situation where the start and end times of an expedition schedule sp
 
 ### Data availability
 
-The following tables summarise which Copernicus product is selected by `virtualship fetch` per combination of time period and variable (see legend below).
+The following tables summarise which Copernicus product is selected by `virtualship run` per combination of time period and variable (see legend below).
 
 For biogeochemical variables `ph` and `phyc`, monthly products are required for hindcast and hindcast interim periods. For all other variables, daily products are available.
 
 #### Physical products
 
-| Period              | Product ID                               | Temporal Resolution | Typical Years Covered               | Variables                  |
+| Period              | Dataset ID                               | Temporal Resolution | Typical Years Covered               | Variables                  |
 | :------------------ | :--------------------------------------- | :------------------ | :---------------------------------- | :------------------------- |
 | Reanalysis          | `cmems_mod_glo_phy_my_0.083deg_P1D-m`    | Daily               | ~30 years ago to ~5 years ago       | `uo`, `vo`, `so`, `thetao` |
 | Reanalysis Interim  | `cmems_mod_glo_phy_myint_0.083deg_P1D-m` | Daily               | ~5 years ago to ~2 months ago       | `uo`, `vo`, `so`, `thetao` |
@@ -40,7 +40,7 @@ For biogeochemical variables `ph` and `phyc`, monthly products are required for
 
 #### Biogeochemical products
 
-| Period                        | Product ID                                 | Temporal Resolution | Typical Years Covered               | Variables                         | Notes                                  |
+| Period                        | Dataset ID                                 | Temporal Resolution | Typical Years Covered               | Variables                         | Notes                                  |
 | :---------------------------- | :----------------------------------------- | :------------------ | :---------------------------------- | :-------------------------------- | :------------------------------------- |
 | Hindcast                      | `cmems_mod_glo_bgc_my_0.25deg_P1D-m`       | Daily               | ~30 years ago to ~5 years ago       | `o2`, `chl`, `no3`, `po4`, `nppv` | Most BGC variables except `ph`, `phyc` |
 | Hindcast (monthly)            | `cmems_mod_glo_bgc_my_0.25deg_P1M-m`       | Monthly             | ~30 years ago to ~5 years ago       | `ph`, `phyc`                      | Only `ph`, `phyc` (monthly only)       |

docs/user-guide/documentation/example_copernicus_download.ipynb

@@ -0,0 +1,193 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "a48322c9",
+   "metadata": {},
+   "source": [
+    "# Example Copernicus data download \n",
+    "\n",
+    "This notebook provides a rough, non-optimised example of how to download Copernicus Marine data using the `copernicusmarine` Python package.\n",
+    "\n",
+    "This will download:\n",
+    "- Global bathymetry data (static)\n",
+    "- Global biogeochemical monthly data (0.25 degree hindcast)\n",
+    "- Global physical daily data (0.25 degree reanalysis)\n",
+    "\n",
+    "For a singular year (2023) and two months (June and July).\n",
+    "\n",
+    "This notebook is intended as a basic example only. Modifications will be needed to adapt this to your own use case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7f5a7cc7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import copernicusmarine\n",
+    "import os\n",
+    "from datetime import datetime"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e7279d5a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "YEAR = \"2023\"\n",
+    "MONTHS = [\"06\", \"07\"]\n",
+    "DAYS = [\n",
+    "    \"01\",\n",
+    "    \"02\",\n",
+    "    \"03\",\n",
+    "    \"04\",\n",
+    "    \"05\",\n",
+    "    \"06\",\n",
+    "    \"07\",\n",
+    "    \"08\",\n",
+    "    \"09\",\n",
+    "    \"10\",\n",
+    "    \"11\",\n",
+    "    \"12\",\n",
+    "    \"13\",\n",
+    "    \"14\",\n",
+    "    \"15\",\n",
+    "    \"16\",\n",
+    "    \"17\",\n",
+    "    \"18\",\n",
+    "    \"19\",\n",
+    "    \"20\",\n",
+    "    \"21\",\n",
+    "    \"22\",\n",
+    "    \"23\",\n",
+    "    \"24\",\n",
+    "    \"25\",\n",
+    "    \"26\",\n",
+    "    \"27\",\n",
+    "    \"28\",\n",
+    "    \"29\",\n",
+    "    \"30\",\n",
+    "    \"31\",\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1a583dba",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "### PHYSICAL DAILY FILES\n",
+    "\n",
+    "os.chdir(\"~/data/phys/\")\n",
+    "DATASET_ID = \"cmems_mod_glo_phy-all_my_0.25deg_P1D-m\"\n",
+    "\n",
+    "for month in MONTHS:\n",
+    "    for day in DAYS:\n",
+    "        # check is valid date\n",
+    "        try:\n",
+    "            datetime(year=int(YEAR), month=int(month), day=int(day), hour=0)\n",
+    "        except ValueError:\n",
+    "            continue\n",
+    "\n",
+    "        filename = f\"{DATASET_ID}_global_fulldepth_{YEAR}_{month}_{day}.nc\"\n",
+    "\n",
+    "        if os.path.exists(filename):\n",
+    "            print(f\"File {filename} already exists, skipping...\")\n",
+    "            continue\n",
+    "\n",
+    "        copernicusmarine.subset(\n",
+    "            dataset_id=DATASET_ID,\n",
+    "            variables=[\"uo_glor\", \"vo_glor\", \"thetao_glor\", \"so_glor\"],\n",
+    "            minimum_longitude=-180,\n",
+    "            maximum_longitude=179.75,\n",
+    "            minimum_latitude=-80,\n",
+    "            maximum_latitude=90,\n",
+    "            start_datetime=f\"{YEAR}-{month}-{day}T00:00:00\",\n",
+    "            end_datetime=f\"{YEAR}-{month}-{day}T00:00:00\",\n",
+    "            minimum_depth=0.5057600140571594,\n",
+    "            maximum_depth=5902.0576171875,\n",
+    "            output_filename=filename,\n",
+    "        )"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "89921772",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "### BIOGEOCHEMICAL MONTHLY FILES\n",
+    "\n",
+    "os.chdir(\"~/data/bgc/\")\n",
+    "DATASET_ID = \"cmems_mod_glo_bgc_my_0.25deg_P1M-m\"\n",
+    "DAY = \"01\"\n",
+    "\n",
+    "for month in MONTHS:\n",
+    "    try:\n",
+    "        datetime(year=int(YEAR), month=int(month), day=int(DAY), hour=0)\n",
+    "    except ValueError:\n",
+    "        continue\n",
+    "\n",
+    "    filename = f\"{DATASET_ID}_global_fulldepth_{YEAR}_{month}_{DAY}.nc\"\n",
+    "\n",
+    "    if os.path.exists(filename):\n",
+    "        print(f\"File {filename} already exists, skipping...\")\n",
+    "        continue\n",
+    "\n",
+    "    copernicusmarine.subset(\n",
+    "        dataset_id=\"cmems_mod_glo_bgc_my_0.25deg_P1M-m\",\n",
+    "        variables=[\"chl\", \"no3\", \"nppv\", \"o2\", \"ph\", \"phyc\", \"po4\"],\n",
+    "        minimum_longitude=-180,\n",
+    "        maximum_longitude=179.75,\n",
+    "        minimum_latitude=-80,\n",
+    "        maximum_latitude=90,\n",
+    "        start_datetime=f\"{YEAR}-{month}-{DAY}T00:00:00\",\n",
+    "        end_datetime=f\"{YEAR}-{month}-{DAY}T00:00:00\",\n",
+    "        minimum_depth=0.5057600140571594,\n",
+    "        maximum_depth=5902.05810546875,\n",
+    "        output_filename=filename,\n",
+    "    )"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8b5495c6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "### BATHYMETRY FILE\n",
+    "os.chdir(\"~/data/bathymetry/\")\n",
+    "DATASET_ID = \"cmems_mod_glo_phy_anfc_0.083deg_static\"\n",
+    "filename = \"cmems_mod_glo_phy_anfc_0.083deg_static_bathymetry.nc\"\n",
+    "\n",
+    "copernicusmarine.subset(\n",
+    "    dataset_id=DATASET_ID,\n",
+    "    dataset_part=\"bathy\",\n",
+    "    variables=[\"deptho\"],\n",
+    "    minimum_longitude=-180,\n",
+    "    maximum_longitude=179.91668701171875,\n",
+    "    minimum_latitude=-80,\n",
+    "    maximum_latitude=90,\n",
+    "    minimum_depth=0.49402499198913574,\n",
+    "    maximum_depth=0.49402499198913574,\n",
+    "    output_filename=filename,\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/user-guide/documentation/pre_download_data.md

@@ -0,0 +1,87 @@
+# Pre-downloading data
+
+By default, VirtualShip will automatically 'stream' data from the Copernicus Marine Service via the [copernicusmarine toolbox](https://github.com/mercator-ocean/copernicus-marine-toolbox?tab=readme-ov-file). However, for users who wish to manage data locally, it is possible to pre-download the required datasets and feed them into VirtualShip simulations.
+
+<!-- TODO: quickstart guide needs full update! -->
+
+As outlined in the [Quickstart Guide](https://virtualship.readthedocs.io/en/latest/user-guide/quickstart.html), the `virtualship run` command supports an optional `--from-data` argument, which allows users to specify a local directory containing the necessary data files.
+
+```{tip}
+See the [for example...](#for-example) section for an example data download workflow.
+```
+
+### Data requirements
+
+When using pre-downloaded data with VirtualShip, the software supports only: daily and monthly resolution physical and biogeochemical data, along with a static bathymetry file.
+
+In addition, all pre-downloaded data must be split into separate files per timestep (i.e. one .nc file per day or month).
+
+```{note}
+**Monthly data**: when using monthly data, ensure that your final .nc file download is for the month *after* your expedition schedule end date. This is to ensure that a Parcels FieldSet can be generated under-the-hood which fully covers the expedition period. For example, if your expedition runs from 1st May to 15th May, your final monthly data file should be in June. Daily data files only need to cover the expedition period exactly.
+```
+
+Further, VirtualShip expects pre-downloaded data to be organised in a specific directory & filename structure within the specified local data directory. The expected structure is as outlined in the subsequent sections.
+
+#### Directory structure
+
+Assuming the local data directory (as supplied in the `--from-data` argument) is named `data/`, the expected subdirectory structure is:
+
+```bash
+.
+└── data
+    ├── bathymetry # containing the singular bathymetry .nc file
+    ├── bgc # containing biogeochemical .nc files
+    └── phys # containing physical .nc files
+```
+
+#### Filename conventions
+
+Within these subdirectories, the expected filename conventions are:
+
+- Physical data files (in `data/phys/`) should be named as follows:
+  - `<COPERNICUS_DATESET_NOMENCLATURE>_<YYYY_MM_DD>.nc`
+    - e.g. `cmems_mod_glo_phy-all_my_0.25deg_P1D-m_1998_05_01.nc` and so on for each timestep.
+- Biogeochemical data files (in `data/bgc/`) should be named as follows:
+  - `<COPERNICUS_DATESET_NOMENCLATURE>_<YYYY_MM_DD>.nc`
+    - e.g. `cmems_mod_glo_bgc_my_0.25deg_P1M-m_1998_05_01.nc` and so on for each timestep.
+- Bathymetry data file (in `data/bathymetry/`) should be named as follows:
+  - `cmems_mod_glo_phy_anfc_0.083deg_static_bathymetry.nc`
+
+```{tip}
+Careful to use an underscore (`_`) as the separator between date components in the filenames (i.e. `YYYY_MM_DD`).
+```
+
+```{note}
+Using the `<COPERNICUS_DATESET_NOMENCLATURE>` in the filenames is vital in order to correctly identify the temporal resolution of the data (daily or monthly). The `P1D` in the example above indicates daily data, whereas `P1M` would indicate monthly data.
+
+See [here](https://help.marine.copernicus.eu/en/articles/6820094-how-is-the-nomenclature-of-copernicus-marine-data-defined#h_34a5a6f21d) for more information on Copernicus dataset nomenclature.
+
+See also our own [documentation](https://virtualship.readthedocs.io/en/latest/user-guide/documentation/copernicus_products.html#data-availability) on the Copernicus datasets used natively by VirtualShip when 'streaming' data if you wish to use the same datasets for pre-download.
+```
+
+```{note}
+**Monthly data**: the `DD` component of the date in the filename for monthly .nc files should always be `01`, representing the first day of the month. This ensures that a Parcels FieldSet can be generated under-the-hood which fully covers the expedition period from the start.
+```
+
+#### Further assumptions
+
+The following assumptions are also made about the data:
+
+1. All pre-downloaded data files must be in NetCDF format (`.nc`).
+2. Physical data files must contain the following variables: `uo`, `vo`, `so`, `thetao`
+   - Or these strings must appear as substrings within the variable names (e.g. `uo_glor` is acceptable for `uo`).
+3. If using BGC instruments (e.g. `CTD_BGC`), the relevant biogeochemical data files must contain the following variables: `o2`, `chl`, `no3`, `po4`, `nppv`, `ph`, `phyc`.
+   - Or these strings must appear as substrings within the variable names (e.g. `o2_glor` is acceptable for `o2`).
+4. Bathymetry data files must contain a variable named `deptho`.
+
+#### Also of note
+
+1. Whilst not mandatory to use data downloaded only from Copernicus Marine (any existing data you may hold can be re-organised accordingly), the assumptions made by VirtualShip regarding directory structure and filename conventions are motivated by alignment with the Copernicus Marine's practices.
+   - If you want to use pre-existing data with VirtualShip, which you may have accessed from a different source, it is possible to do so by restructuring and/or renaming your data files as necessary.
+2. The whole VirtualShip pre-downloaded data workflow should support global data or subsets thereof, provided the data files contain the necessary variables and are structured as outlined above.
+
+#### For example...
+
+Example Python code for automating the data download from Copernicus Marine can be found in [Example Copernicus Download](example_copernicus_download.ipynb).
+
+<!-- TODO: replace with URL? -->

docs/user-guide/index.md

@@ -15,4 +15,6 @@ assignments/index
 :maxdepth: 1
 
 documentation/copernicus_products.md
+documentation/pre_download_data.md
+documentation/example_copernicus_download.ipynb
 ```