Docs/documentation updates oct29

docs/contributing.rst

@@ -0,0 +1,10 @@
+=================
+Contributing
+=================
+
+Placeholder
+
+Pull Requests
+------------------
+
+Placeholder

docs/data_creation.rst

@@ -1,5 +1,5 @@
 =================
-Create Training Data
+Dataset Creation
 =================
 
 We use ufs2arco to generate training, validation, and test datasets. The ufs2arco library helps preprocess weather data and makes it ready to be used to train a machine learning model.
@@ -68,6 +68,8 @@ ufs2arco recipe.yaml
 
 For further information see the ufs2arco `github <https://ufs2arco.readthedocs.io/en/latest/>`_ or `documentation <https://ufs2arco.readthedocs.io/en/latest/>`_
 
+ufs2arco was created by Tim Smith at NOAA Physical Sciences Laboratory.
+
 Helpful quick tips for ufs2arco
 ------------------
 
@@ -95,4 +97,15 @@ Note: it is best practice to ensure that your statistics period only includes da
 Changing Variables
 ~~~~~~~~~~~~~~~~~~~~~~
 
-To change variables or levels desired, simply add or remove variables or levels within the `source` section of a recipe.yaml. Please see ufs2arco documentation for further information on exactly what variables are available.
+To change variables or levels desired, simply add or remove variables or levels within the `source` section of a recipe.yaml. Please see ufs2arco documentation for further information on exactly what variables are available.
+
+MPI Usage
+~~~~~~~~~~~~~~~~~~~~~~
+
+ufs2arco uses MPI to parallelize data preprocessing. If you wish to not use MPI, change the top line of the yaml to say
+
+.. code-block:: yaml
+
+    mover:
+    name: datamover
+    batch_size: 2

docs/index.rst

@@ -12,6 +12,7 @@ Table of Contents
    :maxdepth: 1
    :caption: Full Pipeline
 
+   pipeline_overview
    data_creation
    train_a_model
    inference
@@ -23,3 +24,10 @@ Table of Contents
 
    getting_started
    nested_eagle
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Community
+
+   contributing
+   support

docs/inference.rst

@@ -1,10 +1,10 @@
 =================
-Inference
+Create a Forecast
 =================
 
 We use the anemoi-inference package to create a forecast.
 
-See `anemoi-inference documentation <https://anemoi.readthedocs.io/projects/inference/en/latest/>`_ documentation for further information
+See `anemoi-inference documentation <https://anemoi.readthedocs.io/projects/inference/en/latest/>`_ for further information
 
 Helpful quick tips for using anemoi-inference
 ------------------

docs/nested_eagle.rst

@@ -2,22 +2,30 @@
 Nested Eagle
 =================
 
-Provide brief overview of Nested-Eagle
+The nested-eagle model is a prototype model trained on global NOAA Global Forecast System (GFS) data 
+with High Resolution Rapid Refresh (HRRR) data over the Contiguous United States (CONUS). 
+This builds on previous work from Met Norway (Nipen et al., 2024, arXiv:2409.02891) by creating a 
+nested model with lower resolution global data and high resolution over an area of interest.
 
-Insert image of nested domain...
+TODO - Insert image of nested domain here.
+
+Nested-eagle configurations were created by Tim Smith at NOAA Physical Sciences Laboratory.
 
 Training Data
 ------------------
 
 Datasets:
+
 - GFS convservatively regridded to 1-degree
 - HRRR conservatively regridded to 15-km 
 
 Time period:
+
 - Training dataset: 2015-02-01T06 to 2023-01-31T18
 - Validation dataset: 2023-02-01T06 to 2024-01-31T18
 
 Variables:
+
 - Prognostic: gh, u, v, w, t, q, sp, u10, v10, t2m, t_surface, sh2
 - Diagnostic: u80, v80, accum_tp (use fhr 6)
 - Forcing: lsm, orog, cos_latitude, sin_latitude, cos_longitude, sin_longitude, cos_julian_day, sin_julian_day, cos_local_time, sin_local_time, insolation
@@ -26,12 +34,11 @@ Variables:
 Model Architecture
 ------------------
 
-- Encoder
-    - GraphTransformer
-- Processor
-    - Transformer with 512 channels and a window size of 4320
-- Decoder
-    - GraphTransformer
+Encoder: Graph Transformer
+
+Processor: Shifted Window Transformer with 512 channels
+
+Decoder: Graph Transformer
 
-Graph:
-- Encoder-decoder only
+Graph: Used for the encoder and decoder to connect targets to nodes via nearest neighbors (encoder_knn=12, decode_knn=3).
+Latent mesh is 4 times more coarse than native data.

docs/pipeline_overview.rst

@@ -2,7 +2,7 @@
 Overview of Full Pipeline
 =================
 
-You will learn how to complete a full machine learning pipeline for weather prediction!
+An end-to-end machine learning pipeline for weather prediction.
 
 The steps of this pipeline include:
 

docs/support.rst

@@ -0,0 +1,7 @@
+=================
+Support
+=================
+
+If you encounter any errors please raise an issue in the github repository.
+
+You can also email ___________ for further support.

docs/train_a_model.rst

@@ -2,15 +2,56 @@
 Train a Graph-Based Model
 =================
 
-We use the anemoi-core modules to train a graph-based model.
+We deploy the anemoi-core modules to train a graph-based model.
 
 See Anemoi documentation for further information:
 
 - `anemoi-graphs <https://anemoi.readthedocs.io/projects/graphs/en/latest/>`_
 - `anemoi-training <https://anemoi.readthedocs.io/projects/training/en/latest/>`_
 - `anemoi-models <https://anemoi.readthedocs.io/projects/models/en/latest/index.html>`_
 
+Anemoi was created by the European Center for Medium-Range Weather Forecasting.
+
 Helpful quick tips for using anemoi-core
 ------------------
 
-Insert tips
+Throughout this repository, all anemoi configs are typically provided for you and should work out of the box. 
+See below for various tips and explanations if you wish to learn more about the configs or want to change the configurations.
+
+Brief Config Overview
+~~~~~~~~~~~~~~~~~~~~~~
+
+The configs used by anemoi-training contain a lot of information. At the top of a main config you will see something like
+
+.. code-block:: yaml
+
+    defaults:
+    - data: zarr
+    - dataloader: native_grid
+    - datamodule: single
+    - diagnostics: evaluation
+    - hardware: slurm
+    - graph: encoder_decoder_only
+    - model: transformer
+    - training: stretched
+    - _self_
+
+This points the training process to the appropriate yaml file needed for various steps. 
+For example, the first line points to zarr.yaml within the data folder, 
+which then provides the training process with information on the training data such as variables used and temporal frequency. 
+
+Throughout this repository, we have consolidated a lot of very useful information within the main config.yaml. 
+This makes it so the main config.yaml contains most model configurations that are worth noting, and additionally makes those 
+configurations easy to change.
+
+Generating Configs Yourself
+~~~~~~~~~~~~~~~~~~~~~~
+
+If you wish to use brand new configs and configure a model yourself, run the following command while within 
+a conda environemnt that contains all of the anemoi-core modules:
+
+```
+anemoi-training config generate
+```
+
+This will generate new anemoi configs for you. If you have any questions about the configs go see the anemoi-training documentation.

docs/verification.rst

@@ -1,11 +1,13 @@
 =================
-Validation
+Verify a Forecast
 =================
 
 We use wxvx for forecast verification.
 
 See `wxvx <https://github.com/maddenp-cu/wxvx>`_  for further information.
 
+wxvx was created by Paul Madden at NOAA GSL/CIRES.
+
 Helpful quick tips for using wxvx
 ------------------