Implementing `convert.nemo_to_sgrid()` #2460

VeckoTheGecko · 2026-01-09T15:23:04Z

This PR supercedes #2447
implementing the following additional functionality:

Refactor out the FieldSet.from_nemo() functionality into a convert.py module with a nemo_to_sgrid() function. Also added corresponding tests.
Make the input parameters of the nemo_to_sgrid function more closely align with the input datasets
Update FieldSet.from_sgrid_conventions to infer mesh from the metadata included in the dataset
Add get_value_by_id() function to Grid*DMetadata classes allowing to easily refer to parts of the SGRID metadata model

Future work:

Improve logging and metadata conversion convert.py - its important that we refactor this module so that its clearer, and have logging so that changes to the input dataset are communicated to the user. To support this, we need
Better testing against different NEMO datasets
- Reach out to the NEMO community asking for ncdump output so that better see the types of nemo models out there

I'll make an issue to track this future work.

And update of tutorial_nemo_curvilinear

And updating tutorial

ANd expading comments on loading in the dataset

To use same ordering as in XLinear

This requires smaller selection_dict for the isel, so hopefully faster code

And also separating offset calculation into its own helper function

It's not clear why this is here, nor why removing it causes a test failure. To be investigated another time. EDIT: This was introduced in Parcels-code@c311fba - though we're investigating if this can be implemented another way since there should be no particular difference with NEMO

Gradually reducing the dependency on the `mesh` param

Updates the API for conversion to be more closely aligned with the input data. Also handles the U and V fields separately - correctly assigning the dimension naming before merging into a single dataset.

Update tests

Avoid assuming there's a U and V field. Maybe this should be refactored later...

VeckoTheGecko · 2026-01-09T15:26:01Z

I'll do a full self review next week. Feel free to take a look in the meantime @erikvansebille if you'd like

VeckoTheGecko · 2026-01-09T15:45:18Z

Also still a few failing tests that I need to sort out...

erikvansebille

Looks nice already. Some first comments below

erikvansebille · 2026-01-12T07:28:38Z

docs/user_guide/examples/tutorial_nemo_curvilinear.ipynb

This and other tutorials still need to be updated to use the new convert workflow? Do you want me to do that? Could be a good test-case how user-friendly the convert functions are

erikvansebille · 2026-01-12T07:31:24Z

docs/user_guide/index.md

 :titlesonly:
 examples/explanation_grids.md
 examples/tutorial_nemo_curvilinear.ipynb
+examples/tutorial_nemo_3D.ipynb


But the file is at examples_v3/tutorial_nemo_3D.ipynb so it either has to be moved (which then destroys the git diff), or we use the old directory here and then update it in a later PR?

Suggested change

examples/tutorial_nemo_3D.ipynb

examples_v3/tutorial_nemo_3D.ipynb # TODO move to examples folder

erikvansebille · 2026-01-12T07:32:33Z

src/parcels/_core/utils/sgrid.py

        return _metadata_rename_dims(self, dims_dict)

+    def get_value_by_id(self, id: str) -> str:
+        """In the SGRID specification different parts of the spec are identified to with different "ID"s.


Suggested change

"""In the SGRID specification different parts of the spec are identified to with different "ID"s.

"""In the SGRID specification for 2D grids, different parts of the spec are identified by different "ID"s.

erikvansebille · 2026-01-12T07:32:45Z

src/parcels/_core/utils/sgrid.py

        return _metadata_rename_dims(self, dims_dict)

+    def get_value_by_id(self, id: str) -> str:
+        """In the SGRID specification different parts of the spec are identified to with different "ID"s.


Suggested change

"""In the SGRID specification different parts of the spec are identified to with different "ID"s.

"""In the SGRID specification for 3D grids, different parts of the spec are identified by different "ID"s.

erikvansebille · 2026-01-12T07:35:27Z

src/parcels/_core/fieldset.py

+                vector_interp_method=CGrid_Velocity,
+                # ^Seems to work with AGrid as well? (at least, tests aren't failing -
+                # either logic needs to be added to choose interpolator, or this interpolator should be renamed)


No, this is part of the changes to unit converters in #2455, #2459 and #2461.

In #2461, I propose we attach a new XLinear_Velocity interpolator for A-Grids, for consistency/transparency, but that can be done in a later pR (when the others have merged too)

erikvansebille · 2026-01-12T07:36:30Z

src/parcels/_core/fieldset.py

                fields["W"] = Field("W", ds["W"], grid, XLinear)
-                fields["UVW"] = VectorField("UVW", fields["U"], fields["V"], fields["W"])
+                fields["UVW"] = VectorField(
+                    "UVW", fields["U"], fields["V"], fields["W"], vector_interp_method=CGrid_Velocity


Also here, it would depend on the grid topology whether we use XLinear_Velocity or CGrid_Velocity

erikvansebille · 2026-01-12T07:37:48Z

src/parcels/_core/fieldset.py

+        if mesh == "spherical":
+            if "U" in fieldset.fields:
+                fieldset.U.units = GeographicPolar()
+            if "V" in fieldset.fields:
+                fieldset.V.units = Geographic()


This would all go if we implement #2459. We need to be careful when fixing merge conflicts...

erikvansebille · 2026-01-12T07:39:12Z

src/parcels/convert.py

+These functions use knowledge about the model to attach any missing metadata. The functions
+emit verbose messaging so that the user is kept in the loop. The returned output is an
+Xarray dataset so that users can further provide any missing metadata that was unable to
+be determined.


Suggested change

be determined.

be determined before they pass it to the FieldSet constructor.

erikvansebille · 2026-01-12T07:39:59Z

src/parcels/convert.py

+}
+
+
+def _maybe_bring_UV_depths_to_depth(ds):


Would all the functions in this convert file require verbose logging? Or is that too much?

erikvansebille · 2026-01-12T07:41:50Z

tests/test_advection.py

+        pytest.param(
+            AdvectionRK4_3D,
+            marks=pytest.mark.xfail(
+                reason="from_nemo had 'fieldset.V.units = GeographicPolar()', I'm not sure _why_ this code is needed to get this to pass. To be further investigated.",


Should be fixed in the changes that come with #2455, #2459 and #2461

github-project-automation bot added this to Parcels development Jan 9, 2026

github-project-automation bot moved this to Backlog in Parcels development Jan 9, 2026

erikvansebille and others added 28 commits January 9, 2026 16:24

First implementation of fieldset.from_nemo

18bd8b4

And update of tutorial_nemo_curvilinear

Setting vector_interp_method only when vector fields available

57d4a54

Add error when pset.execute has neither runtime nor endtime

d47400b

Adding support for NEMO 3D

a60c911

And updating tutorial

Fixing negating vertical velocity code

6769a2c

Adding docstring for from_nemo

d9f7a52

ANd expading comments on loading in the dataset

Support for NEMO fields without depth

1fa20b5

Simplifying nemo dataset creation in tutorial

c0f5d3e

Using glamf, gphif and depth as dimensions in from_nemo

c5ad537

Using xgcm coordinates for cgrid interpolation selection

60e2f38

Reordering corner_data in CGrid_Velocity

d36cf82

To use same ordering as in XLinear

Separating U and V interpolation in CGrid_Velocity

0f91c23

This requires smaller selection_dict for the isel, so hopefully faster code

Cleaning up CGrid_Velocity code

17c9d4e

Fixing bug in CGrid_Tracer interpolation

79b4c37

Using offsets in CGrid_Tracer

7343a43

And also separating offset calculation into its own helper function

Move from_nemo body into parcels.convert

24de000

Move selection of vector interpolator to from_sgrid_conventions

26c6d55

Fix from_nemo

c8800a2

Refactor test_nemo_3D_curvilinear_fieldset

ba7a953

Add discovery of mesh from convention metadata

5dc0a5f

Gradually reducing the dependency on the `mesh` param

Fix xfail mark

ea4e993

Add helpers get_value_by_id

fd699f2

Add test_convert.test_nemo_to_sgrid

875fe70

Fix param naming to obey pep8

31c63dd

Update API for convert.nemo_to_sgrid

24357bf

Updates the API for conversion to be more closely aligned with the input data. Also handles the U and V fields separately - correctly assigning the dimension naming before merging into a single dataset.

Remove FieldSet.from_nemo

3670dd5

Update tests

Fix from_sgrid_conventions

0b9c787

Avoid assuming there's a U and V field. Maybe this should be refactored later...

VeckoTheGecko force-pushed the fieldset_from_nemo-nick branch from 6d12f21 to 0b9c787 Compare January 9, 2026 15:25

VeckoTheGecko mentioned this pull request Jan 9, 2026

Improve convert.nemo_to_sgrid with logging and better coverage #2462

Open

2 tasks

erikvansebille reviewed Jan 12, 2026

View reviewed changes

	examples/tutorial_nemo_3D.ipynb
	examples_v3/tutorial_nemo_3D.ipynb # TODO move to examples folder

	"""In the SGRID specification different parts of the spec are identified to with different "ID"s.
	"""In the SGRID specification for 2D grids, different parts of the spec are identified by different "ID"s.

	"""In the SGRID specification different parts of the spec are identified to with different "ID"s.
	"""In the SGRID specification for 3D grids, different parts of the spec are identified by different "ID"s.

	be determined.
	be determined before they pass it to the FieldSet constructor.

Implementing convert.nemo_to_sgrid() #2460

Are you sure you want to change the base?

Implementing convert.nemo_to_sgrid() #2460

Uh oh!

Conversation

VeckoTheGecko commented Jan 9, 2026

Uh oh!

VeckoTheGecko commented Jan 9, 2026

Uh oh!

VeckoTheGecko commented Jan 9, 2026

Uh oh!

erikvansebille left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Implementing `convert.nemo_to_sgrid()` #2460

Implementing `convert.nemo_to_sgrid()` #2460