Domain decomposition and halo construction by halungge · Pull Request #540 · C2SM/icon4py

halungge · 2024-09-06T08:55:34Z

Decompose (global) grid file:

uses pymetis to decompose the global grid (cells) into n patches
after decomposition halos for all dimensions (cell, edge, vertex) are constructed. Halo construction is done in a ICON like fashion: They consist halos of 2 cell levels (one upward and one downward pointing line) and the corresponding vertices and edges on these lines.

Omissions:

LAM grids need to be investigated further:
- tests comparing decomposed vs. single_node computation are only run on the global grid.
- for the LAM grids ICON reorders arrays to arrange halo points on the first boundary layers together with the boundary layers, it should be investigated whether that is essential in the model.
- This PR does only take this into account on the computation of the start_index and end_index not in the halo construction.
the number of halo lines (in terms of cells) is hardcoded to 2, that could be made a parameter.
Not sure it all runs on GPU correctly... most probably there are some numpy cupy issues to fix.

model/common/src/icon4py/model/common/grid/grid_manager.py

# Conflicts: # model/common/src/icon4py/model/common/grid/base.py # model/common/src/icon4py/model/common/grid/grid_manager.py # model/common/src/icon4py/model/common/grid/horizontal.py # model/common/src/icon4py/model/common/grid/refinement.py # model/common/tests/common/decomposition/mpi_tests/test_mpi_decomposition.py # model/common/tests/common/grid/unit_tests/test_refinement.py

# Conflicts: # model/common/src/icon4py/model/common/grid/grid_manager.py # model/testing/src/icon4py/model/testing/grid_utils.py

# Conflicts: # model/atmosphere/diffusion/tests/diffusion/mpi_tests/test_parallel_diffusion.py # model/atmosphere/dycore/tests/dycore/mpi_tests/test_parallel_solve_nonhydro.py # model/common/src/icon4py/model/common/decomposition/definitions.py # model/common/tests/common/decomposition/fixtures.py # model/common/tests/common/decomposition/mpi_tests/test_mpi_decomposition.py # model/common/tests/common/decomposition/unit_tests/test_definitions.py # model/common/tests/common/grid/mpi_tests/test_parallel_icon.py # model/testing/src/icon4py/model/testing/definitions.py # model/testing/src/icon4py/model/testing/grid_utils.py # model/testing/src/icon4py/model/testing/serialbox.py # tools/src/icon4py/tools/py2fgen/wrappers/common.py

# Conflicts: # model/common/pyproject.toml # model/common/src/icon4py/model/common/decomposition/definitions.py # model/common/src/icon4py/model/common/decomposition/mpi_decomposition.py # model/common/src/icon4py/model/common/grid/grid_manager.py # model/testing/src/icon4py/model/testing/grid_utils.py # tach.toml

jcanton · 2026-03-06T16:10:28Z

model/common/src/icon4py/model/common/grid/grid_manager.py

+        my_cell_indices = self._decomposition_info.global_index(dims.CellDim)
+        my_edge_indices = self._decomposition_info.global_index(dims.EdgeDim)
+        my_vertex_indices = self._decomposition_info.global_index(dims.VertexDim)


possibly for later cleanup, make these and their friends in _read_coordinates class members?

Yep, definitely for later.

msimberg · 2026-03-09T11:36:47Z

cscs-ci run default

msimberg · 2026-03-09T11:36:52Z

cscs-ci run distributed

msimberg · 2026-03-09T15:00:45Z

cscs-ci run default

msimberg · 2026-03-09T15:00:50Z

cscs-ci run distributed

jcanton

very important

msimberg · 2026-03-10T09:56:12Z

cscs-ci run default

github-actions · 2026-03-10T09:56:14Z

Mandatory Tests

Please make sure you run these tests via comment before you merge!

cscs-ci run default
cscs-ci run distributed

Optional Tests

To run benchmarks you can use:

cscs-ci run benchmark-bencher

To run tests and benchmarks with the DaCe backend you can use:

cscs-ci run dace

To run test levels ignored by the default test suite (mostly simple datatest for static fields computations) you can use:

cscs-ci run extra

For more detailed information please look at CI in the EXCLAIM universe.

msimberg · 2026-03-10T09:56:19Z

cscs-ci run distributed

* main: Domain decomposition and halo construction (#540)

#540 added mpi4py as required dependency to the typing group. It is in principle required, but this makes mpi4py be installed by default with a regular `uv sync` which is suboptimal. Removing it for now from the typing group. mpi4py is still a required dependency in the distributed group.

* main: (29 commits) Scheduled Halo Exchange (#980) Add missing metrics fields to `test_parallel_grid_manager.py` test (#1114) Muphys: Lowering with single precision (#1101) Add single-rank lsq pseudoinv factory test (#1099) Cleanup Diffusion config (#1060) Fortran bindings: fix numpy allocation and cleanups (#1112) fix: fix gt4py metrics extractor in the StencilTest benchmarking (#1111) py2fgen: don't recompile if unchanged (#1110) CI for standalone_driver (#1070) Update mpi4py and pymetis groups to make them optional (#1100) Bump mshick/add-pr-comment from 2 to 3 (#1109) Use inout fields for full_muphys as well (#1108) Update GPU configuration for graupel (#1104) Move the mask of _q_t_update outside in graupel (#1093) Update gt4py to v1.1.7 (#1105) cleanup for ugly if condition of single node default in lsq coeffs (#1103) Domain decomposition and halo construction (#540) Muphys: Add flag to wait for graupel completion (#1095) Give each gt4py program a return type hint (#1087) Turn data download off for distributed CI (#1092) ...

halungge commented Jul 25, 2025

View reviewed changes

model/common/src/icon4py/model/common/grid/grid_manager.py Outdated Show resolved Hide resolved

halungge force-pushed the halo_construction branch from df9c2ef to 72d4d4b Compare July 25, 2025 16:30

halungge mentioned this pull request Sep 4, 2025

remove the workaround separating vertical and horizontal connectivities #859

Draft

halungge mentioned this pull request Sep 15, 2025

Better torus support for mean cell area etc. #848

Merged

Magdalena Luz added 26 commits September 18, 2025 11:34

fix some typting in test_parallel_grid_manager.py

271d70b

make test run again

eca2a8f

fixes from merge

3698b9a

Merge branch 'main' into halo_construction

7a4170c

# Conflicts: # model/common/src/icon4py/model/common/grid/grid_manager.py # model/testing/src/icon4py/model/testing/grid_utils.py

fix merge: global grid params

c9be1d0

fix typings

0a2d126

remove obsolete TODO

785f7d1

fix parallel diffusion tests

c5ad92b

fix parallel non hydro test

8791be0

delete files unrelated to this PR

ef3b494

delete files unrelated to this PR

c87800c

Merge branch 'main' into halo_construction

ed51c58

get grid tests to run again

935b113

get parallel test to run again

179d4b7

get parallel test to run again

45bbbf6

start/end index for halos (I)

bb190ec

fix parallel test for start_index/end_index

e89c5e5

add tests WIP(1)

464302a

fix typing

b0746c3

pre-commit

12dbe47

add tests (WIP)

88ae0ec

simple fixes

3c35b61

fix reading of additional edge fields

def79a5

jcanton reviewed Mar 6, 2026

View reviewed changes

msimberg added 3 commits March 9, 2026 12:34

Add xfails for wrong halo, interior, local indices with torus grid

56f9cef

Check INTERIOR indices with icon4py domain decomposition

c6cf66b

Update ci/distributed.yml

301c041

msimberg force-pushed the halo_construction branch from acdf71e to 301c041 Compare March 9, 2026 11:47

msimberg added 4 commits March 9, 2026 12:48

Less verbose xfail

d77f5d9

Download r01b01 grid in ci

37b196b

Enable download again

39703b3

Disable download again

6df6af6

msimberg force-pushed the halo_construction branch from 8114383 to 6df6af6 Compare March 9, 2026 13:58

msimberg added 2 commits March 9, 2026 15:13

Explicitly set ICON4PY_TEST_DATA_PATH in github workflows

115aec8

More explicit path

f204644

msimberg force-pushed the halo_construction branch from 914bcbc to f204644 Compare March 9, 2026 14:19

Enable grid download for grid download step

dc339dc

jcanton approved these changes Mar 9, 2026

View reviewed changes

nfarabullini approved these changes Mar 9, 2026

View reviewed changes

Remove todos

fc0a445

msimberg merged commit 6fc8c37 into main Mar 10, 2026
48 checks passed

msimberg deleted the halo_construction branch March 10, 2026 10:55

jcanton added a commit that referenced this pull request Mar 10, 2026

Merge branch 'main' into distributed_standalone_driver

2a71be6

* main: Domain decomposition and halo construction (#540)

msimberg mentioned this pull request Mar 10, 2026

Update mpi4py and pymetis groups to make them optional #1100

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Domain decomposition and halo construction#540

Domain decomposition and halo construction#540
msimberg merged 375 commits intomainfrom
halo_construction

halungge commented Sep 6, 2024 •

edited

Loading

Uh oh!

Uh oh!

jcanton Mar 6, 2026

Uh oh!

msimberg Mar 9, 2026

Uh oh!

msimberg commented Mar 9, 2026

Uh oh!

msimberg commented Mar 9, 2026

Uh oh!

msimberg commented Mar 9, 2026

Uh oh!

msimberg commented Mar 9, 2026

Uh oh!

jcanton left a comment

Uh oh!

msimberg commented Mar 10, 2026

Uh oh!

github-actions bot commented Mar 10, 2026

Uh oh!

msimberg commented Mar 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

halungge commented Sep 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jcanton Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

msimberg Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

msimberg commented Mar 9, 2026

Uh oh!

msimberg commented Mar 9, 2026

Uh oh!

msimberg commented Mar 9, 2026

Uh oh!

msimberg commented Mar 9, 2026

Uh oh!

jcanton left a comment

Choose a reason for hiding this comment

Uh oh!

msimberg commented Mar 10, 2026

Uh oh!

github-actions bot commented Mar 10, 2026

Uh oh!

msimberg commented Mar 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

halungge commented Sep 6, 2024 •

edited

Loading