add io offload prototype #1774

rem1776 · 2025-09-22T17:59:14Z

Description
draft for the io offloading changes

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Please also note
any relevant details for your test configuration (e.g. compiler, OS). Include
enough information so someone can reproduce your tests.

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
Any dependent changes have been merged and published in downstream modules
New check tests, if applicable, are included
make distcheck passes

… test

…resses

vithikashah001

needs documentation for variables, subroutines and functions.

fms2_io/netcdf_io.F90

offloading/metadata_transfer.F90

uramirez8707

Some minor changes, but I think this is a good start

uramirez8707 · 2026-01-08T18:47:17Z

offloading/offloading_io.F90

This needs the apache license

test_fms/offloading/test_io_offloading.F90 and test_fms/offloading/test_metadata_transfer.F90 needs it too

uramirez8707 · 2026-01-08T18:47:49Z

test_fms/offloading/Makefile.am

The license for this file should be apache

test_fms/offloading/test_io_offloading.sh needs it too

uramirez8707 · 2026-01-08T18:52:01Z

offloading/offloading_io.F90

+  integer, parameter :: non_domain_decomposed = 1 !< enumeration for type of domain
+  integer, parameter :: Unstructured_grid = 2 !< enumeration for type of domain
+
+  integer, parameter :: max_files = 10 !< amount of offloaded files to allocate space for


TODO Make this a namelist, runtime constant, etc

uramirez8707 · 2026-01-08T19:14:06Z

offloading/offloading_io.F90

+  !! Registers an axis to a netcdf file on offloaded PEs. File must have been opened with open_file_offload.
+  interface register_axis_offload
+    procedure :: register_netcdf_axis_offload
+    procedure :: register_domain_axis_offload


TODO we need a register_unstructured_axis_offload for the unstructured grid

uramirez8707 · 2026-01-08T19:17:19Z

offloading/offloading_io.F90

+
+      !< The number of tiles must be the same as the number of offloading pes
+      !if (size(pe_out) .ne. ntile ) &
+      !  call mpp_error(FATAL, "The number of offloading PEs must be the same as the number of tiles of the domain")


This should be added back, and it should crash if the ntiles is not a multiple of the ntiles

uramirez8707 · 2026-01-08T19:21:09Z

offloading/offloading_io.F90

+      call mpp_broadcast(filename_out, str_len, pe_in(1))
+      call mpp_broadcast(global_domain_size, size(global_domain_size), pe_in(1))
+      call mpp_broadcast(ntile, pe_in(1))


TODO maybe we can find a way to do this all in one mpp_brodcast call (or mpp_send/mpp_receive)

uramirez8707 · 2026-01-08T19:32:53Z

offloading/offloading_io.F90

+    integer, allocatable :: all_current_pes(:)
+    integer, allocatable :: broadcasting_pes(:)
+    logical :: is_model_pe
+    character(len=255) :: var_info(2)


Update to use the max str length variable

uramirez8707 · 2026-01-08T19:34:16Z

offloading/offloading_io.F90

+    if (.not. is_model_pe) then
+      select type(wut=>this%fileobj)
+        type is(FmsNetcdfDomainFile_t)
+          call register_axis(wut, trim(var_info(1)), trim(var_info(2)))
+      end select
+    endif


Let's clean this up

uramirez8707 · 2026-01-08T19:36:44Z

offloading/offloading_io.F90

+    type(domain2D) :: domain_out
+    type(domain2D) :: domain_in
+    integer :: isc, iec, jsc, jec, nz, redistribute_clock
+    character(len=255) :: varname_tmp(1)


use max str length

J-Lentz · 2026-01-08T19:32:29Z

offloading/offloading_io.F90

+  !> Structure to hold offloading file information
+  type :: offloading_obj_out
+    integer :: id
+    character(len=:), allocatable :: filename !< filename of the offloaded netcdf file


Is there a need to store the filename here, as opposed to getting it from fileobj%path?

No looks like it can be removed, I'll add that change.

J-Lentz · 2026-01-08T19:36:28Z

offloading/offloading_io.F90

+    character(len=:), allocatable :: filename !< filename of the offloaded netcdf file
+    class(FmsNetcdfFile_t), allocatable :: fileobj !< fms2_io file object
+    type(domain2D) :: domain_out !< domain on offloading PEs
+    integer :: type_of_domain !< type of domain (domain_decomposed, non_domain_decomposed, Unstructured_grid)


It doesn't seem that type_of_domain is used. Is it needed?

Should be able to remove this as well.

J-Lentz · 2026-01-08T19:37:45Z

offloading/offloading_io.F90

+    integer :: type_of_domain !< type of domain (domain_decomposed, non_domain_decomposed, Unstructured_grid)
+  end type
+
+  !> Offload equivalent of open_file in fms2_io_mod


This comment seems like it belongs above open_file_offload.

J-Lentz · 2026-01-08T19:39:13Z

offloading/offloading_io.F90

+      type_of_domain = domain_decomposed
+    type is (FmsNetcdfUnstructuredDomainFile_t)
+      type_of_domain = Unstructured_grid
+    end select


type_of_domain is assigned, but never used.

J-Lentz · 2026-01-08T19:45:53Z

offloading/offloading_io.F90

+    endif
+
+    if (.not. is_model_pe) then
+      select type(wut=>this%fileobj)


Since FmsNetcdfDomainFile_t has no types derived from it, perhaps change fileobj from class(FmsNetcdfDomainFile_t) to type(FmsNetcdfDomainFile_t), and remove the select type.

J-Lentz · 2026-01-08T19:47:02Z

offloading/offloading_io.F90

+    endif
+
+    if (.not. is_model_pe) then
+      select type(wut=>this%fileobj)


See above comment, regarding class(FmsNetcdfDomainFile_t) to type(FmsNetcdfDomainFile_t).

J-Lentz · 2026-01-08T19:51:25Z

offloading/offloading_io.F90

+      select type(wut=>this%fileobj)
+        type is(FmsNetcdfDomainFile_t)
+          call register_field(wut, trim(var_info(1)), trim(var_info(2)), var_info(3:))
+      end select


This doesn't seem to handle the case where fileobj is FmsNetcdfFile_t as opposed to FmsNetcdfDomainFile_t - is this case supported?

J-Lentz · 2026-01-08T19:59:08Z

offloading/offloading_io.F90

+  !! This routine should be called from both the model PEs and the offload PEs, with the full list
+  !! of pes for each group being provided. The model PEs will broadcast the filename and domain.
+  subroutine open_file_offload(fileobj, filename, domain_in, pe_in, pe_out)
+    class(FmsNetcdfFile_t), intent(inout) :: fileobj !< fms2_io file object


Some other subroutines (e.g., write_data_offload) only seem to support FmsNetcdfDomainFile_t and not the FmsNetcdfFile_t base type. If non-domain-decomposed files are unsupported, type(FmsNetcdfDomainFile_t) should probably be used consistently for fileobj arguments.

uramirez8707 and others added 23 commits November 3, 2023 11:59

#NOTHING_WORKS

f6ebba0

wut

972bc90

push uncommitted updates

06d2405

separate and move modules to new offloading directory

bdde8ed

Merge remote-tracking branch 'origin/main' into HEAD

130ab90

add routines for metadata transfer type

90c0c8c

add constants for allocatable workaround and expand metadata transfer…

710d887

… test

add inherited class structure for reals and ints

f762986

finish up support for all data types and add to test

fbd739e

add metadata object usage to offload moduld

757764b

free memory after redistribute calls and adjust domains total mem add…

1fe352f

…resses

fix for cubic domains

b3770af

add clocks for offload writes/redistributes

0d54864

update for layouts and fix for more than 1 offload pe per tile

9e31d77

fix bug with different layouts and pe counts

6c56515

Merge branch 'main' of github.com:noaa-gfdl/fms into metadata-transfer

1176605

remove attribute_type from base class and any unused variables/args

ed4c178

comments and linter

890bfd6

undo accidental change to data override test

9541c66

add cast for r8 build option to pass

79a3af4

add ifdefs for for mpi-less build

a9ce72c

fix mpi transfer for intel but not gcc :(

18c7178

fix mpi struct bug with gcc and remove mpp test

ecba40d

rem1776 marked this pull request as ready for review December 1, 2025 17:52

rem1776 requested review from bensonr, uramirez8707 and vithikashah001 as code owners December 1, 2025 17:52

Merge branch 'main' into metadata-transfer

b2be463

vithikashah001 requested changes Dec 29, 2025

View reviewed changes

fms2_io/netcdf_io.F90 Show resolved Hide resolved

fms2_io/netcdf_io.F90 Show resolved Hide resolved

offloading/metadata_transfer.F90 Outdated Show resolved Hide resolved

add more documentation and any missing headers

c0ab0dc

J-Lentz self-requested a review January 5, 2026 18:50

fix typo and white space

08a7da9

uramirez8707 requested changes Jan 8, 2026

View reviewed changes

J-Lentz requested changes Jan 8, 2026

View reviewed changes

add io offload prototype #1774

Are you sure you want to change the base?

add io offload prototype #1774

Uh oh!

Conversation

rem1776 commented Sep 22, 2025

Uh oh!

vithikashah001 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uramirez8707 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants