Fix cross compiling without emulation #5718

hmaarrfk · 2025-08-10T13:47:38Z

Use CC_FOR_BUILD for H5match_types. Truthfully, this is likely better written in a scripting language... but i think this workaround is fine.
Provided override variables for all defined FORTRAN constants.

Documented fotran bypass variables
Documented the variable used for the H5MATCH_TYPES program
I 100% wrote this with the assistance of AI.
I 30% reviewed this prior making this PR.
I have 100% reviewed from AI slop this before asking for your review
- I still need to do a few cleanups, but it seems to be roughly in good shape

See progress in the OSX ARM builds conda-forge/hdf5-feedstock#268

Important

Enhance HDF5 cross-compilation by using CC_FOR_BUILD for H5match_types and adding Fortran constant overrides.

Cross-Compilation Support:
- Use CC_FOR_BUILD for H5match_types in fortran/src/CMakeLists.txt to support cross-compilation.
- Add override variables for Fortran constants in config/HDF5UseFortran.cmake to bypass runtime tests.
Documentation:
- Update release_docs/INSTALL_CMake.txt to include instructions for Fortran cross-compilation support, detailing environment variables and CMake arguments for setting Fortran constants.

^{This description was created by}^{for ad23ff3. You can customize this summary. It will automatically update as commits are pushed.}

hmaarrfk · 2025-08-10T13:49:28Z

Didn't meant to ping you all, i meant to mark it as "review"

This is being tested in conda-forge/hdf5-feedstock#268

hmaarrfk · 2025-08-11T12:24:23Z

fortran/src/CMakeLists.txt

+if(CMAKE_CROSSCOMPILING)
+    set(H5MATCH_TYPES_COMPILER "$ENV{H5MATCH_TYPES_COMPILER}" CACHE STRING "compiler for build system")
+    if (H5MATCH_TYPES_COMPILER)
+      set_target_properties(H5match_types PROPERTIES
+          C_COMPILER ${H5MATCH_TYPES_COMPILER}
+      )
+      message(STATUS "Using build compiler ${H5MATCH_TYPES_COMPILER} for H5match_types")
+    endif()
+endif()
 target_include_directories (H5match_types PRIVATE "${HDF5_SRC_BINARY_DIR};${HDF5_SRC_INCLUDE_DIRS};${HDF5_F90_BINARY_DIR};$<$<BOOL:${HDF5_ENABLE_PARALLEL}>:${MPI_C_INCLUDE_DIRS}>")

 add_custom_command (TARGET H5match_types POST_BUILD
    BYPRODUCTS ${HDF5_F90_BINARY_DIR}/H5f90i_gen.h ${HDF5_F90_BINARY_DIR}/H5fortran_types.F90
-    COMMAND ${CMAKE_CROSSCOMPILING_EMULATOR} $<TARGET_FILE:H5match_types>
+    COMMAND $<TARGET_FILE:H5match_types>


this whole change is bogus and doesn't work.

We had a similar patch for autotools that had the concept of CC_FOR_BUILD (man don't get me started about the distinction between build and host and target).

But I can't find any documentation where CMake makes this explicit distinction.

https://cmake.org/cmake/help/book/mastering-cmake/chapter/Cross%20Compiling%20With%20CMake.html has info on cross compile and toolchain files

byrnHDF · 2025-08-11T13:12:23Z

config/HDF5UseFortran.cmake


 include (CheckFortranFunctionExists)

+# Cross-compilation support variables


Instead of this, let's use the method we used in 1.12 releases to allow the use of a pre-generated file.

why? a pre-generated file is notoriously difficult to make.

These "overrides" mock the build system early, and ensure more "bug" compatibility, and create a straightforward path for generating these on any machine.

The suggested way, of using cmake variables, is often a common "trope" in CMake.

Preference for auto detection of capabilities.

A defined variable as an override.

all the FindPackages do this kind of stuff, as well as compilers, and other aspects of cmake.

Why should this be any different?

The same process is used. run a configure on an actual target (or create a file) to get the values you need, then supply the values for cross-compile. Read one file to supply the values or supply the values one-by-one in env values. Either way you need the values. The configure is going to create a file for Fortran to use anyway. Personally, a file is easier to deal with, which could be checked in as I don't think there are that many values that will change, Plus we only need to keep the necessary values.

byrnHDF · 2025-08-11T13:45:13Z

While you have listed all the variables to be used, there are secondary variables that must be set to indicate not to rerun checks that need to be set. From CMake doc:
Cross-Compiling Considerations: When cross-compiling, try_run will still attempt to compile the executable but will not execute it on the host system. Instead, it expects the user to manually set cache variables (RUN_RESULT_VAR and potentially RUN_RESULT_VAR__COMPILE_RESULT_VAR__TRYRUN_OUTPUT) to represent the expected results if the program were run on the target platform. This requires pre-setting these variables in a CMake script or providing them during configuration.

byrnHDF · 2025-08-11T13:53:18Z

I think a CMake toolchain file with the values and the variables to prevent configure runs should work. Instead of random pre-generated file or a list of ENV variables. It would be more CMake like and allow also for some simple logic if needed.
Reference: https://cmake.org/cmake/help/book/mastering-cmake/chapter/Cross%20Compiling%20With%20CMake.html

hmaarrfk · 2025-08-11T14:45:34Z

I guess i routinely read the CMake documentation and come out more confused than when I start. I've been reading it for 15+years and this hasn't really changed. The documentation is "templated" with many "placeholders" that add indirection to this example.

I would love it if this section wasn't rerun every time, I think I've noticed the symptoms of it rerunning, but I figured they were harmless.

I strongly believe we should avoid "pre-generated" files, but I defer to whatever implementation you want to go with.

Getting an example from you, in the form of a draft PR or patch against the develop branch would go a long way.

byrnHDF · 2025-08-11T14:58:40Z

Getting an example from you, in the form of a draft PR or patch against the develop branch would go a long way.

Yes, I'm working on that. I need to relearn the steps, and figure out the actual variables we do need. Let's table my pregen file idea and see if we can use the toolchain idea first. I will review your list of variables and first create a toolchain file, which target system do you want to start with? See the toolchains in config/toolchain and see if there is one we can copy from.

byrnHDF · 2025-08-11T15:01:10Z

Also, we will need to figure out the actual values, if your target system matches one we already natively create - I can use the native values that build.

hmaarrfk · 2025-08-11T15:03:27Z

which target system do you want to start with

I think the challenge is that I want to target a build system that is quite popular outside the high performance compute space.

I really want to target:

Running code on an Intel Processor + Mac OS
Executing code for Apple M1 processor + Mac OS

This combination cannot be "emulated" so any "emulation tools" just fail, which creates a great "sentinel" for any holes in the system.

However, i will find

Running code on an Intel Processor + Linux
Executing code on an arm64 Processor + Linux

as an acceptable substitute if you can show that the toolchain file variables are being used as an override at cmake configuration time.

byrnHDF · 2025-08-11T15:09:24Z

as an acceptable substitute if you can show that the toolchain file variables are being used as an override at cmake configuration time.

Working on that - Need to do some digging
All the Fortran try-runs use the same variable name!

hmaarrfk · 2025-08-11T15:14:19Z

All the Fortran try-runs use the same variable name!

lol

hmaarrfk · 2025-08-11T15:19:33Z

All the Fortran try-runs use the same variable name!

You know, its things like this, with the CMake documentation templating that get me really confused.

Then the question is:

is it acceptable to rewrite all the fortran detect code......

byrnHDF · 2025-08-11T15:25:07Z

is it acceptable to rewrite all the fortran detect code......

YES. That is exactly my intention in order to use toolchain properly. The other choice is to use toolchain and skip the Fortran try-runs globally by using if(CMAKE_CROSSCOMPILING)

CMAKE_CROSSCOMPILING is one of the side-effects of using toolchain files.

byrnHDF · 2025-08-11T15:39:31Z

if (NOT CMAKE_CROSSCOMPILING)is used in ConfigureChecks.cmake (twice) and then eiter uses a cache variable or a default:

    else ()
      # Use the default if there's no cache variable and cross-compiling
      if (${def})
        message (VERBOSE "${msg}... yes (cross-compile default)")
        set (${TEST} 1 CACHE INTERNAL ${msg})
      else ()
        message (VERBOSE "${msg}... no (cross-compile default)")
        set (${TEST} "" CACHE INTERNAL ${msg})
      endif ()
    endif ()

That may be a quick compromise - still need to create a toolchain file.

byrnHDF · 2025-08-11T16:47:20Z

config/HDF5UseFortran.cmake

 include (CheckFortranFunctionExists)

+# Cross-compilation support variables
+set(HDF5_FORTRAN_VALID_INT_KINDS "$ENV{HDF5_FORTRAN_VALID_INT_KINDS}" CACHE STRING "Valid Fortran INTEGER kinds (comma-separated)")


So these names are the H5fort_type_defines.h names not the configure variables. We need to set the configure variables (either in HDF5UseFortran.cmake or in a file)

#define H5_FORTRAN_NATIVE_INTEGER_KIND @PAC_FORTRAN_NATIVE_INTEGER_KIND@ #define H5_FORTRAN_NATIVE_INTEGER_SIZEOF @PAC_FORTRAN_NATIVE_INTEGER_SIZEOF@ #define H5_FORTRAN_NATIVE_REAL_KIND @PAC_FORTRAN_NATIVE_REAL_KIND@ #define H5_FORTRAN_NATIVE_REAL_SIZEOF @PAC_FORTRAN_NATIVE_REAL_SIZEOF@ #define H5_FORTRAN_NATIVE_DOUBLE_KIND @PAC_FORTRAN_NATIVE_DOUBLE_KIND@ #define H5_FORTRAN_NATIVE_DOUBLE_SIZEOF @PAC_FORTRAN_NATIVE_DOUBLE_SIZEOF@ #define H5_FORTRAN_NUM_INTEGER_KINDS @PAC_FORTRAN_NUM_INTEGER_KINDS@ #define H5_FORTRAN_INTEGER_KINDS @PAC_FC_ALL_INTEGER_KINDS@ #define H5_FORTRAN_INTEGER_KINDS_SIZEOF @PAC_FC_ALL_INTEGER_KINDS_SIZEOF@ #define H5_FORTRAN_REAL_KINDS @PAC_FC_ALL_REAL_KINDS@ #define H5_FORTRAN_REAL_KINDS_SIZEOF @PAC_FC_ALL_REAL_KINDS_SIZEOF@ #define H5_HAVE_Fortran_INTEGER_SIZEOF_16 @H5_HAVE_Fortran_INTEGER_SIZEOF_16@ #define H5_FORTRAN_HAVE_C_LONG_DOUBLE @H5_FORTRAN_HAVE_C_LONG_DOUBLE@ #define H5_FORTRAN_C_LONG_DOUBLE_IS_UNIQUE @H5_FORTRAN_C_LONG_DOUBLE_IS_UNIQUE@

PAC_FORTRAN_NATIVE_INTEGER_KIND instead H5_FORTRAN_NATIVE_INTEGER_KIND

I would really like to keep this PR as a "reference" since it is the one I'm including in the conda-forge PR, but happy to consider an other solution and drop this PR in its entirety.

No problem, I'm using it mostly as communication - I intend to create a PR or to post concrete solutions..

byrnHDF · 2025-08-11T20:43:33Z

#5720 PR has anoter solution for the HDF5UseFortran file.

byrnHDF · 2025-09-04T19:15:24Z

@hmaarrfk please see PR #5720 for an extensive solution. The CI runner has an issue, but my local fedora system is able to use mingw/wine64 to configure, build and test hdf5 with only 4 failing tests. The tests that fail seem to be with file locking.

byrnHDF · 2025-09-15T12:57:12Z

#5720 incorporates this issue

hmaarrfk requested review from brtnfld, derobins, epourmal, lrknox, byrnHDF, fortnern, jhendersonHDF, qkoziol and vchoi-hdfgroup as code owners August 10, 2025 13:47

github-project-automation bot added this to HDF5 - TRIAGE & TRACK Aug 10, 2025

hmaarrfk requested review from bmribler, glennsong09 and mattjala as code owners August 10, 2025 13:47

github-project-automation bot moved this to To be triaged in HDF5 - TRIAGE & TRACK Aug 10, 2025

hmaarrfk marked this pull request as draft August 10, 2025 13:49

Use cc_for_build

b5615e5

hmaarrfk force-pushed the fix_cross_compiling_without_emulation branch from 2134914 to 69854c9 Compare August 10, 2025 14:06

Provide a means to override Fortran runtime determined constants

e3ca87c

This is being tested in conda-forge/hdf5-feedstock#268

hmaarrfk force-pushed the fix_cross_compiling_without_emulation branch from 69854c9 to e3ca87c Compare August 10, 2025 14:10

Try to fixup patch again

364853a

hmaarrfk mentioned this pull request Aug 10, 2025

Hdf5 2 conda-forge/hdf5-feedstock#268

Draft

15 tasks

hmaarrfk added 4 commits August 10, 2025 11:18

fixup braces

0278c73

Fixup patch

6081aa9

add new line

a47d20a

Fixup again

7cdee28

hmaarrfk mentioned this pull request Aug 11, 2025

Cross compilation with CMake + other conda-forge wishlist items #5709

Open

hmaarrfk commented Aug 11, 2025

View reviewed changes

byrnHDF reviewed Aug 11, 2025

View reviewed changes

byrnHDF mentioned this pull request Aug 11, 2025

Rework Fortran configure to allow cross compile overrides #5720

Open

glennsong09 assigned byrnHDF Aug 12, 2025

byrnHDF closed this Sep 15, 2025

github-project-automation bot moved this from To be triaged to Done in HDF5 - TRIAGE & TRACK Sep 15, 2025


		include (CheckFortranFunctionExists)

		# Cross-compilation support variables

Uh oh!

Fix cross compiling without emulation #5718

Fix cross compiling without emulation #5718

Uh oh!

Conversation

hmaarrfk commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hmaarrfk commented Aug 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

byrnHDF commented Aug 11, 2025

Uh oh!

byrnHDF commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hmaarrfk commented Aug 11, 2025

Uh oh!

byrnHDF commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

byrnHDF commented Aug 11, 2025

Uh oh!

hmaarrfk commented Aug 11, 2025

Uh oh!

byrnHDF commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hmaarrfk commented Aug 11, 2025

Uh oh!

hmaarrfk commented Aug 11, 2025

Uh oh!

byrnHDF commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

byrnHDF commented Aug 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

byrnHDF commented Aug 11, 2025

Uh oh!

byrnHDF commented Sep 4, 2025

Uh oh!

byrnHDF commented Sep 15, 2025

Uh oh!

Uh oh!

hmaarrfk commented Aug 10, 2025 •

edited

Loading

byrnHDF commented Aug 11, 2025 •

edited

Loading

byrnHDF commented Aug 11, 2025 •

edited

Loading

byrnHDF commented Aug 11, 2025 •

edited

Loading

byrnHDF commented Aug 11, 2025 •

edited

Loading