Compiling hoomd for rocm 6.3

[MartinGirard]

My institution just got access to a MI300 cluster; I managed to compile hoomd with rocm 6.3 and performance is good. However, this is quite hacky, and I thought I'd leave problems / solutions with the code base here so we can get this fixed later. This is done with the latest trunk-minor branch, commit 23ebdda.

1. CMake HIP_PLATFORM

To compile on rocm, hoomd sets the HIP_PLATFORM to "hip-clang". This causes issues with the rocm cmake files, which expects this variable to be set to "amd". This requires changing all occurrences of "hip-clang" to "amd".

Files:

• /CMake/hoomd/FindCUDALibs.cmake
• /CMake/hoomd/HOOMDCUDASetup.cmake
• /CMake/hoomd/HOOMDHIPSetup.cmake
• /hoomd/CMakeLists.txt

I have also modified /hoomd/extern/HIP/bin/hipconfig on L99, but it is very unclear whether that was necessary.

This leads to the C++/cuda macro __HIP_PLATFORM_HCC to not be defined. Technically, we could rename all macros to __HIP_PLATFORM_AMD__ but this is tedious. Instead, this can be added directly as definitions manually. I have added to HOOMDCUDASetup.cmake on L58:

• set(CMAKE_HIP_FLAGS "{$CMAKE_HIP_FLAGS} -Wno-unused-result -D__HIP_PLATFORM_HCC__")
• set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -D__HIP_PLATFORM_HCC__")

2. hipfft includes

For reasons detailed in ROCm/hipFFT#90 , hipfft files are not properly added to includes. I had to manually include them to fix this; in FindCUDALibs.cmake L108-111:

find_package(hipfft REQUIRED)
include_directories("${hipfft_INCLUDE_DIR}")
include_directories("${hipfft_INCLUDE_DIR}/hipfft")

3. hip vector types

The cuda vector types, eg float3, have equivalents in hip through hip_vector_type. Multiplication & division operations are now defined on this, which clashes with additions in HOOMDMath.h; the functions inline hoomd::Scalar3 operator* and inline hoomd::Scalar3 operator/ need to be placed with inline hoomd::Scalar3 operator+=, under the macro #if !defined(ENABLE_HIP) || defined(__HIP_PLATFORM_NVCC__)

[joaander]

Thanks for figuring this out and providing a nice summary! Those changes are reasonable. Are they available on a branch that I could clone? I would like to test them with rocm 6.3 on Frontier. If your branch works on Frontier, then we can drop support for rocm5 and merge the changes. If not, we can make the changes conditional on the rocm version.

[MartinGirard]

I'm setup on a weird remote configuration; can you make me a branch called rocm6 from trunk-minor and give me rights to push to it?

[mphoward]

^ It might be easier for you to just make a patch from your diff and drop it here.

[MartinGirard]

patch_rocm.txt

[MartinGirard]

This was compiled with amdclang 18.0; I can't test whether this works correctly on other compilers
AMD clang version 18.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-6.3.0 24455 f24aa3b4a91f6ee2fcd15629ba0b49fa545d8d6b)

[joaander]

Your patch works on Frontier. HOOMD-blue builds with no problem and passes all tests.

Feel free to make a PR from your fork. The changes are not hacks at all, they are necessary changes to adapt to a breaking release of rocm. Once this is merged, we can bump the required rocm to 6.0.

[joaander]

Do you currently have a minimum version of CUDA?

@mphoward Yes, 12.x. I think the x >0, but I don't remember exactly. The CI system builds and tests with 12.4.