Skip to content

Improves test throughput, especially for threaded tests #1749

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kennyweiss wants to merge 10 commits into
base: develop
Choose a base branch
from
Open

Improves test throughput, especially for threaded tests #1749

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
4e0b17d
Adds new CMake option for the default number of OpenMP threads -- AXO…
kennyweiss Dec 21, 2025
1b593b8
Adds NUM_OMP_THREADS keyword to omp-based tests
kennyweiss Dec 21, 2025
144380b
Reduces resolution of quest_shape_in tests
kennyweiss Dec 22, 2025
414e1c2
Adds OMP_NUM_THREADS=1 to non-omp tests in omp configurations
kennyweiss Dec 22, 2025
2c97d4f
Prefer axom_add_test to blt_add_test
kennyweiss Dec 22, 2025
807cd57
Improves consistency of OpenMP setup in threaded tests
kennyweiss Dec 22, 2025
d1d1d39
Improves consistency of MPI setup in parallel tests
kennyweiss Dec 22, 2025
a10bf77
Improves consistency in expressing num threads and ranks
kennyweiss Dec 22, 2025
e3e5fa3
Updates RELEASE-NOTES
kennyweiss Dec 22, 2025
db0ae2b
Reduces resolution for heroic_roses tests
kennyweiss Dec 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions RELEASE-NOTES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ The Axom project release numbers follow [Semantic Versioning](http://semver.org/
or when `components="all"`), all components are enabled. All components can be disabled via `components=none`.
- Adds a `conduit` spack variant. Conduit was previously a required dependency in our spack package, and is now enabled by default.
- Adds spack variants for `adiak` and `caliper`. These replace the previous `profiling` variant which enabled both at the same time.
- Adds the `AXOM_TEST_NUM_OMP_THREADS` configuration variable to control the default OpenMP thread count for tests.
Copy link
Member

@rhornung67 rhornung67 Dec 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Adds the `AXOM_TEST_NUM_OMP_THREADS` configuration variable to control the default OpenMP thread count for tests.
- Adds the `AXOM_TEST_OMP_NUM_THREADS` configuration variable to control the default OpenMP thread count for tests.

Minor nit. I think it's better to be consistent with the OMP_NUM_THREADS env variable or OpenMP library function with a similar name. Here and throughout.

Also, the terms ranks and tasks are used interchangeably as they relate to MPI. Should that be made consistent also?

Copy link
Member Author

@kennyweiss kennyweiss Dec 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for consistency, but it might be tricky here.

The keywords/parameters for blt_add_test are NUM_MPI_TASKS and NUM_OMP_THREADS, while the ENV variable for OpenMP is OMP_NUM_THREADS. It's not clear that renaming the blt parameters would be appropriate or an improvement, and would affect all blt users.

What do you suggest?

Copy link
Contributor

@Arlie-Capps Arlie-Capps Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aargh--- naming is important and consistency is a desirable quality. I would say, let's match blt_add_test, and in any discussion let's point out the discrepancy to OpenMP environment variables.

Copy link
Member

@rhornung67 rhornung67 Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably best to discuss with Chris and others. I would defer to OpenMP which is a common, well-known standard, whereas BLT it not.

Copy link
Member Author

@kennyweiss kennyweiss Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be confusing to switch it. This is an existing macro that's been in the code for a long time and is entirely Axom developer focused.

We're calling axom_add_test() which directly calls blt_add_test() and all other keywords/parameters are the same.

Copy link
Member

@white238 white238 Dec 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My vote is to match BLT since it is influencing a BLT macro. The reason it is named that way is because it follows the pattern of the BLT MPI variable which was created long before the OpenMP parameter which was before I knew that OMP_NUM_THREADS environment variable existed.


### Changed
- Treatment of materials on strided-structured Blueprint meshes has changed in `axom::mir`.
Expand Down
4 changes: 2 additions & 2 deletions src/axom/bump/tests/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ foreach(test ${gtest_bump_tests})
DEPENDS_ON ${bump_tests_depends_on}
FOLDER axom/bump/tests )

blt_add_test( NAME ${test_name}
COMMAND ${test_name}_test )
axom_add_test( NAME ${test_name}
COMMAND ${test_name}_test )
endforeach()

30 changes: 20 additions & 10 deletions src/axom/core/examples/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,26 +73,36 @@ if(AXOM_ENABLE_TESTS)
# Run the core array performance example on N ranks for each enabled policy
set(_policies "seq")
if(RAJA_FOUND)
blt_list_append(TO _policies ELEMENTS "omp" IF AXOM_ENABLE_OPENMP)
blt_list_append(TO _policies ELEMENTS "omp" IF AXOM_ENABLE_OPENMP)
blt_list_append(TO _policies ELEMENTS "cuda" IF AXOM_ENABLE_CUDA)
blt_list_append(TO _policies ELEMENTS "hip" IF AXOM_ENABLE_HIP)
blt_list_append(TO _policies ELEMENTS "hip" IF AXOM_ENABLE_HIP)
endif()
foreach(_pol ${_policies})
# sets the number of threads for the omp policy; leave empty for non-omp policies
set(_num_threads)
if(_pol STREQUAL "omp")
set(_num_threads ${AXOM_TEST_NUM_OMP_THREADS})
endif()

axom_add_test(
NAME "core_array_perf_1d_${_pol}"
COMMAND core_array_perf_ex --policy ${_pol} --shape 1200000 -r 10)
NAME core_array_perf_1d_${_pol}
COMMAND core_array_perf_ex --policy ${_pol} --shape 1200000 -r 10
NUM_OMP_THREADS ${_num_threads})
Comment on lines +81 to +90
Copy link
Member Author

@kennyweiss kennyweiss Dec 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: This PR relies on allowing empty arguments to NUM_OMP_THREADS, in which case if(arg_NUM_OMP_THREADS) will evaluate to false

axom_add_test(
NAME "core_array_perf_2d_${_pol}"
COMMAND core_array_perf_ex --policy ${_pol} --shape 1000 1200 -r 10)
NAME core_array_perf_2d_${_pol}
COMMAND core_array_perf_ex --policy ${_pol} --shape 1000 1200 -r 10
NUM_OMP_THREADS ${_num_threads})
axom_add_test(
NAME "core_array_perf_3d_${_pol}"
COMMAND core_array_perf_ex --policy ${_pol} --shape 100 100 120 -r 10)
NAME core_array_perf_3d_${_pol}
COMMAND core_array_perf_ex --policy ${_pol} --shape 100 100 120 -r 10
NUM_OMP_THREADS ${_num_threads})
if(NOT RAJA_FOUND)
# RAJA provides support for up to 3D tiled nested loops,
# so run 4D test only for non-RAJA builds.
axom_add_test(
NAME "core_array_perf_4d_${_pol}"
COMMAND core_array_perf_ex --policy ${_pol} --shape 50 20 30 6 )
NAME core_array_perf_4d_${_pol}
COMMAND core_array_perf_ex --policy ${_pol} --shape 50 20 30 6
NUM_OMP_THREADS ${_num_threads})
endif()
endforeach()
endif()
Expand Down
20 changes: 11 additions & 9 deletions src/axom/core/tests/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,9 +140,9 @@ if (AXOM_ENABLE_OPENMP AND RAJA_FOUND)

foreach( test_suite ${core_openmp_tests} )
get_filename_component( test_suite ${test_suite} NAME_WE )
axom_add_test( NAME ${test_suite}_openmp
COMMAND core_openmp_tests --gtest_filter=${test_suite}*
NUM_OMP_THREADS 24 )
axom_add_test( NAME ${test_suite}_openmp
COMMAND core_openmp_tests --gtest_filter=${test_suite}*
NUM_OMP_THREADS ${AXOM_TEST_NUM_OMP_THREADS} )
endforeach()
endif()

Expand Down Expand Up @@ -183,14 +183,16 @@ axom_add_executable(NAME utils_annotations_test

foreach(_mode none report counts gputx)
set(_test_name utils_annotations_${_mode}_test)

set(_num_ranks 1)
if(AXOM_ENABLE_MPI)
axom_add_test(NAME ${_test_name}
COMMAND utils_annotations_test -m ${_mode}
NUM_MPI_TASKS 2)
else()
axom_add_test(NAME ${_test_name}
COMMAND utils_annotations_test -m ${_mode})
set(_num_ranks 2)
endif()
Comment on lines 186 to 193
Copy link
Member Author

@kennyweiss kennyweiss Dec 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: A recent BLT PR allows for NUM_MPI_TASKS to be 1 in non-mpi configs.
This allows us to simplify some test setup logic.


axom_add_test(NAME ${_test_name}
COMMAND utils_annotations_test -m ${_mode}
NUM_MPI_TASKS ${_num_ranks})

set_property(TEST ${_test_name}
APPEND PROPERTY ENVIRONMENT "CALI_LOG_VERBOSITY=2")
endforeach()
Expand Down
2 changes: 1 addition & 1 deletion src/axom/klee/tests/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ foreach(test ${gtest_klee_tests})
OUTPUT_DIR ${TEST_OUTPUT_DIRECTORY}
DEPENDS_ON klee_test_utils
FOLDER axom/klee/tests )
blt_add_test( NAME ${test_name}
axom_add_test(NAME ${test_name}
COMMAND ${test_name}_test )
endforeach()

83 changes: 44 additions & 39 deletions src/axom/mint/tests/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,57 +7,62 @@
#------------------------------------------------------------------------------

set( mint_tests
## execution model tests
mint_execution_cell_traversals.cpp
mint_execution_node_traversals.cpp
mint_execution_face_traversals.cpp
mint_execution_xargs.cpp

## execution model tests
mint_execution_cell_traversals.cpp
mint_execution_node_traversals.cpp
mint_execution_face_traversals.cpp
mint_execution_xargs.cpp

## mesh tests
mint_mesh.cpp
mint_mesh_cell_types.cpp
mint_mesh_blueprint.cpp
mint_mesh_connectivity_array.cpp
mint_mesh_coordinates.cpp
mint_mesh_curvilinear_mesh.cpp
mint_mesh_field.cpp
mint_mesh_field_data.cpp
mint_mesh_field_types.cpp
mint_mesh_field_variable.cpp
mint_mesh_particle_mesh.cpp
mint_mesh_rectilinear_mesh.cpp
mint_mesh_uniform_mesh.cpp
mint_mesh_unstructured_mesh.cpp
mint_mesh_face_relation.cpp

## fem tests
mint_fem_shape_functions.cpp
mint_fem_single_fe.cpp

## util tests
mint_su2_io.cpp
mint_util_write_vtk.cpp

## container tests
mint_deprecated_mcarray.cpp
## mesh tests
mint_mesh.cpp
mint_mesh_cell_types.cpp
mint_mesh_blueprint.cpp
mint_mesh_connectivity_array.cpp
mint_mesh_coordinates.cpp
mint_mesh_curvilinear_mesh.cpp
mint_mesh_field.cpp
mint_mesh_field_data.cpp
mint_mesh_field_types.cpp
mint_mesh_field_variable.cpp
mint_mesh_particle_mesh.cpp
mint_mesh_rectilinear_mesh.cpp
mint_mesh_uniform_mesh.cpp
mint_mesh_unstructured_mesh.cpp
mint_mesh_face_relation.cpp

## fem tests
mint_fem_shape_functions.cpp
mint_fem_single_fe.cpp

## util tests
mint_su2_io.cpp
mint_util_write_vtk.cpp

## container tests
mint_deprecated_mcarray.cpp
)

foreach( test ${mint_tests} )

get_filename_component( test_name ${test} NAME_WE )
get_filename_component( test_name ${test} NAME_WE )

axom_add_executable(
axom_add_executable(
NAME ${test_name}_test
SOURCES ${test}
OUTPUT_DIR ${TEST_OUTPUT_DIRECTORY}
DEPENDS_ON mint gtest
FOLDER axom/mint/tests
)

axom_add_test( NAME ${test_name}
COMMAND ${test_name}_test
NUM_OMP_THREADS 4
)
# sets the number of threads for the omp policy; leave empty for non-omp policies
set(_num_threads)
if(AXOM_ENABLE_OPENMP)
set(_num_threads ${AXOM_TEST_NUM_OMP_THREADS})
endif()

axom_add_test( NAME ${test_name}
COMMAND ${test_name}_test
NUM_OMP_THREADS ${_num_threads}
)

endforeach()
9 changes: 8 additions & 1 deletion src/axom/mir/examples/concentric_circles/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,15 +39,22 @@ if(AXOM_ENABLE_TESTS)
endif()

foreach(_policy ${_policies})
# sets the number of threads for the omp policy; leave empty for non-omp policies
set(_num_threads)
if(_policy STREQUAL "omp")
set(_num_threads ${AXOM_TEST_NUM_OMP_THREADS})
endif()

set(_testname "mir_concentric_circles_${_policy}")

axom_add_test(
NAME ${_testname}
COMMAND mir_concentric_circles
--gridsize 100
--numcircles 5
--policy ${_policy}
--disable-write
)
NUM_OMP_THREADS ${_num_threads})

set_tests_properties(${_testname} PROPERTIES
PASS_REGULAR_EXPRESSION "Material interface reconstruction time:")
Expand Down
9 changes: 7 additions & 2 deletions src/axom/mir/examples/heavily_mixed/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ if(AXOM_ENABLE_TESTS)
endif()

foreach(_policy ${_policies})
set(_num_threads)
if(_policy STREQUAL "omp")
set(_num_threads ${AXOM_TEST_NUM_OMP_THREADS})
endif()

set(_testname "mir_heavily_mixed_2D_${_policy}")
axom_add_test(
NAME ${_testname}
Expand All @@ -37,7 +42,7 @@ if(AXOM_ENABLE_TESTS)
--materials 10
--policy ${_policy}
--disable-write
)
NUM_OMP_THREADS ${_num_threads})

set_tests_properties(${_testname} PROPERTIES
PASS_REGULAR_EXPRESSION "Material interface reconstruction time:")
Expand All @@ -51,7 +56,7 @@ if(AXOM_ENABLE_TESTS)
--materials 10
--policy ${_policy}
--disable-write
)
NUM_OMP_THREADS ${_num_threads})

set_tests_properties(${_testname} PROPERTIES
PASS_REGULAR_EXPRESSION "Material interface reconstruction time:")
Expand Down
7 changes: 6 additions & 1 deletion src/axom/mir/examples/tutorial_simple/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@ if(AXOM_ENABLE_TESTS)
endif()

foreach(_policy ${_policies})
set(_num_threads)
if(_policy STREQUAL "omp")
set(_num_threads ${AXOM_TEST_NUM_OMP_THREADS})
endif()

foreach(_testnum ${_test_numbers})
set(_testname "mir_tutorial_simple_${_policy}_${_testnum}")
axom_add_test(
Expand All @@ -56,7 +61,7 @@ if(AXOM_ENABLE_TESTS)
--test-case ${_testnum}
--policy ${_policy}
--disable-write
)
NUM_OMP_THREADS ${_num_threads})

set_tests_properties(${_testname} PROPERTIES
PASS_REGULAR_EXPRESSION "Reconstruction time:")
Expand Down
4 changes: 2 additions & 2 deletions src/axom/mir/tests/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ foreach(test ${gtest_mir_tests})
DEPENDS_ON ${mir_tests_depends_on}
FOLDER axom/mir/tests )

blt_add_test( NAME ${test_name}
COMMAND ${test_name}_test )
axom_add_test( NAME ${test_name}
COMMAND ${test_name}_test )
endforeach()

2 changes: 1 addition & 1 deletion src/axom/multimat/examples/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ foreach(example_source ${multimat_example_sources})
)
unset(deps)

blt_add_test(
axom_add_test(
NAME multimat_${example_name}
COMMAND multimat_${example_name}_ex
)
Expand Down
4 changes: 2 additions & 2 deletions src/axom/multimat/tests/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,6 @@ foreach(test ${gtest_multimat_tests})
OUTPUT_DIR ${TEST_OUTPUT_DIRECTORY}
DEPENDS_ON multimat gtest
FOLDER axom/multimat/tests )
blt_add_test( NAME ${test_name}
COMMAND ${test_name}_test )
axom_add_test( NAME ${test_name}
COMMAND ${test_name}_test )
endforeach()
12 changes: 8 additions & 4 deletions src/axom/primal/examples/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,13 +51,17 @@ if (RAJA_FOUND AND UMPIRE_FOUND)
blt_list_append(TO _policies ELEMENTS "hip" IF AXOM_ENABLE_HIP)

foreach(_policy ${_policies})
# sets the number of threads for the omp policy; leave empty for non-omp policies
set(_num_threads)
if(_policy STREQUAL "omp")
set(_num_threads ${AXOM_TEST_NUM_OMP_THREADS})
endif()

set(_testname "primal_hex_tet_volume_ex_${_policy}")
axom_add_test(
NAME ${_testname}
COMMAND hex_tet_volume_ex
--policy ${_policy}
)
NAME ${_testname}
COMMAND hex_tet_volume_ex --policy ${_policy}
NUM_OMP_THREADS ${_num_threads})

set_tests_properties(${_testname} PROPERTIES
PASS_REGULAR_EXPRESSION "Difference between sums")
Expand Down
Loading