Framework: Enable building of tests but skip running tests in cuda-uvm PR builds

cmake/SimpleTesting/README.md

@@ -1,20 +1,3 @@
-CMake File Structure
-====================
-This CMake structure attempts to capture the structure of the original
-`TFW_testing_single_configure_prototype` that is used by the current / old Trilinos
-Pull-Request (PR) framework.
-
-**Note**: _Trilinos uses the **`pull_request_changes`** branch from this repository_.
-
-Specifically, this new configuration will replace the existing `simple_testing.cmake`
-script.
-
-This diagram describes the current structure of the CMake files and how they
-interact with each other.
-<center>
-     <img src="img/CMake-structure.png" alt="CMake Structure Diagram" height="600"/>
-</center>
-
 CMake Files
 -----------
 
@@ -32,42 +15,6 @@ CMake Files
 The _guarded_ files use the CMake command [`include_guard()`][1] which should prevent that file
 from being include more than once in an include chain.
 
-Options and Variables (`simple_testing.cmake`)
-----------------------------------------------
-The `simple_testing.cmake` file has a number of optional parameters that can be sent into the
-CMake system via `-D<VARNAME>:BOOL=<ON|OFF>` parameters:
-
-| Option                     | Required? | Default                                     | PR Override | Purpose                          |
-|----------------------------|:---------:|---------------------------------------------|-------------|----------------------------------|
-| `build_name`               |    YES    | N/A                                         | YES         | Sets `CTEST_BUILD_NAME`          |
-| `ctest_submit_retry_count` |    NO     | 5                                           |             |                                  |
-| `ctest_submit_retry_delay` |    NO     | 3                                           |             |                                  |
-| `dashboard_model`          |    NO     | `Experimental`                              | YES but NO  | Set to the same value as default |
-| `dashboard_track`          |    NO     | `Experimental`                              | YES but NO  | Set to the same value as default |
-| `skip_by_parts_submit`     |    NO     | `ON`                                        | YES         |                                  |
-| `skip_clean_build_dir`     |    NO     | `ON`                                        |             |                                  |
-| `skip_single_submit`       |    NO     | `ON`                                        |             |                                  |
-| `skip_update_step`         |    NO     | `OFF`                                       | YES         |                                  |
-| `skip_upload_config_files` |    NO     | `OFF`                                       |             |                                  |
-| `PARALLEL_LEVEL`           |    NO     | _num cores_                                 | YES         |                                  |
-| `TEST_PARALLEL_LEVEL`      |    NO     | `${PARALLEL_LEVEL}`                         | YES         |                                  |
-| `build_root`               |    NO     | `${CTEST_SOURCE_DIRECTORY}/nightly_testing` |             |                                  |
-| `build_dir`                |    NO     | `${build_root}/${CTEST_BUILD_NAME}`         | YES         | The CMake build dir              |
-| `configure_script`         |    YES    | N/A                                         | YES         | See note below table             |
-| `package_enables`          |    YES    | N/A                                         | YES         | `packageEnables.cmake`           |
-| `subprojects_file`         |    YES    | N/A                                         | YES         | `package_subproject_list.cmake`  |
-
-1. `configure_script` points to the `cmake/std/PullRequestLinux<COMPILER><COMPILER_VERSION>TestingSettings.cmake` file.
-    - Example: `${WORKSPACE}/Trilinos/cmake/std/PullRequestLinuxGCC8.3.0TestingSettings.cmake`
-
-See `TrilinosPRConfigurationStandard.py`[2] for information on what options are set to something
-other than the default during normal Trilinos PR operations.
-
-Expected Operation
-------------------
-The expected operation of this set of files to replace the old `simple_testing.cmake` is to load
-the [`ctest-driver.cmake`](ctest-driver.cmake) file in its place.
-
 
 `ctest-driver.cmake` Options
 ----------------------------
@@ -92,20 +39,21 @@ the [`ctest-driver.cmake`](ctest-driver.cmake) file in its place.
 | `build_dir`                | STRING |    NO     | `${build_root}/${CTEST_BUILD_NAME}`           | Path to the build directory.                                       |
 | `PARALLEL_LEVEL`           | STRING |    NO     | `<num cores>`                                 |                                                                    |
 | `TEST_PARALLEL_LEVEL`      | STRING |    NO     | `${PARALLEL_LEVEL}`                           |                                                                    |
+| `SKIP_RUN_TESTS`           | BOOL   |    NO     | OFF                                           | Skip running any tests (any tests enabled will still compile)      |
+
 
 1. It might worthwhile to remove `build_root` since it's only used to create `build_dir` IF `build_dir` is not passed in
    via a `-D` option.
 2. Related to (1), we might also change `build_dir` to be `BUILD_DIR` and pass that in.
 
 
-
 Example CTest call from a Trilinos PR
 -------------------------------------
-This is an example, for reference, of how the `ctest` command is invoked in the current/old Trilinos
+This is an example, for reference, of how the `ctest` command is invoked in the current Trilinos
 PR test driver.
 ```bash
 ctest \
-   -S simple_testing.cmake \
+   -S ctest-driver.cmake \
    -Dsource_dir=${WORKSPACE}/Trilinos \
    -Dbuild_name=PR-9495-test-Trilinos_pullrequest_gcc_8.3.0-5164 \
    -Dskip_by_parts_submit=OFF \
@@ -120,6 +68,9 @@ ctest \
    -Dsubprojects_file=../package_subproject_list.cmake
 ```
 
+See `TrilinosPRConfigurationStandard.py`[2] for information on what options are set to something
+other than the default during normal Trilinos PR operations.
+
 Additional Notes and Pitfalls
 =============================
 
@@ -136,4 +87,4 @@ when only documentation or perhaps the testing framework itself is modified and
 not need to spend O(5 hours) for the test suite to run.
 
 [1]: https://cmake.org/cmake/help/latest/command/include_guard.html
-[2]: https://github.com/trilinos/Trilinos/blob/master/cmake/std/trilinosprhelpers/TrilinosPRConfigurationStandard.py
+[2]: https://github.com/trilinos/Trilinos/blob/master/packages/framework/pr_tools/trilinosprhelpers/TrilinosPRConfigurationStandard.py

cmake/SimpleTesting/cmake/ctest-common.cmake

@@ -76,6 +76,9 @@ if( NOT DEFINED skip_upload_config_files )
     set( skip_upload_config_files OFF )
 endif()
 
+if( NOT DEFINED SKIP_RUN_TESTS )
+    set (SKIP_RUN_TESTS OFF)
+endif()
 
 # -----------------------------------------------------------
 # -- Miscellaneous Settings

cmake/SimpleTesting/cmake/ctest-stage-test.cmake

@@ -13,20 +13,26 @@ banner("START test step")
 
 set(STAGE_TEST_ERROR OFF)
 
-if(CTEST_BUILD_NAME MATCHES .*_asan_.*)
-    set(CTEST_MEMORYCHECK_TYPE "AddressSanitizer")
-    set(ENV{LSAN_OPTIONS} "suppressions=${CTEST_SOURCE_DIRECTORY}/packages/framework/asan_assets/lsan.supp")
-    set(ENV{LD_PRELOAD} ${CTEST_SOURCE_DIRECTORY}/packages/framework/asan_assets/dummy_dlclose.so)
-    ctest_memcheck(PARALLEL_LEVEL ${TEST_PARALLEL_LEVEL}
+if(NOT SKIP_RUN_TESTS)
+    if(CTEST_BUILD_NAME MATCHES .*_asan_.*)
+        set(CTEST_MEMORYCHECK_TYPE "AddressSanitizer")
+        set(ENV{LSAN_OPTIONS} "suppressions=${CTEST_SOURCE_DIRECTORY}/packages/framework/asan_assets/lsan.supp")
+        set(ENV{LD_PRELOAD} ${CTEST_SOURCE_DIRECTORY}/packages/framework/asan_assets/dummy_dlclose.so)
+        ctest_memcheck(PARALLEL_LEVEL ${TEST_PARALLEL_LEVEL}
+                       CAPTURE_CMAKE_ERROR captured_cmake_error
+                       RETURN_VALUE test_error)
+        unset(ENV{LD_PRELOAD})
+        submit_by_parts( "MemCheck" )
+    else()
+        ctest_test(PARALLEL_LEVEL ${TEST_PARALLEL_LEVEL}
                    CAPTURE_CMAKE_ERROR captured_cmake_error
                    RETURN_VALUE test_error)
-    unset(ENV{LD_PRELOAD})
-    submit_by_parts( "MemCheck" )
+        submit_by_parts( "Test" )
+    endif()
 else()
-    ctest_test(PARALLEL_LEVEL ${TEST_PARALLEL_LEVEL}
-               CAPTURE_CMAKE_ERROR captured_cmake_error
-               RETURN_VALUE test_error)
-    submit_by_parts( "Test" )
+    message(">>> SKIPPED RUNNING TESTS (SKIP_RUN_TESTS=${SKIP_RUN_TESTS})")
+    set(test_error 0)
+    submit_by_parts("Test")
 endif()
 
 # Print out final stage banner

packages/framework/ini-files/config-specs.ini

@@ -1517,16 +1517,17 @@ use PACKAGE-ENABLES|NO-EPETRA
 use COMMON_SPACK_TPLS
 use SEMS_CUDA
 
-opt-set-cmake-var Trilinos_ENABLE_TESTS BOOL FORCE : OFF
+# If run through PR testing, PR scripts will control whether tests
+# should build and run, or ONLY build (TRILFRAME-675)
+opt-set-cmake-var Trilinos_ENABLE_TESTS BOOL FORCE : ON
 
 [rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_all]
 use rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
-use PACKAGE-ENABLES|ALL-NO-EPETRA
+use PACKAGE-ENABLES|ALL
 
 [rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_all-no-epetra]
-use rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_all
-
-opt-set-cmake-var Trilinos_ENABLE_TESTS BOOL FORCE : ON
+use rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
+use PACKAGE-ENABLES|ALL-NO-EPETRA
 
 [rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables]
 # uses sems-v2 modules

packages/framework/pr_tools/PullRequestLinuxDriver.sh

@@ -279,6 +279,11 @@ then
     test_cmd_options+=( "--skip-create-packageenables ")
 fi
 
+if [[ ${GENCONFIG_BUILD_NAME} == *"_uvm_"* && ${GENCONFIG_BUILD_NAME} == *"no-package-enables"* ]]
+then
+    test_cmd_options+=( "--skip-run-tests" )
+fi
+
 test_cmd="${PYTHON_EXE:?} ${REPO_ROOT:?}/packages/framework/pr_tools/PullRequestLinuxDriverTest.py ${test_cmd_options[@]}"
 
 # Call the script to launch the tests

packages/framework/pr_tools/PullRequestLinuxDriverTest.py

@@ -258,6 +258,12 @@ def parse_args():
                           default=False,
                           help="Enable dry-run mode. Script will run but not execute the build steps. Default = %(default)s")
 
+    optional.add_argument("--skip-run-tests",
+                          dest="skip_run_tests",
+                          action="store_true",
+                          default=False,
+                          help="Skip running tests in SimpleTesting test stage. Default = %(default)s")
+
     optional.add_argument("--extra-configure-args",
                           dest="extra_configure_args",
                           action="store",
@@ -299,6 +305,7 @@ def parse_args():
     print("| - [O] req-mem-per-core            : {req_mem_per_core}".format(**vars(arguments)))
     print("| - [O] test-mode                   : {test_mode}".format(**vars(arguments)))
     print("| - [O] workspace-dir               : {workspace_dir}".format(**vars(arguments)))
+    print("| - [O] skip-run-tests              : {skip_run_tests}".format(**vars(arguments)))
     print("| - [O] extra_configure_args        : {extra_configure_args}".format(**vars(arguments)))
     print("| - [O] dashboard_build_name        : {dashboard_build_name}".format(**vars(arguments)))
     print("| - [O] use_explicit_cachefile       : {use_explicit_cachefile}".format(**vars(arguments)))

packages/framework/pr_tools/trilinosprhelpers/TrilinosPRConfigurationBase.py

@@ -352,6 +352,15 @@ def arg_ccache_enable(self):
         """Is ccache enabled?"""
         return self.args.ccache_enable
 
+    @property
+    def arg_skip_run_tests(self):
+        """
+        Control whether tests should run for this build. Used for cases
+        where resources are limited such that you choose to only compile tests
+        but skip running them.
+        """
+        return self.args.skip_run_tests
+
     # --------------------
     # P R O P E R T I E S
     # --------------------

packages/framework/pr_tools/trilinosprhelpers/TrilinosPRConfigurationStandard.py

@@ -87,6 +87,7 @@ def execute_test(self):
                f"-Dsubprojects_file:FILEPATH={self.arg_filename_subprojects}",
                f"-DCTEST_DROP_SITE:STRING={self.arg_ctest_drop_site}",
                 "-DUSE_EXPLICIT_TRILINOS_CACHEFILE:BOOL=" + ("ON" if self.arg_use_explicit_cachefile else "OFF"),
+                "-DSKIP_RUN_TESTS:BOOL=" + ("ON" if self.arg_skip_run_tests else "OFF"),
              ]
 
         if self.arg_extra_configure_args:

packages/framework/pr_tools/trilinosprhelpers/unittests/test_TrilinosPRConfigurationStandard.py

@@ -175,7 +175,8 @@ def dummy_args(self):
             ccache_enable=False,
             dry_run = False,
             use_explicit_cachefile = False,
-            extra_configure_args = ""
+            extra_configure_args = "",
+            skip_run_tests = False
         )
         return output
 

packages/framework/pr_tools/unittests/test_PullRequestLinuxDriverTest.py

@@ -81,6 +81,7 @@ def setUp(self):
                                          filename_packageenables='../packageEnables.cmake',
                                          filename_subprojects='../package_subproject_list.cmake',
                                          skip_create_packageenables=False,
+                                         skip_run_tests=False,
                                          test_mode='standard',
                                          req_mem_per_core=3.0,
                                          max_cores_allowed=12,