[Question] Select the compiler used to generate tests

[josephgarnier]

Hello dead dev,

On Windows, the compiler selected by CMakeTest to generate or run tests is MSVC.

How can I select a different one? I couldn't figure out how to change this parameter despite 2 hours of digging through the code.

I'm using GCC to compile a static lib for testing CMake modules and the Makefile generator. It therefore has the '.a' extension. But then my tests fail, because MSVC expects to find a lib with the extension '.lib'.

Here's an example of how the choice of compiler affects CMake variable values:

# With GCC
# CMAKE_SHARED_LIBRARY_PREFIX: lib
# CMAKE_SHARED_LIBRARY_SUFFIX: .dll
# CMAKE_STATIC_LIBRARY_PREFIX: lib
# CMAKE_STATIC_LIBRARY_SUFFIX: .a
# CMAKE_FIND_LIBRARY_PREFIXES: lib;
# CMAKE_FIND_LIBRARY_SUFFIXES: .dll.a;.a;.lib

# With MSVC
# CMAKE_SHARED_LIBRARY_PREFIX: 
# CMAKE_SHARED_LIBRARY_SUFFIX: .dll
# CMAKE_STATIC_LIBRARY_PREFIX: 
# CMAKE_STATIC_LIBRARY_SUFFIX: .lib
# CMAKE_FIND_LIBRARY_PREFIXES: ;lib
# CMAKE_FIND_LIBRARY_SUFFIXES: .dll.lib;.lib;.a

[ryanmrichard]

Just to check, you are setting CMAKE_C_COMPILER and CMAKE_CXX_COMPILER to the compiler you want when you call cmake, right?

[josephgarnier]

Sure, I use my preset "x64-Release-Win-GCC".

Here is a part of CMakeCache.txt:

//Path to a program.
CMAKE_ADDR2LINE:FILEPATH=C:/msys64/mingw64/bin/addr2line.exe

//Path to a program.
CMAKE_AR:FILEPATH=C:/msys64/mingw64/bin/ar.exe

//Choose the type of build, options are: None Debug Release RelWithDebInfo
// MinSizeRel ...
CMAKE_BUILD_TYPE:STRING=

//Enable/Disable color output during build.
CMAKE_COLOR_MAKEFILE:BOOL=ON

//A wrapper around 'ar' adding the appropriate '--plugin' option
// for the GCC compiler
CMAKE_CXX_COMPILER_AR:FILEPATH=C:/msys64/mingw64/bin/gcc-ar.exe

And here is the console log:

-- The CXX compiler identification is GNU 14.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/msys64/mingw64/bin/g++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done

I use a cmake toolchain file, couldn't the cause come from these lines?

In build\x64-Release-Win-GCC\_deps\cmakepp_lang-subbuild\CMakeLists.txt and build\x64-Release-Win-GCC\_deps\cmaize-subbuild\CMakeLists.txt:

# Reject any attempt to use a toolchain file. We must not use one because
# we could be downloading it here. If the CMAKE_TOOLCHAIN_FILE environment
# variable is set, the cache variable will have been initialized from it.
unset(CMAKE_TOOLCHAIN_FILE CACHE)
unset(ENV{CMAKE_TOOLCHAIN_FILE})

[ryanmrichard]

Thanks! Try passing CMAKE_CXX_COMPILER/CMAKE_C_COMPILER explicitly to ct_add_dir (see here). If that works, we can look into automating it.

Automating the passing of the CMake cache or variables would have to be done very carefully to avoid corrupting the cleanliness of the testing environment. We could define a variable CMAKE_TEST_ENVIRONMENT_VARIABLES, or something like that, that is a list of variables to pass to each test.

[josephgarnier]

No change, unless I've misimplemented the proposed solution. I used these instructions:

message("-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER};-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}") # display "-DCMAKE_CXX_COMPILER=C:/msys64/mingw64/bin/g++.exe;-DCMAKE_C_COMPILER=gcc"
ct_add_dir("./" USE_REL_PATH_NAMES CMAKE_OPTIONS "-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}" "-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}") # I also tried with semicolon CMake style

Console still displaying:

[ctest] -- Building for: Visual Studio 16 2019
[ctest] -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19045.
[ctest] -- The C compiler identification is MSVC 19.29.30133.0

[ryanmrichard]

What you have looks right, but for sanity, try this:

ct_add_dir("./" USE_REL_PATH_NAMES CMAKE_OPTIONS "-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}")

and see if it at least fixes the C compiler. Since it looks for the C compiler first, this should work, regardless of whether there is a problem with passing a list of arguments. That said, I'm on MacOS, and doing:

ct_add_dir(
    "./" CMAKE_OPTIONS "-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}" 
                       "-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}"
)

ensures that the tests are finding gcc and g++ instead of the system default Apple Clang.

So if you're still getting MSVC as the C compiler, I think this is a Windows problem and I'll have to punt to @zachcran because he's our resident Windows guru.

FWIW, there seems to be precedent that this could be a Windows problem example and example.

[josephgarnier]

I'm going crazy, no solution works :(. The strange thing is that on all my other C++ projects, I can easily change compilers (GCC, CLang or MSVC). I just have to change toolchain file or preset...

[ryanmrichard]

CMakeTest runs CMake in a nested fashion. Based on your outputs above, the most user-facing CMake gets the right compiler; it's the nested runs that are somehow finding MSVC despite being told not to.

The only other thing I can think of is that, despite how I wrote the code snippet above, I actually call ct_add_dir up one directory from the tests, i.e.,

ct_add_dir(
    "./tests" CMAKE_OPTIONS "-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}" 
                       "-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}"
)

Maybe by running ct_add_dir("./" ...) in the same directory as the tests, the nested settings are conflicting and causing CMake to throw out the options it found and search again? This is purely speculation though.

[josephgarnier]

I tried your suggestions, but without success. I'm going to investigate further. What surprises me is that on my other projects with CTest and GTest I've never encountered this problem.

Actually, here the problem only occurs with CMAKE_C_COMPILER:

[ctest] -DCMAKE_CXX_COMPILER=C:/msys64/mingw64/bin/g++.exe;-DCMAKE_C_COMPILER=C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe

[josephgarnier]

I did it !

[ctest] -- The CXX compiler identification is GNU 14.2.0
[ctest] -- Detecting CXX compiler ABI info
[ctest] -- Detecting CXX compiler ABI info - done
[ctest] -- Check for working CXX compiler: C:/msys64/mingw64/bin/g++.exe - skipped
[ctest] -- Detecting CXX compile features
[ctest] -- Detecting CXX compile features - done
[ctest] Test w/ friendly name "test_directory_find_lib_operation" has ID "cpp_l6yco_1749639684" and file "C:/dev/tests/DirectoryFindLib.cmake"
[ctest] -- CMAKE_CXX_COMPILER=C:/msys64/mingw64/bin/g++.exe
[ctest] -- CMAKE_C_COMPILER=gcc
[ctest] -- CMAKE_GENERATOR=MinGW Makefiles

I've found the cause of the problem, and as I suspected, its a bug coming from CMakeTest.

In the template file build\x64-Release-Win-GCC\_deps\cmake_test-src\cmake\cmake_test\templates\test_CMakeLists.txt.in, there is:

project(@_CT_CMAKELISTS_TEMPLATE_PROJECT_NAME@ LANGUAGES C)

Replace with :

project(@_CT_CMAKELISTS_TEMPLATE_PROJECT_NAME@ LANGUAGES CXX)

Then call ct_add_dir like this:

# Without Toolchain file
ct_add_dir("./" USE_REL_PATH_NAMES CMAKE_OPTIONS "-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}" "-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}" "-DCMAKE_GENERATOR=${CMAKE_GENERATOR}")

# Or with Toolchain file
ct_add_dir("./" USE_REL_PATH_NAMES CMAKE_OPTIONS "-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}" "-DCMAKE_GENERATOR=${CMAKE_GENERATOR}")

It is essential to specify a generator.

[ryanmrichard]

CMakeTest was envisioned as testing CMake modules and not the code built by them. Thus, by design, we wanted to keep CMakeTest somewhat language agnostic. In theory, CMake can be used with more languages than C/C++ (though I haven't really seen such use in the wild). The C language is only specified in that template to silence a CMake warning about not providing a linking language (why CMake provides the NONE option and then warns you if you use it is beyond me).

That said, what we could do is in ct_add_dir check to see what languages the outer project enables, and then ensure the same languages are enabled in the tests. If a user wanted the two language sets to differ, they could use enable_language to enable additional languages in the tests that are not enabled in the outer project. We could also implement the CMAKE_TEST_ENVIRONMENT_VARIABLES solution to grab variables like CMAKE_GENERATOR automatically.

@josephgarnier, if that sounds good to you, I can try to work on this later this week or early next week.

[josephgarnier]

Thus, by design, we wanted to keep CMakeTest somewhat language agnostic

yeah, I understand. Perhaps using the project() command to remove the warning had unexpected side effects (although I didn't see anything in its documentation to that effect).

if that sounds good to you, I can try to work on this later this week or early next week.

Okay, no pressure. I'll use my fix in the meantime.

[ryanmrichard]

@zachcran what are your thoughts on the CMAKE_TEST_ENVIRONMENT_VARIABLES idea? I'm wondering if it's more trouble than it's worth. It's relatively easy for the user to do:

set(
    common_options 
    "-DCMAKE_CXX_COMPILER=/path/to/cxx/compiler"
    "-CMAKE_C_COMPILER=/path/to/c/compiler"
)
ct_add_dir(some_dir CMAKE_OPTIONS "${common_options}")

and they have fine-grained control over what options they share with what tests.

If we auto-capture certain options (say CMAKE_CXX_COMPILER), I worry about what happens if the user also tries to manually pass -DCMAKE_CXX_COMPILER=... to ct_add_dir. I'd assume the intent would be to use the provided value and not the auto-captured value (assuming they are different). This means we'd have to do parsing on the list, which gets ugly.

Another worry is that users will not read the documentation to know that we pass certain variables, which could then lead to subtle bugs.

[zachcran]

@ryanmrichard I like the idea of being able to generate a predictable, explicit set of options that tests written with CMakeTest will use. It removes guesswork by giving users something that they can inspect to easily know exactly what their tests are using. Making it able to be edited/overridden is also great, because that (in theory) gives users better control over the conditions their tests will run under. It seems really hard to gather a complete list of default values that we may want to forward to the tests.

To auto-capture the values we want, I see us needing to do this in one of a few ways (I'm open to other options, too, but this is what I came up with):

1. Somewhat arbitrarily choose certain options to include as we generate the set of options. I'll just refer back up to your worries about this approach, and add that this opens the door for a ton of feature requests in the future for another value to be added. It is probably better to capture everything from the start or not do it at all.
2. On the other end, somehow capture all build/CMakeCache.txt entries and ensure they are applied in each test project directory. There are a few ways I found, although both options seem pretty error-prone: a) Copy build/CMakeCache.txt into each test project. The CMake Cache documentation explicitly discourages "initializing" with a pre-populated CMakeCache.txt file though (see paragraph 7, by my count, of the docs). b) Use the -C flag of cmake (docs) in the command invoking each test. This requires "a CMake script containing SET commands that use the CACHE option, not a cache-format file", so we would need to generate that ourselves.
   • One extra note here is that we probably only need to actually collect the "EXTERNAL" cache entries. Trying to do so might only make the problem harder, though.
3. Somehow gather all of the explicit arguments provided to CMake when building the main project that is being tested. There are some suggestions on how to do this in this SO post, but they seem fragile and don't seem to work in all desired cases.

For the record, I think the most promising option is 2.b), generating a cache-initializing CMake script based on the values in build/CMakeCache.txt (potentially only the "EXTERNAL" entries to help avoid some hard-coded path issues, although we likely still have to comb through and fix or ignore some cache paths).

Whatever the solution is for collecting the values, perhaps we should provide a function or family of functions that could generate CMake options to pass to ct_add_dir() instead of setting a global like CMAKE_TEST_ENVIRONMENT_VARIABLES. This way the user can choose the level of control they want/need a bit easier. I'm imagining something like this:

• If no CMAKE_OPTIONS kwarg is provided to ct_add_dir(), default either to what is happening right now, which is that the user rolls the dice on what defaults they are getting (I don't like the default being uncertain), or use a method (listed above or otherwise) to collect every CMake option explicitly set on the project to mirror the project config as closely as we can. Mirroring the explicit configurations of the main project seems like a reasonable default for tests, and is what I would expect (and already thought was happening when developing CMaize).
• If CMAKE_OPTIONS is defined with no value, default to what is happening right now where no explicit options are given to the project.
• If the user chooses to call one of the option-generating functions, it is up to them to then add, modify, or remove options. This would come close to your proposed alternative above of just defining common_options to pass to ct_add_dir(). Maybe we could make the add/modify/remove more straightforward by using a cpp_map() that is converted into an option list right before add_test() is called. It would be best to allow CMAKE_OPTIONS to accept a list or cpp_map in that case.

One final thought, I think the name CMAKE_TEST_ENVIRONMENT_VARIABLES might be a bit misleading. Unless I am misunderstanding, these are CMake options, not environment variables, that it would contain. Maybe this could be something like CMAKE_TEST_DEFAULT_CMAKE_OPTIONS.