[BUG] The EXPECTFAIL negative test does not fail

[josephgarnier]

Describe the bug
According to the documentation, we can test that an exception has indeed been raised by using the EXPECTFAIL keyword.

So I assumed that the inverse is also true.

However, the code below passes, when it should fail:

ct_add_test(NAME "test_throw_exception")
function(${CMAKETEST_TEST})

	ct_add_section(NAME "should_fail" EXPECTFAIL)
	function(${CMAKETEST_SECTION})
		message(STATUS "test_throw_exception/should_fail")
	endfunction()
endfunction()

Is this the expected behavior or is it a bug?

[ryanmrichard]

That's a bug. We test for the inverse here. I'll try to look into this more in a couple hours.

[ryanmrichard]

Update: So I figured out the problem, but haven't worked out a fix yet. The problem is that the CMakeLists.txt that gets generated when the EXPECTFAIL flag is present always has an error in it (this also means that CMakeTest has NOT been correctly testing for expected failures). It looks like the error is caused by the generated file not seeing the definition of the function.

[ryanmrichard]

The bug has to do with the use of ${CMAKETEST_TEST} (so CMakeTest has only been incorrectly testing expected failures when the tests rely on ${CMAKETEST_TEST}, which is not as bad as I had feared). For now, if you do:

ct_add_test(NAME "test_throw_exception")
function(${test_throw_exception})

	ct_add_section(NAME "should_fail" EXPECTFAIL)
	function(${CMAKETEST_SECTION})
		message(STATUS "test_throw_exception/should_fail")
	endfunction()
endfunction()

your test should correctly fail.

Unfortunately, because EXPECTFAIL has to be run in a subprocess, I'm not sure if there's a good way to get ${CMAKETEST_TEST} to be set correctly. I'm wondering if we may need to deprecate support for ${CMAKETEST_TEST} (and probably should also deprecate ${CMAKETEST_SECTION} out of an abundance of caution) to avoid this problem.

[josephgarnier]

Thank you for your feedback.

Concerning the temporary fix, in my tests I use one ct_add_test per function to test, so could it cause a bug if I use function(${test_throw_exception}) for all my tests (even if this syntax is deprecated)?

[ryanmrichard]

I forgot that using the test names was deprecated. Thanks for pointing that out. I'll see if I can come up with a solution that uses CMAKETEST_TEST, but it may take a bit.