Skip to content

Conversation

jurahul
Copy link
Contributor

@jurahul jurahul commented Sep 30, 2025

Remove UnitTests from LLVM_TEST_DEPENDS_COMMON and create a specialized lit suite for unit-tests and that depends only on UnitTests.

@jurahul jurahul marked this pull request as ready for review October 1, 2025 14:41
Copy link
Member

@mshockwave mshockwave left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jurahul jurahul merged commit 0e14973 into llvm:main Oct 1, 2025
11 checks passed
@jurahul jurahul deleted the nfc_specialize_unit_tests branch October 1, 2025 20:06
@llvm-ci
Copy link
Collaborator

llvm-ci commented Oct 1, 2025

LLVM Buildbot has detected a new failure on builder clang-m68k-linux-cross running on suse-gary-m68k-cross while building llvm at step 5 "ninja check 1".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/27/builds/16894

Here is the relevant piece of the build log for the reference
Step 5 (ninja check 1) failure: stage 1 checked (failure)
...
[325/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTests/MemberPointerTypeLoc.cpp.o
[326/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTests/TraversalScope.cpp.o
[327/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTestDeclVisitor.cpp.o
[328/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/ReplacementsYamlTest.cpp.o
[329/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/ParsedSourceLocationTest.cpp.o
[330/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/FixedPointString.cpp.o
[331/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RefactoringActionRulesTest.cpp.o
[332/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTestTypeLocVisitor.cpp.o
[333/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/Syntax/MutationsTest.cpp.o
[334/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/AST/ASTImporterTest.cpp.o
FAILED: tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/AST/ASTImporterTest.cpp.o 
/usr/bin/c++ -DGTEST_HAS_RTTI=0 -DLLVM_BUILD_STATIC -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GLIBCXX_USE_CXX11_ABI=1 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/var/lib/buildbot/workers/suse-gary-m68k-cross/clang-m68k-linux-cross/stage1/tools/clang/unittests -I/var/lib/buildbot/workers/suse-gary-m68k-cross/clang-m68k-linux-cross/llvm/clang/unittests -I/var/lib/buildbot/workers/suse-gary-m68k-cross/clang-m68k-linux-cross/llvm/clang/include -I/var/lib/buildbot/workers/suse-gary-m68k-cross/clang-m68k-linux-cross/stage1/tools/clang/include -I/var/lib/buildbot/workers/suse-gary-m68k-cross/clang-m68k-linux-cross/stage1/include -I/var/lib/buildbot/workers/suse-gary-m68k-cross/clang-m68k-linux-cross/llvm/llvm/include -I/var/lib/buildbot/workers/suse-gary-m68k-cross/clang-m68k-linux-cross/llvm/clang/unittests/Tooling -I/var/lib/buildbot/workers/suse-gary-m68k-cross/clang-m68k-linux-cross/llvm/third-party/unittest/googletest/include -I/var/lib/buildbot/workers/suse-gary-m68k-cross/clang-m68k-linux-cross/llvm/third-party/unittest/googlemock/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-uninitialized -Wno-nonnull -Wno-class-memaccess -Wno-dangling-reference -Wno-redundant-move -Wno-pessimizing-move -Wno-array-bounds -Wno-stringop-overread -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wno-misleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -O3 -DNDEBUG -std=c++17  -Wno-variadic-macros -fno-exceptions -funwind-tables -fno-rtti -UNDEBUG -Wno-suggest-override -MD -MT tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/AST/ASTImporterTest.cpp.o -MF tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/AST/ASTImporterTest.cpp.o.d -o tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/AST/ASTImporterTest.cpp.o -c /var/lib/buildbot/workers/suse-gary-m68k-cross/clang-m68k-linux-cross/llvm/clang/unittests/AST/ASTImporterTest.cpp
c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
[335/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/ReparseWorkingDirTest.cpp.o
[336/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTests/InitListExprPostOrder.cpp.o
[337/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/Syntax/SynthesisTest.cpp.o
[338/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/ASTUnitTest.cpp.o
[339/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/SearchPathTest.cpp.o
[340/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/Syntax/BuildTreeTest.cpp.o
[341/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTests/InitListExprPostOrderNoQueue.cpp.o
[342/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/Syntax/TreeTest.cpp.o
[343/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/Syntax/TokensTest.cpp.o
[344/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RefactoringCallbacksTest.cpp.o
[345/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTests/CallbacksBinaryOperator.cpp.o
[346/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/CompilerInvocationTest.cpp.o
[347/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RefactoringTest.cpp.o
[348/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/PCHPreambleTest.cpp.o
[349/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/TextDiagnosticTest.cpp.o
[350/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/FrontendActionTest.cpp.o
[351/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/SourceCodeBuildersTest.cpp.o
[352/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/StencilTest.cpp.o
[353/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/Syntax/TreeTestBase.cpp.o
[354/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/CompilerInstanceTest.cpp.o
[355/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/ToolingTest.cpp.o
[356/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/CodeGenActionTest.cpp.o
[357/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/OutputStreamTest.cpp.o
[358/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/UtilsTest.cpp.o
[359/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTests/CallbacksCallExpr.cpp.o
[360/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTests/CallbacksCompoundAssignOperator.cpp.o
[361/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/ASTMatchers/ASTMatchersTraversalTest.cpp.o
[362/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTests/CallbacksUnaryOperator.cpp.o
[363/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTests/CallbacksLeaf.cpp.o
[364/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/RecursiveASTVisitorTestPostOrderVisitor.cpp.o
[365/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Frontend/NoAlterCodeGenActionTest.cpp.o
[366/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/SourceCodeTest.cpp.o
[367/466] Building CXX object tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Tooling/TransformerTest.cpp.o
ninja: build stopped: subcommand failed.

@chapuni
Copy link
Contributor

chapuni commented Oct 2, 2025

I wonder this would suppress check-llvm-unit out of both check-llvm and check-all. Intentional?

@jurahul
Copy link
Contributor Author

jurahul commented Oct 2, 2025

It should not, so not intentional. I think these tests should still be a part of check-all atleast and check-llvm as well. Let me test a little bit to confirm tomorrow

@jurahul
Copy link
Contributor Author

jurahul commented Oct 2, 2025

IIRC check-llvm and check-all are their own standalone targets independent of the check-llvm-* targets, but will check

@jurahul
Copy link
Contributor Author

jurahul commented Oct 2, 2025

Here's the command generated for check-llvm:

cd /xyz/llvm-project/build/test && /usr/bin/python3 /xyz/llvm-project/build/./bin/llvm-lit -vv /xyz/llvm-project/build/test

So, this will run everything under llvm/test, including the unit tests. Locally I do see several LLVM-Unit tests executing when I run ninja -C build check-llvm. check-llvm is the target that GitHib CI uses. check-all is similar.

@jurahul
Copy link
Contributor Author

jurahul commented Oct 2, 2025

But for additional verification I created this PR with a broken unit test, expecting that it will fail the checks: #161623

@chapuni
Copy link
Contributor

chapuni commented Oct 2, 2025

Could you try again with cleaning unittests?

find unittests -type f -name '*.o' | xargs rm
ninja llvm-test-depends # Does it rebuild unittests?

@jurahul
Copy link
Contributor Author

jurahul commented Oct 2, 2025

It does not. And isn't that expected? It's no longer a part of LLVM_TEST_DEPENDS and hence not a part of llvm-test-depends target.

@chapuni
Copy link
Contributor

chapuni commented Oct 2, 2025

I expect *-test-depends should arrange required targets before lit.

e.g. https://lab.llvm.org/buildbot/#/builders/199/builds/6235/steps/12/logs/stdio
This doesn't test LLVM-Unit with check-all.

llvm-lit: /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/llvm/llvm/utils/lit/lit/discovery.py:250: warning: test suite 'LLVM-Unit' contained no tests

@jurahul
Copy link
Contributor Author

jurahul commented Oct 2, 2025

I see, so I think we need to add UnitTest as a explicit dependency for both llvm-test-depend as well as check-llvm. I think without that, check-llvm will not rebuild the unit tests and either fail (like above) or execute stale tests (like what I saw locally I guess). Let's see if my PR above which is doing a part of it atleast fails.

@jurahul
Copy link
Contributor Author

jurahul commented Oct 2, 2025

Ok. in the latest test PR version, I do see that the 2 faulty unit tests do fail:

FAIL: LLVM-Unit :: Support/./SupportTests/34/104 (8615 of 61658)
******************** TEST 'LLVM-Unit :: Support/./SupportTests/34/104' FAILED ********************
Script(shard):
--
GTEST_OUTPUT=json:/home/gha/actions-runner/_work/llvm-project/llvm-project/build/unittests/Support/./SupportTests-LLVM-Unit-1199807-34-104.json GTEST_SHUFFLE=0 GTEST_TOTAL_SHARDS=104 GTEST_SHARD_INDEX=34 /home/gha/actions-runner/_work/llvm-project/llvm-project/build/unittests/Support/./SupportTests
--
Script:
--
/home/gha/actions-runner/_work/llvm-project/llvm-project/build/unittests/Support/./SupportTests --gtest_filter=CastingTest.isa
--
Classof: 0xa741f0
/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/unittests/Support/Casting.cpp:151: Failure
Value of: isa<foo>(B1)
  Actual: true
Expected: false
Classof: 0xa741f0
Classof: 0xa741f0
Classof: 0xa741f0
/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/unittests/Support/Casting.cpp:151
Value of: isa<foo>(B1)
  Actual: true
Expected: false
********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.
FAIL: LLVM-Unit :: IR/./IRTests/90/103 (42923 of 61658)
******************** TEST 'LLVM-Unit :: IR/./IRTests/90/103' FAILED ********************

@@ -0,0 +1,5 @@
add_lit_testsuite(check-llvm-unit "Running lit suite for LLVM unit tests"
${CMAKE_CURRENT_BINARY_DIR}
EXCLUDE_FROM_CHECK_ALL
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it unconditional EXCLUDE_FROM_CHECK_ALL? It looks conditional on the left.

Copy link
Contributor Author

@jurahul jurahul Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See similar PR for the earlier specialized suites: #159781

Without EXCLUDE_FROM_CHECK_ALL, these tests run twice, which is wasteful and sometimes leading to failure due to concurrent execution (which will likely not affect these unit tests).

jurahul added a commit that referenced this pull request Oct 2, 2025
… (#161623)

Add `UnitTests` as an explicit dependency for `check-llvm` and
`llvm-test-depends`. In
#161442, the intent was to
remove `UnitTests` as a dependency for the individual per-directory
`check-llvm-*` test suites created but not to drop it from `check-llvm`
or `llvm-test-depends`. This missing dependency will cause LLVM unit
tests to be not rebuilt and resulting in either `warning: test suite
'LLVM-Unit' contained no tests` or running stale running versions of the
unit tests when running `check-llvm`.
llvm-sync bot pushed a commit to arm/arm-toolchain that referenced this pull request Oct 2, 2025
…ed by 0e14973 (#161623)

Add `UnitTests` as an explicit dependency for `check-llvm` and
`llvm-test-depends`. In
llvm/llvm-project#161442, the intent was to
remove `UnitTests` as a dependency for the individual per-directory
`check-llvm-*` test suites created but not to drop it from `check-llvm`
or `llvm-test-depends`. This missing dependency will cause LLVM unit
tests to be not rebuilt and resulting in either `warning: test suite
'LLVM-Unit' contained no tests` or running stale running versions of the
unit tests when running `check-llvm`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants