Crashing tests result in empty xunit output with Swift Testing

[jqsilver]

Is it reproducible with SwiftPM command-line tools: swift build, swift test, swift package etc?

• Confirmed reproduction steps with SwiftPM CLI. The description text must include reproduction steps with either of command-line SwiftPM commands, swift build, swift test, swift package etc.

Description

I wrote a small test in both XCTest and Swift Testing

import XCTest

class SwiftTestingCrashResultXCTestCase: XCTestCase {
  func testAlwaysPass() {
    XCTAssert(true)
  }

  func testAlwaysFail() {
    XCTAssert(false)
  }

  func testCrash() {
    fatalError("This is a crash")
  }
}

import Testing

struct TestSuite {
  @Test
  func always_pass()
    #expect(Bool(true))
  }

  @Test
  func always_fail() {
    #expect(Bool(false))
  }

  @Test
  func crashing_test() {
    assertionFailure("testing an assertionFailure")
  }
}

I put these in a small Swift Package as a test target, and ran using swift test --parallel --xunit-output xunit.xml

Expected behavior

The resulting xunit-swift-testing.xml should report always_pass() as passing, always_fail() as failing, and crashing_test() as failing as well. This would match the behavior of the equivalent XCTestCase.

Actual behavior

The xunit-swift-testing.xml contains only:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>

While the xunit.xml file contains:

<?xml version="1.0" encoding="UTF-8"?>

<testsuites>
  <testsuite name="TestResults" errors="0" tests="3" failures="2" time="0.156307999">
    <testcase classname="SwiftTestingCrashResultTests.SwiftTestingCrashResultXCTestCase"
      name="testCrash" time="0.052093333">
      <failure message="failed"></failure>
    </testcase>
    <testcase classname="SwiftTestingCrashResultTests.SwiftTestingCrashResultXCTestCase"
      name="testAlwaysPass" time="0.052104958">
</testcase>
    <testcase classname="SwiftTestingCrashResultTests.SwiftTestingCrashResultXCTestCase"
      name="testAlwaysFail" time="0.052109708">
      <failure message="failed"></failure>
    </testcase>
  </testsuite>
</testsuites>

Steps to reproduce

Here's my package file:

// swift-tools-version: 6.0
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "SwiftTestingCrashResult",
    dependencies: [
        .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.2.0"),
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .executableTarget(
            name: "SwiftTestingCrashResult",
            dependencies: [
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
            ],
            path: "Sources"
        ),
        .testTarget(
          name: "SwiftTestingCrashResultTests",
          dependencies: ["SwiftTestingCrashResult"],
          path: "Tests"
        )
    ]
)

The two tests are contained in a Tests/ folder in separate files, while the Sources/ folder contains the default starter code for a command-line tool:

// The Swift Programming Language
// https://docs.swift.org/swift-book
// 
// Swift Argument Parser
// https://swiftpackageindex.com/apple/swift-argument-parser/documentation

import ArgumentParser

@main
struct SwiftTestingCrashResult: ParsableCommand {
    mutating func run() throws {
        print("Hello, world!")
    }
}

Swift Package Manager version/commit hash

6.0

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.115.1 Apple Swift version 6.0.3 (swiftlang-6.0.3.1.10 clang-1600.0.30.1)
Target: arm64-apple-macosx15.0
Darwin Andys-MacBook-Pro.local 24.3.0 Darwin Kernel Version 24.3.0: Thu Jan 2 20:24:22 PST 2025; root:xnu-11215.81.4~3/RELEASE_ARM64_T6041 arm64

[jqsilver]

Here's the full command line output when running swift test

SwiftTestingCrashResult ()$ swift test --xunit-output xunit.xml --parallel
Building for debugging...
[2/2] Write swift-version-5BDAB9E9C0126B9D.txt
Build complete! (0.12s)
error: Exited with unexpected signal code 5
[3/3] Testing SwiftTestingCrashResultTests.SwiftTestingCrashResultXCTestCase/testAlwaysFail
Test Suite 'Selected tests' started at 2025-05-07 16:07:21.904.
Test Suite 'SwiftTestingCrashResultPackageTests.xctest' started at 2025-05-07 16:07:21.905.
Test Suite 'SwiftTestingCrashResultXCTestCase' started at 2025-05-07 16:07:21.905.
Test Case '-[SwiftTestingCrashResultTests.SwiftTestingCrashResultXCTestCase testCrash]' started.
SwiftTestingCrashResultTests/SwiftTestingCrashResultXCTest.swift:16: Fatal error: This is a crash

Test Suite 'Selected tests' started at 2025-05-07 16:07:21.904.
Test Suite 'SwiftTestingCrashResultPackageTests.xctest' started at 2025-05-07 16:07:21.905.
Test Suite 'SwiftTestingCrashResultXCTestCase' started at 2025-05-07 16:07:21.905.
Test Case '-[SwiftTestingCrashResultTests.SwiftTestingCrashResultXCTestCase testAlwaysFail]' started.
/Users/andy_bartholomew/Code/SwiftTestingCrashResult/Tests/SwiftTestingCrashResultXCTest.swift:12: error: -[SwiftTestingCrashResultTests.SwiftTestingCrashResultXCTestCase testAlwaysFail] : XCTAssertTrue failed
Test Case '-[SwiftTestingCrashResultTests.SwiftTestingCrashResultXCTestCase testAlwaysFail]' failed (0.006 seconds).
Test Suite 'SwiftTestingCrashResultXCTestCase' failed at 2025-05-07 16:07:21.911.
	 Executed 1 test, with 1 failure (0 unexpected) in 0.006 (0.006) seconds
Test Suite 'SwiftTestingCrashResultPackageTests.xctest' failed at 2025-05-07 16:07:21.911.
	 Executed 1 test, with 1 failure (0 unexpected) in 0.006 (0.006) seconds
Test Suite 'Selected tests' failed at 2025-05-07 16:07:21.911.
	 Executed 1 test, with 1 failure (0 unexpected) in 0.006 (0.007) seconds

◇ Test run started.
↳ Testing Library Version: 102 (arm64e-apple-macos13.0)
◇ Suite TestSuite started.
◇ Test always_true() started.
◇ Test crashing_test() started.
◇ Test always_fail() started.
SwiftTestingCrashResultTests/SwiftTestingCrashResultTests.swift:19: Fatal error: testing an assertionFailure
✔ Test always_true() passed after 0.001 seconds.
✘ Test always_fail() recorded an issue at SwiftTestingCrashResultTests.swift:14:5: Expectation failed: Bool(false)
error: Exited with unexpected signal code 5

[jqsilver]

Verified that this also repros with Xcode 16.3

SwiftTestingCrashResult ()$ swift --version; uname -a
swift-driver version: 1.120.5 Apple Swift version 6.1 (swiftlang-6.1.0.110.21 clang-1700.0.13.3)
Target: arm64-apple-macosx15.0
Darwin Andys-MacBook-Pro.local 24.3.0 Darwin Kernel Version 24.3.0: Thu Jan  2 20:24:22 PST 2025; root:xnu-11215.81.4~3/RELEASE_ARM64_T6041 arm64

[plemarquand]

swift-package-manager delegates the creation of the XML results to swift-testing. Because results are written by swift-testing itself, when the process crashes it cannot write out results. I'm going to move this issue to the swift-testing repository.

[bachand]

Thanks @plemarquand for the details and triaging.

swift-package-manager delegates the creation of the XML results to swift-testing. Because results are written by swift-testing itself, when the process crashes it cannot write out results. I'm going to move this issue to the swift-testing repository.

By chance, do you happen to have a pointer to where swift-testing writes out the XML report? If not, no big deal, but I was curious to look a bit deeper.

[plemarquand]

@bachand sure, the code that produces the xml can be found here https://github.com/swiftlang/swift-testing/blob/main/Sources/Testing/Events/Recorder/Event.JUnitXMLRecorder.swift#L98

[stmontgomery]

We have some plans to rearchitect how JUnit XML is produced for Swift Testing tests such that it would be done via a supervisory process rather than in the same process where tests are running. That should address this problem by ensuring that a crash in the test runner process is correctly represented in the results XML file and that the final file is still valid. We should dupe this to the issue tracking that effort, or create an issue to track that if none exists yet.

[bachand]

We have some plans to rearchitect how JUnit XML is produced for Swift Testing tests such that it would be done via a supervisory process rather than in the same process where tests are running. That should address this problem by ensuring that a crash in the test runner process is correctly represented in the results XML file and that the final file is still valid. We should dupe this to the issue tracking that effort, or create an issue to track that if none exists yet.

Nice! This would be great. 😃

[bachand]

In an effort to be helpful, I can share the context that this limitation of Swift Testing has made it challenging for us to roll out Swift Testing at scale, as we have a limited ability to disable crashing tests in our large codebase as we try to keep CI builds for the app green. The supervisory process approach makes a lot of sense to me. Thanks for taking a look at this issue — we appreciate it!