Skip to content

Conversation

@bart-vmware
Copy link
Member

@bart-vmware bart-vmware commented Nov 25, 2025

Description

This PR migrates existing xUnit v3 tests to run on Microsoft Testing Platform v2, instead of VSTest. This enables running tests both via dotnet test and as direct executables.

What's changed

  • Replaced coverlet with Microsoft.Testing.Extensions.CodeCoverage, because it is incompatible
  • Removed xunit.runner.visualstudio and Microsoft.NET.Test.Sdk, which are no longer needed
  • Alternate libraries used for hang/crashdump and TRX reports
  • Added global.json, which is required for MTP v2 when using the .NET 10 SDK
  • Different file format for coverage settings (.runsettings replaced by coverage.config)
  • Different coverage output format (OpenCover is no longer available, Cobertura can't be read by Sonar, so what's left is Visual Studio XML format)
  • Different command-line arguments to filter tests and take crash/hangdumps
  • In CI build, run tests against all target frameworks in a single command (because TestTfmsInParallel now actually works)
  • Make xUnit use MTP2 via UseMicrosoftTestingPlatformRunner
  • Keep using xunit.runner.json instead of testconfig.json, because it works with multiple runners
  • Not using the superior GitHubActionsTestLogger, due to license terms about Russia/Ukraine
  • When no tests are found, dotnet test exists with an error (can be overruled with --minimum-expected-tests 0 or catching the related exit code)
  • We lost branch coverage and risk hotspots, because ReportGenerator does not support it with VS coverage format
  • The CI build log does not tell which test(s) failed (only the assembly), and there is no stacktrace. Need to search for it on the GitHub Actions summary tab

Verification

  • Tests are discovered (5520 in total, 1840 per target framework)
    • In Visual Studio 2026 Test Explorer
    • In Visual Studio 2026 Test Explorer on WSL2 with Ubuntu-22.04
    • In Visual Studio 2026 R# Unit Tests
    • In dotnet test on the command line
  • Tests can be run
    • In Visual Studio 2026 Test Explorer
    • In Visual Studio 2026 Test Explorer on WSL2 with Ubuntu-22.04
    • In Visual Studio 2026 R# Unit Tests
    • In dotnet test on the command line
    • Test outcomes are listed on the GitHub Actions summary tab
  • Tests can be debugged
  • All tests succeed
    • In Visual Studio 2026 Test Explorer
    • In Visual Studio 2026 Test Explorer on WSL2 with Ubuntu-22.04
    • In Visual Studio 2026 R# Unit Tests
    • In dotnet test on the command line
    • In CI build on Ubuntu/Windows/macOS, split into regular/memorydump testruns
  • Coverage is reported
    • Coverage files contain source-link paths
    • Coverage on PR diff in SonarCloud and Sonar PR comment
      image
    • Coverage on full branch displayed on GitHub Actions summary tab and PR comment (only Steeltoe libraries)
      image
      ✋ We lost statement branch coverage here
    • Coverage on full branch roughly matches outcome from main branch (only Steeltoe libraries)
      image
      ✋ We lost statement branch coverage here
  • Test failures
    • Crashdump is captured and uploaded during CI build
      ✋ Works, but outputs a 160 MB text log, which makes the GitHub Actions UI completely unresponsive; tracked at Excessive output when crashdump kicks in microsoft/testfx#7045
    • Hangdump is captured and uploaded during CI build
      ✋ This doesn't currently work, tracked at [MTP]: Timeout doesn't abort hanging tests microsoft/testfx#6776
    • Failed test is reported
      • In Visual Studio 2026 Test Explorer
      • In Visual Studio 2026 Test Explorer on WSL2 with Ubuntu-22.04
      • In Visual Studio 2026 R# Unit Tests
      • In dotnet test on the command line
      • On GitHub Actions summary tab
      • In CI build log
    • Skipped test is reported
      • In Visual Studio 2026 Test Explorer
      • In Visual Studio 2026 Test Explorer on WSL2 with Ubuntu-22.04
      • In Visual Studio 2026 R# Unit Tests
      • In dotnet test on the command line
      • On GitHub Actions summary tab
      • In CI build log

Quality checklist

  • Your code complies with our Coding Style.
  • You've updated unit and/or integration tests for your change, where applicable.
  • You've updated documentation for your change, where applicable.
    If your change affects other repositories, such as Documentation, Samples and/or MainSite, add linked PRs here.
  • There's an open issue for the PR that you are making. If you'd like to propose a new feature or change, please open an issue to discuss the change or find an existing issue.
  • You've added required license files and/or file headers (explaining where the code came from with proper attribution), where code is copied from StackOverflow, a blog, or OSS.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 25, 2025

Summary - All Code Coverage (ubuntu-latest)

Line coverage Branch coverage

Assembly Line coverage Branch coverage
Steeltoe.Bootstrap.AutoConfiguration 97.4% 100%
Steeltoe.Common 84.3% 77.7%
Steeltoe.Common.Certificates 96.2% 85.7%
Steeltoe.Common.Hosting 84% 65%
Steeltoe.Common.Http 100% 82.3%
Steeltoe.Common.Logging 81.3% 41.6%
Steeltoe.Common.Net 64.5% 66.6%
Steeltoe.Configuration.Abstractions 98.5% 91.6%
Steeltoe.Configuration.CloudFoundry 99.1% 92.3%
Steeltoe.Configuration.ConfigServer 97.4% 92.2%
Steeltoe.Configuration.Encryption 97.7% 92.4%
Steeltoe.Configuration.Kubernetes.ServiceBindings 93.3% 87.5%
Steeltoe.Configuration.Placeholder 91.9% 78.8%
Steeltoe.Configuration.RandomValue 93.2% 87.5%
Steeltoe.Configuration.SpringBoot 98.3% 95%
Steeltoe.Connectors 94.3% 88.7%
Steeltoe.Connectors.EntityFrameworkCore 81.5% 75%
Steeltoe.Discovery.Configuration 92.8% 100%
Steeltoe.Discovery.Consul 97.8% 96.2%
Steeltoe.Discovery.Eureka 91.5% 84.5%
Steeltoe.Discovery.HttpClients 95% 95.8%
Steeltoe.Logging.Abstractions 99.4% 97%
Steeltoe.Logging.DynamicConsole 100% 96.8%
Steeltoe.Logging.DynamicSerilog 99.1% 95.8%
Steeltoe.Management.Abstractions 100% 100%
Steeltoe.Management.Endpoint 95.5% 89.3%
Steeltoe.Management.Prometheus 95.9% 92.3%
Steeltoe.Management.Tasks 100% ****
Steeltoe.Management.Tracing 100% 83.3%
Steeltoe.Security.Authentication.JwtBearer 100% 100%
Steeltoe.Security.Authentication.OpenIdConnect 74.4% 57.6%
Steeltoe.Security.Authorization.Certificate 96.3% 80%
Steeltoe.Security.DataProtection.Redis 100% ****

@bart-vmware bart-vmware force-pushed the update-xunit-mtp2 branch 15 times, most recently from 52252c4 to fa40ecf Compare December 4, 2025 13:23
@bart-vmware bart-vmware force-pushed the update-xunit-mtp2 branch 3 times, most recently from 39f0e96 to bba338e Compare December 5, 2025 15:34
@sonarqubecloud
Copy link

sonarqubecloud bot commented Dec 5, 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants