Code coverage reporting

[sevenam]

For xunit we've normally been creating code coverage reports with:

dotnet test --collect:"XPlat Code Coverage"

Is there a way to do this with TUnit or is this a feature that's not been added yet?

[thomhurst]

Heya @sevenam - Details are here: https://thomhurst.github.io/TUnit/docs/extensions/

[sevenam]

Thanks! I missed this when going through the documentation, sorry! It would be great to have an example of how to use it as well.
Was hoping to make it work and to maybe do a PR with an example, but I didn't quite manage to get it to work.

Added the nuget in the .csproj file:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
    <CollectCoverage>true</CollectCoverage>
    <CoverletOutputFormat>cobertura</CoverletOutputFormat>
    <CoverletOutput>$(OutputPath)coverage.cobertura.xml</CoverletOutput>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="FluentAssertions" Version="6.12.0" />
    <PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" Version="17.12.5" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="8.0.8" />
    <PackageReference Include="Microsoft.AspNetCore.TestHost" Version="8.0.8" />
    <!-- had to remove this reference as TUnit tests would not run when it was included -->
    <!--<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.11.1" />-->
    <PackageReference Include="TUnit" Version="0.1.897" />
  </ItemGroup>

Then tried running the tests with:
dotnet test ProjectFolder --collect:"Code Coverage"
and
dotnet test ProjectFolder --collect:"XPlat Code Coverage"

Also tried adding a runsettings file:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage">
        <Configuration>
          <Format>cobertura</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

and pointing to this with dotnet test:
dotnet test ProjectFolder --collect:"Code Coverage" --settings codecoverage.runsettings

Maybe there's something obvious I missed...

[thomhurst]

The flag is --coverage. See here https://thomhurst.github.io/TUnit/docs/tutorial-extras/command-line-flags

[sevenam]

Great - thanks a lot! So there was something obvious I'd missed...

Got it to work with this input to dotnet test (I had initially missed the additional "--" that was required).

dotnet test ProjectFolder -- --coverage --coverage-output coverage.cobertura.xml --coverage-output-format cobertura --results-directory .

It didn't output any .xml until I included the --results-directory in addition to the --coverage flags.

[thomhurst]

I believe by default it goes into bin/Test results/coverage.xml

[gioce90]

The --collect:"XPlat Code Coverage" is the command for vstest driver to activate coverlet.collector, and --collect:"Code Coverage" instead uses the dotnet-coverage tool.
The --coverage in TUnit is using this last one as default, and it's great, but there is a way to change the collector?

[thomhurst]

Coverlet isn't compatible currently so I think you're stuck with that donet collector for now.

[gioce90]

What about inline params? There is a way to put simple settings directly in the command?

For example, for years in my projects on Azure DevOps the coverage was including also the Test projects and the third-party libraries coverage... and this was a bit frustrating because our pipelines where showing a false percentage of code coverage.
Luckily then was introduced IncludeTestAssembly=false to be used, for example, in this way:

- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: 'test'
    projects: '**/*.csproj'  #or *.sln
    arguments: '--no-restore --configuration $(buildConfiguration) --collect "Code Coverage;IncludeTestAssembly=false"' 

This would be a convenient way to quickly pass parameters without using a XML settings file (runsettings.xml can be really frustrating sometimes).

[thomhurst]

The coverage extension isn't actually part of TUnit, it's a Microsoft tool. So if you want features for it you're better off raising it with them as I can't do anything. The docs on their extension is here https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-extensions-code-coverage

[gioce90]

Sorry, maybe it seems I was focused too much on coverage features, but the goal here is the way how TUnit pass parameters to the underline MS coverage tool.

I read your docs and I saw:

--coverage-settings
XML code coverage settings

And maybe I'm wrong, but my worries is about that there is no way to pass quick input to the collector (like --collect "Code Coverage;IncludeTestAssembly=false" but could be everything else) and the only way is to use the XML settings file.

[thomhurst]

TUnit doesn't pass anything explicitly. The extension point is all owned by Microsoft and if you want to be able to pass stuff, you'd still need to raise an issue with them.

[gioce90]

What about default exclusions from coverage?

As mentioned, test assemblies and third-party libraries are usually included in coverage, which can be confusing when you need to focus on the % coverage of your codebase.

With the dotnet collector, you can exclude test assemblies with --collect "Code Coverage;IncludeTestAssembly=false".
Now is also possible to exclude third-party libraries with the ExcludeAssembliesWithoutSources=MissingAll workaround .
In the past I asked the introduction of a parameter, but with no luck.

However, parameters aside, I think excluding test assemblies and third-party libraries should be the default. People usually want to focus on their codebase.

So, in my opinion:

• the --collect command in TUnit should allow parameters, like --collect "IncludeTestAssembly=true" or something like that.
• test assemblies should be excluded from coverage by default. But it should be possible to include them with IncludeTestAssembly=true.
• third-party libraries should also be excluded from coverage by default. Coverlet is already doing this. But should be possible to include them with "IncludeThirdPartyLibraries=true", or something like that.

[thomhurst]

As mentioned, the collect flags etc aren't part of TUnit so I can't physically change anything about them. You need to raise with MS 😄

[gioce90]

Oh sorry, I saw --coverage, --coverage-output, etc. in the docs regarding command-line-flags, and this led me to the wrong conclusion.

Indeed If I try from the CLI to run dotnet run -c Release --report-trx --coverage I see this error:

The strange thing is that the part --report-trx --coverage is wrote also here.
I'm a bit confused right now.

[thomhurst]

Sorry, if you want to use coverage or trx you need to install the extension packages: https://thomhurst.github.io/TUnit/docs/extensions/