[BUG]: Inconsistency in displaying code coverage with PublishCodeCoverageResults

[alex-zyl]

Task name

PublishCodeCoverageResults

Task version

2

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu

Task log

2023-11-04T01:02:19.9558822Z ##[section]Starting: Publish Test Coverage Report
2023-11-04T01:02:19.9567109Z ==============================================================================
2023-11-04T01:02:19.9567392Z Task         : Publish code coverage results v2
2023-11-04T01:02:19.9567599Z Description  : Publish any of the code coverage results from a build
2023-11-04T01:02:19.9567915Z Version      : 2.229.0
2023-11-04T01:02:19.9568026Z Author       : Microsoft Corporation
2023-11-04T01:02:19.9568200Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/publish-code-coverage-results
2023-11-04T01:02:19.9568677Z ==============================================================================
2023-11-04T01:02:20.1096746Z Found 2 result(s) matching pattern: /home/vsts/work/_temp/build-artifacts/**/*.xml
2023-11-04T01:02:20.1100194Z /home/vsts/work/_temp/build-artifacts/test-results/05593de1-f4e0-4ba3-bbf0-59f240135536/coverage.opencover.xml
2023-11-04T01:02:20.1101805Z /home/vsts/work/_temp/build-artifacts/test-results/_fv-az460-120_2023-11-04_01_01_27/In/fv-az460-120/coverage.opencover.xml
2023-11-04T01:02:20.1203191Z [command]/home/vsts/work/_tasks/PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97/2.229.0/node_modules/coveragepublisher/CoveragePublisher/linux-x64/CoveragePublisher.Console /home/vsts/work/_temp/build-artifacts/test-results/05593de1-f4e0-4ba3-bbf0-59f240135536/coverage.opencover.xml /home/vsts/work/_temp/build-artifacts/test-results/_fv-az460-120_2023-11-04_01_01_27/In/fv-az460-120/coverage.opencover.xml --reportDirectory /home/vsts/work/_temp/704ae2ed-c1e5-4675-906a-291c690c3051 --sourceDirectory .
2023-11-04T01:02:21.6809791Z 2023-11-04T01:02:21: Loading report '/home/vsts/work/_temp/build-artifacts/test-results/05593de1-f4e0-4ba3-bbf0-59f240135536/coverage.opencover.xml' 1/2 in memory
2023-11-04T01:02:21.8191598Z 2023-11-04T01:02:21: Preprocessing report
2023-11-04T01:02:21.8231521Z 2023-11-04T01:02:21: Initiating parser for OpenCover
2023-11-04T01:02:21.8342626Z 2023-11-04T01:02:21: Current Assembly: XXX
2023-11-04T01:02:21.9080956Z 2023-11-04T01:02:21: Finished parsing '/home/vsts/work/_temp/build-artifacts/test-results/05593de1-f4e0-4ba3-bbf0-59f240135536/coverage.opencover.xml' 1/2
2023-11-04T01:02:21.9082916Z 2023-11-04T01:02:21: Loading report '/home/vsts/work/_temp/build-artifacts/test-results/_fv-az460-120_2023-11-04_01_01_27/In/fv-az460-120/coverage.opencover.xml' 2/2 in memory
2023-11-04T01:02:21.9084338Z 2023-11-04T01:02:21: Starting merging result 1
2023-11-04T01:02:21.9104355Z 2023-11-04T01:02:21: Finished merging result 1
2023-11-04T01:02:22.0183685Z 2023-11-04T01:02:22: Preprocessing report
2023-11-04T01:02:22.0198813Z 2023-11-04T01:02:22: Initiating parser for OpenCover
2023-11-04T01:02:22.0228361Z 2023-11-04T01:02:22: Current Assembly: XXX
2023-11-04T01:02:22.0960970Z 2023-11-04T01:02:22: Finished parsing '/home/vsts/work/_temp/build-artifacts/test-results/_fv-az460-120_2023-11-04_01_01_27/In/fv-az460-120/coverage.opencover.xml' 2/2
2023-11-04T01:02:22.0962032Z 2023-11-04T01:02:22: Starting merging result 2
2023-11-04T01:02:22.0962492Z 2023-11-04T01:02:22: Parsing of 2 files completed
2023-11-04T01:02:22.1085239Z 2023-11-04T01:02:22: Finished merging result 2
2023-11-04T01:02:22.1379422Z 2023-11-04T01:02:22: Initializing report builders for report types: HtmlInline_AzurePipelines
2023-11-04T01:02:22.1432257Z 2023-11-04T01:02:22: Analyzing 145 classes
2023-11-04T01:02:22.1434418Z 2023-11-04T01:02:22: Creating report 1/145 (Assembly: XXX, Class: XXX)
2023-11-04T01:02:22.1570610Z 2023-11-04T01:02:22: Writing report file 'XXX.html'
...
2023-11-04T01:02:22.8407851Z 2023-11-04T01:02:22: Creating report 145/145 (Assembly: XXX, Class: XXX)
2023-11-04T01:02:22.8409170Z 2023-11-04T01:02:22: Writing report file 'XXX.html'
2023-11-04T01:02:22.8409887Z 2023-11-04T01:02:22: Creating summary
2023-11-04T01:02:22.8410680Z 2023-11-04T01:02:22: Writing report file '/home/vsts/work/_temp/704ae2ed-c1e5-4675-906a-291c690c3051/index.html'
2023-11-04T01:02:22.8413116Z Publishing file coverage data.
2023-11-04T01:02:23.1305313Z Publishing code coverage HTML report.
2023-11-04T01:02:23.1383789Z Uploading 148 files.
2023-11-04T01:02:28.1428805Z Total file: 148 ---- Processed file: 32 (21%).
2023-11-04T01:02:38.1434140Z Total file: 148 ---- Processed file: 101 (68%).
2023-11-04T01:02:44.4422703Z File upload succeed.
2023-11-04T01:02:44.5083450Z Published '/home/vsts/work/_temp/704ae2ed-c1e5-4675-906a-291c690c3051' as artifact 'Code Coverage Report_105922'.
2023-11-04T01:02:44.5259752Z ##[section]Finishing: Publish Test Coverage Report

Relevant log output

no relevant logs

Aditional info

I'm observing discrepancy in how Code Coverage reports are displayed when build finishes normally and when interrupted (cancelled).

Configuration:

• Code coverage file is generated in 'OpenCover' format.
• Coverage file is published with PublishCodeCoverageResults@2 task
• No report generation tools/tasks are used

Task configuration:

  - task: PublishCodeCoverageResults@2
    displayName: Publish Test Coverage Report
    inputs:
      summaryFileLocation: "${{ parameters.outputDirectory }}/**/*.xml"
      pathToSources: .
    continueOnError: true
    condition: succeededOrFailed()

Coverlet configuration:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
            <Format>opencover</Format>
            <Exclude>[NS.Shared.*]*,</Exclude>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Case 1: normal build

Scenario: build runs to the end and is not interrupted.

PublishCodeCoverageResults task generates code coverage reports and uploads into builds artifacts store

...
2023-11-04T11:38:41: Writing report file 'xxx.html'
2023-11-04T11:38:41: Creating summary
2023-11-04T11:38:41: Writing report file '/home/vsts/work/_temp/1707ff5b-a2f5-45b7-bc6b-b4ed54644e87/index.html'
Publishing file coverage data.
Publishing code coverage HTML report.
Uploading 148 files.
Total file: 148 ---- Processed file: 42 (28%).
Total file: 148 ---- Processed file: 122 (82%).
File upload succeed.
Published '/home/vsts/work/_temp/1707ff5b-a2f5-45b7-bc6b-b4ed54644e87' as artifact 'Code Coverage Report_105925'.

Basic code coverage info is then visible on build summary page. Detailed info from generated html files are not shown on the tab:

On build summary page code coverage percentage is displayed correctly:

Case 2: cancelled build

Scenario: build is manually cancelled after PublishCodeCoverageResults task is executed.

PublishCodeCoverageResults task generates code coverage reports and uploads into builds artifacts store

Task execution logs and produced artifacts are exactly the same as in Case 1.

Now Code Coverage info on build summary page is displayed from htmls generated by PublishCodeCoverageResults task

On build summary page code coverage percentage value is invalid, however report shows correct coverage percentage

[rikrak]

I'm seeing similar behaviour insofar as I only see the basic coverage report on a successful build, even though the detailed report has been uploaded as an artefact on the build.

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[alex-zyl]

The comment to remove 'stale' label

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[alex-zyl]

Still exists