(fix) Fix code coverage reporting by JennyPng · Pull Request #48822 · Azure/azure-sdk-for-python

jenny (JennyPng) · 2026-08-31T22:18:53Z

storage run that now outputs code coverage:
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6786508&view=codecoverage-tab

code coverage for core: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6785738&view=codecoverage-tab

Summary

discover package-local sdk/**/.coverage files directly instead of the obsolete _coverage staging directory
combine coverage data deterministically while preserving the package files
gate XML report generation on the merged root .coverage file
add regression coverage for discovery, combination, and XML generation

Testing

python -m pytest -q scripts/devops_tasks/test_create_coverage.py
python -m black --check scripts/devops_tasks/create_coverage.py scripts/devops_tasks/test_create_coverage.py

Read package-local .coverage files directly when generating the combined CI coverage report. Add regression tests for discovery, combination, and XML generation gating.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-08-31T22:19:34Z

Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

azure-pipelines · 2026-09-03T22:32:44Z

Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

🟡 Changes recommended

Coverage path normalization fails for duplicate package names across service directories.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes Azure Pipelines coverage collection, normalization, reporting, and cleanup.

Changes:

Adds per-check coverage collection and opt-out support.
Preserves test environments until reporting, then cleans them up.
Adds regression tests for coverage and cleanup behavior.

File summaries

File	Description
`scripts/devops_tasks/test_create_coverage.py`	Tests coverage reporting behavior.
`scripts/devops_tasks/create_coverage.py`	Combines and normalizes coverage data; package resolution does not handle duplicate package names across services.
`eng/tools/azure-sdk-tools/tests/test_install_and_test_coverage.py`	Tests coverage arguments and opt-out behavior.
`eng/tools/azure-sdk-tools/tests/test_dispatch_checks.py`	Tests environment finalization.
`eng/tools/azure-sdk-tools/tests/test_cleanup_isolate_dirs.py`	Tests isolated-environment cleanup.
`eng/tools/azure-sdk-tools/azpysdk/main.py`	Adds the coverage opt-out option.
`eng/tools/azure-sdk-tools/azpysdk/install_and_test.py`	Enables per-check package coverage.
`eng/scripts/run_coverage.py`	Accepts an explicit coverage data file.
`eng/scripts/dispatch_checks.py`	Conditionally preserves coverage environments.
`eng/scripts/cleanup_isolate_dirs.py`	Adds deferred environment cleanup.
`eng/pipelines/templates/steps/build-test.yml`	Runs cleanup after coverage generation.

Review details

Files reviewed: 11/11 changed files
Comments generated: 1
Review effort level: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

+    if len(matches) == 1:
+        return matches[0]
+
+    logging.warning(
+        "Unable to map coverage paths for %s: expected one package directory, found %d",
+        package_name,
+        len(matches),
+    )
+    return None


Package names are not unique across sdk/ (e.g. azure-ai-textanalytics exists under both sdk/textanalytics and sdk/cognitivelanguage). The previous find_package_directory search matched by basename globally, so ambiguous packages resolved to None and their .venv site-packages paths were left unnormalized, breaking source mapping in the coverage report. Record each package's originating directory as its .coverage file is discovered in find_coverage_files (unambiguous, since the coverage file lives directly under the actual package directory that produced it), and use that mapping first in find_package_directory. Retain the prior filesystem search as a fallback for callers that don't run coverage file discovery first. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

The two moderate coverage attribution and isolate-retention issues must be addressed before approval.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

eng/scripts/dispatch_checks.py:768

This now preserves isolate directories for every Azure Pipelines invocation unless --disablecov was supplied, even when the selected checks cannot produce coverage. Existing callers such as run_pylint.yml, run_pyright.yml, and the samples invocation in build-test.yml omit that flag; their environments will therefore no longer be deleted, and those jobs do not all run the new cleanup step. This can accumulate large virtual environments for the rest of the job. Restrict preservation to the checks that actually emit package coverage (currently whl, sdist, devtest, and whl_no_aio) or to detected coverage output.

scripts/devops_tasks/create_coverage.py:49

This does not actually keep duplicate package names unambiguous: the dictionary is keyed only by the basename, and on the second directory it warns but retains the first path. The repository currently has both sdk/cognitivelanguage/azure-ai-textanalytics and sdk/textanalytics/azure-ai-textanalytics, so if both produce data, every .venv/azure-ai-textanalytics/... XML path is rewritten to whichever directory sorted first and coverage is attributed to the wrong source. Please carry a unique service/package identity through the isolate path and normalization (or otherwise rewrite each data file using its originating full directory before combining).

                existing_directory = package_directories.get(package_name)
                if existing_directory and existing_directory != root:
                    logging.warning(
                        "Multiple package directories produced coverage for %s: %s and %s. "
                        "Coverage paths for this package may not map back to repository sources.",
                        package_name,
                        existing_directory,
                        root,
                    )

Files reviewed: 11/11 changed files
Comments generated: 0 new
Review effort level: Balanced

Resolve two issues raised in Copilot PR review: 1. create_coverage.py: after "coverage combine", identically named isolate paths (e.g. azure-ai-textanalytics under both textanalytics and cognitivelanguage) collapse into one mis-attributed entry, since the recorded venv path only carries the package basename. Combine the per-package data files ourselves and relocate each file's paths to its originating package directory *before* merging, so duplicate package names stay unambiguous. Revert find_package_directory to a filesystem safety-net that returns None on ambiguity rather than guessing. 2. dispatch_checks.py: isolate directories were preserved for every CI invocation unless --disablecov was passed, even for checks that emit no coverage (pylint, pyright, samples), leaking large virtual environments in jobs without a cleanup step. Restrict preservation to the checks that actually produce package coverage (whl, whl_no_aio, sdist, devtest, optional). Also run black on test_cleanup_isolate_dirs.py and update affected tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The static-analysis workflow runs black with eng/black-pyproject.toml (line-length 120); reformat the touched files to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

The optional-only path can retain isolates without producing coverage data and requires correction or human review.

Review details

Files reviewed: 11/11 changed files
Comments generated: 1
Review effort level: Balanced

+    "whl_no_aio",
+    "sdist",
+    "devtest",
+    "optional",


Copilot

🟡 Changes recommended

Duplicate packages can corrupt XML output, and optional-only dispatches do not produce the coverage data expected by the reporting flow.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

eng/scripts/dispatch_checks.py:67

optional does not currently emit package coverage: its overridden run/prepare_and_test_optional path builds pytest arguments without --cov=<namespace>, never sets the new COVERAGE_FILE, and never calls check_coverage (azpysdk/optional.py:55-215). Consequently an optional-only dispatch is treated as coverage-producing here and preserves its isolate, but create_coverage.py still finds no data or report. Either remove optional from this set or wire its override into the same coverage flow as the other listed checks.

    "optional",

Files reviewed: 11/11 changed files
Comments generated: 1
Review effort level: Balanced

-    generate_coverage_xml()
-    create_coverage_report()
+    if collect_coverage_files() and generate_coverage_xml():
+        create_coverage_report()


Fix code coverage report collection

47defc0

Read package-local .coverage files directly when generating the combined CI coverage report. Add regression tests for discovery, combination, and XML generation gating.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jenny (JennyPng) changed the title ~~Fix code coverage report collection~~ (fix) Fix coverage report collection Aug 31, 2026

jenny (JennyPng) changed the title ~~(fix) Fix coverage report collection~~ (fix) Fix code coverage reporting Aug 31, 2026

jenny (JennyPng) added 6 commits September 2, 2026 10:57

propagate disablecov, set cov= to pytest

03d845a

fix

5cc12c8

Merge branch 'main' into fix-code-coverage

330513f

gh actions pip run should cleanup uv envs

400d2a2

add CI step for cleanup

8eb41cf

fix

1ae14ee

jenny (JennyPng) marked this pull request as ready for review September 3, 2026 22:32

jenny (JennyPng) requested review from Ben Broderick Phillips (benbp), Daniel Jurek (danieljurek) and Mike Harder (mikeharder) as code owners September 3, 2026 22:32

Copilot AI balanced review requested due to automatic review settings September 3, 2026 22:32

Copilot started reviewing on behalf of jenny (JennyPng) September 3, 2026 22:32 View session

Copilot AI reviewed Sep 3, 2026

View reviewed changes

Comment thread scripts/devops_tasks/create_coverage.py

Comment on lines +84 to +92

if len(matches) == 1:

return matches[0]

logging.warning(

"Unable to map coverage paths for %s: expected one package directory, found %d",

package_name,

len(matches),

)

return None

Copilot AI review requested due to automatic review settings September 4, 2026 19:01

Copilot started reviewing on behalf of jenny (JennyPng) September 4, 2026 19:02 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 4, 2026 19:49

Copilot started reviewing on behalf of jenny (JennyPng) September 4, 2026 19:50 View session

style: apply repo black config (line-length 120) to changed files

4773f8d

The static-analysis workflow runs black with eng/black-pyproject.toml (line-length 120); reformat the touched files to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread eng/scripts/dispatch_checks.py

"whl_no_aio",

"sdist",

"devtest",

"optional",

Copilot AI review requested due to automatic review settings September 4, 2026 19:56

Copilot started reviewing on behalf of jenny (JennyPng) September 4, 2026 19:57 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread scripts/devops_tasks/create_coverage.py

generate_coverage_xml()

create_coverage_report()

if collect_coverage_files() and generate_coverage_xml():

create_coverage_report()

Navigation Menu

Uh oh!

(fix) Fix code coverage reporting - #48822

(fix) Fix code coverage reporting#48822
jenny (JennyPng) wants to merge 10 commits into
Azure:mainfrom
JennyPng:fix-code-coverage

jenny (JennyPng) commented Aug 31, 2026 •

edited

Loading

Uh oh!

azure-pipelines Bot commented Aug 31, 2026

Uh oh!

azure-pipelines Bot commented Sep 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jenny (JennyPng) commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

azure-pipelines Bot commented Aug 31, 2026

Uh oh!

azure-pipelines Bot commented Sep 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Suppressed comments (2)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jenny (JennyPng) commented Aug 31, 2026 •

edited

Loading