Implemented --fail-under flag #(514) #2051

Victowolf · 2025-03-25T06:46:11Z

Thanks for your contribution! Please replace this text with a description of what this PR is changing or adding and why, list any relevant issues, and review the contribution guidelines below.

I’ve reviewed the contributor guide and applied the relevant portions to this PR.

Contribution guidelines:

See our contributor guide for general expectations for PRs.
Larger or significant changes should be discussed in an issue before creating a PR.
Contributions to our repos should follow the Dart style guide and use dart format.
Most changes should add an entry to the changelog and may need to rev the pubspec package version.
Changes to packages require corresponding tests.

Note that many Dart repos have a weekly cadence for reviewing PRs - please allow for some latency before initial review feedback.

I was trying to do this implementation, but got stuck in the last.

(First approach):-

This is all what I did:

Added --fail-under flag to ArgParser ( parsing CLI arguments) this function is present in bin/format_coverage.dart

Added this logic to extract and validate --fail-under after parsing.

    final failUnder = args['fail-under'] as String?;
    int? failUnderThreshold;

    if (failUnder != null) {
      failUnderThreshold = int.tryParse(failUnder);
      if (failUnderThreshold == null || failUnderThreshold < 0 || failUnderThreshold > 100) {
         fail('Invalid value for --fail-under. It must be a number between 0 and 100.');
        }
    }

Modified the Environment constructor and included failUnderThreshold so that it can be accessed later. then passed failUnderThreshold in parseArgs return statement.

I found that coverage calculation is happening inside the formatLcov function, which is present in lib/src/formatter.dart and added this logic to check fail-under:

   buf.write('LF:${lineHits.length}\n');
   buf.write('LH:${lineHits.values.where((v) => v > 0).length}\n');

   int totalLines = lineHits.length;
   int coveredLines = lineHits.values.where((v) => v > 0).length;
   double coveragePercentage = totalLines == 0 ? 0.0 : (coveredLines / totalLines) * 100;

   //Adds the coverage percentage to output.
   buf.write('Coverage: ${coveragePercentage.toStringAsFixed(2)}%\n');

   //Fail-under to check
   if (failUnderThreshold != null && coveragePercentage < failUnderThreshold) {
       throw Exception(
          'Coverage check failed! Coverage (${coveragePercentage.toStringAsFixed(2)}%) '
          'is below the required threshold ($failUnderThreshold%).'
        );
   }

Testing:

Ran dart test --coverage=coverage and gathered coverage data. data was successfully gathered with all tests passed.
Converted the coverage data to .json file by running: dart run coverage:format_coverage -i coverage/test -o coverage/coverage.json --report-on=lib
Then tested the --fail-under by running: dart bin/format_coverage.dart -i coverage/coverage.json --fail-under=90
Test was conducted in VScode terminal (Powershell).

Result:

no output was recevied

   PS C:\Users\HP\tools\pkgs\coverage> dart bin/format_coverage.dart -i coverage/coverage.json --fail-under=90
   PS C:\Users\HP\tools\pkgs\coverage>

Conclusion:

Not able to find the cause for which output is empty.

Possible cause:

As the fail under check is one file (lib/src/formatter.dart) and the parser is in another file (bin/format_coverage.dart), it is not getting executed when I run: dart bin/format_coverage.dart -i coverage/coverage.json --fail-under=90

(Secondary approach):-

Following the possible cause of previous approach.
I tried to shift the coveragePercentage calculation to bin/format_coverage.dart to get it executed.

This is what I did:

Declared a global variable double coveragePercentage = 0.0;, in bin/format_coverage.dart.
Defined a new function to calculated the coverage and called this function just after the hitmap calculation calculateCoverage(hitmap); :

    // Compute overall coverage percentage
    void calculateCoverage(Map<String, HitMap> hitmap) {
      int totalLines = 0;
      int coveredLines = 0;

      for (var entry in hitmap.values) {
        totalLines += entry.lineHits.length;
        coveredLines += entry.lineHits.values.where((v) => v > 0).length;
     }

      coveragePercentage = totalLines == 0 ? 0.0 : (coveredLines / totalLines) * 100;
      print('Overall Coverage: ${coveragePercentage.toStringAsFixed(2)}%');
   }

Added fail-under check inside bin/format_coverage.dart instead of adding it in lib/src/formatter.dart.

 // fail-under check.
 if (failUnderThreshold != null && coveragePercentage < failUnderThreshold) {
   print(
      'Coverage check failed! Coverage (${coveragePercentage.toStringAsFixed(2)}%) '
      'is below the required threshold ($failUnderThreshold%).');
   exit(1); // Fail the CI/CD pipeline
  }

Testing:

Ran dart test --coverage=coverage and gathered coverage data. data was successfully gathered with all tests passed.
Converted the coverage data to .json file by running: dart run coverage:format_coverage -i coverage/test -o coverage/coverage.json --report-on=lib
Then tested the --fail-under by running: dart bin/format_coverage.dart -i coverage/coverage.json --fail-under=90
Test was conducted in VScode terminal (Powershell).

Result:

Output was received

   PS C:\Users\HP\tools\pkgs\coverage> dart bin/format_coverage.dart -i coverage/coverage.json --fail-under=90
   Parsed failUnderThreshold: 90
   Coverage check failed! Coverage (0.00%) is below the required threshold (90%).
   PS C:\Users\HP\tools\pkgs\coverage>

CoveragePercentage was not calculated, the debuging statemenets that i had added in calculateCoverage function didnt get executed. It used predefinded value double coveragePercentage = 0.0; .

Conclusion:

calculateCoverage function didnt get executed.
failunder check used predefined value of coveragePercentage to check and return output.

Possible cause:

Still trying to figure it out.
Any suggestions, feedback will be helpfull.

…art-lang#533

github-actions · 2025-04-17T12:55:59Z

PR Health

Breaking changes

⚠️

Package	Change	Current Version	New Version	Needed Version	Looking good?
coverage	Non-Breaking	1.12.0	1.12.0-wip	1.13.0 Got "1.12.0-wip" expected >= "1.13.0" (non-breaking changes)	⚠️

This check can be disabled by tagging the PR with skip-breaking-check.

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

Coverage

⚠️

File	Coverage
pkgs/coverage/bin/format_coverage.dart	💔 45 % ⬇️ 7 %
pkgs/coverage/lib/src/collect.dart	💔 86 % ⬇️ 1 %
pkgs/coverage/lib/src/formatter.dart	💔 82 % ⬇️ 15 %
pkgs/coverage/lib/src/resolver.dart	💚 96 % ⬆️ 0 %

This check for test coverage is informational (issues shown here will not fail the PR).

This check can be disabled by tagging the PR with skip-coverage-check.

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package	Leaked API symbols

License Headers ✔️

// Copyright (c) 2025, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers

Files
pkgs/bazel_worker/benchmark/benchmark.dart
pkgs/bazel_worker/example/client.dart
pkgs/bazel_worker/example/worker.dart
pkgs/benchmark_harness/integration_test/perf_benchmark_test.dart
pkgs/boolean_selector/example/example.dart
pkgs/clock/lib/clock.dart
pkgs/clock/lib/src/clock.dart
pkgs/clock/lib/src/default.dart
pkgs/clock/lib/src/stopwatch.dart
pkgs/clock/lib/src/utils.dart
pkgs/clock/test/clock_test.dart
pkgs/clock/test/default_test.dart
pkgs/clock/test/stopwatch_test.dart
pkgs/clock/test/utils.dart
pkgs/coverage/lib/src/coverage_options.dart
pkgs/coverage/test/collect_coverage_config_test.dart
pkgs/coverage/test/config_file_locator_test.dart
pkgs/html/example/main.dart
pkgs/html/lib/dom.dart
pkgs/html/lib/dom_parsing.dart
pkgs/html/lib/html_escape.dart
pkgs/html/lib/parser.dart
pkgs/html/lib/src/constants.dart
pkgs/html/lib/src/encoding_parser.dart
pkgs/html/lib/src/html_input_stream.dart
pkgs/html/lib/src/list_proxy.dart
pkgs/html/lib/src/query_selector.dart
pkgs/html/lib/src/token.dart
pkgs/html/lib/src/tokenizer.dart
pkgs/html/lib/src/treebuilder.dart
pkgs/html/lib/src/utils.dart
pkgs/html/test/dom_test.dart
pkgs/html/test/parser_feature_test.dart
pkgs/html/test/parser_test.dart
pkgs/html/test/query_selector_test.dart
pkgs/html/test/selectors/level1_baseline_test.dart
pkgs/html/test/selectors/level1_lib.dart
pkgs/html/test/selectors/selectors.dart
pkgs/html/test/support.dart
pkgs/html/test/tokenizer_test.dart
pkgs/pubspec_parse/test/git_uri_test.dart
pkgs/stack_trace/example/example.dart
pkgs/watcher/test/custom_watcher_factory_test.dart
pkgs/yaml_edit/example/example.dart

Victowolf added 6 commits March 6, 2025 04:13

Added uncovered files detection issue dart-lang#529

a75eb7e

Updated documentation for collect function regarding isolate groups d…

f7a6cc8

…art-lang#533

Refined filtering functionality for collect function dart-lang#529

d126883

Fix BRDA syntax in lcov files dart-lang#517

da955fd

Update changelog for LCOV BRDA fix

e5ccb70

Implemented --fail-under flag

d42db2d

github-actions bot added the package:coverage label Mar 25, 2025

Victowolf changed the title ~~Implemented --fail-under flag #issue 514~~ Implemented --fail-under flag #(514) Mar 25, 2025

mosuem requested a review from liamappelbe April 17, 2025 12:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implemented --fail-under flag #(514) #2051

Implemented --fail-under flag #(514) #2051

Victowolf commented Mar 25, 2025

github-actions bot commented Apr 17, 2025

Implemented --fail-under flag #(514) #2051

Are you sure you want to change the base?

Implemented --fail-under flag #(514) #2051

Conversation

Victowolf commented Mar 25, 2025

(First approach):-

This is all what I did:

Testing:

Result:

Conclusion:

Possible cause:

(Secondary approach):-

This is what I did:

Testing:

Result:

Conclusion:

Possible cause:

github-actions bot commented Apr 17, 2025

PR Health