ci: reintroduce code coverage reporting with cargo-llvm-cov

[buraksenn]

Which issue does this PR close?

• Closes Coverage jobs are failing on master: Failed to compile tests! Error: sql_integration: linking with cc failed: exit status: 1" #3678.

Rationale for this change

Code coverage has been disabled since #3678. The reason was it failed because of sql_integration test binary. This PR reintroduces it

What changes are included in this PR?

Reintroduce code coverage

Are these changes tested?

I've been successfully make it work in my fork: buraksenn#345

I needed to add CODECOV_TOKEN as a secret variable to the fork.

You can check #23336 (comment) comment of codecov for successful run

Are there any user-facing changes?

no

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.0.0 (current)
       Built [ 110.916s] (current)
     Parsing datafusion v54.0.0 (current)
      Parsed [   0.037s] (current)
    Building datafusion v54.0.0 (baseline)
       Built [ 107.513s] (baseline)
     Parsing datafusion v54.0.0 (baseline)
      Parsed [   0.037s] (baseline)
    Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   1.013s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 221.561s] datafusion
    Building datafusion-ffi v54.0.0 (current)
       Built [  59.655s] (current)
     Parsing datafusion-ffi v54.0.0 (current)
      Parsed [   0.064s] (current)
    Building datafusion-ffi v54.0.0 (baseline)
       Built [  59.967s] (baseline)
     Parsing datafusion-ffi v54.0.0 (baseline)
      Parsed [   0.064s] (baseline)
    Checking datafusion-ffi v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.356s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure feature_missing: package feature removed or renamed ---

Description:
A feature has been removed from this package's Cargo.toml. This will break downstream crates which enable that feature.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#cargo-feature-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/feature_missing.ron

Failed in:
  feature tarpaulin_include in the package's Cargo.toml

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 121.839s] datafusion-ffi

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@d0304b3). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #23336   +/-   ##
=======================================
  Coverage        ?   80.61%           
=======================================
  Files           ?     1086           
  Lines           ?   364976           
  Branches        ?   364976           
=======================================
  Hits            ?   294216           
  Misses          ?    53234           
  Partials        ?    17526           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[buraksenn]

As expected it could not compare since there is no code coverage uploaded for main branch. After this PR merges it will be ready

[Jefffrey]

👀

[alamb]

will this consume more of our ASF runner time?

maybe we can try it for a while and make sure it is useful before we spend too much xtra time.

[buraksenn]

this is correct can you check #23336 (comment)

[alamb]

Looks like the coverage run takes 19m and the main tests take 20m, so this adds a lot of resource usage, but it won't increase testing latancy

[Jefffrey]

would it also be an option to have it run occasionally on main only, to at least give us a general idea on the health of the codebase in terms of coverage, without consuming too many resources? 🤔

[buraksenn]

I've first introduced it as uncommenting the coverage step but cargo-llvm-cov internally runs cargo test and calculates coverage with it so instead of this I've changed cargo test (amd64) to use llvm-cov. With this it actually does not cause any resource usage:

cargo test amd64 with llvm-cov:

unchanged cargo test amd64:

Another approach would be adding it as nightly/weekly. @alamb @Jefffrey

[2010YOUY01]

LGTM overall—thank you!

Andrew mentioned the resource-usage concern, so here is some background: our CI runner resources are provided by the ASF, and they have noted that DataFusion consumes a significant amount of compute. We are still within the hard limit, IIRC, but it would be good to continue reducing our usage as they suggested.

(I think the biggest contributor is PRs with dozens of commits, but we could address that separately.)

I would prefer to run this check on every PR. It is true that 100% line coverage does not necessarily imply good test quality; however, low coverage strongly suggests that we are missing test cases. I'd want to check the coverage metrics regularly when reviewing.

[2010YOUY01]

I think they're not needed:

datafusion/datafusion/core/tests/fuzz.rs

Line 21 in f87e817

#[cfg(feature = "extended_tests")]

Those tests are under extended_tests attribute, and now we run llvm-cov in the regular test suite, so those slow tests won't be included.

In the future if there are other tests that are too expensive to run for llvm-cov, I think we should also put them in extended_tests feature, then we don't have to manage this additional coverage attribute to keep things simpler.

[buraksenn]

Thanks for the heads up, I did not think if they were necessary and blindly applied. Now they are removed