Fix code coverage collection & reporting

[sarthak-19]

Summary

This PR fixes the currently broken automated codecov coverage in github workflow.

Fixes #1284

Problem Reference - #1652 (comment)

Additional Technical Details Required

As mentioned here - #1586 (comment)

Additionally, the coverage report for each plugin is getting overwritten to the same XML file. There needs to be separate XML files generated for each plugin.
Lovekesh has some additional thoughts.

Currently the xml file name is being generated like coverage-multisite-${{ github.sha }}.xml, so if it's getting overwritten for each plugin, what different identifier can we use here so that it's unique for each plugin ?

cc : @thelovekesh @westonruter

[sarthak-19]

The test is still failing, even after adding an additional arg seperator --, I ran a ls -la command to debug, the coverage file is not getting stored at root of the project.
Is that the cause or any other.

I'm not able to debug and stuck here, any help or insights would be instrumental.

cc : @thelovekesh @westonruter

[westonruter]

I see this https://github.com/WordPress/performance/actions/runs/11825633049/job/32949852740?pr=1653#step:12:49 which seems to be the problem:

warning - 2024-11-13 20:58:58,206 -- Some files were not found --- {"not_found_files": ["coverage-6c3bb1148143cc38a09cb04f2cf7400b806d21c2.xml"]}

[sarthak-19]

I see this https://github.com/WordPress/performance/actions/runs/11825633049/job/32949852740?pr=1653#step:12:49 which seems to be the problem:

warning - 2024-11-13 20:58:58,206 -- Some files were not found --- {"not_found_files": ["coverage-6c3bb1148143cc38a09cb04f2cf7400b806d21c2.xml"]}

Yes, there's no coverage file to upload and hence the test is failing, however the WorkFlow before uploading is running (where xml file is generated ) and the hash is also same I verified.
So is there a way to determine that if the file is properly getting generated or not or is this some directory path issue.
Maybe file is getting stored somewhere else, and I'm trying to upload the file from root of the project.

cc : @westonruter

[westonruter]

I believe @thelovekesh has in mind what is wrong there.

[codecov]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

[sarthak-19]

Issue with Codecov upload report job :

Currently Creating report in codecov is failing :

==> Running create-report
      ./codecov  create-report --fail-on-error --git-service github --sha c56fc50c9c306134344d9b64e5d00d789b90dc64
info - 2024-12-03 17:56:44,090 -- ci service found: github-actions
info - 2024-12-03 17:56:44,464 -- Process Report creating complete
error - 2024-12-03 17:56:44,464 -- Report creating failed: {"message":"Token required because branch is protected"}
==> Failed to create-report

Latest Commit : https://github.com/WordPress/performance/actions/runs/12145298761/job/33866607688?pr=1653

However it was running fine in previous commits :

• https://github.com/WordPress/performance/actions/runs/12145028699/job/33865747938
• https://github.com/WordPress/performance/actions/runs/12144752300/job/33864838801

==> Running create-report
      ./codecov  create-report --fail-on-error --git-service github --sha 72183d4acc81ffb64510050ff793f4bbfa568427
info - 2024-12-03 17:40:20,844 -- ci service found: github-actions
info - 2024-12-03 17:40:21,379 -- Process Report creating complete
info - 2024-12-03 17:40:21,379 -- Finished creating report successfully --- {"response": "{\"status\":\"queued\"}\n"}

Error says token is required, but previously it was working, any idea why it might be the case here ?

cc : @thelovekesh @westonruter

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @[email protected].

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Unlinked contributors: [email protected].

Co-authored-by: sarthak-19 <[email protected]>
Co-authored-by: thelovekesh <[email protected]>
Co-authored-by: westonruter <[email protected]>
Co-authored-by: swissspidy <[email protected]>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[westonruter]

I re-ran the Props Bot job that had failed due to codecov not being recognized where my PR to fix this has been merged. The job is now passing. ✅

[sarthak-19]

I'll fix the platform ^7.2 issue and indentation issue soon, then we it'll be completed. 😄

[westonruter]

Is develop a value for --xdebug? I only see profile, trace, and debug listed: https://developer.wordpress.org/block-editor/reference-guides/packages/packages-env/#using-xdebug

[sarthak-19]

Yes develop is also a value for --xdebug along with coverage as mentioned here : https://xdebug.org/docs/code_coverage#mode

But we no longer need develop so removing it.

cc : @westonruter

[westonruter]

Looks great! I'll defer to @felixarntz or @swissspidy to give final approval and merge.

[westonruter]

When running tests locally in wp-env I'm now seeing this warning:

  Warning - The configuration file did not pass validation!
  The following problems have been detected:

  Line 44:
  - Element 'coverage': This element is not expected.

  Test results may not be as expected.

I guess this is because it is running PHPUnit 8.5.39 which is a version prior to this element being introduced.

[westonruter]

Note: I noticed that code coverage was being reported for test files, so I've excluded them in ca0ca13.