Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test compatibility with other tools #17

Closed
irongut opened this issue Oct 16, 2021 · 11 comments
Closed

Test compatibility with other tools #17

irongut opened this issue Oct 16, 2021 · 11 comments
Assignees
@irongut
Labels
enhancement New feature or request help wanted Extra attention is needed Parsing An issue reletad to parsing the coverage xml. stale

Comments

@irongut
Copy link
Owner

irongut commented Oct 16, 2021 •
edited
Loading

Feature Request

CCS has been tested with Coverlet and gcov, test with other tools that output Cobertura format.

Expected Behaviour

Obtain sample Coberura output from as many testing tools as possible, CCS should parse them correctly.

Additional Context

Linked To

#20 What does complexity mean? (issue with coverage.py)
#22 Allow hiding some output values
@irongut irongut added the enhancement New feature or request label Oct 16, 2021
@irongut irongut self-assigned this Oct 16, 2021
@irongut irongut added the Parsing An issue reletad to parsing the coverage xml. label Oct 16, 2021
This was referenced Oct 29, 2021
Closed
Closed
@irongut
Copy link
Owner Author

irongut commented Oct 29, 2021

coverage.py Sample

<coverage branch-rate="0" branches-covered="0" branches-valid="0" complexity="0" line-rate="0.6500" lines-covered="65" lines-valid="100" timestamp="1635484070186" version="6.0.2">
    <packages>
        <package branch-rate="0" complexity="0" line-rate="0.6500" name="one" />
        <package branch-rate="0" complexity="0" line-rate="0.6000" name="two" />
        <package branch-rate="0" complexity="0" line-rate="0.7000" name="three" />
    </packages>
</coverage>

(Details not relevant to CCS omitted.)

@irongut irongut pinned this issue Oct 29, 2021
@irongut irongut added the help wanted Extra attention is needed label Jan 6, 2022
@pboling
Copy link

pboling commented Feb 17, 2022 •
edited
Loading

Tried it with simplecov & simplecov-cobertura, and it failed, but that may be caused by the options I have enabled not being supported by simplecov? If that's the case this error should be handled more gracefully or avoided altogether.

Here's the log:

Run irongut/CodeCoverageSummary@v1.[2](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:2).0
  with:
    filename: ./coverage/coverage.xml
    badge: true
    fail_below_min: true
    format: markdown
    hide_branch_rate: false
    hide_complexity: true
    indicators: true
    output: both
    thresholds: 95 97
  env:
    CC_TEST_REPORTER_ID: ***
    GIT_COMMIT_SHA: 5e7798ac8bfdc4f[3](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:3)d8537d[4](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:4)23af23be[5](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:5)f33ec[6](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:6)81
    GIT_BRANCH: 1-4-stable
/usr/bin/docker run --name ghcrioirongutcodecoveragesummaryv120_db1[7](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:7)64 --label 7db2d7 --workdir /github/workspace --rm -e CC_TEST_REPORTER_ID -e GIT_COMMIT_SHA -e GIT_BRANCH -e INPUT_FILENAME -e INPUT_BADGE -e INPUT_FAIL_BELOW_MIN -e INPUT_FORMAT -e INPUT_HIDE_BRANCH_RATE -e INPUT_HIDE_COMPLEXITY -e INPUT_INDICATORS -e INPUT_OUTPUT -e INPUT_THRESHOLDS -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/oauth2/oauth2":"/github/workspace" ghcr.io/irongut/codecoveragesummary:v1.2.0  "--files" "./coverage/coverage.xml" "--badge" "true" "--fail" "true" "--format" "markdown" "--hidebranch" "false" "--hidecomplexity" "true" "--indicators" "true" "--output" "both" "--thresholds" "[9](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:9)5 97"
Code Coverage File: ./coverage/coverage.xml
Parse Error: Sequence contains no elements
Error: System.NullReferenceException - Object reference not set to an instance of an object.

Going to tweak options and see if I can get it working.

@pboling
Copy link

pboling commented Feb 17, 2022 •
edited
Loading

Simplecov Sample (Ruby, branch coverage not enabled)

<?xml version='1.0'?> 
<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<!-- Generated by simplecov-cobertura version 2.1.0 (https://github.com/dashingrocket/simplecov-cobertura) -->
<coverage line-rate="0.97" lines-covered="350" lines-valid="362" complexity="0" version="0" timestamp="1645060052">
  <sources>
    <source>/Users/pboling/src/my/oauth2</source>
  </sources>
  <packages>
    <package name="oauth2" line-rate="0.97" complexity="0">
      <classes>
        <class name="oauth2" filename="lib/oauth2.rb" line-rate="1.0" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="2" hits="1"/>
            <line number="3" hits="1"/>
            <line number="4" hits="1"/>
            <line number="5" hits="1"/>
            <line number="6" hits="1"/>
            <line number="7" hits="1"/>
            <line number="8" hits="1"/>
            <line number="9" hits="1"/>
            <line number="10" hits="1"/>
            <line number="11" hits="1"/>
            <line number="12" hits="1"/>
          </lines>
        </class>
        <class name="access_token" filename="lib/oauth2/access_token.rb" line-rate="0.96" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="2" hits="1"/>
            <line number="3" hits="1"/>
            <line number="4" hits="1"/>
            <line number="7" hits="1"/>
            <line number="13" hits="1"/>
            <line number="14" hits="7"/>
            <line number="15" hits="7"/>
            <line number="23" hits="1"/>
            <line number="24" hits="1"/>
            <line number="41" hits="1"/>
            <line number="42" hits="151"/>
            <line number="43" hits="151"/>
            <line number="44" hits="151"/>
            <line number="45" hits="151"/>
            <line number="46" hits="453"/>
            <line number="48" hits="151"/>
            <line number="49" hits="151"/>
            <line number="50" hits="151"/>
            <line number="51" hits="151"/>
            <line number="52" hits="151"/>
            <line number="55" hits="151"/>
            <line number="61" hits="1"/>
            <line number="62" hits="6"/>
            <line number="68" hits="1"/>
            <line number="69" hits="11"/>
            <line number="75" hits="1"/>
            <line number="76" hits="3"/>
            <line number="83" hits="1"/>
            <line number="84" hits="2"/>
            <line number="86" hits="2"/>
            <line number="87" hits="2"/>
            <line number="88" hits="2"/>
            <line number="89" hits="2"/>
            <line number="90" hits="2"/>
            <line number="91" hits="2"/>
            <line number="97" hits="1"/>
            <line number="98" hits="1"/>
            <line number="107" hits="1"/>
            <line number="108" hits="20"/>
            <line number="109" hits="20"/>
            <line number="115" hits="1"/>
            <line number="116" hits="2"/>
            <line number="122" hits="1"/>
            <line number="123" hits="18"/>
            <line number="129" hits="1"/>
            <line number="130" hits="2"/>
            <line number="136" hits="1"/>
            <line number="137" hits="0"/>
            <line number="143" hits="1"/>
            <line number="144" hits="2"/>
            <line number="148" hits="1"/>
            <line number="149" hits="8"/>
            <line number="152" hits="1"/>
            <line number="154" hits="1"/>
            <line number="155" hits="20"/>
            <line number="157" hits="8"/>
            <line number="158" hits="8"/>
            <line number="160" hits="8"/>
            <line number="161" hits="8"/>
            <line number="163" hits="4"/>
            <line number="164" hits="4"/>
            <line number="165" hits="4"/>
            <line number="167" hits="0"/>
            <line number="171" hits="0"/>
            <line number="175" hits="1"/>
            <line number="176" hits="13"/>
            <line number="178" hits="12"/>
          </lines>
        </class>
        <class name="authenticator" filename="lib/oauth2/authenticator.rb" line-rate="1.0" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="3" hits="1"/>
            <line number="4" hits="1"/>
            <line number="5" hits="1"/>
            <line number="7" hits="1"/>
            <line number="8" hits="106"/>
            <line number="9" hits="106"/>
            <line number="10" hits="106"/>
            <line number="22" hits="1"/>
            <line number="23" hits="106"/>
            <line number="25" hits="14"/>
            <line number="27" hits="89"/>
            <line number="29" hits="1"/>
            <line number="31" hits="1"/>
            <line number="33" hits="1"/>
            <line number="37" hits="1"/>
            <line number="38" hits="15"/>
            <line number="41" hits="1"/>
            <line number="45" hits="1"/>
            <line number="46" hits="89"/>
            <line number="51" hits="1"/>
            <line number="52" hits="1"/>
            <line number="57" hits="1"/>
            <line number="58" hits="14"/>
            <line number="59" hits="14"/>
            <line number="60" hits="14"/>
            <line number="64" hits="1"/>
            <line number="65" hits="14"/>
          </lines>
        </class>
        <class name="client" filename="lib/oauth2/client.rb" line-rate="0.95" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="2" hits="1"/>
            <line number="4" hits="1"/>
            <line number="6" hits="1"/>
            <line number="7" hits="1"/>
            <line number="9" hits="1"/>
            <line number="10" hits="1"/>
            <line number="11" hits="1"/>
            <line number="31" hits="1"/>
            <line number="32" hits="204"/>
            <line number="33" hits="204"/>
            <line number="34" hits="204"/>
            <line number="35" hits="204"/>
            <line number="36" hits="204"/>
            <line number="39" hits="204"/>
            <line number="49" hits="204"/>
            <line number="55" hits="1"/>
            <line number="56" hits="0"/>
            <line number="57" hits="0"/>
            <line number="61" hits="1"/>
            <line number="62" hits="423"/>
            <line number="63" hits="158"/>
            <line number="64" hits="158"/>
            <line number="65" hits="134"/>
            <line number="66" hits="134"/>
            <line number="69" hits="158"/>
            <line number="76" hits="1"/>
            <line number="77" hits="11"/>
            <line number="78" hits="11"/>
            <line number="84" hits="1"/>
            <line number="85" hits="100"/>
            <line number="100" hits="1"/>
            <line number="101" hits="141"/>
            <line number="103" hits="141"/>
            <line number="105" hits="141"/>
            <line number="106" hits="141"/>
            <line number="107" hits="141"/>
            <line number="109" hits="141"/>
            <line number="111" hits="141"/>
            <line number="113" hits="3"/>
            <line number="114" hits="3"/>
            <line number="115" hits="3"/>
            <line number="117" hits="2"/>
            <line number="118" hits="1"/>
            <line number="119" hits="1"/>
            <line number="121" hits="2"/>
            <line number="124" hits="127"/>
            <line number="126" hits="11"/>
            <line number="127" hits="11"/>
            <line number="129" hits="2"/>
            <line number="130" hits="2"/>
            <line number="132" hits="0"/>
            <line number="133" hits="0"/>
            <line number="143" hits="1"/>
            <line number="144" hits="97"/>
            <line number="145" hits="182"/>
            <line number="146" hits="2"/>
            <line number="148" hits="180"/>
            <line number="151" hits="97"/>
            <line number="153" hits="97"/>
            <line number="154" hits="97"/>
            <line number="155" hits="97"/>
            <line number="156" hits="97"/>
            <line number="157" hits="79"/>
            <line number="158" hits="79"/>
            <line number="160" hits="18"/>
            <line number="161" hits="18"/>
            <line number="163" hits="97"/>
            <line number="164" hits="97"/>
            <line number="167" hits="97"/>
            <line number="169" hits="0"/>
            <line number="172" hits="97"/>
            <line number="173" hits="3"/>
            <line number="174" hits="3"/>
            <line number="176" hits="94"/>
            <line number="182" hits="1"/>
            <line number="183" hits="48"/>
            <line number="189" hits="1"/>
            <line number="190" hits="5"/>
            <line number="196" hits="1"/>
            <line number="197" hits="11"/>
            <line number="203" hits="1"/>
            <line number="204" hits="21"/>
            <line number="207" hits="1"/>
            <line number="208" hits="9"/>
            <line number="227" hits="1"/>
            <line number="228" hits="55"/>
            <line number="229" hits="2"/>
            <line number="231" hits="53"/>
            <line number="235" hits="1"/>
            <line number="236" hits="91"/>
            <line number="237" hits="91"/>
            <line number="240" hits="1"/>
            <line number="242" hits="1"/>
            <line number="243" hits="97"/>
            <line number="244" hits="97"/>
            <line number="246" hits="93"/>
            <line number="250" hits="93"/>
            <line number="251" hits="1"/>
            <line number="253" hits="92"/>
          </lines>
        </class>
        <class name="error" filename="lib/oauth2/error.rb" line-rate="1.0" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="2" hits="1"/>
            <line number="3" hits="1"/>
            <line number="7" hits="1"/>
            <line number="8" hits="14"/>
            <line number="9" hits="14"/>
            <line number="11" hits="14"/>
            <line number="12" hits="7"/>
            <line number="13" hits="7"/>
            <line number="14" hits="7"/>
            <line number="17" hits="14"/>
            <line number="23" hits="1"/>
            <line number="24" hits="14"/>
            <line number="26" hits="14"/>
            <line number="28" hits="14"/>
            <line number="29" hits="7"/>
            <line number="30" hits="7"/>
            <line number="32" hits="7"/>
            <line number="35" hits="14"/>
            <line number="37" hits="14"/>
          </lines>
        </class>
        <class name="mac_token" filename="lib/oauth2/mac_token.rb" line-rate="0.95" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="2" hits="1"/>
            <line number="3" hits="1"/>
            <line number="4" hits="1"/>
            <line number="6" hits="1"/>
            <line number="7" hits="1"/>
            <line number="14" hits="1"/>
            <line number="15" hits="3"/>
            <line number="18" hits="1"/>
            <line number="30" hits="1"/>
            <line number="31" hits="19"/>
            <line number="32" hits="19"/>
            <line number="34" hits="18"/>
            <line number="43" hits="1"/>
            <line number="44" hits="4"/>
            <line number="46" hits="4"/>
            <line number="47" hits="4"/>
            <line number="49" hits="4"/>
            <line number="53" hits="1"/>
            <line number="54" hits="1"/>
            <line number="61" hits="1"/>
            <line number="62" hits="9"/>
            <line number="63" hits="9"/>
            <line number="65" hits="9"/>
            <line number="67" hits="8"/>
            <line number="69" hits="7"/>
            <line number="71" hits="7"/>
            <line number="80" hits="1"/>
            <line number="82" hits="8"/>
            <line number="91" hits="8"/>
            <line number="97" hits="1"/>
            <line number="98" hits="19"/>
            <line number="101" hits="1"/>
            <line number="103" hits="0"/>
            <line number="107" hits="17"/>
            <line number="109" hits="0"/>
            <line number="112" hits="1"/>
            <line number="116" hits="1"/>
            <line number="120" hits="1"/>
            <line number="124" hits="1"/>
            <line number="125" hits="8"/>
          </lines>
        </class>
        <class name="response" filename="lib/oauth2/response.rb" line-rate="1.0" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="2" hits="1"/>
            <line number="3" hits="1"/>
            <line number="5" hits="1"/>
            <line number="7" hits="1"/>
            <line number="8" hits="1"/>
            <line number="9" hits="1"/>
            <line number="14" hits="62"/>
            <line number="15" hits="44"/>
            <line number="16" hits="3"/>
            <line number="20" hits="1"/>
            <line number="32" hits="1"/>
            <line number="33" hits="3"/>
            <line number="34" hits="3"/>
            <line number="35" hits="3"/>
            <line number="36" hits="6"/>
            <line number="46" hits="1"/>
            <line number="47" hits="147"/>
            <line number="48" hits="147"/>
            <line number="52" hits="1"/>
            <line number="53" hits="6"/>
            <line number="57" hits="1"/>
            <line number="58" hits="150"/>
            <line number="62" hits="1"/>
            <line number="63" hits="154"/>
            <line number="69" hits="1"/>
            <line number="70" hits="134"/>
            <line number="72" hits="131"/>
            <line number="76" hits="1"/>
            <line number="77" hits="244"/>
            <line number="81" hits="1"/>
            <line number="82" hits="244"/>
            <line number="84" hits="244"/>
            <line number="89" hits="1"/>
            <line number="90" hits="1"/>
          </lines>
        </class>
        <class name="assertion" filename="lib/oauth2/strategy/assertion.rb" line-rate="0.89" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="3" hits="1"/>
            <line number="4" hits="1"/>
            <line number="23" hits="1"/>
            <line number="27" hits="1"/>
            <line number="28" hits="1"/>
            <line number="45" hits="1"/>
            <line number="46" hits="8"/>
            <line number="47" hits="8"/>
            <line number="50" hits="1"/>
            <line number="51" hits="8"/>
            <line number="53" hits="8"/>
            <line number="60" hits="1"/>
            <line number="62" hits="8"/>
            <line number="67" hits="8"/>
            <line number="68" hits="8"/>
            <line number="69" hits="0"/>
            <line number="70" hits="0"/>
          </lines>
        </class>
        <class name="auth_code" filename="lib/oauth2/strategy/auth_code.rb" line-rate="1.0" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="2" hits="1"/>
            <line number="6" hits="1"/>
            <line number="10" hits="1"/>
            <line number="11" hits="3"/>
            <line number="17" hits="1"/>
            <line number="18" hits="3"/>
            <line number="27" hits="1"/>
            <line number="28" hits="44"/>
            <line number="30" hits="44"/>
          </lines>
        </class>
        <class name="base" filename="lib/oauth2/strategy/base.rb" line-rate="1.0" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="2" hits="1"/>
            <line number="3" hits="1"/>
            <line number="4" hits="1"/>
            <line number="5" hits="95"/>
          </lines>
        </class>
        <class name="client_credentials" filename="lib/oauth2/strategy/client_credentials.rb" line-rate="1.0" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="2" hits="1"/>
            <line number="6" hits="1"/>
            <line number="10" hits="1"/>
            <line number="11" hits="1"/>
            <line number="18" hits="1"/>
            <line number="19" hits="20"/>
            <line number="20" hits="20"/>
          </lines>
        </class>
        <class name="implicit" filename="lib/oauth2/strategy/implicit.rb" line-rate="1.0" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="2" hits="1"/>
            <line number="6" hits="1"/>
            <line number="10" hits="1"/>
            <line number="11" hits="3"/>
            <line number="17" hits="1"/>
            <line number="18" hits="3"/>
            <line number="24" hits="1"/>
            <line number="25" hits="1"/>
          </lines>
        </class>
        <class name="password" filename="lib/oauth2/strategy/password.rb" line-rate="1.0" complexity="0">
          <methods/>
          <lines>
            <line number="1" hits="1"/>
            <line number="2" hits="1"/>
            <line number="6" hits="1"/>
            <line number="10" hits="1"/>
            <line number="11" hits="1"/>
            <line number="19" hits="1"/>
            <line number="20" hits="10"/>
            <line number="23" hits="10"/>
          </lines>
        </class>
      </classes>
    </package>
  </packages>
</coverage>

@irongut
Copy link
Owner Author

irongut commented Feb 17, 2022

Hi @pboling thanks for the sample, that's always useful.

Looking at the sample there are no branch metrics on coverage or package nodes. Unfortunately the hide_branch_rate option only hides it from the report, the action still attempts to calculate branch metrics.

It's funny how many different ways different tools can output what is supposed to be the same format. I'll need to make some changes to prevent attempting to calculate metrics that don't exist in the data.

@pboling
Copy link

pboling commented Feb 17, 2022 •
edited
Loading

@irongut Branch coverage can be turned on, but I've never tried it. I'll see what that generates now. But, yeah, definitely good to support the lack of any specific feature. Always assume the null case :)

@irongut
Copy link
Owner Author

irongut commented Feb 17, 2022

@pboling Looking at the docs, enable branch coverage but leave primary coverage set to the default of line. That should be compatible with the current version of CCS.

I'll add a note to the CCS docs and include a fix for missing branch rate in the next version.

@irongut irongut mentioned this issue Feb 17, 2022
Closed
@github-actions
Copy link

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days.

@github-actions github-actions bot added the stale label May 28, 2022
@github-actions
Copy link

This issue was closed because it has been stale for 30 days with no activity.

@pboling
Copy link

pboling commented Jun 27, 2022

NOTE: the suggested config is working. I wrote up a how to on dev.to.

Screen Shot 2022-06-28 at 5 28 32 AM

@irongut irongut removed the stale label Jul 24, 2022
@irongut irongut reopened this Jul 24, 2022
@github-actions
Copy link

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days.

@github-actions github-actions bot added the stale label Oct 23, 2022
@github-actions
Copy link

This issue was closed because it has been stale for 30 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@irongut irongut

Labels
enhancement New feature or request help wanted Extra attention is needed Parsing An issue reletad to parsing the coverage xml. stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pboling @irongut