This directory contains all the Scorecard probes. Each probe has its own subdirectory.
A probe is an individual heuristic, which provides information about a distinct behavior a project under analysis may or may not be doing. The probes follow a camelcase naming convention that describe the exact heuristic a particular probe assesses. The name should be phrased in a way that can be answered by "true" or "false".
Probes can return one or more findings, where a finding is a piece of data with an outcome, message, and optionally a location where the behavior was observed.
The primary outcomes are finding.OutcomeTrue and finding.OutcomeFalse, but other outcomes are available as well.
For example, the finding.OutcomeNotAvailable is often used for scenarios, where Scorecard cannot assess a behavior because there is no data to analyze.
A probe consists of three files:
def.yml: The documentation of the probe.impl.go: The actual implementation of the probe.impl_test.go: The probe's test.
Most probes read raw data that a check collects, such as the commits the Maintained check lists. A probe can also collect its own data; these are called independent probes. To add a check as well, see checks/write.md.
The steps use a probe named myProbe that reads the Maintained check's raw
data. Replace both with yours.
-
Create the files. From the repository root, run:
make setup-probe probeName=myProbe
This creates
probes/myProbe/withdef.yml,impl.go, andimpl_test.go. -
Fill in
def.yml. Scorecard checksid,lifecycle,remediation, andecosystemwhen the probe creates its first finding, so a mistake there shows up in the probe's own tests:id: the probe's name,myProbe.lifecycle:experimentalfor a new probe. See Lifecycle.short,motivation,implementation: what the probe checks, why it matters, and how it decides. These appear in docs/probes.md.outcome: a list with one entry per outcome the probe can return.remediation:onOutcome: the outcome that needs fixing,TrueorFalse.effort:Low,Medium, orHigh.text, and optionallymarkdown: the steps to fix it.
ecosystem:languages: the languages the probe applies to, fromclients/languages.go, orall.clients: where the probe works:github,gitlab,localdir.
hasRecentCommits/def.ymlis a complete example. -
Write
Runinimpl.go. It reads the raw data and returns findings, the probe's name, and an error. The scaffold already declaresfs(the embeddeddef.yml) and theProbename constant;uerrorisgithub.com/ossf/scorecard/v5/probes/internal/utils/uerror:func Run(raw *checker.RawResults) ([]finding.Finding, string, error) { if raw == nil { return nil, "", fmt.Errorf("%w: raw", uerror.ErrNil) } outcome := finding.OutcomeFalse if len(raw.MaintainedResults.DefaultBranchCommits) > 0 { outcome = finding.OutcomeTrue } f, err := finding.NewWith(fs, Probe, "explain the outcome here", nil, outcome) if err != nil { return nil, Probe, fmt.Errorf("create finding: %w", err) } return []finding.Finding{*f}, Probe, nil }
Pass a
*finding.Locationinstead ofnilwhen the finding points to a file. Attach data for the check's evaluation withf.WithValues(...).hasRecentCommits/impl.gois a complete example. -
Register the probe. Uncomment the
init()inimpl.gothat fits, and add the imports its comment names. A probe that reads a check's raw data names that check:func init() { probes.MustRegister(Probe, Run, []checknames.CheckName{checknames.Maintained}) }
An independent probe has
func Run(c *checker.CheckRequest) ([]finding.Finding, string, error), reads its data throughc.RepoClientas a collector does, and registers withprobes.MustRegisterIndependent(Probe, Run). Registered probes can be run by name withscorecard --probes=myProbe. -
List the probe in
entries.go. Import the package, then addmyProbe.Runto one list:- a check's group, such as
Maintained, if the check should use it to score. Also addmyProbe.Probeto the list of probe names that the check's evaluation inchecks/evaluation/passes tofinding.UniqueProbesEqual, and score its findings there. Uncategorized, if no check uses it yet.Independent, for an independent probe.
Without this step the probe's package is never loaded, and
--probes=myProbefails withprobe "myProbe" not found. No test catches it. - a check's group, such as
-
Test the probe in
impl_test.go. Buildchecker.RawResultsby hand for each case and compare the outcomes.test.AssertOutcomesininternal/utils/testdoes the comparison;hasRecentCommits/impl_test.gois an example. -
Regenerate the docs and check your work. Run
make generate-docsand commit the updateddocs/probes.md; CI fails if it is out of date. Then runmake unit-testandmake check-linter.
Probes can exist in several different lifecycle states:
experimental: The semantics of the probe may change, and there are no stability guarantees.stable: The probe behavior and semantics will not change. There may be bug fixes as needed.deprecated: The probe is no longer supported and callers should not expect it to be maintained.
When multiple probes use the same code, the reused code can be placed in a package under probes/internal/
In general, browsing through the Scorecard GitHub issues is the best way to find new probes to add. Requests for support for new tools, fuzzing engines or other heuristics can often be converted into specific probes.
Probe definitions can display links following standard markdown format.
Probe definitions can display dynamic content. This requires modifications in def.yml and impl.go and in the evaluation steps.
The following snippet in def.yml will display dynamic data provided by impl.go:
${{ metadata.dataToDisplay }}And then in impl.go add the following metadata:
f, err := finding.NewWith(fs, Probe,
"Message", nil,
finding.OutcomeTrue)
f = f.WithRemediationMetadata(map[string]string{
"dataToDisplay": "this is the text we will display",
})Consider a probe with following line in its def.yml:
The project ${{ metadata.oss-fuzz-integration-status }} integrated into OSS-Fuzz.
and the probe sets the following metadata:
f, err := finding.NewWith(fs, Probe,
"Message", nil,
finding.OutcomeTrue)
f = f.WithRemediationMetadata(map[string]string{
"oss-fuzz-integration-status": "is",
})The probe will then output the following text:
The project is integrated into OSS-Fuzz.
The remediation data must be set in the probe.