The scorecard command-line tool runs Scorecard security checks or probes
against one or more source repositories and writes the results to standard
output or a file. This spec covers the command tree, input selection, the
scan flag list, feature-flag environment variables, validation errors,
progress messages, result ordering, logging verbosity, and exit status. The
behavior activated by check, probe, output, and policy flags is defined in
the capability each flag names.
The system SHALL provide a root scorecard command that runs a scan, a
serve subcommand that starts the HTTP server, and a version subcommand
that prints build information. The scan flags SHALL be accepted only by the
root command; subcommands SHALL reject them as unknown flags. serve SHALL
accept --listen, whose behavior is defined in http-server.
- WHEN a user runs
scorecard --repo=github.com/owner/repo - THEN the system scans that repository and prints the results
- WHEN a user runs
scorecard serve --verbosity=debug - THEN the system fails with
unknown flag: --verbosity
- WHEN a user runs
scorecard serve --listen 0.0.0.0 - THEN the system starts the HTTP server on all interfaces
The system MUST require exactly one of --repo, --repos, --org,
--local, --npm, --pypi, --rubygems, or --nuget. When zero or more
than one is set, the system MUST fail validation with the message
"exactly one of repo, repos, org, npm, pypi, rubygems, nuget
or local must be set".
- WHEN a user runs
scorecardwith no input flag - THEN the system reports the "exactly one of" validation error
- AND prints usage and exits with a non-zero status
- WHEN a user runs
scorecard --repo=owner/repo --local=. - THEN the system reports the "exactly one of" validation error and exits with a non-zero status
The system SHALL validate all options before scanning and SHALL report every failed validation together, prefixed with "validating options: some options could not be validated". On a validation failure the system SHALL print command usage. Once validation succeeds, the system SHALL NOT print usage for later runtime errors.
- WHEN a user sets no input source and
--file-mode=svn - THEN the error lists both the input-source error and "unsupported file mode"
- GIVEN options that pass validation
- WHEN check selection fails because a check name is invalid
- THEN the system prints the error without printing usage
The system SHALL accept a repository via --repo in the forms owner/repo,
github.com/owner/repo, or a full URL. The system SHALL try to interpret
the value as a GitHub repository first and as a GitLab repository otherwise.
When SCORECARD_EXPERIMENTAL is set, the system SHALL also try Azure DevOps
between GitHub and GitLab, and SHALL fail immediately for a value on an
Azure DevOps host that cannot be parsed.
- WHEN a user runs
scorecard --repo=ossf/scorecard - THEN the system scans
github.com/ossf/scorecard
- WHEN a user runs
scorecard --repo=::bad:: - THEN the system prints
Skipping ::bad::: unable to parse as github, gitlab, or azuredevops: ...to standard error
- GIVEN
SCORECARD_EXPERIMENTALis set - WHEN a user passes a
dev.azure.comor legacyvisualstudio.comrepository URL - THEN the system scans it as an Azure DevOps repository
The system SHALL accept a comma-separated list of repositories via
--repos, trimming whitespace and ignoring empty entries. The system SHALL
scan each repository in order and SHALL continue with the remaining
repositories when one cannot be parsed or scanned.
- WHEN a user runs
scorecard --repos=ossf/scorecard,ossf/scorecard-action - THEN the system scans both repositories in that order
- GIVEN the second of three repositories cannot be scanned
- WHEN the scan runs
- THEN the system prints
Skipping <uri>: <error>to standard error for that repository - AND still scans the third repository
The system SHALL accept a GitHub organization via --org as owner,
github.com/owner, http://github.com/owner, or https://github.com/owner.
The system SHALL scan every non-archived repository in that organization,
following API pagination. When the organization's repositories cannot be
listed, the system MUST fail with a non-zero exit status.
- WHEN a user runs
scorecard --org=github.com/ossf - THEN the system scans every non-archived repository owned by
ossf - AND skips archived repositories
- WHEN listing repositories for the organization returns an error
- THEN the system fails with
listing repositories for org "<org>": ...and a non-zero exit status
The system SHALL accept a local folder via --local and scan its files
without contacting a repository host for repository data.
- WHEN a user runs
scorecard --local=./project --checks=License - THEN the system runs the License check against the files in
./project
The system SHALL accept a package name via --npm, --pypi, --rubygems,
or --nuget and SHALL scan the source repository that the package registry
associates with that package. Failure to resolve the package MUST stop the
command with a non-zero exit status. Resolution rules are defined in the
package manager resolution spec.
- WHEN a user runs
scorecard --npm=left-pad - THEN the system resolves the package's source repository and scans it
- WHEN the registry lookup for the package fails
- THEN the system fails with an error naming the package manager resolution step and exits with a non-zero status
The root command SHALL register the scan flags below. Each flag's behavior SHALL be the one defined by the capability named for it:
--checks(comma-separated, default empty): check selection, defined incheck-registry.--probes(comma-separated, default empty): probe execution, defined inprobes.--format(defaultdefault): format names and gating, defined inresult-formats.--output(shorthand-o, default empty): output destination, defined inresult-formats.--show-details(boolean, default false): detail rendering, defined inresult-formats.--show-annotations(boolean, default false): annotation rendering, defined inresult-formats; annotation discovery is defined inmaintainer-annotations.--policy(default empty): policy file and policy-driven selection, defined inpolicy.
The system SHALL register --policy only when ENABLE_SARIF is set;
without it, --policy MUST be rejected as an unknown flag.
- WHEN a user runs
scorecard --repo=owner/repo -o results.json - THEN the system treats
-o results.jsonas--output=results.json
- GIVEN
ENABLE_SARIFis not set - WHEN a user runs
scorecard --repo=owner/repo --policy=policy.yml - THEN the command fails with
unknown flag: --policy
The system SHALL analyze the commit given by --commit, defaulting to
HEAD. An empty --commit value MUST fail validation with
"commit should be non-empty". Which checks can run at a commit other than
HEAD is defined in check-registry.
- WHEN a user runs
scorecard --repo=owner/repo --commit=<sha> - THEN the system analyzes the repository at
<sha>
- WHEN a user runs
scorecard --repo=owner/repo --commit= - THEN validation fails with "commit should be non-empty"
The system SHALL limit commit-history analysis to the number of commits
given by --commit-depth, counting back from the analyzed commit. A value of
zero or less SHALL use the default depth of 30 commits.
- WHEN a user omits
--commit-depth - THEN commit-history checks examine up to the 30 most recent commits
- WHEN a user runs with
--commit-depth=100 - THEN commit-history checks examine up to 100 commits
The system SHALL accept --file-mode values archive (default) and git,
case-insensitively. In git mode the system SHALL fetch repository files
with git instead of the host's export archive. Any other value MUST fail
validation with "unsupported file mode".
- WHEN a user runs with
--file-mode=git - THEN repository files are fetched with git
- WHEN a user runs with
--file-mode=svn - THEN validation fails with "unsupported file mode"
The system SHALL append each comma-separated value of --metadata to the
metadata of every result it produces.
- WHEN a user runs with
--metadata=team=infra,tier=1 - THEN each result's metadata includes
team=infraandtier=1
In default format only, the system SHALL print
Starting (<uri>) [<check>] for each enabled check before a scan and
Finished (<uri>) [<check>] after it, followed by a RESULTS banner, all
to standard error. When probes are selected, the messages SHALL read
Starting (<uri>) probe [<probe>] and Finished (<uri>) probe <probe>
without the banner. The check messages SHALL name the checks enabled before
the repository-type filter is applied, so they can name a check that the
scan does not run, and SHALL appear in no defined order.
- WHEN a user runs
scorecard --local=. --checks=License - THEN standard error shows
Starting (.) [License],Finished (.) [License], andRESULTS
- WHEN a user runs
scorecard --local=. --probes=hasLicenseFile - THEN standard error shows
Starting (.) probe [hasLicenseFile]andFinished (.) probe hasLicenseFile - AND no
RESULTSbanner is printed
- WHEN a user runs with
--format=json - THEN no
StartingorFinishedmessages are printed
The CLI SHALL sort each repository's check results by check name, in
ascending byte order, before formatting them. Formatters do not sort; see
result-formats.
- GIVEN concurrent checks finish in a non-alphabetic order
- WHEN the CLI formats the repository's results
- THEN the check results appear in ascending order of check name
- GIVEN results for checks named
SASTandFuzzing - WHEN the CLI formats the results
- THEN
Fuzzingappears beforeSAST
The system MUST exit with status 1 and print
error during command execution: <error> when validation fails, the policy
file is invalid, check selection fails, the organization cannot be listed,
or package resolution fails. When a repository cannot be parsed or scanned,
the system SHALL print Skipping <uri>: <error> to standard error and
continue without changing the exit status. When formatting or writing a
repository's results fails, the system SHALL print
Failed to format results for <uri>: <error> to standard error, continue
with the remaining repositories, and leave the exit status unchanged. When
any check reports a runtime error, the system SHALL print
Check <name> failed for <uri>: <error> to standard error and finish all
repositories. It SHALL then exit with status 1 and
"one or more checks failed during execution". Check scores and policy
thresholds SHALL NOT affect the exit status.
- GIVEN a check returns a runtime error for one repository
- WHEN the scan of all repositories completes
- THEN the system exits with status 1 and "one or more checks failed during execution"
- WHEN a user runs
scorecard --repo=::bad:: - THEN the system prints a
Skippingmessage and exits with status 0
- WHEN a user runs
scorecard --local=. --probes=nope - THEN the system prints
Skipping .: run: getting probe "nope": ...to standard error and exits with status 0
- GIVEN a scan of two repositories with
--repos - WHEN formatting the first repository's results fails
- THEN
Failed to format results for <uri>: <error>is printed on standard error - AND the second repository is still scanned and formatted
- AND the exit status is not changed by the formatting failure
- WHEN every check succeeds but scores 0
- THEN the system exits with status 0
The system SHALL set the log level from --verbosity, defaulting to
info. Accepted values SHALL be trace, debug, info, warn, error,
panic, and fatal, case-insensitively; any other value SHALL fall back to
info without an error.
- WHEN a user runs with
--verbosity=debug - THEN debug-level log messages and details are emitted
- WHEN a user runs with
--verbosity=loud - THEN the system logs at
infolevel and does not fail
The system SHALL treat each feature-flag variable as enabled when it is
present in the environment, regardless of its value. ENABLE_SARIF enables
SARIF output and the --policy flag. SCORECARD_V6 enables raw output.
SCORECARD_EXPERIMENTAL enables experimental checks and Azure DevOps
repositories. Scan flags SHALL have no
environment-variable equivalents.
- GIVEN
ENABLE_SARIF=falseis set - WHEN a user runs with
--format=sarif - THEN SARIF output is accepted
When CI is true, SCORECARD_INTERNAL_GITHUB_INTEGRATION is 1, and
GITHUB_EVENT_NAME is dynamic, the system SHALL replace the selected
checks with the comma-separated, whitespace-trimmed value of
SCORECARD_INTERNAL_GITHUB_CHECKS. The override SHALL apply to the --checks
list of a CLI scan and to the checks list of a serve request.
- GIVEN the three integration variables are set as described and
SCORECARD_INTERNAL_GITHUB_CHECKS=License, SAST - WHEN the command runs with
--checks=Fuzzing - THEN the system runs
LicenseandSAST
- GIVEN the three integration variables are set as described for a
running
scorecard serve - WHEN a request names checks in its
checkslist - THEN the server runs the checks from
SCORECARD_INTERNAL_GITHUB_CHECKSinstead
When the user runs scorecard version, the system SHALL print the tool
name, description, git version, git commit, tree state, build date, Go
version, compiler, and platform. With --json, the system SHALL print the
same data as JSON.
- WHEN a user runs
scorecard version - THEN the output includes
GitVersion:,GitCommit:, andPlatform:
- WHEN a user runs
scorecard version --json - THEN the output is a JSON object with the version fields