From b2926cfe07214657970a41c71cbc736d4f23be64 Mon Sep 17 00:00:00 2001 From: marcusquinn <6428977+marcusquinn@users.noreply.github.com> Date: Sat, 10 Oct 2026 23:56:37 +0100 Subject: [PATCH] GH#34293: docs: keep Qlty on organisation repos and skip its out-of-minutes check --- .agents/reference/ci-gate-policy.md | 5 ++++- .agents/tools/code-review/qlty.md | 13 +++++++++++++ .agents/tools/code-review/setup.md | 6 +++++- .agents/tools/wordpress/wp-plugin-new.md | 2 +- 4 files changed, 23 insertions(+), 3 deletions(-) diff --git a/.agents/reference/ci-gate-policy.md b/.agents/reference/ci-gate-policy.md index d91431efba..e3242595c4 100644 --- a/.agents/reference/ci-gate-policy.md +++ b/.agents/reference/ci-gate-policy.md @@ -163,7 +163,10 @@ run those directly too, without `-k` (they do not provide unittest filtering). not rerun it, change code, file an issue or wait for it. Merge when the other checks pass, and note the skipped check in the PR body. A qlty failure with real findings still counts, and an out-of-credits failure on - a public personal-account repository is unexpected: report it. + a public personal-account repository is unexpected: report it. The free + minutes return monthly, so keep the app installed (it is not a reason to + remove Qlty) and never make "qlty check" a required check on a limited + repository. Setup guidance: `tools/code-review/qlty.md`. ## Private repositories and public launch diff --git a/.agents/tools/code-review/qlty.md b/.agents/tools/code-review/qlty.md index 1efd73666b..f60c8ad1e8 100644 --- a/.agents/tools/code-review/qlty.md +++ b/.agents/tools/code-review/qlty.md @@ -28,9 +28,22 @@ tools: - Commands: `bash .agents/scripts/qlty-cli.sh install|init|check|fmt --all|smells --all [ORG]` - Storage: `~/.config/aidevops/api-keys` (600 permissions) - Path: Ensure `~/.qlty/bin` in PATH +- GitHub App on organisation or private repos: monthly minutes run out; an out-of-minutes "qlty check" is billing, not a finding. Keep the app and skip the check (`reference/ci-gate-policy.md` rule 14; see "GitHub App: monthly minutes") +## GitHub App: monthly minutes + +Qlty's GitHub App posts a "qlty check" on pull requests. Its analysis minutes are limited on private repositories and on every organisation-owned repository, public or private; only public repositories on personal accounts are unlimited. The free minutes return each month, so early-month PRs get real findings and later ones may fail with "Qlty did not run because you are out of minutes". + +When setting up or managing repositories: + +- Keep the app installed for its free minutes. A red out-of-minutes check is not a reason to uninstall Qlty, remove it from a repository's docs, buy credits or ask the owner, unless the owner asks about cost. +- Skip the out-of-minutes failure: don't rerun it, change code, file an issue or wait for it. Merge when the other checks pass and note the skipped check in the PR body. The merge helpers do this already (`gh-checks-wait-helper.sh` `apply_qlty_billing_policy`, `pulse-merge-required-checks.sh` `_pmrc_repo_has_qlty_credit_limit`). +- Never make "qlty check" a required status check or ruleset check on a credit-limited repository: once the minutes run out it would block every merge for the rest of the month. +- A qlty check with real findings still counts, and an out-of-minutes failure on a public personal-account repository is unexpected: report it. +- CI jobs that run the qlty CLI in a repository's own Actions (such as the aidevops Qlty Regression Gate and Qlty Smell Threshold) are separate from the app and do not use its minutes; their failures are real. + ## Credentials Three types, selected in priority order: diff --git a/.agents/tools/code-review/setup.md b/.agents/tools/code-review/setup.md index fb0d5288be..43a57ec534 100644 --- a/.agents/tools/code-review/setup.md +++ b/.agents/tools/code-review/setup.md @@ -24,8 +24,11 @@ Setup time: ~5 min per platform via GitHub OAuth. Targets: CodeFactor A+, Codacy | CodeFactor | → add repo → enable GitHub Checks | A-F grade, cyclomatic complexity, technical debt, trends | — | | Codacy | → import repo | Security scanning, quality metrics, test coverage, standards | `.codacy.yml` | | SonarCloud | → create org → import project → add `SONAR_TOKEN` GitHub secret | Security hotspots, bugs, code smells, duplication, quality gate | `sonar-project.properties` | +| Qlty | Install the Qlty GitHub App for the account or organisation | Multi-linter findings and smells on PRs; monthly minutes limited on organisation and private repos | `.qlty/qlty.toml` (optional) | -Deep dives: `coderabbit.md`, `codacy.md`, `tools.md`, `.agents/scripts/sonarcloud-cli.sh` +Deep dives: `coderabbit.md`, `codacy.md`, `qlty.md`, `tools.md`, `.agents/scripts/sonarcloud-cli.sh` + +Qlty's out-of-minutes failure is billing: skip it, keep the app, and never make "qlty check" a required check on a credit-limited repository (`reference/ci-gate-policy.md` rule 14). @@ -48,3 +51,4 @@ Replace `{owner}/{repo}` with your repository slug. | CodeRabbit not reviewing | Ensure repo is connected, app permissions granted, and PR triggers enabled. | | CodeFactor not updating | Check repo connection, webhook/GitHub Checks status, and repo authorization. | | Codacy analysis issues | Check `.codacy.yml`, confirm import succeeded, and verify file types are supported. | +| "qlty check" fails: "out of minutes" | Billing, not a finding. Skip it and merge on the other checks; keep the app; never make it required (`qlty.md`, `reference/ci-gate-policy.md` rule 14). | diff --git a/.agents/tools/wordpress/wp-plugin-new.md b/.agents/tools/wordpress/wp-plugin-new.md index dc3963e7e5..f1dfc09f51 100644 --- a/.agents/tools/wordpress/wp-plugin-new.md +++ b/.agents/tools/wordpress/wp-plugin-new.md @@ -64,7 +64,7 @@ For **public repos**: - **Codacy**: inject `CODACY_API_TOKEN` securely (`aidevops secret set CODACY_API_TOKEN`), then run `quality`. It adds the repository with API v3 `POST /repositories` (`provider: gh`, `repositoryFullPath: OWNER/SLUG`), tolerates an already-added 409, fetches `GET /organizations/gh/OWNER/repositories/SLUG`, and restores the badge from `data.badges.grade`, never a copied project ID. Missing token, access or pending badge is reported; rerun after recovery. - **SonarCloud**: inject `SONAR_TOKEN`, optionally `SONAR_ORGANIZATION` / `SONAR_PROJECT_KEY` when they differ from the GitHub owner / `OWNER_SLUG`; verify they match this plugin's scanner configuration, not another repo's environment. The helper provisions a public project when missing and only restores its badge after a quality-gate measure exists. **Project creation and measures do not prove GitHub binding**: the current published Web API exposes no GitHub import endpoint. The sole SonarCloud human fallback is: [Import a project](https://sonarcloud.io/projects/create) — an org admin imports this GitHub repo and configures its analysis method, then the AI reruns `quality`. For starter v1.0.7+ with `.github/workflows/sonarcloud.yml`, keep **Automatic Analysis off** and configure the repository `SONAR_TOKEN` securely for the Actions scanner (see the copied `DEVELOPMENT.md` "Services setup"); older copies without the scanner may enable Automatic Analysis. Never run both methods or claim binding from project existence alone. -- **Apps**: Codacy and CodeFactor must appear on the first PR; also verify CodeRabbit, Qlty and Socket. A check/status proves integration visibility, not a passing result. The helper reports each missing service; the AI diagnoses existing app selection and configuration, and an app admin grants repository access only if necessary. Do not silently accept missing public Codacy/CodeFactor checks. The helper restores CodeFactor's badge only after its public endpoint serves a grade SVG; restore a latest-release badge only after an actual release exists. +- **Apps**: Codacy and CodeFactor must appear on the first PR; also verify CodeRabbit, Qlty and Socket. A check/status proves integration visibility, not a passing result; a "qlty check" failing with "out of minutes" still proves Qlty is connected, and is skipped, never a reason to remove Qlty (`reference/ci-gate-policy.md` rule 14, `tools/code-review/qlty.md`). The helper reports each missing service; the AI diagnoses existing app selection and configuration, and an app admin grants repository access only if necessary. Do not silently accept missing public Codacy/CodeFactor checks. The helper restores CodeFactor's badge only after its public endpoint serves a grade SVG; restore a latest-release badge only after an actual release exists. - **Full review**: after the plugin is its own, create/deduplicate the **Code Audit Routines** issue using the signed framework issue wrapper and dashboard pattern in `scripts/stats-quality-sweep-issues.sh` (`_ensure_quality_issue`). Include repo scripts, scope and verification; mention `@coderabbitai` to request a **full codebase review**, not merely the PR diff (`tools/code-review/coderabbit.md`, "Daily Code Quality Review"). Confirm the review request was accepted; report app/plan limitations rather than inventing a review. For **private repos**, run local Composer/PHPCS/PHPStan, ShellCheck and Docker checks and generate metrics now. Defer hosted onboarding by default: Codacy and CodeFactor free tiers are public-only; the current starter documents limited private SonarCloud free-plan capacity (50,000 organization-wide lines), so verify current organization limits before using an existing entitlement. Do not assume all hosted services are free for private code. CodeRabbit, Qlty and Socket work only when their installed app's repository selection and plan allow private repos; verify actual first-PR checks/statuses, not assumed free access. Do not purchase plans or make the repo public to fix missing checks. At owner-approved public launch, repeat this checklist (`wp-plugin-release.md`).