Skip to content

Review agent: suppress content-security URL findings in documentation files #1629

@fullsend-ai-retro

Description

@fullsend-ai-retro

What happened

On PR #1483, the review agent flagged a Cloud Run URL in docs/guides/admin/install.md as [high] content-security, recommending it be replaced with an environment variable. The URL was intentionally hardcoded — the install guide needed to show users the actual endpoint to configure. The human author replied explaining this, but the agent re-flagged the same URL on the next review pass. This false positive consumed author attention across two review cycles.

What could go better

The content-security heuristic that detects hardcoded URLs/secrets does not distinguish between executable code (where hardcoded credentials are a real risk) and documentation (where showing actual URLs is often the point). A URL in a markdown file under docs/ cannot be exploited the same way as one in source code — it is descriptive, not operative. The review agent should calibrate its content-security scanning based on file type: markdown/docs files should have a much higher bar for URL-based security findings. Confidence: high — this is a clear category error in the heuristic. The agent correctly identifies that hardcoded URLs can be a security concern, but misapplies that rule to documentation where hardcoded URLs are expected and appropriate.

Proposed change

In the code-review skill (internal/scaffold/fullsend-repo/skills/code-review/SKILL.md), add guidance to the content-security evaluation dimension: when the file under review is documentation (markdown, RST, or similar non-executable format under docs/), URLs and service endpoints should not be flagged as content-security findings unless they contain obvious secrets (tokens, API keys, passwords). Hardcoded URLs in documentation are expected and should be treated differently from hardcoded URLs in source code. This could be implemented as a file-type-aware severity modifier: content-security findings about URLs in docs/**/*.md files should be capped at [info] severity unless the URL contains path segments that look like secrets.

Validation criteria

On the next 5 documentation-only PRs reviewed by the review agent, none should receive a [high] or [medium] content-security finding solely for containing a URL in a markdown file. If a docs PR does contain an actual secret (API key, token), it should still be flagged. Timeframe: next 30 days or 5 docs PRs, whichever comes first.


Generated by retro agent from #1483

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/retroRetro agentagent/reviewReview agentcomponent/harnessAgent harness, config, and skills loadingfeatureFeature-category issue awaiting human prioritizationpriority/mediumNormal priority, plan for next cycletriagedTriaged but awaiting human prioritizationtype/featureNew capability request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions