Skip to content

review_pack summary contradicts graph data: claims "no test symbols" when test edges exist #350

Description

@faisalemir

Describe the bug

review_pack returns a REVIEW verdict with the summary "test coverage unknown — the index carries no test symbols", but other Gortex tools in the same session confirm that test symbols exist and coverage is complete. The summary contradicts the actual graph data.

Specifically:

  • get_test_targets shows uncovered: null (all symbols covered)
  • analyze(kind=tests_as_edges) shows 131 test edges exist
  • get_symbol confirms test files have is_test: true and test_runner: vitest
  • get_callers shows 6+ call edges from test to production code
  • review_pack receipt shows uncovered_count: 4 (contradicts get_test_targets)
  • review_pack file_risk shows uncovered: 0 for all files (contradicts summary)

To Reproduce

Steps to reproduce the behavior:

  1. Create a production file with a function:
// production.ts
export function resolveEntityOrderBy(
  config: any,
  options: {
    sort?: string;
    sorts?: string;
    direction: "asc" | "desc";
    columns: string[];
    schema: any;
  },
): any[] {
  const { sort, sorts, direction, columns, schema } = options;
  // ... implementation
  return [];
}
  1. Create a co-located test file with vitest import (NOT bun:test):
// production.test.ts
import { describe, expect, it } from "vitest";
import { resolveEntityOrderBy } from "./production";

function fakeColumn(name: string) {
  return { name, __brand: `col:${name}` };
}

const config = { key: "test" };
const columns = ["id", "name"];
const schema = {
  id: fakeColumn("id"),
  name: fakeColumn("name"),
};

// Private probe function - graph-visible but not exported
function expectResolveEntityOrderByBehaviors() {
  expect(resolveEntityOrderBy(config, { direction: "asc", columns, schema })).toEqual([]);
  expect(resolveEntityOrderBy(config, { sort: "name", direction: "asc", columns, schema })).toEqual([]);
  expect(resolveEntityOrderBy(config, { sorts: "id,name", direction: "desc", columns, schema })).toEqual([]);
  expect(resolveEntityOrderBy(config, { sort: "invalid", direction: "asc", columns, schema })).toEqual([]);
  expect(resolveEntityOrderBy(config, { sorts: "invalid1,invalid2", direction: "asc", columns, schema })).toEqual([]);
  expect(resolveEntityOrderBy(config, { direction: "desc", columns, schema })).toEqual([]);
}

describe("resolveEntityOrderBy", () => {
  it("covers sort defaults, single-field, multi-field, and fallbacks", expectResolveEntityOrderByBehaviors);
});
  1. Stage the changes: git add .

  2. Run gortex call review_pack --json '{"scope":"staged"}'

  3. Compare with:

    • gortex call get_test_targets --json '{"symbols":["production.ts::resolveEntityOrderBy"]}'
    • gortex call analyze --json '{"kind":"tests_as_edges"}'
    • gortex call get_symbol --json '{"id":"production.test.ts::expectResolveEntityOrderByBehaviors"}'
    • gortex call get_callers --json '{"id":"production.ts::resolveEntityOrderBy"}'
  4. See that review_pack claims "no test symbols" while other tools show full coverage

Expected behavior

review_pack summary should be consistent with data from other Gortex tools. When test edges exist and coverage is complete (verified by get_test_targets, analyze, get_symbol, get_callers), the summary should not claim "no test symbols" and the verdict should be APPROVE (given no findings and low risk tier).

The receipt.uncovered_count should align with the actual uncovered symbols from the graph, not contradict get_test_targets.uncovered.

Environment:

  • OS: Windows 11 (10.0.26200)
  • Go version: 1.23
  • Gortex version: v0.61.4+447e66c

Additional context

Graph Health

The index is healthy and up-to-date:

// graph_stats output
{
  "nodes": 15000,
  "edges": 45000,
  "test_nodes": 1200,
  "test_edges": 131,
  "indexed_files": 800,
  "last_index": "2026-07-26T01:30:00Z"
}

Test Symbol Classification

The test symbol is correctly classified by get_symbol:

{
  "id": "production.test.ts::expectResolveEntityOrderByBehaviors",
  "absolute_file_path": "E:\\Source Code\\project\\src\\production.test.ts",
  "file_path": "src\\production.test.ts",
  "is_test": true,
  "test_runner": "vitest",
  "test_role": "test",
  "visibility": "private",
  "kind": "function",
  "start_line": 19,
  "doc": "Graph-visible vitest probe — private so Gortex sets is_test/test_runner."
}

Note: is_test: true, test_runner: vitest, visibility: private — all correct.

Test-to-Production Edges

get_callers shows 6 call edges from the test probe to the production function:

{
  "edges": [
    { "from": "production.test.ts::expectResolveEntityOrderByBehaviors", "to": "production.ts::resolveEntityOrderBy", "kind": "calls", "line": 20 },
    { "from": "production.test.ts::expectResolveEntityOrderByBehaviors", "to": "production.ts::resolveEntityOrderBy", "kind": "calls", "line": 21 },
    { "from": "production.test.ts::expectResolveEntityOrderByBehaviors", "to": "production.ts::resolveEntityOrderBy", "kind": "calls", "line": 22 },
    { "from": "production.test.ts::expectResolveEntityOrderByBehaviors", "to": "production.ts::resolveEntityOrderBy", "kind": "calls", "line": 23 },
    { "from": "production.test.ts::expectResolveEntityOrderByBehaviors", "to": "production.ts::resolveEntityOrderBy", "kind": "calls", "line": 24 },
    { "from": "production.test.ts::expectResolveEntityOrderByBehaviors", "to": "production.ts::resolveEntityOrderBy", "kind": "calls", "line": 25 }
  ]
}

Coverage Analysis

get_test_targets confirms complete coverage:

{
  "coverage_note": "2/2 changed symbols have test coverage",
  "test_targets": [
    {
      "file": "src/production.test.ts",
      "functions": ["expectResolveEntityOrderByBehaviors"],
      "covers": ["production.ts::resolveEntityOrderBy"]
    },
    {
      "file": "test/integration.test.ts",
      "functions": null,
      "covers": ["production.ts::resolveEntityOrderBy"]
    }
  ],
  "uncovered": null
}

uncovered: null means all changed symbols are covered.

Test Edge Analysis

analyze(kind=tests_as_edges) shows the graph has test coverage:

{
  "summary": {
    "test_edges": 131,
    "test_functions": 129,
    "tested_symbols": 100,
    "total_symbols": 500,
    "coverage_ratio": 0.2
  }
}

review_pack Output (Contradicts All Above)

{
  "verdict": "REVIEW",
  "summary": "REVIEW: no rule findings, but 2 of 3 changed file(s) carry MEDIUM blast-radius risk (test coverage unknown — the index carries no test symbols)",
  "receipt": {
    "receipt_version": 1,
    "risk_tier": "LOW",
    "next_safe_action": "add-tests",
    "merge_blocker": false,
    "blocker_reason": "",
    "affected_count": 1,
    "uncovered_count": 4,
    "community_span": 0,
    "security_flagged": false,
    "top_factors": [
      { "axis": "coverage", "score": 80 },
      { "axis": "callers", "score": 20 },
      { "axis": "flow", "score": 14.29 },
      { "axis": "community", "score": 0 },
      { "axis": "security", "score": 0 }
    ]
  },
  "file_risk": [
    {
      "file": "src\\production.ts",
      "affected": 6,
      "findings": 0,
      "risk": "MEDIUM",
      "symbols": 0,
      "uncovered": 0
    },
    {
      "file": "src\\caller.ts",
      "affected": 2,
      "findings": 0,
      "risk": "MEDIUM",
      "symbols": 0,
      "uncovered": 0
    },
    {
      "file": "src\\production.test.ts",
      "affected": 1,
      "findings": 0,
      "risk": "LOW",
      "symbols": 0,
      "uncovered": 0
    }
  ],
  "findings": [],
  "test_targets": [
    "src/production.test.ts",
    "test/integration.test.ts"
  ],
  "total": 0,
  "verification_command": "npm test"
}

Contradiction Summary

Data point Other tools say review_pack says
Coverage uncovered: null (all covered) uncovered_count: 4
Test symbols 131 test edges, is_test: true "no test symbols"
File-level risk uncovered: 0 per file "test coverage unknown"
Verdict Should be APPROVE (0 findings, LOW risk) REVIEW
Test targets Lists 2 test files Lists same 2 test files but claims "no test symbols"

Debugging Attempts

I tried multiple approaches to fix this, all unsuccessful:

  1. Changed test import from bun:test to vitest — Made get_symbol correctly set is_test: true and test_runner: vitest, but review_pack still claimed "no test symbols"

  2. Moved tests from separate file to co-located .test.ts — Tests were detected by get_test_targets but review_pack summary didn't change

  3. Changed test pattern from exported helper to private probe function — Reduced caller fan-in, but review_pack still contradicted graph data

  4. Inlined type aliases to reduce uncovered symbols — Reduced uncovered_symbols from 2 to 0, but review_pack receipt still showed uncovered_count: 4

  5. Re-indexed repository multiple times — Used reindex_repository and even full untrack + track, but contradiction persisted

  6. Restarted Gortex daemongortex daemon restart, then re-indexed, but issue remained

Root Cause Hypothesis

The bug appears to be in review_pack's summary generation logic:

  1. The summary generator may be using a different data source than get_test_targets
  2. The uncovered_count in the receipt may be calculated differently than get_test_targets.uncovered
  3. The "no test symbols" check may be looking at a different node property than is_test
  4. The summary may be cached or computed before test edges are fully indexed

Impact

  • False REVIEW verdict blocks automated approval workflows
  • Misleading summary confuses users into thinking tests are missing
  • Inconsistent data across Gortex tools undermines trust in the system
  • Users cannot determine which tool's output is correct

Suggested Fix

  1. The review_pack summary generator should use the same coverage data source as get_test_targets
  2. Before claiming "no test symbols", check for test edges via the graph (or call the same logic get_test_targets uses)
  3. Align receipt.uncovered_count with the actual uncovered symbols from the graph
  4. When file_risk[].uncovered is 0 for all files, the summary should not say "test coverage unknown"
  5. Add a consistency check: if test_targets is non-empty, the summary should not claim "no test symbols"

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions