Skip to content

fix(cypher): classify aggregation columns by is_aggregate_func, not a bare func check - #1221

Merged
DeusData merged 2 commits into
DeusData:mainfrom
AmirF194:fix/1111-aggregate-scalar-func-misclassification
Jul 31, 2026
Merged

fix(cypher): classify aggregation columns by is_aggregate_func, not a bare func check#1221
DeusData merged 2 commits into
DeusData:mainfrom
AmirF194:fix/1111-aggregate-scalar-func-misclassification

Conversation

@AmirF194

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes the type(r)/count(*) half of #1111: RETURN/WITH aggregation misroutes non-aggregate functions into the aggregate-value branch whenever the query also carries an aggregate column, silently substituting the row count for the actual value.

MATCH ()-[r]->() RETURN type(r) AS tipo, count(*) AS n ORDER BY n DESC
# before: {"tipo":"176178","n":"176178"}   (row count in both columns)
# after:  {"tipo":"CALLS","n":176178}

Root cause

ret_agg_build_key/ret_agg_emit_row (RETURN) and with_agg_build_key/with_agg_find_or_create/with_agg_accumulate (WITH) classify each projected column with a bare if (item->func) check to decide "is this the aggregate column or the group-key column". That check is wrong: item->func is set for any function call, aggregate or not, so type(r), labels(n), id(n), keys(n), and properties(n) read as truthy and land in the aggregate branch next to real aggregates like count(*). There they are formatted by format_agg_value's default case, which emits the accumulated row count, not the value the query asked for.

The correct predicate already exists and is already used correctly one level up, where execute_return_clause/execute_with_clause decide whether a query needs aggregation at all: is_aggregate_func(), which checks the function name against the fixed aggregate set (COUNT/SUM/AVG/MIN/MAX/COLLECT). The per-column classification inside the aggregation-execution path never adopted it.

The fix

Swap the five item->func truthy checks for is_aggregate_func(item->func). A group-key column that carries a non-aggregate function now needs to actually evaluate that function instead of reading a raw variable/property, so its value comes from the existing project_item() helper (already used by the non-aggregated RETURN/WITH paths) rather than binding_get_virtual().

Verification

Docker (ubuntu:24.04, gcc, ASan+UBSan build):

  • Reverted only the src/cypher/cypher.c hunk with the two new tests kept: both fail exactly as reported (type(r) and count(*) both return the row count).
  • Restored the fix: both new tests pass; full cypher suite is 162/162.
  • scripts/lint.sh --ci CLANG_FORMAT=clang-format-20 (cppcheck 2.20.0 + clang-format-20, matching _lint.yml): clean.
  • scripts/security-audit.sh: passed, no new findings touching the changed files.
  • Full unit suite (make -f Makefile.cbm test-par): 6770 passed, 12 failed, 4 skipped. The 12 failures (in subprocess, cli, mcp, index_supervisor, all process-tree/daemon-lifecycle tests unrelated to cypher.c) reproduce identically on unmodified main in the same container, so they predate this change.

Not verified: the other claim in #1111, an inline property map in a MATCH pattern silently matching 0 rows, does not reproduce on current HEAD and is unrelated to this fix. Leaving that open for you to split or close separately.

Fixes #1111

@AmirF194
AmirF194 requested a review from DeusData as a code owner July 23, 2026 04:22
@DeusData DeusData added this to the 0.9.1-rc milestone Jul 24, 2026
@DeusData DeusData added bug Something isn't working cypher Cypher query language parser/executor bugs priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. labels Jul 24, 2026
@DeusData

Copy link
Copy Markdown
Owner

Thanks, this is focused and the reported RETURN/WITH RED cases are good. One correctness regression remains before approval: scalar node functions now reach the existing bare-node carry path. labels(n), id(n), keys(n), or properties(n) can therefore leave the projected scalar alias carrying the original node ID, after which alias.property can incorrectly re-fetch the source node. Please make the carry predicate explicitly require a bare node projection, such as excluding items with func, and add a WITH-aggregation regression proving a scalar introspection alias cannot expose properties from the original node. The current head is otherwise DCO-clean, fully green, and introduces no new public surface.

AmirF194 added a commit to AmirF194/codebase-memory-mcp that referenced this pull request Jul 24, 2026
… id carry

The bare-node-carry check in with_agg_find_or_create only tested
!property && variable, so labels(n)/id(n)/keys(n)/properties(n) aliases
(variable set, property NULL, func set) were also tagged with the source
node's id. A later alias.property then hit node_prop's stub re-fetch
heuristic and silently returned the source node's real property instead
of empty for the non-node alias.

Adds cypher_issue1111_with_scalar_func_alias_no_node_leak, which fails
on the prior code (returns the source node's file_path) and passes with
this fix.

Addresses DeusData's review on DeusData#1221.
@AmirF194

Copy link
Copy Markdown
Contributor Author

Good catch, fixed in 5fb4c49. with_agg_find_or_create's bare-node-carry check only tested !property && variable, so labels(f) AS l (variable set, property NULL, func set) also got tagged with f's node id, and a later l.file_path hit node_prop's stub re-fetch heuristic and returned f's real file_path instead of empty.

Excluded items with func set from that carry path (only bare node variables like WITH g, count(*) should tag the id). Added cypher_issue1111_with_scalar_func_alias_no_node_leak: confirmed it fails on the prior commit (l.file_path returns "handler.go" instead of "") and passes with this fix. Full cypher suite (164 tests) green, lint and security-audit clean on the touched files.

AmirF194 added a commit to AmirF194/codebase-memory-mcp that referenced this pull request Jul 24, 2026
… id carry

The bare-node-carry check in with_agg_find_or_create only tested
!property && variable, so labels(n)/id(n)/keys(n)/properties(n) aliases
(variable set, property NULL, func set) were also tagged with the source
node's id. A later alias.property then hit node_prop's stub re-fetch
heuristic and silently returned the source node's real property instead
of empty for the non-node alias.

Adds cypher_issue1111_with_scalar_func_alias_no_node_leak, which fails
on the prior code (returns the source node's file_path) and passes with
this fix.

Addresses DeusData's review on DeusData#1221.

Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
@AmirF194
AmirF194 force-pushed the fix/1111-aggregate-scalar-func-misclassification branch from 5fb4c49 to c218e0e Compare July 24, 2026 17:01
@AmirF194

Copy link
Copy Markdown
Contributor Author

The test-windows-guards failure here (test_windows_launcher.py, managed update secure-open error) is not from this branch: main failed the identical test the same way on 2026-07-24 (https://github.com/DeusData/codebase-memory-mcp/actions/runs/30067754814), and this PR only touches src/cypher/cypher.c and tests/test_cypher.c. Cause unknown beyond that, just flagging so it doesn't read as something this fix broke.

@DeusData

Copy link
Copy Markdown
Owner

A quick note so this does not sit here looking like your problem: your red CI is not caused by your change.

It is a stale-base failure — the thing your run tripped over was fixed on main after your branch point. Your diff is not implicated.

One thing worth knowing, because it will save you a wasted click: "Re-run failed jobs" will not clear it. A re-run re-tests the same recorded merge commit and fails identically. Only a fresh push refreshes the merge ref, so a rebase on current main (or git merge origin/main) and a push is what picks up the fix.

Apologies for the delay. We are working through a large review backlog oldest-first — the queue is real, not a judgement on your PR. Once it is green I will pick it up properly.

AmirF194 added 2 commits July 31, 2026 06:18
… bare func check

RETURN/WITH aggregation classified each projected column with a bare
`if (item->func)` check to decide group-key vs. aggregate value. That
check is true for any function call, so type()/labels()/id()/keys()/
properties() were routed into the aggregate branch alongside real
aggregates and formatted via format_agg_value's default case, which
emits the row count instead of the function's actual value.

Swap the five call sites (ret_agg_build_key, ret_agg_emit_row,
with_agg_build_key, with_agg_find_or_create, with_agg_accumulate) to
is_aggregate_func(), the predicate already used correctly one level up
to decide whether a query needs aggregation at all. Group-key columns
carrying a non-aggregate function now project through the existing
project_item() helper instead of binding_get_virtual(), since they
need to evaluate the function rather than read a raw property.

Fixes DeusData#1111 (the type(r)/count(*) half; the inline-property-map half
of that issue does not reproduce on current HEAD)

Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
… id carry

The bare-node-carry check in with_agg_find_or_create only tested
!property && variable, so labels(n)/id(n)/keys(n)/properties(n) aliases
(variable set, property NULL, func set) were also tagged with the source
node's id. A later alias.property then hit node_prop's stub re-fetch
heuristic and silently returned the source node's real property instead
of empty for the non-node alias.

Adds cypher_issue1111_with_scalar_func_alias_no_node_leak, which fails
on the prior code (returns the source node's file_path) and passes with
this fix.

Addresses DeusData's review on DeusData#1221.

Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
@DeusData

Copy link
Copy Markdown
Owner

Following up with the specific signature, since my note above was vaguer than it needed to be — I have now pulled your failing job log.

The failure in test-windows-guards is:

RED: managed update failed:
codebase-memory-mcp-launcher: private staged copy failed secure open (open failed (error 32); error 32):
  \\?\C:\Users\runneradmin\...\.cbm\runtime\activation-…payload.exe
codebase-memory-mcp: Windows launcher completion failed: launcher rejected or timed out completing payload authentication

Windows error 32 is ERROR_SHARING_VIOLATION — the antivirus scanner still had the freshly staged executable open when the launcher tried its secure open. It is a timing window on the runner, entirely on our side, and it has since been dealt with twice over on main: first by 040c7f4 ("absorb the first-touch AV scan window on staged-copy secure open"), which I confirmed is an ancestor of current main, and then more permanently by #1316, which removed the launcher stub altogether in favour of a single binary.

So the guidance stands and is now specific: rebase and push — a re-run alone re-tests the same recorded merge commit and will reproduce it. Nothing here is yours to fix.

@AmirF194
AmirF194 force-pushed the fix/1111-aggregate-scalar-func-misclassification branch from c218e0e to 2f522b4 Compare July 31, 2026 06:39
@AmirF194

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (no conflicts) and pushed. All 167 cypher tests pass locally.

@DeusData

Copy link
Copy Markdown
Owner

Reviewed in full now that the rebase is in, and this is a model fix. Approved — merging as soon as CI concludes green.

The bug is real and I verified it on current main. Seven sites classify "aggregate column vs group-key column" with a bare item->func truthy test, which is true for any function call: with_agg_build_key (cypher.c:3745), with_agg_find_or_create (:3784), with_agg_accumulate (:3806), execute_with_aggregate's emit (:3901), ret_agg_accumulate (:4200), ret_agg_build_key (:4252) and ret_agg_emit_row (:4276). The correct predicate is_aggregate_func() already existed at :3643 and already gated the aggregation path one level up at :4018 and :4674 — it was simply never adopted per column.

The symptom is as bad as your report says. RETURN type(r) AS t, count(*) AS n excludes type(r) from the grouping key, collapsing every row into one group, and then formats it through format_agg_value's default case at :4164, which emits the row count. So both columns come back as the same number. Same story for labels(), id(), keys() and properties() in any projection that also carries a real aggregate.

Three things I want to credit specifically.

First, the diff is more complete than its own commit message. It says five sites; it actually fixes seven — ret_agg_accumulate and execute_with_aggregate are in there too. I grepped every remaining bare .func reference on main to check for a straggler: the strcmp(func, "COUNT"/"COLLECT") sites at :3811, :3903, :4213, :4218 and :4286 are all downstream of the now-fixed check, and :4342:4359 build display names rather than classify. No sibling site is left inconsistent.

Second, you reused project_item() instead of inventing a parallel evaluation path. Non-aggregate function group keys now have to be evaluated rather than read raw, and project_item is exactly the helper ret_agg_build_key already used at :4259. It is NULL-func-safe, returns identical values for plain variable and property columns, and its stack scratch is heap_strdup'd immediately. Converging the WITH path onto the RETURN path's existing behaviour is the right shape.

Third — and this is the part I liked most — commit 2 chases a follow-on that commit 1 exposes. The bare-node id carry in with_agg_find_or_create tested only !property && variable, so labels(f) AS l got tagged with the source node's id, and a later l.file_path hit the stub re-fetch at :3913 and leaked HandleOrder's real file_path instead of returning "" for a non-node alias. Your !func guard is correct, and the existing bare-node test is unaffected by it.

The tests bind at each stage, which is unusual and welcome. cypher_issue1111_with_scalar_func_alias_no_node_leak fails on main and fails on commit-1-only — so it discriminates the second commit specifically, not just the pair. And its "" negative sits alongside positive controls in the same row, so it cannot pass by the query having failed.

On the behaviour change: row counts will increase for queries mixing a non-aggregate function with an aggregate, since one collapsed row becomes N real groups. Anyone depending on the old output was consuming a row count mislabelled as a type or label, so there is no legitimate reliance to protect.

And on the timing: #1177 merged into cypher.c shortly after your rebase, so I checked whether it lands on you. It does not — it bounds the OPTIONAL-fallback hop buffer inside expand_pattern_rels, which produces bindings, while yours changes aggregation classification, which consumes them. A merge-tree against current main comes back clean, including both PRs' additions to test_cypher.c. Nothing for you to do.

Security review clean: no new dependencies, no CI changes, no fopen(, no new API surface. Scope is exactly two files with no drive-bys.

Thank you as well for the honest note that the inline-property-map half of #1111 does not reproduce — we will keep that claim open separately rather than letting this PR close it by implication.

@DeusData
DeusData merged commit 9595822 into DeusData:main Jul 31, 2026
28 checks passed
@DeusData

Copy link
Copy Markdown
Owner

Merged as 95958222c — 28/28 green, re-diffed at merge (2 commits, 2 files, +87/−11, unchanged from what was reviewed).

Thank you for this one, and for the patience while a launcher failure that was entirely ours sat on your PR looking like your problem.

Worth restating what landed, because the diff is quietly bigger than its own commit message: seven classification sites now use is_aggregate_func() instead of a bare item->func truthy test, and RETURN type(r) AS t, count(*) AS n returns real groups instead of the row count in both columns. The second commit — the !func guard on the bare-node id carry — is the part I liked most, because it fixes a problem your first commit exposed rather than one you were asked about, and its test discriminates that commit specifically rather than just the pair.

Enjoy the rest of your weekend, and thanks again.

@AmirF194

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review and for catching the launcher noise for me, that would have taken a while to track down on my own. Glad the second commit landed well, that carry bug was a fun one to chase down once the first fix surfaced it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cypher Cypher query language parser/executor bugs priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inline property map in MATCH pattern silently matches nothing — should error if unsupported

2 participants