fix(cross-repo): emit HTTP_CALLS for unindexed client libs and normalize URLs for route matching (#523)

[RithvikReddy0-0]

Fixes two root causes of cross-repo-intelligence returning 0 edges (#523).

pass_calls.c

HTTP client calls (requests, httpx, axios, etc.) were silently dropped when
the client library wasn't indexed (external pip/npm dep). The callee resolved
to a QN but cbm_gbuf_find_by_qn returned NULL, so the call was discarded
before HTTP classification.

Fix: detect known HTTP/async patterns via cbm_service_pattern_match and emit
the edge even without a target node in the graph.

pass_cross_repo.c

Three issues in match_http_routes:

1. Consumer url_path carried full URL (scheme+host+port); provider Route has
   bare path. Added cr_url_path() to strip scheme+authority before QN lookup.

2. Concrete paths (/v2/orders/123) never matched templated routes
   (/v2/orders/{id}). Added cr_path_matches_template() and
   find_route_handler_fuzzy() for segment-level template matching.

3. match_http_routes only searched HTTP_CALLS in the src project. When
   cross-repo is run from the provider side, HTTP_CALLS live in the consumer
   DB. Added reverse direction call so both orientations are covered.

Repro

Verified manually: FastAPI provider + requests consumer, cross-repo-intelligence
now returns cross_http_calls: 1 where it previously returned 0.

Checklist

• Every commit is signed off (git commit -s) — required, CI rejects
  unsigned commits (DCO, see CONTRIBUTING.md)
• Tests pass locally (make -f Makefile.cbm test)
• Lint passes (make -f Makefile.cbm lint-ci)
• New behavior is covered by a test (reproduce-first for bug fixes)

Note: build and manual repro verified in WSL (cross_http_calls: 1 confirmed).
Test suite and lint require Unix toolchain — ran in WSL but full make test
timed out; happy to add a regression test if the maintainer points me to the
right test file.

[azuttre]

Built this branch on macOS (arm64) and traced resolve_single_call to see exactly what happens. The two halves behave differently:

pass_cross_repo.c (URL normalize + template match): works. Holding an HTTP_CALLS edge constant (url_path = the full URL http://order-service:8080/v2/orders/123), main links nothing and this branch links it to the templated route /v2/orders/{id}. Clean before/after.

pass_calls.c (emit without the client lib indexed): doesn't fire for a genuinely external requests. The call reaches resolve_single_call and first_string_arg holds the URL, so that part is fine. But with requests not installed or vendored anywhere indexable, the import map is empty (imp_count=0), so cbm_registry_resolve returns an empty qualified_name and the function returns at if (!res.qualified_name) return 0; before your new svc==HTTP emit. There is no QN for cbm_service_pattern_match to match. Trace:

ENTER    callee='requests.get' first_arg='http://svc:8080/v2/orders/123' imp_count=0
RESOLVED callee='requests.get' -> res_qn='(EMPTY)' svc=-1   # returns here

The moment requests is locally resolvable (a vendored stub or an installed venv), imp_count=1, res_qn becomes '...requests.get', svc=1, and it emits, but that is the resolved path, which main emits on too.

So the emit-without-target path seems to help only when the callee resolves to a QN that has no node, not when the external client resolves to nothing. Was requests pip-installed in your WSL repro? If so, cross_http_calls: 1 is the matcher fix plus a resolvable call, and the index-just-my-service case (the #523 scenario) is still 0. Happy to share the repro.

[RithvikReddy0-0]

Good catch you were exactly right. The emit sat after the empty-QN early return, so a genuinely external requests (empty import map → empty resolved QN) bailed before reaching it. My WSL repro had a vendored requests stub, which created a node and went through the resolved path so it never actually exercised the index-just-my-service case. That's on me.

Fixed in the latest commit: the detection now lives in the empty-QN branch and classifies from the raw callee name (requests.get → HTTP, GET) rather than the resolved QN. Re-verified with no stub and no install requests external, consumer indexes only its own service:

• HTTP_CALLS edge now appears on the consumer (fetch_order → http://order-service:8080/v2/orders/123, method GET)
• cross-repo links it to the provider's templated route: cross_http_calls: 1

One caveat worth flagging separately: a single-file provider still returns 0, but for an unrelated reason FastAPI route extraction (@app.get → Route node) only fires on the parallel pipeline path (>50 files); on the sequential path the decorator is captured but no Route node is created. That's the same "decorator captured but no route" gap you noted originally, and it's independent of this PR's matcher/emission changes. Happy to open a separate issue for it.

thanks for tracing this on your end ;)

[azuttre]

Re-validated 0a8a44f on macOS (arm64), with a genuinely external requests (no stub, no install, consumer indexes only its own service):

• consumer now emits HTTP_CALLS for both calls (was 0 on main)
• cross-repo links them to the provider: cross_http_calls: 2
  • fetch_order -> get_order (/v2/orders/123 -> /v2/orders/{id})
  • place_order -> create_order (/v2/orders)

So the empty-QN path is fixed. Confirmed on my end.

On the single-file caveat: I'm not reproducing it here. My provider is a single app.py with two routes (@app.get + @app.post), both Route nodes extracted fine, which is why the end-to-end run above links. So the no-route-on-single-file behavior may be platform-specific or a narrower trigger than file count, rather than a general <50-file thing. Didn't block the cross-repo case for me, but happy to share details if you open a separate issue for it.

Nice work turning this around so fast.

[RithvikReddy0-0]

Thanks for re-validating ;)
Glad it holds on macOS too, and good to see both calls linking.

You're right to push back on the single-file theory if your single app.py with two routes extracts both Route nodes fine, then it is not a general parallel-vs-sequential thing. On my WSL setup the single-file provider consistently produced no Route node, but that's clearly a narrower or platform-specific trigger, not the file-count threshold I guessed at. I will dig into what is actually different on my end before claiming a cause, and open a separate issue with a proper repro if it is real rather than a local artifact.

Appreciate the thorough trace throughout it made both fixes tighter.

[DeusData]

Thanks @RithvikReddy0-0 — the unindexed-client + URL-normalization direction is right. Two things before this can land:

1. Duplicate edges (blocking). The entry point now calls match_http_routes in both directions (src→tgt and tgt→src), but delete_cross_edges runs once and emit_cross_route_bidirectional already writes both sides — so the reverse pass re-emits the same CROSS_HTTP_CALLS pair, producing duplicates and inflating http_edges. Please dedupe (single direction, or guard against re-emitting an existing edge).
2. Test. Add a reproduce-first test (refs cross-repo-intelligence returns 0 edges for a byte-identical call/route #523): a byte-identical client call/route that must yield exactly one edge, asserting no duplicates.

Also, emit_http_async_edge(ctx, call, source_node, source_node, ...) passes the source node as both source and target — please add a comment confirming that's intentional for the unindexed-external case. 🙏

[RithvikReddy0-0]

Thanks for the review. Addressed all three in 4817d79.

1. Duplicate edges. Fixed via an idempotency guard in insert_cross_edge: it skips insertion when an identical (source_id, target_id, type) edge already exists. Verified no inflation across both repeated runs and runs initiated from either project:

• run twice from provider side → cross_http_calls: 1 both times
• run from consumer side → cross_http_calls: 1
• exactly one edge per DB in all cases (fetch_order → url in consumer, get_order → route in provider)

One thing worth raising on the "single direction" suggestion: I don't think dropping the reverse match_http_routes works on its own. match_http_routes reads HTTP_CALLS from the source project, and emit_cross_route_bidirectional only runs inside a successful match. In the reporter's invocation (run from the provider with target_projects=[consumer]), the provider has no outbound HTTP_CALLS — it serves routes — so the forward pass matches nothing and the bidirectional write never fires. The reverse pass is what lets a provider-initiated run discover the consumer's calls. So I kept both directions and made emission idempotent instead.

Tradeoff I want to flag: the guard adds a find_edges_by_source_type lookup per candidate edge. On large graphs that's a non-trivial cost. If you'd prefer, I can switch to deduping once at the end of the pass (collect, unique, then insert) or a dedicated cbm_store_edge_exists to avoid pulling the full edge list , happy to go whichever way fits the codebase.

3. Self-pass comment. Added documents that the external client has no graph node, so source_node is passed as both source and target, and that the URL/topic path links source → Route without dereferencing the duplicated arg.

2. Test. This is the one I'd like a pointer on. cbm_cross_repo_match resolves project DBs via cbm_resolve_cache_dir() + project name rather than an explicit path, so the existing test_pipeline.c harness (which passes explicit temp db_paths) doesn't sandbox it cleanly a naive test would write into the real cache dir. Is there an existing pattern for overriding the cache dir in tests, or would you prefer I add the dedup assertion at the store level instead? Happy to write the reproduce-first byte-identical test once I know which approach fits.

[DeusData]

Thanks @RithvikReddy0-0 — the two-pronged diagnosis (emit HTTP_CALLS for unindexed clients + bidirectional route matching) is right, and this is close. A few things before it can land:

1. Self-loop CALLS edge (src/pipeline/pass_calls.c): the external-client block passes source_node as both source and target — emit_http_async_edge(ctx, call, source_node, source_node, ...). emit_http_async_edge takes its CALLS fallback whenever !is_url && !is_topic, and the new guard only requires first_string_arg be non-empty — so requests.get("orders") (non-empty, not URL-shaped) falls through and emits a source → source self-edge. Please guard on the same URL/topic predicate, or skip emission when the arg isn't URL-shaped.
2. Edge double-count (src/pipeline/pass_cross_repo.c): result.http_edges += match_http_routes(...) runs for both the forward and reverse direction, so a pair matched both ways inflates the count by 2 while the dedup guard keeps only 1 DB row. The returned counter should reflect rows actually inserted.
3. Rebase: the branch now conflicts with main (based on a pre-cross-repo-intelligence returns 0 edges for a byte-identical call/route #523 pass_cross_repo.c).
4. Reproduce-first test: for a 176-line change to edge emission/counting we'd really like a regression test — a FastAPI provider + requests consumer asserting cross_http_calls goes 0→1 — which would also have caught (1) and (2). Happy to point you at the closest existing test under tests/.

Solid work overall — looking forward to the revision.

[RithvikReddy0-0]

Rebased onto latest main and adapted to the new cbm_route_canon_path in match_http_routes. Both review items from the last pass are fixed:

• Self-loop: the external-client emit now requires the arg be URL/topic-shaped (same predicate as emit_http_async_edge), so requests.get("orders") no longer produces a source → source edge. Verified: non-URL arg emits nothing.
• Double-count: insert_cross_edge / emit_cross_route_bidirectional return whether a row was actually inserted, and the match loops count real inserts. A pair matched from both directions yields exactly one row per DB and no count inflation.

On the rebase: main's cbm_route_canon_path canonicalizes placeholder syntax ({id} → {}) but doesn't strip scheme/host or reduce concrete values, so I kept a host-strip step before it and a segment-wise template fallback (/v2/orders/123 against /v2/orders/{}) for when the exact QN lookup misses. End-to-end on fresh DBs: cross_http_calls 0 → 1, one row per side.

For the regression test — cbm_cross_repo_match resolves project DBs via cbm_resolve_cache_dir() + project name rather than an explicit path, so the test_pipeline.c harness (explicit temp db_paths) doesn't sandbox it without writing into the real cache dir. You mentioned pointing me at the closest existing test , where would you like this to live? If there's a cache-dir override pattern I should use, I'll write the FastAPI-provider + requests-consumer test asserting cross_http_calls 0 → 1 with no duplicates. Otherwise I can add the dedup/match assertions at the store level.

[DeusData]

Thank you for the deep dive on #523 — half of this PR is still exactly what's needed. Status after verifying against current main: the pass_calls.c root cause was real, but main landed a broader variant in beaa776 two days after your last push (it also covers the parallel path), so that hunk is now superseded — dropping it resolves your merge conflict entirely. The pass_cross_repo.c half is NOT superseded and is genuinely still broken on main: no scheme/host stripping, no concrete-path-vs-template matching, forward-direction only. That's the valuable part. Could you: (1) rebase dropping the pass_calls.c hunk (cite beaa776 in the commit if you like); (2) add reproduce-first tests for the three cross-repo behaviors (scheme-stripped URL match, template fuzzy match, provider-side reverse run + dedup idempotence)? One perf note for the rework: the fuzzy fallback currently enumerates every Route node per unmatched edge — fine for now, but worth a comment acknowledging the O(calls×routes) bound. Thanks — the cross-repo matching fixes are wanted.

---

Update: to keep momentum on the bug backlog, we're going to carry the changes above over the line ourselves shortly — a distilled follow-up on current main implementing the notes in this thread, with you credited as Co-authored-by on the commit, and this PR closed referencing it. If you'd prefer to push the update yourself, just reply within the next couple of days and we'll gladly take yours instead. Thanks again for the contribution!

[RithvikReddy0-0]

Thanks I saw the offer and pushed the update just now (7736348), so no need to carry it yourselves. All three notes are addressed:

1. Dropped the pass_calls.c hunk — beaa776 supersedes it (broader: registry mis-resolution + both pipeline paths), and dropping it clears the merge conflict. The branch now touches only pass_cross_repo.c + the test file.

2. Reproduce-first tests — extended tests/repro/repro_issue523.c using the existing rh_* harness (which indexes into the cache dir the way cbm_cross_repo_match reads it — that was the sandboxing I couldn't find before). Three cases: scheme/host-stripped URL match, concrete→template fuzzy match, and provider-side reverse run + dedup idempotence (asserts exactly one CROSS_HTTP_CALLS row after two runs). All four cross-repo-intelligence returns 0 edges for a byte-identical call/route #523 repro cases pass locally.

3. O(calls×routes) note — added on find_route_handler_fuzzy, flagging it scans every Route per unmatched edge (only on the exact-match-miss path) with a note that a prefix index would bound it.

Rebased on current main, review requested. Happy to adjust anything, thanks for the detailed steer , this project is really interesting ;) and for the co-author offer either way.

[DeusData]

Thank you for the deep dive on #523 — half of this PR was still exactly what was needed. The pass_calls.c half had been superseded by beaa776 (which also covers the parallel path), but the pass_cross_repo.c half was genuinely still broken on main: no scheme/host stripping, no concrete-vs-template matching, forward-direction only. We carried that half over the line as 4cd8d3e (PR #810) with you credited as co-author, plus the reproduce-first tests the review asked for — and one correction: the pre-insert dedup guard turned out unnecessary (the store's upsert already prevents duplicates) and its row-counting semantics would have re-reported the exact 0-edges symptom on provider-side re-runs, so idempotence is asserted differently. Refs #523 — the reporter should see cross-repo edges now. Closing in favor of the distill — thanks again!