Skip to content

Implement RFC 025 typed sketch logical values#55

Merged
dannymeijer merged 7 commits into
mainfrom
feature/51-rfc025-typed-sketch-values
Jun 4, 2026
Merged

Implement RFC 025 typed sketch logical values#55
dannymeijer merged 7 commits into
mainfrom
feature/51-rfc025-typed-sketch-values

Conversation

@dannymeijer

@dannymeijer dannymeijer commented May 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Implements RFC 025 typed sketch logical values with the first concrete HyperLogLog family. Sketch state is modeled as typed InQL metadata instead of strings or binary payloads, with registry policy metadata, Substrait function options, and a DataFusion adapter capability diagnostic.

This branch is updated onto current origin/main after PR #58 and uses the same Incan rc47 CI baseline from the merged typed-scalar work. The RFC025 public helper surface now also follows the post-#58 typed value-or-column conventions: hll_sketch(...) accepts primitive values or scalar expressions, hll_deserialize(...) accepts string payload values or scalar expressions, and sketch_value(...) accepts scalar values or expressions before attaching sketch metadata.

Type of change

  • Bug fix
  • New feature
  • Refactor / maintenance
  • Documentation
  • CI / tooling
  • RFC (adds/updates docs/rfcs/*)

Area(s)

  • Package & tests
  • Specification (RFCs)
  • Documentation
  • Automation & repo config
  • Other

Key details

  • User-facing behavior: adds SketchLogicalType, SketchExpr, hll_type(...), sketch_col(...), sketch_value(...), hll_sketch(...), hll_merge(...), hll_estimate(...), hll_serialize(...), and hll_deserialize(...); RFC 025 is marked implemented.
  • Typed helper inputs: aligns RFC025 with PR Feature: validate typed scalar expression inputs (#57) #58’s typed surfaces by using ScalarValueOrColumn for source sketch values and StrValueOrColumn for serialized sketch payloads.
  • Internals: adds typed sketch registry policy metadata, HLL Substrait anchors, scalar and aggregate sketch function options, and DataFusion backend planning diagnostics when a typed sketch function reaches an adapter without sketch runtime support.
  • Merge fix: preserves precise Substrait extension declaration traversal for IfThen/case_when, so plans declare only functions they actually use instead of pulling typed-sketch helpers into unrelated scalar plans.
  • Risks: DataFusion intentionally reports backend planning errors for HLL sketch execution because it has no sketch runtime implementation; the InQL plan remains typed and backend-neutral.

Testing / verification

  • make ci (or make fmt-check, make build, make test)
  • Manual verification described below

Manual verification notes:

Using INCAN=/Users/danny/Development/encero/tmp/incan-750-primitive-type-tokens/target/debug/incan (0.3.0-rc47):

  • make fmt
  • make fmt-check
  • make test-style
  • make registry-metadata (204 helpers)
  • make build
  • make test (230 passed)
  • git diff --check
  • git diff --cached --check
  • Focused reruns while resolving and polishing the main merge:
    • incan test tests/test_sketch_functions.incn (9 passed)
    • incan test tests/test_function_registry.incn (20 passed)
    • incan test tests/test_substrait_plan.incn (53 passed)
    • incan test tests/test_session_aggregates.incn (8 passed)
    • incan test tests/test_session_projection.incn (17 passed)

Docs impact

  • No docs changes needed
  • Docs updated
  • Docs follow Divio intent (tutorial/how-to/reference/explanation) where applicable

If docs updated:

  • Link(s): docs/rfcs/025_typed_sketch_logical_values.md, docs/language/reference/functions/sketches.md, docs/language/reference/functions/approximate.md, docs/language/reference/functions/index.md, docs/language/reference/builders/aggregates.md, docs/release_notes/v0_1.md

Checklist

  • I kept public docs user-focused and moved internals to contributing docs when appropriate
  • I avoided duplicating canonical install/run instructions in multiple places
  • I added/updated tests where it materially reduces regressions

Closes #51

@incan-triage-bot incan-triage-bot Bot added documentation Improvements or additions to documentation package Library source, tests, incan.toml specification docs/rfcs/ normative RFCs labels May 30, 2026
@dannymeijer dannymeijer added this to @InQL Jun 2, 2026
@github-project-automation github-project-automation Bot moved this to Todo in @InQL Jun 2, 2026
@dannymeijer dannymeijer marked this pull request as ready for review June 4, 2026 06:55
@dannymeijer dannymeijer merged commit 36ee5a0 into main Jun 4, 2026
3 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in @InQL Jun 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation package Library source, tests, incan.toml specification docs/rfcs/ normative RFCs

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

RFC 025: Typed sketch logical values

1 participant