Native JSON events: cutover readiness tracker

[aspicer]

Track the work required before enabling native JSON event reads and, later, retiring the legacy events table.

Merging compatibility code is separate from enabling it. For #91506, batch exports remain on legacy storage and no teams are enabled as part of the merge. The export follow-up below is deferred from that PR, not waived for cutover. This issue records the intended rollout boundary; it does not assert that deployment settings have been inspected.

Before enabling native reads

• Land the base feature-flag compatibility layer: feat(hogql): flag compatibility for native events and a $false sentinel #91506. Verify direct flag reads, active-flag comparisons, presence checks, restrictions, and the "false" variant sentinel. Legacy-table compilation must remain unchanged. Since 7932d0a the layer runs only on native reads, and the PR description records a legacy-SQL comparison against master whose one difference is that HogQL now accepts mapExists. The flag API also rejects $false as a variant key for every team. Both approvals date from Sep 1 and Sep 10, before the sentinel and the native-only change.
• Finish JSON-function compatibility: rebase fix(hogql): read rebuilt flags through every native JSON function #106152 onto the final feat(hogql): flag compatibility for native events and a $false sentinel #91506 and remove inherited legacy-table rewrites. Cover arrayJoin(JSONExtractArrayRaw(properties, '$active_feature_flags')), JSONLength, JSONType, boolean extraction, and toString(properties) wrappers. The base PR alone does not cover these query shapes.
• Decide the whole-document flag contract: key enumeration, computed JSON keys, and properties passed through a subquery still miss reconstructed flag properties after fix(hogql): read rebuilt flags through every native JSON function #106152. Decide whether whole-document reads reconstruct the legacy shape or expose the native map, then implement and test the chosen contract. Reconstruction also changes what SELECT properties and the event properties panel return on native, where flags now appear as one $feature_flags object, so it is a schema-contract decision, not only a HogQL fix. Include property restrictions and generated column aliases. Sources: feat(hogql): flag compatibility for native events and a $false sentinel #91506 and fix(hogql): read rebuilt flags through every native JSON function #106152.
• Restore sent types for declared String paths: posthog/clickhouse/events_json.py declares 382 property paths as String, and that column is the only stored copy. Numbers, booleans and objects sent to those keys come back as text when the whole document is rebuilt (false becomes "false", 1.5 becomes "1.5"). Single properties.<key> reads still match because HogQL casts by property type. On those keys JSONExtractBool returns false for a sent true, JSONExtractRaw and toJSONString(properties) return quoted values, and JSONType reports String. What breaks: the replay inspector shows an LLM event that sent $ai_is_error: false as an error, because "false" is truthy; the AI trace runners' events-table fallback returns the text (test_trace_query_runner.py pins it); native exports would write strings to destinations. Choose the storage for keys that carry non-string values (for example Dynamic) and re-backfill alongside the "false"-variant reinsertion, or explicitly accept and document the text form. Test whole-document reads, raw JSON functions and exports.
• Resolve flag-type ambiguities and old data: deploy the feat(hogql): flag compatibility for native events and a $false sentinel #91506 cleaner before reinserting historical rows whose variant was literally "false"; the old map representation cannot distinguish that variant from boolean false. The reinsertion reads the legacy table, so it must finish before retirement. Decide how JSON functions distinguish a string variant named "true" from boolean true, which fix(hogql): read rebuilt flags through every native JSON function #106152 identifies as still ambiguous. Verify sentinel rejection and reader/writer compatibility during rollout. The map-derived $active_feature_flags also differs from what some senders wrote: posthog-go before 1.13.2 listed off flags as active, and a $feature/<key> set by hand to the string "false" reads as active. Add both to the expected-differences list.
• Fix batch-export source/column mismatches before moving exports: custom export columns are persisted as compiled ClickHouse expressions. Runtime recompilation in resolve_batch_exports_model follows the team setting, but scheduled runs and recent backfills can still read legacy storage. Saving an export after enablement also persists native expressions. Schemas saved with property restrictions omit their original HogQL, so merely moving recompilation to runtime does not handle them. Cover exports created before and after enablement, restricted and unrestricted schemas, scheduled runs, recent and historical backfills, stable aliases, and parameter bindings. Also cover $active_feature_flags fields/filters and the exported whole-properties contract. Native exports serialize the declared array paths as [] on every event, adding keys the legacy export never had (test_http_batch_export_workflow.py skips its insert test on native for this). The native field rewrite relocates $feature/<key> paths but not JSON-function calls such as JSONExtractString(properties, '$feature/<key>'). Until this is solved, enforce legacy export routing when any other reads are enabled. Sources: feat(hogql): flag compatibility for native events and a $false sentinel #91506 and fix(hogql): read rebuilt flags through every native JSON function #106152.
• Choose and enforce who can override the rollout: feat(hogql): add a useNewEventsSchema query modifier #106274's useNewEventsSchema overrides the instance setting and team allowlist. Per-query input and persisted team modifiers are separate routes; a restriction on team-setting writes alone does not control query input. Decide the authorization policy before shipping the override, and test explicit true/false, unset, team defaults, saved queries, and cache separation. Do not enable teams as part of merging feat(hogql): flag compatibility for native events and a $false sentinel #91506.
• Require native-only result regression coverage before the first production trial: chore(tests): prove event queries read native-only fixtures #106380 adds native-only single/bulk fixtures, checks legacy emptiness, and asserts results for grouping, filtered aggregation, subqueries, person joins, boolean filters, and trends. Local tests passed in both modes; forcing legacy reads failed all nine query cases on expected results. Restoring native reads passed again. Before any trial, land chore(ci): test native JSON events without doubling the suite #106280, rebase chore(tests): prove event queries read native-only fixtures #106380 onto it, and land it with the selected native lane passing. The query modules are already in chore(ci): test native JSON events without doubling the suite #106280's target list. This leaves the broader reader audit, ingestion, deletion, performance, and deliberate export/historical-property legacy dependencies as separate requirements.
• Restore continuous native-schema CI: finish chore(ci): test native JSON events without doubling the suite #106280 after fix(ci): gate native-json tests by label #105879 re-gated the full duplicated matrix. Until it lands, only labeled PRs and manual dispatches run native-mode tests; the merge queue and hourly master runs do not. Add experiments to the curated target list once feat(hogql): flag compatibility for native events and a $false sentinel #91506 lands. Keep the directory list current as new event-reading tests appear. The full test-new-events-schema label remains available for broader validation. The agreed approach is the hand-maintained list, not a new periodic full run or mandatory drift-guard plugin. Sources: fix(hogql): run CI on the native-JSON events table and fix what broke #104783, fix(ci): gate native-json tests by label #105879, chore(ci): test native JSON events without doubling the suite #106280.
• Deploy and validate the executable-UDF reliability fix: fix(clickhouse): close inherited descriptors in udf wrapper scripts #105651. Verify fresh/reloaded UDF pools and repeated client requests on the deployed ClickHouse version. Track the longer-term ClickHouse fix separately from the wrapper workaround.
• Decide remaining datetime compatibility: fix(hogql): run CI on the native-JSON events table and fix what broke #104783 fixes zoned scalar datetime reads, but naive datetime strings inferred as UTC can differ from legacy team-local interpretation, and datetimes nested inside objects still lack a zone. Separately, the JSON column infers Date and DateTime types for date-looking strings in undeclared paths at insert: offsets are converted to UTC and dropped, formats such as Jan 15 2024 are rewritten, and whether a value converts depends on the other rows in the same insert. Turning off input_format_try_infer_dates and input_format_try_infer_datetimes for ingestion and the backfill kept strings as sent in a local ClickHouse 26.6 test. Fix ingestion/reading or explicitly accept and document the differences, with tests for non-UTC teams and nested values.
• Confirm replacement sources for omitted payloads and relocated events: the native cleaner omits AI input/output and feature-flag payload properties. Verify the affected product and query paths have the intended replacement behavior. Coordinate the $feature_flag_called transition with Stop advertising $feature_flag_called in taxonomy, autocomplete, and product surfaces #88126; hiding new references in taxonomy/autocomplete does not migrate existing insights, cohorts, actions, or destination filters. Sources: feat(clickhouse): update native JSON events schema #91515, the event cleaner, and Stop advertising $feature_flag_called in taxonomy, autocomplete, and product surfaces #88126.
• Complete native deletion coverage and catch-up: feat(deletes): skip sharded_events_json, configurable per run #102195 deliberately skips sharded_events_json in deletes_job. Re-enabling the target does not revisit deletion requests already marked verified, so plan and verify a backfill sweep as well. Confirm all native storage copies and clusters are reached; chore(clickhouse): test adhoc deletes across events clusters #99416 (draft) adds a two-cluster test for the job. See deletion coverage.
• Restore person_id squash coverage: chore(clickhouse): drop events_json from the person_id squash targets #105148 removed sharded_events_json from the weekly person_id squash. The squash rewrites person_id for merged persons on the legacy events and flag-evaluation tables, then deletes the overrides that recorded the mapping, so native rows keep the absorbed person_id and nothing corrects them at query time. Add the table back once its cluster resolves reliably, and repair rows stranded in the meantime, for example by copying person_id from the squashed legacy rows by uuid. That repair needs the legacy table. chore(clickhouse): make squash coverage a checked invariant #98865 (open) records the exclusion as an explicit exemption. See the deletion-coverage document.
• Resolve native property-removal behavior: feat(hogql): flag compatibility for native events and a $false sentinel #91506 does not add native JSON property rewriting. Decide how removal covers permanent properties, temporary properties, and retained parsing diagnostics, and verify the complete behavior before declaring parity. See feat(clickhouse): update native JSON events schema #91515 and the existing deletion-coverage document.
• Verify the deployed schema and full ingestion/erasure path: feat(clickhouse): update native JSON events schema #91515 requires a manual rollout for existing tables. Confirm the deployed table definitions, ingestion views, UDF versions, temporary-property TTL, and distributed aliases agree. Complete Kafka-to-native ingestion and end-to-end erasure validation, which that PR did not run. Recheck its previously reported nested-null payload and equal-timestamp attribution cases against the final code; do not assume the historical findings remain open without reproducing them.
• Record correctness and performance acceptance: compare representative query shapes against both tables using synthetic fixtures and authorized validation data. Keep an explicit expected-differences list for empty/null values, JSON scalar types, number formatting (1.50 is stored as 1.5), flag arrays/order, omitted properties, datetime normalization, and deduplication. HogQL reads every native path through the generic Dynamic expression, even declared typed paths such as $browser (resolve_json_subcolumn_source); measure that before signing off on performance. feat(hogql): add a useNewEventsSchema query modifier #106274's modifier is the intended way to run one query against each table. Record rollback criteria and the decision to start a limited rollout here. Do not put customer queries or private operational data in this public issue.

Before retiring the legacy events table

• Remove the historical-person-property dependency: on master, posthog/models/person/point_in_time_properties.py reads $set from legacy events with raw SQL. fix(persons): build properties_at_time via HogQL instead of raw SQL #91452 ports it to HogQL and intentionally pins properties_at_time and historical flag test evaluation to legacy events; its $set/$set_once test runs on both tables and fails on native without the pin. Native $set/$set_once live in temporary_properties with a 60-day TTL, which cannot serve the longer replay window. Choose a durable source or an explicitly changed product contract before dropping legacy storage. The mutation log in feat(clickhouse): retain event person mutation payloads on aux #97215 has a 30-day retention window and is not automatically a replacement.
• Finish the event-details/mutation-payload contract: feat(events): show retained person property mutations #97217 remains open and depends on the storage introduced by feat(clickhouse): retain event person mutation payloads on aux #97215. Validate live ingestion, retention, access control, loading/failure behavior, and compatibility with the native temporary column. fix(hogql): rewrite person update property reads #79128 is closed, not merged; reassess its proposed snapshot substitutions and unsupported-payload errors rather than treating them as shipped.
• Audit direct SQL and deliberate legacy pins: inventory readers outside the HogQL schema switch and either migrate or retire them. Starting points include products/feature_flags/backend/flag_analytics.py and products/growth/backend/temporal/health_checks/sdk_outdated.py, plus the historical-person and export paths above. The MCP analytics metrics in posthog/tasks/usage_report.py read events directly because their dedup expression matches the legacy sort key. A semgrep rule against new raw events SQL outside the switch was proposed and not started. Native-only result regression coverage is required before the first production trial, as tracked above; the complete reader inventory and replacement of deliberate legacy dependencies must still finish before retirement.
• Close legacy fallbacks and verify rollback/retention: do not stop legacy writes or drop storage until exports, historical property replay, deletion catch-up, and remaining readers have an approved replacement, and until the repairs that read legacy rows are done: the "false"-variant reinsertion, the person_id repair, and any typed-value re-backfill. Record completion evidence for each dependency here.

Completed foundations

• feat(clickhouse): update native JSON events schema #91515: native schema and base property-reading support merged. Deployment and broader validation remain separate tasks above.
• fix(hogql): run CI on the native-JSON events table and fix what broke #104783: native-mode correctness fixes and initial test coverage merged. Continuous CI coverage is tracked in chore(ci): test native JSON events without doubling the suite #106280.
• feat(clickhouse): retain event person mutation payloads on aux #97215: retained mutation-log storage merged. The application layer and live deployment validation remain separate.

Keeping this tracker current

When working on native JSON events, consult this issue first. Add newly confirmed cutover blockers with a public code/PR reference and a concrete completion condition. Check an item off only when its implementation and required rollout/validation are complete; a merged PR alone may not satisfy both. Link follow-up PRs back here. Keep no-rollout merge decisions distinct from permission to enable teams or retire storage.

Initial inventory: #91506, #91515, #104783, #105879, #106152, #106274, #106280, #105651, #91452, #97215, #97217, and #79128. The deletion dependency comes from #102195 and the repository's deletion-coverage documentation. No production settings were changed while assembling this tracker.

[aspicer]

CI coverage for native reads: #106280 (draft) runs the test paths in .github/new-events-schema-targets.txt (12 directories plus 28 files, about 21% of recorded test time) on the JSON table on every backend run, as ~7 extra jobs instead of the 51-job doubling. The full doubling stays available through the test-new-events-schema label. The list is maintained by hand; a new directory of tests that read event properties needs adding.

Confirmed gaps the test lanes cannot see, because test fixtures always write the legacy table: hand-written SQL that reads events without going through the schema switch passes in both modes and breaks only when legacy storage is retired. Known sites: products/feature_flags/backend/flag_analytics.py (JSONExtract on properties), products/growth/backend/temporal/health_checks/sdk_outdated.py (mat_$lib), posthog/models/person/point_in_time_properties.py (reads $set, which the ingest cleaner moves to temporary_properties). A lint over raw events SQL would catch new ones.

Also: posthog/dags/tests/test_data_deletion_requests.py cannot run in the Django CI job in either mode because that job loads only the versioned JSONDropKeys_v12 UDF, so the JSON leg covers deletes through test_deletes.py and test_deletion_targets.py only.