Roadmap: Hosted sync: upload raw agent artifacts and derive sessions and embeddings server-side

[salmonumbrella]

Summary

Make hosted sync raw-first. A laptop should watch agent source data and transport authenticated incremental raw artifacts to the hosted server. The server should own durable raw storage, parsing, PostgreSQL projection, and embeddings.

The authoritative pipeline is:

laptop watcher → authenticated incremental raw upload → server object storage → server parsing → PostgreSQL → server embeddings

This issue is the umbrella for that end-to-end change. The original request for server-side embedding remains required, but it is now the final derived stage rather than a standalone change to pg serve.

We are starting work on this architecture.

Delivery plan

This is the shipping sequence. Update each item with its PR link or links as the work opens. One item may take more than one PR, but every PR should stay inside one of these boundaries.

1. Raw custody core — canonical manifests, tenant-scoped immutable object and manifest storage, exact verification, PostgreSQL acceptance metadata, source-head fencing, idempotency, and parse-job creation. PR #1396
2. Device enrollment and authenticated raw transport — revocable device credentials, scoped upload tokens, missing-object negotiation, resumable checksummed uploads, manifest-last commit, and durable receipts. PR #1459, PR #1473, PR #1488
3. Laptop watcher and uploader — provider discovery, append-only fast paths, immutable-file and SQLite capture, bounded spool and backpressure, local checkpoints, and reconciliation. PR #1498, PR #1532, PR #1544
4. Server parsing and PostgreSQL projection — isolated manifest materialization, reused provider parsers, leased processing, fenced transactional projection, and explicit source provenance. PR #1607 merged the parse-worker foundation: verified manifest materialization, provider-parser reuse, and generation-fenced job leases. PR #1730 adds transactional PostgreSQL session projection, public session identities, and opt-in runtime activation. It is open for review and needs reconciliation with newer main changes; this item remains open until merge.
5. Server embeddings and search activation — PostgreSQL embeddable units, server-owned encoders, versioned vector generations, complete-coverage activation, and continued service from the previous generation during rebuilds. PR #1736 adds the server worker, durable retries, generation storage and search activation. It depends on PR feat: derive hosted raw captures into PostgreSQL sessions #1730 and currently includes that parent diff; this item remains open until merge.
6. Recovery, retention, and operations — retries and dead-letter handling, reconciliation sweeps, tombstones, garbage collection, health and lag reporting, quotas, and object-store-only disaster rebuilds. PR #1737 adds bounded recovery of terminal embedding failures as the first slice, depending on feat: build hosted embeddings and activate complete search generations #1736 and feat: derive hosted raw captures into PostgreSQL sessions #1730. Raw-job recovery, reconciliation, retention, garbage collection, health reporting, quotas and disaster rebuilds remain; this item stays open.
7. Migration and cutover — raw backfill, shadow parsing and embedding parity, per-tenant and per-provider rollout controls, reversible activation, and retirement of local parsing and embedding in hosted raw mode. First slice: feat: build hosted embeddings in shadow before cutover #1740 adds durable shadow embedding builds and explicit coverage-checked activation. Backfill, normalized and embedding parity, cohort rollout, rollback, and local-work retirement remain open.

This issue stays open until every delivery item is complete and the production acceptance criteria below pass.

Why

Today, hosted PostgreSQL sync still makes every pushing machine maintain a local parsed SQLite archive and, unless explicitly disabled, build embeddings locally before pushing vectors. That is the wrong resource boundary for laptops:

• Long embedding runs consume CPU, memory, battery, API quota, and uptime.
• A MacBook or similar development machine sleeps, changes networks, and should not coordinate server corpus indexing.
• Embedding configuration and provider credentials have to be duplicated on every pusher.
• The hosted service receives derived chat rows but does not hold the authoritative source artifacts needed to reparse after parser fixes or rebuild all derived state after loss.
• Local and server embedding fingerprints can drift, leaving semantic search unavailable even though the server already has the text.

The hosted system should retain the raw source generation and make PostgreSQL and vector indexes reproducible derivatives.

Required architecture

Laptop watcher and uploader

Hosted raw mode must not run local parsing, classification, vector construction, or embedding.

The client should:

• Reuse provider-owned discovery, source identity, companion-file, and reconciliation rules.
• Track stable device identity, file fingerprints, acknowledged byte offsets, pending uploads, and server receipts in a small local state database.
• Use an append fast path for active JSONL files so ordinary updates read and upload only new completed bytes.
• Upload immutable files as content-addressed blobs.
• Use content-defined chunking only for large rewritten files where the append path cannot apply.
• Capture SQLite-backed providers through a consistent online backup rather than copying a live database file.
• Bound concurrency, CPU, memory, bandwidth, and spool size, with explicit degraded-coverage state when offline or full.

Authenticated raw transport

The transport should combine content-addressed objects with immutable manifests:

1. The client negotiates which {sha256, size} objects are missing.
2. Missing objects use resumable, checksummed upload sessions.
3. The server independently verifies final object length and SHA-256.
4. The client commits a canonical manifest only after every referenced object exists.
5. The server returns a durable receipt.
6. The client advances its local checkpoint only after receiving that receipt.

A manifest identifies the tenant, immutable device ID, provider, configured source root, logical source path, generation, parent generation, exact length, ordered objects, source metadata, and any tombstone. Generation commits must use compare-and-swap semantics so delayed clients cannot move a source head backward.

Object storage needs a common interface with filesystem storage for self-hosting and S3-compatible storage for hosted deployments. Raw objects and manifests are immutable. PostgreSQL stores artifact metadata and processing state, not the authoritative raw bytes.

Server parsing and PostgreSQL projection

Manifest acceptance must atomically record the accepted generation and enqueue durable processing work. Object-store events may wake workers but must not be a correctness dependency.

A parse worker should:

• Materialize the manifest into an isolated read-only provider tree.
• Reuse the existing provider parsers and stable path rewriting.
• Apply CPU, memory, byte, file-count, time, path-traversal, and no-network limits.
• Fence writes against the current source generation so an older worker cannot overwrite a newer manifest.
• Write the complete normalized source projection to PostgreSQL in one transaction.
• Preserve the last successful projection if a newer generation fails to parse.
• Record manifest, parser-data, and projection versions for replay and diagnosis.

The normalization and prepared-write portion of the current sync engine should become reusable with SQLite and PostgreSQL writers. We should not create a second hosted-only parser implementation or make a permanent server-side SQLite archive part of the hosted data path.

Source provenance must be explicit. Identical or conflicting provider session identities appearing on multiple devices cannot use arbitrary last-writer-wins behavior.

Server embeddings

After a normalized PostgreSQL projection commits, the server should enqueue idempotent embedding work.

This requires:

• A PostgreSQL implementation of the embeddable-unit source contract.
• Shared document and chunk construction semantics with the local vector system.
• Server-owned encoder configuration and credentials.
• Jobs keyed by the relational revision, document-builder version, chunker version, model, and encoder fingerprint.
• Versioned PostgreSQL vector generations.
• Atomic activation only after a generation has complete corpus coverage.
• Continued query availability from the previous active generation during rebuilds.

Push clients no longer send vectors in hosted raw mode. PostgreSQL content can be re-embedded entirely on the server.

Security and tenancy

Hosted ingestion needs device-scoped authentication rather than the existing single shared bearer token.

• A user enrolls a device and receives a revocable device credential stored in the platform credential store.
• The credential is exchanged for short-lived tokens scoped to object negotiation, upload, manifest commit, and status.
• Tenant identity is derived from authentication, never trusted from a request body.
• Machine names are display metadata, not authorization identities.
• Object namespaces and every ingestion key are tenant-scoped.
• Deduplication stays within one tenant to prevent cross-tenant object-existence leaks.
• PostgreSQL row-level security provides defense in depth.
• Raw content never appears in logs.
• TLS protects transit; object storage, PostgreSQL, backups, and scratch storage are encrypted at rest.

The server must read raw chats to parse them, so this is not end-to-end encryption from the server.

Failure recovery, retention, and operations

Processing state should be durable and observable:

accepted → parsing → projected → embedding → complete

A generation may instead be retrying, failed, or superseded.

Workers use leased, idempotent jobs with bounded retry, backoff, and dead-letter state. Reconciliation sweeps must detect accepted manifests missing jobs or projections, missing objects, stale leases, incomplete embeddings, and unreferenced objects. Duplicate or out-of-order notifications must be harmless.

Source disappearance creates a tombstone manifest. It does not immediately destroy raw history. Garbage collection marks from retained manifests, active upload sessions, and processing jobs, then sweeps unreachable objects only after a grace period.

A supported disaster-recovery command must recreate all derived PostgreSQL and vector state from retained manifests alone.

Client and server status need upload lag, parse lag, embedding lag, queued bytes, retry/failure state, degraded source coverage, queue age, parser failure rates, embedding throughput/cost, quota rejections, and orphan detection.

Migration and compatibility

Local-only AgentsView remains supported. Existing pg push behavior remains available while hosted raw mode rolls out.

The production rollout should be reversible:

1. Deploy the final auth, object, manifest, and job contracts behind tenant/provider controls.
2. Backfill existing raw provider roots without local parsing or embedding.
3. Shadow-parse retained raw artifacts and compare normalized results with existing PostgreSQL data.
4. Enable server projection per tenant, device, and provider after parity passes.
5. Build server embeddings in shadow and atomically activate them.
6. Disable local pg push and local embedding scheduling for hosted raw mode.

Existing PostgreSQL sessions whose original raw source no longer exists should be preserved as legacy-derived records. They must not be represented as rebuildable raw generations.

Production acceptance criteria

• Every supported provider produces equivalent normalized output when parsed from a reconstructed manifest tree and from its original local source.
• Ordinary appends read and upload only appended bytes.
• Hosted raw mode invokes no local parser, classifier, vector builder, or encoder.
• Rotation, truncation, rename, atomic replacement, partial files, multi-file sources, and live SQLite updates are lossless.
• Upload interruption, duplication, reordering, checksum failure, revocation, quota exhaustion, and spool exhaustion never advance an unsafe checkpoint.
• Worker crashes at every transaction boundary are replay-safe.
• Superseded processing cannot overwrite a newer generation.
• Tenant isolation covers APIs, PostgreSQL, object storage, upload URLs, caches, and logs.
• A clean derived database can be rebuilt from object storage with equivalent normalized corpus hashes and complete vector coverage.
• Backfill and sustained append workloads meet measured laptop resource and server queue budgets.

Implementation should be decomposed into independently reviewable changes, but every change must preserve these final contracts. There should be no disposable protocol or reduced-correctness “v1” path.

[salmonumbrella]

@rodboev would love your help with this roadmap. It lets users offload processing to a server.