fix(sync): avoid starving windowed full scans

.github/workflows/ci.yml

@@ -23,7 +23,7 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
         with:
-          node-version: 26
+          node-version: 26.12.1
       - name: Test cloud publisher
         run: node --test scripts/publish-cloud.test.mjs
       - name: Test documentation build

.github/workflows/pages.yml

@@ -35,7 +35,7 @@ jobs:
       - name: Set up Node
         uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
         with:
-          node-version: 26
+          node-version: 26.12.1
 
       - name: Test site builder
         run: node --test scripts/build-docs-site.test.mjs

CHANGELOG.md

@@ -2,6 +2,21 @@
 
 ## Unreleased
 
+- Fix `sync --full --since` repeatedly resuming incomplete stored history and never syncing the guild's other channels; windowed full syncs now scan the normal channel catalog. Thanks @rszrszrsz.
+
+- Settle channel message failures answered Unknown Channel only after an exact channel lookup corroborates them, as a provider outcome that keeps history and the retry marker; ordinary repair also settles earlier failures superseded by a recorded deletion or exclusion. Public-only snapshots withhold channels with an unavailable marker until a successful crawl re-admits them, and routine syncs no longer panic when only marked channels remain.
+- Apply Gateway `GUILD_CREATE` and `THREAD_LIST_SYNC` channel catalogs in one transaction, so large guilds no longer exceed the tail handler deadline at startup under FULL synchronization or leave a partially applied catalog.
+- Retain channel scope metadata independently of content, fail closed on unknown ancestry, and protect public snapshots from excluded ancestors; resolve allowed new channels without a restart.
+- Honor channel and ancestor tombstones before page, exact-message and attachment requests even without configured exclusions, resolve every scope rejection as a policy outcome, and count only actionable attachment failures in status. Desktop cache imports no longer clear channel tombstones, and text repair keeps author and channel names in multilingual lexical indexes.
+- Preserve attributed embed fields and whitespace boundaries in derived text; add durable attachment extraction status and retain successful text through transient failures.
+- Reconcile bounded exact-message and metadata failures during ordinary repair, expose failures and resumable text-repair progress in status, and preserve vectors when text is unchanged.
+- Capture supported bulk message deletions, channel/thread tombstones, role changes and thread-list metadata; include voice/stage text in scoped REST repair.
+- Open every archive and metrics writer connection with FULL synchronization and macOS full-fsync checkpoints, including pool replacements, while keeping read-only connections unchanged.
+- Reconcile full guild members inside the running tail owner with separately bounded pagination waits, without treating cached authors as complete enumeration.
+- Keep native Components V2 text as attributed parts and fence recovery pages so a strictly newer edit or deletion is never replaced by a delayed page.
+- Accept protected owner-only credential files for the bot token and embedding key in unattended services, with no symlinks, writable ancestors or granting ACLs, and no Keychain fallback once selected.
+- Add opt-in measurement of native OpenAI embedding attempts and usage.
+
 ## 0.16.0 - 2026-10-20
 
 - Give private checkpoint upload, full authenticated SQLite verification, and promotion independent step budgets, preserving every custody check and the promotion reserve when verification is slow.

Dockerfile

@@ -5,6 +5,7 @@ ARG GO_IMAGE_DIGEST=sha256:f2c872bcee949b7be9c490efd02b48a8a2fbf947cdf533f1c5852
 ARG ALPINE_VERSION=3.24
 
 FROM golang:${GO_VERSION}-alpine@${GO_IMAGE_DIGEST} AS build
+RUN apk add --no-cache --upgrade ca-certificates=20261020-r0 ca-certificates-bundle=20261020-r0
 RUN apk add --no-cache ca-certificates git
 WORKDIR /src
 COPY go.mod go.sum ./
@@ -16,6 +17,7 @@ RUN CGO_ENABLED=0 GOOS=linux go build -trimpath \
     -o /out/discrawl ./cmd/discrawl
 
 FROM alpine:${ALPINE_VERSION}
+RUN apk add --no-cache --upgrade ca-certificates=20261020-r0 ca-certificates-bundle=20261020-r0
 RUN apk add --no-cache ca-certificates git openssh-client tzdata \
     && adduser -D -u 10001 -h /home/discrawl discrawl \
     && mkdir -p /data/config /data/data /data/cache /data/state \

docs/commands/members.md

@@ -68,5 +68,6 @@ last_message=2026-03-08T15:59:58Z
 
 ## See also
 
+- [Full member refresh](../guides/member-refresh.html)
 - [`channels`](channels.html)
 - [Data layout](../guides/data-storage.html)

docs/commands/status.md

@@ -22,6 +22,20 @@ discrawl status
 - remote endpoint/archive metadata when `remote.mode = "cloud"`
 - embeddings status if `[search.embeddings]` is enabled
 
+## macOS scheduled reporter
+
+A launchd `spawn failed` / `EX_CONFIG` result can occur before Discrawl runs.
+Use current launchd diagnostic timestamps to establish the cause; an old stderr
+file does not establish a new database failure. A `needs LWCR update` error means
+launchd could not refresh its managed lightweight code requirement.
+
+After an authorized signed-binary replacement, verify every job that invokes
+that executable, including idle status reporters. Preserve its plist and signing
+requirements, then use the job owner's supported unload/bootstrap registration
+path when a stale managed requirement needs refreshing. Verify a new successful
+JSON report and the original scheduling interval. Re-register only the affected
+job; preserve the running collector and existing identity/security policy.
+
 ## See also
 
 - [`coverage`](coverage.html) - per-guild/channel archive readiness and wiretap skip counts

docs/commands/sync.md

@@ -73,6 +73,7 @@ discrawl sync --with-media
 - `--with-media` records expired or removed Discord CDN URLs as failed fetches with the HTTP status, commonly `404`.
 - `--with-media` updates the local cache only; run `publish --push` afterward to include cached non-DM media in the Git backup as gzip-compressed files.
 - `--since` does not mark older history as complete, so a later `sync --full` without `--since` can continue the backfill.
+- Untargeted `--full --since` runs discover the normal channel catalog even when stored channels have incomplete older history; only unwindowed `--full` runs resume that backlog before discovery.
 - `--full --since <RFC3339>` reads from each channel's current head to that boundary and verifies current history permissions before and after the scan. Successful scans record their `[since, until)` coverage even when no messages exist in that window. The `channel:<id>:history_window:v1:<since>` sync-state record binds the guild, channel, native type, UTC bounds, completion time, and terminal pagination boundary. Distinct lower bounds retain separate records; routine latest-only syncs do not extend them. These records do not mark older history complete.
 - Voice and stage text chats use the same history path as text channels. Verifying their history requires `VIEW_CHANNEL`, `READ_MESSAGE_HISTORY`, and `CONNECT`.
 - Targeted bot syncs (`--source discord --channels <ids>`) refresh Desktop-imported channel placeholders and their parent metadata from Discord before crawling messages. Unavailable metadata or missing parent or permission evidence fails the sync visibly; ordinary cached channels keep their existing behavior. Ancestor metadata refreshes do not crawl ancestor messages or reset history and archive cursors. This is still a message sync: `--since` limits initial history and backfill, not fetching new messages after an existing cursor.

docs/configuration.md

@@ -55,6 +55,9 @@ token_source = "env" # use "none" for Git-only read access
 token_env = "DISCORD_BOT_TOKEN"
 token_keyring_service = "discrawl"
 token_keyring_account = "discord_bot_token"
+# Optional unattended macOS/Linux service credential:
+# token_source = "file"
+# token_file = "/absolute/private/directory/discord-bot-token"
 
 [sync]
 source = "both" # "discord" for bot-only sync, "wiretap" for desktop-cache-only import
@@ -93,6 +96,9 @@ api_key_env = "OPENAI_API_KEY"
 # api_key_source = "keyring"
 # api_key_keyring_service = "discrawl/embeddings"
 # api_key_keyring_account = "api-key"
+# Optional unattended macOS/Linux service credential:
+# api_key_source = "file"
+# api_key_file = "/absolute/private/directory/openai-embeddings.key"
 dimensions = 512 # optional OpenAI projection; omit for provider default
 batch_size = 64
 max_input_chars = 12000
@@ -168,10 +174,49 @@ Set `discord.token_source = "keyring"` if you want to require keyring lookup and
 - `[share.filter]` narrows only `publish` output; sync can still keep a richer local archive
 - `share.filter.public_only` exports only channels visible to the guild
   `@everyone` role after category/channel permission overwrites; private
-  threads are excluded
+  threads and channels with an unavailable marker (and their threads) are
+  excluded
 - `share.filter.include_channel_ids` and
   `share.filter.exclude_channel_ids` accept Discord channel ids; exclusions win,
   and including a forum parent also includes its allowed public threads
 - filtered publishes cannot write generated README reports, and remove older
   generated Discrawl share READMEs and the producer-owned
   `reports/latest-field-notes.md` / `.json` artifacts before committing
+
+## Unattended service credentials
+
+On macOS and Linux, select `discord.token_source = "file"` with an absolute
+`discord.token_file`, and `search.embeddings.api_key_source = "file"` with an
+absolute `search.embeddings.api_key_file`. This supports ordinary service
+startup after login without an interactive Keychain unlock. Environment and
+Keychain modes keep their existing behavior.
+
+Place each credential in a regular, singly linked file owned by the service
+user with mode `0600`, inside an owned `0700` directory. Ancestors must be
+owned by that user or root and have no group/world write permission. Use a
+canonical absolute path with no symlink components. Discrawl opens each
+component relative to a verified directory descriptor and validates the
+opened file before reading at most 16 KiB. Missing, unsafe, empty, or oversized
+files fail with redacted diagnostics. Explicit file mode selects only that
+file; it never tries another credential source.
+
+On macOS, extended ACLs are checked through each opened descriptor; only
+absent or deny-only ACLs are accepted. On Linux, the POSIX ACL mask reflected
+in the mode bits constrains named-user/group access. Safe execute-only
+ancestors can be traversed without requiring directory-listing permission.
+
+File bytes receive the same normalization as existing Keychain values:
+Discord trims whitespace and an optional `Bot ` prefix; embedding keys trim
+surrounding whitespace and otherwise remain unchanged. Document and query
+embeddings use the same resolver, without exporting the key to the process
+environment.
+
+For a service migration, transfer the existing resolved Keychain value directly
+in memory into the protected file, verify byte equality privately, and retain
+the original Keychain item and private config/binary backup for rollback. Keep
+the collector running during preparation. Validate file selection with a
+failing mock Keychain and local provider fixtures, then gracefully replace the
+installed binary/config at an idle boundary and verify ordinary collection
+and embedding success. Retain the service's existing identity, schedule and
+archive state. Protected files remove the Keychain-unlock dependency; they
+remain readable by their owning user and privileged local administrators.

docs/guides/components-v2-recovery.md

@@ -0,0 +1,44 @@
+# Components V2 preservation and scoped recovery
+
+The native collector retains Discord's component tree in `messages.raw_json`.
+For messages with `IS_COMPONENTS_V2` (`32768`), the native text-parts path extracts
+text displays, button labels and URLs, and thumbnail/gallery descriptions, with
+their JSON-pointer paths. The additive part kinds are `component_text`,
+`component_label`, `component_url`, and `component_description`. The text-v2
+format remains unchanged. Authored `content` stays exactly as Discord supplied
+it, including an empty string. Ordinary button-only messages keep their existing
+normalized text; unchanged normalized text reuses existing vectors.
+
+Older native builds dropped components while serializing the Discord SDK object.
+An absent component tree cannot be reconstructed from that stored row. Check
+the actual installed source and retained evidence before attempting a local
+text rebuild. A version label alone does not prove that a custom build includes
+this preservation fix.
+
+Prepare recovery under the source owner's normal maintenance process:
+
+1. Freeze a finite candidate list with message/guild/channel IDs, hashes and
+   component-presence metadata. Keep message bodies out of diagnostic output.
+2. Prepare and review a compatible signed build, preserving active source
+   permissions, member refresh, durability and other installed fixes.
+3. Take a consistent private backup and record the exact original cursors and
+   installed identity before an approved graceful native cutover.
+4. Rewind only approved affected channel cursors to just before their oldest
+   candidate, following the scoped upstream recovery procedure. Keep unrelated
+   cursors, history and exclusions intact. Existing normal startup repair can
+   recover those ranges while Gateway capture runs.
+5. REST pages carry their request-start observation time into the existing
+   transactional revision guard. A deletion or a strictly newer provider edit
+   wins; when edit timestamps are equal or absent, a write recorded after the
+   page started wins over the stale page. The guard
+   covers current text and dependent projections together. A page's fetched-row
+   count does not prove every row was applied: verify exact candidates afterward.
+6. Verify components, attributed text, search and source frontiers. Report
+   inaccessible/deleted provider responses explicitly; missing data is not a
+   fabricated deletion. Coordinate downstream publication after source acceptance.
+
+Retain binary rollback and before-state evidence. Roll back only under native
+ownership, preserving observations made since the backup. Never lower a cursor
+that has advanced beyond its recorded original frontier or blindly restore an
+old whole database over new collection. Source recovery and downstream database
+migration are separate operations; capture-feed activation holds remain separate.

docs/guides/embeddings.md

@@ -94,6 +94,45 @@ The snapshot stores vectors under `embeddings/<provider>/<model>/<input_version>
 - [`embed`](../commands/embed.html)
 - [Configuration](../configuration.html)
 
+## Optional local inference measurement
+
+Set `CLAW_INFERENCE_STATSD=127.0.0.1:18125` in the embedding process environment
+to send best-effort DogStatsD metadata to a local collector. An unset, malformed,
+or non-loopback destination disables measurement. This applies only to direct
+OpenAI embeddings at the standard public endpoint; local and compatible
+providers keep their existing behavior. Collection, model settings, credentials,
+quotas and schedules are independent of measurement.
+
+Each actual HTTP attempt emits `claw.inference.requests`, `duration` (milliseconds)
+and `last_event` (Unix seconds). Tags are `service:discrawl`, `workload:discrawl`,
+`operation:embedding_document` or `embedding_query`, `provider:openai`, bounded
+returned `model` (configured model if absent), `tier`, and `outcome`. Outcomes distinguish success, HTTP
+error, transport/body-read error and invalid response. Duration covers transport
+through the existing body reads, excluding queue wait and database publication.
+Retries count separately; skipped inputs and vector reuse do not count.
+
+Known numeric `usage.prompt_tokens` emits `input_tokens` and `usage_known`;
+`usage.prompt_tokens_details.cached_tokens` emits `cached_tokens` when valid.
+Missing, malformed or out-of-range usage stays unknown; reported zero is retained.
+No output/reasoning/cache-write zeros are invented for embedding responses.
+Usage may remain known when a decoded response fails vector validation.
+The standard `text-embedding-3-small` default tier is estimated at USD 0.02 per
+million input tokens (OpenAI pricing checked 2026-10-12), emitted as `cost_usd`
+and `cost_known`. A different returned model or explicit unpriced tier leaves
+cost unknown. An omitted tier defaults to `default` only for this public
+embeddings endpoint, whose request does not select a service tier. This fallback
+does not apply to custom endpoints or Responses/classifier requests. These lossy
+measurements are standard list-price estimates, not billing receipts or proof
+of collection coverage.
+
+The observer retains only bounded metadata while the existing decoder reads;
+it does not buffer vectors, read ahead, change response limits or log payloads,
+identifiers, credentials or error bodies. It sends at most one 4096-byte local
+UDP datagram per attempt with a 2 ms write deadline and no retries, persistence,
+background tasks or shutdown flush. An unavailable receiver does not change the
+provider result. Native query embeddings are measured only when that command is
+explicitly used; ordinary shared archive research remains with Clawmmunity.
+
 ## Continuous embedding during capture
 
 `discrawl tail --embed-live` opts into crawlkit's background worker runtime and

docs/guides/git-snapshots.md

@@ -29,6 +29,10 @@ Filter rules:
 - `--public-only` keeps only channels where the guild `@everyone` role has
   `VIEW_CHANNEL` after category and channel permission overwrites
 - private threads are excluded
+- channels whose last message crawl Discord refused (an Unknown Channel or
+  Missing Access unavailable marker), and their threads, are withheld because
+  their stored permissions can no longer be verified; a later successful crawl
+  clears the marker and re-admits them
 - `--include-channels` and `--exclude-channels` accept comma-separated channel
   ids; exclusions win
 - including a forum parent also includes its allowed public threads