feat(skill): add /drt-troubleshoot — full top-to-bottom diagnostic checklist (closes #369)

.claude/commands/drt-troubleshoot.md

@@ -0,0 +1,139 @@
+
+Systematically isolate which layer of a drt setup is failing, then hand off
+the specific fix.
+
+`/drt-debug` is reactive — the user has an error message and wants it fixed.
+`/drt-troubleshoot` is for when the symptom is vague ("nothing happens",
+"the data looks wrong", "I just set this up and want to confirm it works").
+Walk the layers **top to bottom**, confirming each one is green before
+moving on. The first layer that fails is where the problem lives — stop
+there and fix it (or invoke `/drt-debug` for a known error pattern).
+
+## The checklist
+
+Run these in order. Each step has a "✅ green when" condition — only proceed
+to the next step once the current one is green.
+
+### 1. Environment
+
+```bash
+drt doctor
+```
+
+Checks Python version, the `~/.drt/profiles.yml` file, the active
+`drt_project.yml`, installed extras, and common env-var gotchas (v0.7+).
+
+- **✅ green when:** every line is ✅ (extras you don't use can be ❌ — only
+  the ones your source/destination need must be installed).
+- **🔴 common failures:** missing `drt-core[<extra>]` for your connector;
+  `~/.drt/profiles.yml` not found (run `drt init`); wrong Python (<3.10).
+
+### 2. Profile + credentials
+
+```bash
+cat ~/.drt/profiles.yml          # confirm the profile name matches drt_project.yml
+echo "$YOUR_PASSWORD_ENV"        # confirm the referenced env vars are actually set
+```
+
+- **✅ green when:** the profile named in `drt_project.yml` exists in
+  `profiles.yml`, and every `${VAR}` / `*_env` it references resolves to a
+  non-empty value in the shell.
+- **🔴 common failures:** profile name mismatch (`drt run --profile <name>`
+  or `DRT_PROFILE` to override); env var unset or exported in a different
+  shell; secret hardcoded in YAML instead of an env reference (`drt validate`
+  flags this since v0.7.5).
+
+### 3. Connectivity
+
+```bash
+drt validate                     # also surfaces connection issues where supported
+```
+
+For SQL sources/destinations, a connection round-trip is the cheapest way to
+separate "can't reach the warehouse" from "the data is wrong". If `drt
+validate` doesn't exercise a live connection for your connector, a one-off
+manual check works: `psql` / `curl -X POST <url>` / the warehouse console.
+
+- **✅ green when:** the source warehouse and the destination both accept a
+  connection with the configured credentials.
+- **🔴 common failures:** wrong host/port/account; firewall / VPN; expired
+  token; for BigQuery, `GOOGLE_APPLICATION_CREDENTIALS` unset or pointing at
+  a stale keyfile; for non-US BigQuery datasets, a missing `location`.
+
+### 4. Config validity
+
+```bash
+drt validate                     # JSON-Schema + semantic checks on every sync YAML
+drt list                         # confirm your sync is actually discovered
+```
+
+- **✅ green when:** `drt validate` reports 0 errors and `drt list` shows the
+  sync you expect.
+- **🔴 common failures:** sync file outside `syncs/`; YAML indentation; a
+  `model: ref('table')` pointing at a table that doesn't exist; an
+  `upsert`/`mirror` mode without the required `upsert_key`; deprecation
+  warnings (v0.7.2+) that will become errors.
+
+### 5. Dry run (the data preview)
+
+```bash
+drt run --select <name> --dry-run            # config parses, rows extract, nothing is written
+drt run --select <name> --dry-run --diff     # record-level preview for queryable destinations (v0.7.1+)
+```
+
+This is the single most useful step for "the sync runs but the data looks
+wrong" — it shows exactly what would be written without touching the
+destination.
+
+- **✅ green when:** the row count is what you expect and the previewed
+  records / `--diff` look correct.
+- **🔴 common failures:** 0 rows (the source query / `model` filters
+  everything out, or `mode: incremental` already consumed the watermark —
+  check `drt status`, replay with `--cursor-value`); `{{ row.field }}`
+  referencing a column the source doesn't return (use `tojson_safe` for
+  datetime/Decimal/UUID, v0.7.6+); wrong column names in the template.
+
+### 6. First real run
+
+```bash
+drt run --select <name> --verbose            # row-level error detail
+drt status                                   # what actually happened
+drt status --output json                     # machine-readable, for CI
+```
+
+- **✅ green when:** `result.success` equals the dry-run row count and
+  `result.failed` is 0.
+- **🔴 common failures:** rate limit (429 — lower `rate_limit.requests_per_second`;
+  HubSpot max 9/s, GitHub Actions 5/s); per-row auth/permission errors
+  (`on_error: skip` to see the full failure count instead of stopping at the
+  first); partial success where some rows fail validation downstream.
+
+### 7. Post-sync correctness
+
+```bash
+drt test --select <name>                     # freshness / unique / accepted_values tests, if defined
+```
+
+- **✅ green when:** all declared tests pass (or there are none — that's not a
+  failure, just no assertions).
+- **🔴 common failures:** the sync reported success but downstream `unique` /
+  `freshness` tests fail — the data moved but isn't what was expected. This
+  usually points back to the source query (step 5), not the sync itself.
+
+## When you've found the failing layer
+
+- **A specific error message** → switch to `/drt-debug` for the known-pattern
+  fix.
+- **Silent / wrong-data** → the dry-run step (5) almost always localises it;
+  fix the source query or template and re-run the checklist from step 5.
+- **Still stuck after the checklist** → capture `drt doctor`, the sync YAML,
+  `drt_project.yml`, and `drt run --verbose --output json` output, and open a
+  discussion / issue with that bundle.
+
+## Reference
+
+- `docs/llm/CONTEXT.md` — architecture and key concepts
+- `docs/llm/API_REFERENCE.md` — all config fields
+- `docs/connectors/` — per-connector auth, sync modes, and gotchas
+- `/drt-debug` — the companion skill for fixing a specific error once this
+  checklist has localised it

CHANGELOG.md

@@ -50,6 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 - **`sync.field_mappings` — declarative column renaming** ([#415](https://github.com/drt-hub/drt/issues/415)): a `{source_column: destination_field}` map in the sync's `sync:` block renames columns in flight, so you no longer have to alias in the source SQL (`SELECT user_id AS id`). This decouples the rename from the query — the same query can feed multiple syncs with different destination shapes — matching the first-class field-mapping feature Census / Hightouch / Polytomic expose. Implemented as a **pure transform** (`drt/engine/field_mappings.py`, `apply_field_mappings`) that the engine applies **after extraction, cursor tracking, and lookups, immediately before the record reaches the destination**. The ordering is deliberate and load-bearing: `cursor_field` and `lookups` reference **source** column names (they run before the rename, so incremental watermark state and FK resolution use the names the query produces), while `upsert_key`, destination columns, and the `--diff` engine see the **mapped** names. The remap is a single pass — order-independent and non-chaining (`{a: b, b: c}` renames `a→b` and `b→c` from the original keys, it does not turn `a` into `c`) — and best-effort per record (a mapping whose source column is absent from a row is simply not applied to that row). Because the transform is pure (no I/O, no observer side effects), it lives outside `engine/sync.py`'s `SyncObserver` boundary and is unit-testable in isolation; the existing `tests/unit/test_engine_observer.py` boundary checks still pass. New `drt/engine/field_mappings.py` ships `apply_field_mappings` plus `unmapped_source_columns` (a best-effort `drt validate` helper that flags mapping keys absent from the source schema — wired and ready, but only fires once schema introspection is available, since `drt validate` is a static check today). 17 tests across `tests/unit/test_field_mappings.py` (14 — no-op identity, rename-keeps-others, no-mutation, absent-column skip, single-pass-no-chain, collision last-write-wins, validate-helper edge cases) and `tests/unit/test_engine.py` (3 — destination receives mapped keys, no-mappings passthrough, cursor-field-uses-source-name while destination-gets-mapped). `field_mappings: dict[str, str] | None` is a new optional field on `SyncOptions` (auto-surfaced in the generated JSON Schema). New `docs/guides/field-mappings.md` covers the ordering model, the cursor-remap example, the rules/edge cases, and what it deliberately does **not** do (type coercion / computed fields / nested extraction — out of scope, tracked separately). `docs/llm/API_REFERENCE.md` sync block + the `/drt-create-sync` skill updated. Closes [#415](https://github.com/drt-hub/drt/issues/415).
+- **`/drt-troubleshoot` Claude Code skill** ([#369](https://github.com/drt-hub/drt/issues/369)): a fifth user-facing skill that complements `/drt-debug`. Where `/drt-debug` is reactive (the user has a specific error and wants it fixed via known-pattern matching), `/drt-troubleshoot` is for vague symptoms — "drt isn't working", a sync that silently does nothing, results that look wrong, or verifying a fresh setup end-to-end. It walks the setup **top to bottom** as a 7-layer isolation checklist (environment → profile/credentials → connectivity → config validity → dry-run → first real run → post-sync correctness), each layer with a "✅ green when" gate and "🔴 common failures" list, so the first failing layer localises the problem; it then hands off to `/drt-debug` for the specific fix. Synced to `.claude/commands/drt-troubleshoot.md` via `make sync-skills`. README.md / README.ja.md skill tables and `skills/drt/.claude-plugin/plugin.json` description updated to the 5-skill set. The skill references the current CLI surface throughout (`drt doctor`, `drt validate`, `drt run --dry-run --diff`, `--cursor-value`, `tojson_safe`, `drt test`), so it lands already aligned with the v0.7+ feature set rather than needing the catch-up the other skills just went through.
 
 - **Databricks Delta Lake destination** ([#167](https://github.com/drt-hub/drt/issues/167)): the third DWH destination alongside Snowflake ([#353](https://github.com/drt-hub/drt/pull/353)) and BigQuery ([#584](https://github.com/drt-hub/drt/pull/584), in flight) — completes the major-DWH lineup. Supports **INSERT** (append), **MERGE** (upsert via Delta Lake's native `MERGE INTO`), and **`sync.mode: mirror`** (upsert + end-of-sync DELETE-missing — joins the Postgres / MySQL / ClickHouse / Snowflake mirror family from [#340](https://github.com/drt-hub/drt/issues/340)). Auth via the **Databricks SQL Connector**: `host_env` resolves to the workspace hostname (e.g. `dbc-abc12345-1234.cloud.databricks.com`), `http_path_env` to the SQL warehouse HTTP path (e.g. `/sql/1.0/warehouses/abc123def456`), `token_env` to a personal access token (PAT, `dapi*`). The token-bearing principal needs `USAGE` on the catalog + schema, `MODIFY` on the target table, and `CREATE` on the schema for the merge-path staging. Unity Catalog three-part names (`catalog.schema.table`) are the default; legacy workspaces use `catalog: hive_metastore`. **Merge implementation**: because Delta Lake doesn't have session-local temp tables (no `CREATE TEMP TABLE` syntax), drt creates a uniquely-named Delta scratch table `catalog.schema.__drt_staging_<table>` cloned from the target's schema, stages rows via per-row `INSERT`, executes the `MERGE INTO target USING staging ON <upsert_key>` statement, and `DROP TABLE`s the staging at the end. The `__drt_staging_*` prefix makes the scratch table identifiable in audit logs; a `CREATE OR REPLACE TABLE` on the next run cleanly overwrites any interrupted-mid-sync remnant. Composite keys are supported (`upsert_key: [tenant_id, user_id]`) — the `ON` clause becomes AND-joined. When every column is in `upsert_key`, the MERGE skips the `UPDATE` clause (effectively INSERT-IF-NOT-EXISTS). **Mirror semantics**: `sync.mode: mirror` forces the MERGE write path regardless of `config.mode`, then issues `DELETE FROM <table> WHERE upsert_key NOT IN (observed)` at end-of-sync. Composite keys use the `WHERE (c1, c2) NOT IN ((v1a, v1b), (v2a, v2b), ...)` form (same shape as Snowflake's #599 leg). Safety guard: if no batch ever produced records, the DELETE is skipped entirely so a transient empty source doesn't wipe the destination. Empty batches short-circuit before any `databricks.sql` import or warehouse call — the same implicit "no driver was imported" contract used by the other SQL destinations ([#595](https://github.com/drt-hub/drt/pull/595)). Row-level errors during INSERT are captured in `result.row_errors`; `on_error: skip` (default) continues the sync, `on_error: fail` raises. Requires `pip install drt-core[databricks]` (depends on `databricks-sql-connector>=3.0`). New `docs/connectors/databricks.md` covers all three modes, the auth flow with PAT generation steps, the Unity Catalog vs Hive Metastore catalog field, the merge-path staging design (why a Delta scratch table and not `CREATE TEMP TABLE`), and a sync-mode compatibility table. 22 unit tests in `tests/unit/test_databricks_destination.py` cover config validation (including the `schema:` YAML alias for the mypy-strict `schema_` field, three-part FQN in `describe()`, and Hive Metastore catalog), the empty-batch short-circuit, the `databricks.sql.connect()` kwargs shape (protects against silent template-copy drift from the Snowflake destination, since the two share most other code), INSERT / MERGE happy paths, the `upsert_key`-required ValueError, on_error skip vs fail behaviour, composite-key MERGE `ON` clause, all-columns-are-key MERGE (no UPDATE clause), all mirror invariants (`upsert_key` validation, MERGE-write-path forcing, single-column DELETE, composite-key DELETE tuple form, skip-when-no-records safety guard, no-op `finalize_sync` for non-mirror modes), and the `test_connection` round-trip. `databricks.sql` is mocked via `sys.modules` injection (no real Databricks workspace or `databricks-sql-connector` install required). Closes [#167](https://github.com/drt-hub/drt/issues/167). BigQuery destination ([#584](https://github.com/drt-hub/drt/pull/584), contributor PR) follows the same DWH MERGE shape and remains in flight.
 - **Google Cloud Storage destination** ([#169](https://github.com/drt-hub/drt/issues/169)): the natural pair for BigQuery users and the second of the v0.8 cloud-storage trio (S3 / GCS / Azure Blob). Same shape as the S3 destination from [#613](https://github.com/drt-hub/drt/pull/613) — four formats (**csv** / **json** / **jsonl** / **parquet**) with optional **gzip** compression for the text formats (parquet keeps its own column-level compression via `parquet_compression`). Authentication defaults to **Application Default Credentials** (the `GOOGLE_APPLICATION_CREDENTIALS` env var → `gcloud auth application-default login` → GCE/GKE/Cloud Run attached service account); for non-GCP CI / cron environments where ADC isn't available, point `credentials_path` at a service-account JSON keyfile. `project_id` is optional — the credentials file usually carries one. Default object name is `<prefix><UTC ISO8601 basic>.<ext>` — timestamped so re-runs land at a fresh object rather than overwriting, matching the S3 convention and making downstream "new objects since last check" polling trivial. Override the file name via `key_template` (supports the `{timestamp}` placeholder); for per-sync routing, the recommended pattern is a sync-specific `prefix`. Empty batches short-circuit before any `google.cloud` import or GCS call — the same implicit "no driver was imported" contract used by the SQL destinations ([#595](https://github.com/drt-hub/drt/pull/595)), so a run with zero source rows produces zero GCS objects. Failure semantics distinguish missing-extras (the `[gcs]` missing-extras `ImportError` bubbles up so the engine surfaces the deployment mistake once at the top; the `[parquet]` missing-extras `ImportError` flows through the shared serialiser and is recorded as a row failure with the `drt-core[parquet]` install hint preserved) from network / auth / permission errors (recorded as `result.failed = len(records)` so other batches keep going) — see `drt/destinations/gcs.py`. Implementation is a thin `_client` + `blob.upload_from_string` shim on top of `drt/destinations/_blob_serializer.py` from the prior refactor PR ([#622](https://github.com/drt-hub/drt/pull/622)) — the csv / json / jsonl / parquet + gzip + key-naming logic is shared with S3, not duplicated. Requires `pip install drt-core[gcs]` (and additionally `drt-core[parquet]` for `format: parquet`). New `docs/connectors/gcs.md` covers all four formats, the ADC / service-account auth flow, the object-naming convention with `key_template` examples, and notes on BigQuery external-table interop. 17 unit tests in `tests/unit/test_gcs_destination.py` cover config validation, the implicit no-`google.cloud`-import empty-batch path, `blob.upload_from_string` call shape per format, gzip → `blob.content_encoding = "gzip"` + `.gz` extension, ADC vs `project_id` vs `credentials_path` threading, and the failure paths (upload error → row failures, `[gcs]` missing-extras → `ImportError` bubble-up, `[parquet]` missing-extras → row failure with install hint preserved). `google-cloud-storage` is mocked via `sys.modules` injection (no real GCP account or `google-cloud-storage` install required). The parquet orchestration test mocks `pandas` + `pyarrow` rather than calling them end-to-end — the S3 destination's parquet test already validates the real PAR1 binary, and double-registering the pyarrow type extension across two test classes would raise `A type extension with name pandas.period already defined`. Closes [#169](https://github.com/drt-hub/drt/issues/169). Azure Blob ([#170](https://github.com/drt-hub/drt/issues/170)) is the natural follow-up — the same serialiser / key-naming layer is reusable, only the upload client differs.

README.ja.md

@@ -1,4 +1,4 @@
-<!-- i18n-sync: base=README.md, hash=e1b1d150934f03d7e2c30fab260873f66a063c00 -->
+<!-- i18n-sync: base=README.md, hash=fe801bea667943e5cad36ff5fff3b13c48b41ef2 -->
 
 [English](./README.md) | [日本語](./README.ja.md)
 
@@ -217,7 +217,8 @@ Claude Codeの公式スキルをインストールすると、チャットイン
 | スキル | トリガー | 説明 |
 |-------|---------|-------------|
 | `/drt-create-sync` | "create a sync" | インテントから有効な同期YAMLを生成 |
-| `/drt-debug` | "sync failed" | エラーを診断し、修正方法を提案 |
+| `/drt-debug` | "sync failed" | 特定のエラーを診断し、修正方法を提案 |
+| `/drt-troubleshoot` | "drt isn't working" | 上から下まで全体を診断するチェックリストを案内 |
 | `/drt-init` | "set up drt" | プロジェクト初期化を案内 |
 | `/drt-migrate` | "migrate from Census" | 既存の設定をdrt YAMLに変換 |
 

README.md

@@ -242,10 +242,11 @@ Copy the files from `.claude/commands/` into your drt project's `.claude/command
 
 | Skill              | Trigger               | What it does                               |
 | ------------------ | --------------------- | ------------------------------------------ |
-| `/drt-create-sync` | "create a sync"       | Generates valid sync YAML from your intent |
-| `/drt-debug`       | "sync failed"         | Diagnoses errors and suggests fixes        |
-| `/drt-init`        | "set up drt"          | Guides through project initialization      |
-| `/drt-migrate`     | "migrate from Census" | Converts existing configs to drt YAML      |
+| `/drt-create-sync`  | "create a sync"        | Generates valid sync YAML from your intent       |
+| `/drt-debug`        | "sync failed"          | Diagnoses a specific error and suggests fixes    |
+| `/drt-troubleshoot` | "drt isn't working"    | Walks a full top-to-bottom diagnostic checklist  |
+| `/drt-init`         | "set up drt"           | Guides through project initialization            |
+| `/drt-migrate`      | "migrate from Census"  | Converts existing configs to drt YAML            |
 
 ---