From ec5aed9b9c896d21f862466c9ab0c220af10190d Mon Sep 17 00:00:00 2001 From: masukai Date: Tue, 9 Jun 2026 21:05:43 +0900 Subject: [PATCH] =?UTF-8?q?chore(skill):=20refresh=20/drt-migrate=20?= =?UTF-8?q?=E2=80=94=20add=20mirror/replace=20modes=20+=20post-v0.7=20mapp?= =?UTF-8?q?ings?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Skill was last touched 2026-03-30 (#60) and missed the sync-mode expansions that landed since. Drift fixed: 1. **`sync.mode: mirror`** (v0.7.7+) — Census "Full Sync with Deletion" / Hightouch "Mirror" equivalent, supported on postgres / mysql / clickhouse / snowflake. Now in the concept map. 2. **`sync.mode: replace`** + `replace_strategy: swap` (zero-downtime on Postgres / Snowflake) — was missing entirely from the mode mapping table. 3. **`failure_alerts`** (v0.7.0+) — replaces the old "Error notifications" row that pointed at `on_error: skip + drt status` (which is the per-row policy, not the sync-level alerting). 4. **Hardcoded-secret detection** (v0.7.5) — added to the auth migration table as "never hardcode — `drt validate` flags hardcoded secrets". 5. **Connection-string profiles** — added to the auth migration table as the equivalent for connection strings from other tools (resolved via `${VAR}` in `~/.drt/profiles.yml`). New section: **sync-mode picking guide** — a "user wants X, drt mode is Y" table so the LLM can pick the right mode from the contributor's intent rather than guessing from the old tool's vocabulary. Covers full / incremental / upsert / mirror / replace. New "summary" guidance: - Whether the migration changes semantics (Census "Full Sync with Deletion" → `mirror` is same shape; Census "Full Sync without Deletion" → `full` keeps stale rows, so user may want `upsert` instead) — the LLM now surfaces this distinction explicitly. New Reference entry: `examples/postgres_to_postgres_mirror/` — runnable example so the user can confirm the shape before adopting. `make sync-skills` ran; `.claude/commands/drt-migrate.md` matches. `make check-skills` clean. Co-authored-by: Claude Opus 4.7 --- .claude/commands/drt-migrate.md | 35 +++++++++++++++++++------- skills/drt/skills/drt-migrate/SKILL.md | 35 +++++++++++++++++++------- 2 files changed, 52 insertions(+), 18 deletions(-) diff --git a/.claude/commands/drt-migrate.md b/.claude/commands/drt-migrate.md index 6543fd7a..e10e4c12 100644 --- a/.claude/commands/drt-migrate.md +++ b/.claude/commands/drt-migrate.md @@ -5,7 +5,7 @@ Help the user migrate from an existing Reverse ETL tool (Census, Hightouch, Poly 1. Ask the user to share their existing sync configuration (screenshot, YAML, JSON, or description). -2. Map their existing config to drt equivalents using the table below. +2. Map their existing config to drt equivalents using the tables below. 3. Generate a valid `syncs/.yml` for each sync. @@ -19,19 +19,33 @@ Help the user migrate from an existing Reverse ETL tool (Census, Hightouch, Poly |---------------------------|----------------| | Source (BigQuery model) | `model: ref('table')` or raw SQL | | Destination connection | `destination.type` + auth config | -| Sync behavior: Full | `sync.mode: full` | -| Sync behavior: Append | `sync.mode: incremental` + `cursor_field` | -| Field mappings | `body_template` / `properties_template` (Jinja2) | -| Run schedule | `drt run` via cron or CI | -| Error notifications | `on_error: skip` + `drt status` | +| Sync behavior: Full | `sync.mode: full` (every run, no dedup) | +| Sync behavior: Append (incremental) | `sync.mode: incremental` + `cursor_field` | +| Sync behavior: Mirror (upsert + delete-removed) | `sync.mode: mirror` (v0.7.7+) + `upsert_key` — supported on postgres / mysql / clickhouse / snowflake | +| Sync behavior: Replace (overwrite table) | `sync.mode: replace` (TRUNCATE + INSERT, zero-downtime via `replace_strategy: swap` on supported DWHs) | +| Field mappings (UI column picker) | `body_template` / `properties_template` (Jinja2) | +| Run schedule | `drt run` via cron, CI, Dagster, Airflow, or Prefect | +| Error notifications | `failure_alerts` (Slack / webhook, v0.7.0+) — fires on sync-level failures | +| Per-row error policy | `on_error: skip` (continue past failures) vs `on_error: fail` (stop at first) | + +### Sync-mode picking guide (which `sync.mode` matches the source semantic) + +| User wants | drt mode | Notes | +|------------|----------|-------| +| "Re-send everything every run" | `full` | Default. Idempotent destinations only — REST API / Slack / file outputs. | +| "Append new rows since last run" | `incremental` + `cursor_field` | Watermark-based. `--cursor-value` overrides for backfill. | +| "Upsert by key" | `upsert` + `upsert_key` | Census's most common "Update" shape. | +| "Upsert by key AND delete rows removed from source" | `mirror` + `upsert_key` | Census's "Full Sync with Deletion" / Hightouch's "Mirror" semantic. v0.7.7+, SQL destinations. Source key cardinality fits in memory. | +| "Overwrite the destination table each run" | `replace` | TRUNCATE + INSERT. Set `replace_strategy: swap` (Postgres / Snowflake) for zero-downtime via staging-table swap. | ### Auth migration | Old tool style | drt equivalent | |---------------|----------------| -| Stored API key in UI | `token_env: MY_TOKEN` + `export MY_TOKEN=...` | +| Stored API key in UI | `token_env: MY_TOKEN` + `export MY_TOKEN=...` (never hardcode — `drt validate` flags hardcoded secrets) | | OAuth app | Use token from OAuth flow → `token_env` | | Service account JSON | Set `GOOGLE_APPLICATION_CREDENTIALS` for BigQuery source | +| Connection string | Source profile field in `~/.drt/profiles.yml` (env-var-substituted via `${VAR}`) | ## Output Format @@ -48,14 +62,17 @@ destination: # ... fields sync: - mode: full # or incremental + mode: full # or incremental / upsert / mirror / replace # ... ``` Then summarize: - What manual steps are needed (env vars, `~/.drt/profiles.yml`) - Any features the old tool had that drt doesn't support yet (flag these clearly) +- Whether the migration changes semantics (e.g. Census "Full Sync with Deletion" → drt `mirror` is the same shape; Census "Full Sync without Deletion" → drt `full` keeps stale rows in the destination, so the user may want `upsert` instead) ## Reference -See `docs/llm/API_REFERENCE.md` for all destination types and fields. +- `docs/llm/API_REFERENCE.md` — all destination types and fields +- `docs/connectors/` — per-destination details (auth, supported modes) +- `examples/postgres_to_postgres_mirror/` — runnable example for the `mirror` mode shape diff --git a/skills/drt/skills/drt-migrate/SKILL.md b/skills/drt/skills/drt-migrate/SKILL.md index ca57011d..0920df9f 100644 --- a/skills/drt/skills/drt-migrate/SKILL.md +++ b/skills/drt/skills/drt-migrate/SKILL.md @@ -12,7 +12,7 @@ Help the user migrate from an existing Reverse ETL tool (Census, Hightouch, Poly 1. Ask the user to share their existing sync configuration (screenshot, YAML, JSON, or description). -2. Map their existing config to drt equivalents using the table below. +2. Map their existing config to drt equivalents using the tables below. 3. Generate a valid `syncs/.yml` for each sync. @@ -26,19 +26,33 @@ Help the user migrate from an existing Reverse ETL tool (Census, Hightouch, Poly |---------------------------|----------------| | Source (BigQuery model) | `model: ref('table')` or raw SQL | | Destination connection | `destination.type` + auth config | -| Sync behavior: Full | `sync.mode: full` | -| Sync behavior: Append | `sync.mode: incremental` + `cursor_field` | -| Field mappings | `body_template` / `properties_template` (Jinja2) | -| Run schedule | `drt run` via cron or CI | -| Error notifications | `on_error: skip` + `drt status` | +| Sync behavior: Full | `sync.mode: full` (every run, no dedup) | +| Sync behavior: Append (incremental) | `sync.mode: incremental` + `cursor_field` | +| Sync behavior: Mirror (upsert + delete-removed) | `sync.mode: mirror` (v0.7.7+) + `upsert_key` — supported on postgres / mysql / clickhouse / snowflake | +| Sync behavior: Replace (overwrite table) | `sync.mode: replace` (TRUNCATE + INSERT, zero-downtime via `replace_strategy: swap` on supported DWHs) | +| Field mappings (UI column picker) | `body_template` / `properties_template` (Jinja2) | +| Run schedule | `drt run` via cron, CI, Dagster, Airflow, or Prefect | +| Error notifications | `failure_alerts` (Slack / webhook, v0.7.0+) — fires on sync-level failures | +| Per-row error policy | `on_error: skip` (continue past failures) vs `on_error: fail` (stop at first) | + +### Sync-mode picking guide (which `sync.mode` matches the source semantic) + +| User wants | drt mode | Notes | +|------------|----------|-------| +| "Re-send everything every run" | `full` | Default. Idempotent destinations only — REST API / Slack / file outputs. | +| "Append new rows since last run" | `incremental` + `cursor_field` | Watermark-based. `--cursor-value` overrides for backfill. | +| "Upsert by key" | `upsert` + `upsert_key` | Census's most common "Update" shape. | +| "Upsert by key AND delete rows removed from source" | `mirror` + `upsert_key` | Census's "Full Sync with Deletion" / Hightouch's "Mirror" semantic. v0.7.7+, SQL destinations. Source key cardinality fits in memory. | +| "Overwrite the destination table each run" | `replace` | TRUNCATE + INSERT. Set `replace_strategy: swap` (Postgres / Snowflake) for zero-downtime via staging-table swap. | ### Auth migration | Old tool style | drt equivalent | |---------------|----------------| -| Stored API key in UI | `token_env: MY_TOKEN` + `export MY_TOKEN=...` | +| Stored API key in UI | `token_env: MY_TOKEN` + `export MY_TOKEN=...` (never hardcode — `drt validate` flags hardcoded secrets) | | OAuth app | Use token from OAuth flow → `token_env` | | Service account JSON | Set `GOOGLE_APPLICATION_CREDENTIALS` for BigQuery source | +| Connection string | Source profile field in `~/.drt/profiles.yml` (env-var-substituted via `${VAR}`) | ## Output Format @@ -55,14 +69,17 @@ destination: # ... fields sync: - mode: full # or incremental + mode: full # or incremental / upsert / mirror / replace # ... ``` Then summarize: - What manual steps are needed (env vars, `~/.drt/profiles.yml`) - Any features the old tool had that drt doesn't support yet (flag these clearly) +- Whether the migration changes semantics (e.g. Census "Full Sync with Deletion" → drt `mirror` is the same shape; Census "Full Sync without Deletion" → drt `full` keeps stale rows in the destination, so the user may want `upsert` instead) ## Reference -See `docs/llm/API_REFERENCE.md` for all destination types and fields. +- `docs/llm/API_REFERENCE.md` — all destination types and fields +- `docs/connectors/` — per-destination details (auth, supported modes) +- `examples/postgres_to_postgres_mirror/` — runnable example for the `mirror` mode shape