From fa3c38cc7bfd901c6cb3590760f8ce2d7492b5fe Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 12 Mar 2026 18:18:05 -0500 Subject: [PATCH 1/8] docs: point vana-connect skill at CLI --- skills/vana-connect/SETUP.md | 53 +++++++++-------- skills/vana-connect/SKILL.md | 109 +++++++++++++++++++---------------- 2 files changed, 86 insertions(+), 76 deletions(-) diff --git a/skills/vana-connect/SETUP.md b/skills/vana-connect/SETUP.md index e330829..8668447 100644 --- a/skills/vana-connect/SETUP.md +++ b/skills/vana-connect/SETUP.md @@ -1,61 +1,64 @@ # Connect -- Setup -Skip if `~/.dataconnect/playwright-runner/index.cjs` and `~/.dataconnect/run-connector.cjs` both exist. +This setup exists to let the skill use the built local CLI at: + +```bash +node /home/tnunamak/code/vana-connect/dist/cli/bin.js +``` + +Skip runtime setup if `node /home/tnunamak/code/vana-connect/dist/cli/bin.js status --json` reports `"runtime":{"installed":true,...}`. ## Prerequisites - Node.js v18+ - Git -## Install +## Build the CLI -Run the setup script from the data-connectors repo root: +From `/home/tnunamak/code/vana-connect`: ```bash -bash skills/vana-connect/scripts/setup.sh +pnpm install +pnpm build ``` -This installs the playwright-runner, Chromium, and run-connector.cjs in a single step. If the user needs to approve commands, this is one approval instead of many. +Verify: -**Before running**, tell the user: setup will download a browser engine and some Node.js dependencies into `~/.dataconnect/`. This is a one-time step. +```bash +ls /home/tnunamak/code/vana-connect/dist/cli/bin.js +``` -## Manual install +## Install the runtime -If the setup script doesn't work for your environment, follow these steps individually: +Once the CLI is built, use it to provision the runtime: ```bash -mkdir -p ~/.dataconnect/connectors -cd ~/.dataconnect - -git clone --depth 1 --filter=blob:none --sparse --branch main \ - https://github.com/vana-com/data-connect.git _data-connect -cd _data-connect && git sparse-checkout set playwright-runner -cp -r playwright-runner ../playwright-runner -cd .. && rm -rf _data-connect -cd ~/.dataconnect/playwright-runner && npm install -npx playwright install chromium +node /home/tnunamak/code/vana-connect/dist/cli/bin.js setup --yes ``` -Then copy run-connector.cjs from the skill's scripts/ directory: +Before running, tell the user this downloads a browser engine and some dependencies into `~/.dataconnect/`. This is a one-time step. + +## Verify ```bash -cp skills/vana-connect/scripts/run-connector.cjs ~/.dataconnect/run-connector.cjs +node /home/tnunamak/code/vana-connect/dist/cli/bin.js status ``` -> **Do not** use `curl` to fetch this file from GitHub — the repo root contains a symlink that GitHub raw serves as a text pointer, not the actual script. +You should see `Runtime: installed`. If not, inspect the log path surfaced by the CLI and only fall back to the older script-level flow if the CLI setup path is blocked. -## Verify +## Legacy fallback + +Only use this if the CLI setup path is broken and you are debugging the underlying runtime: ```bash -ls ~/.dataconnect/playwright-runner/index.cjs ~/.dataconnect/run-connector.cjs +bash skills/vana-connect/scripts/setup.sh ``` -Both files should exist. - ## File Locations | Path | Purpose | |------|---------| +| `/home/tnunamak/code/vana-connect/dist/cli/bin.js` | Local development CLI entrypoint | | `~/.dataconnect/playwright-runner/` | Runner process | | `~/.dataconnect/run-connector.cjs` | Batch-mode runner wrapper | | `~/.dataconnect/connectors/` | Connector scripts | diff --git a/skills/vana-connect/SKILL.md b/skills/vana-connect/SKILL.md index 777eec7..a2771dc 100644 --- a/skills/vana-connect/SKILL.md +++ b/skills/vana-connect/SKILL.md @@ -10,85 +10,91 @@ description: > # Connect -Connect personal data from web platforms using local browser automation. +Connect personal data from web platforms using the `vana` CLI and local browser automation. ## Setup -If `~/.dataconnect/playwright-runner/index.cjs` or `~/.dataconnect/run-connector.cjs` does not exist, setup is needed. Tell the user: "I need to do a one-time setup first — this downloads a browser engine and some dependencies to `~/.dataconnect/`. It'll take about a minute." Then follow `SETUP.md` (co-located with this file). - -## Flow - -### 1. Find a connector +Use the built local CLI directly during development: ```bash -node scripts/fetch-connector.cjs +node /home/tnunamak/code/vana-connect/dist/cli/bin.js ``` -This searches the registry and downloads the connector + metadata + schemas in one step. It prints JSON: `{ "found": true, "connectorPath": "..." }` on success, `{ "found": false }` if no connector exists. +If `/home/tnunamak/code/vana-connect/dist/cli/bin.js` does not exist, follow `SETUP.md` in this folder to build it first. -If found, let the user know there's an existing connector and this should be quick. +Before connecting a source, check runtime state with: + +```bash +node /home/tnunamak/code/vana-connect/dist/cli/bin.js status --json +``` -**If no connector exists for the platform,** tell the user you'll build one — this involves researching the platform's data APIs, writing the extraction code, and testing it. Let them know it'll take a bit and they're welcome to do something else while you work. Then read `CREATE.md` and follow it. Continue from step 2 with the newly created connector. +If the runtime is missing, tell the user: "I need to do a one-time setup first. This downloads a browser engine and some dependencies into `~/.dataconnect/` and usually takes about a minute." Then run: -### 2. Read the connector +```bash +node /home/tnunamak/code/vana-connect/dist/cli/bin.js setup --yes +``` -Before running, read the connector script to understand: -- What URL it starts from (`page.goto()` or `connectURL` in metadata) -- Whether it uses `requestInput` (batch-compatible) or `showBrowser`/`promptUser` (browser login) -- What data it collects +## Flow -### 3. Run it +### 1. Explore available sources ```bash -node ~/.dataconnect/run-connector.cjs [start-url] -node ~/.dataconnect/run-connector.cjs [start-url] --inputs '{"username":"x","password":"y"}' +node /home/tnunamak/code/vana-connect/dist/cli/bin.js sources --json ``` -**Stdout** is line-delimited JSON: +This is the source of truth for what the CLI can currently connect. Prefer it over inspecting repo files manually. -| type | meaning | action | -|------|---------|--------| -| `need-input` | Connector needs credentials or 2FA | Ask user, write response file (see below) | -| `legacy-auth` | Legacy auth, can't run headless | See legacy section | -| `result` | Data saved to `resultPath` | Read the file | -| `error` | Failure | Report to user | +If the requested platform is present, use the CLI flow below. -Exit codes: 0 = success, 2 = needs input, 3 = legacy auth, 1 = error. +**If no connector exists for the platform,** tell the user you'll build one — this involves researching the platform's data APIs, writing the extraction code, and testing it. Let them know it'll take a bit and they're welcome to do something else while you work. Then read `CREATE.md` and follow it. -### 4. Handle auth +### 2. Connect with the CLI -1. Check if `~/.dataconnect/browser-profiles/{script-filename}/` exists -- try without `--inputs` first (session may still be valid) -2. If `need-input` appears in stdout: the connector is paused, waiting for input. Two options: +Start with the agent-safe probe: - **Option A (preferred): File-based response.** The `need-input` message includes `pendingInputPath` and `responseInputPath`. Ask the user for the requested fields, then write the response as JSON to `responseInputPath`. The connector resumes automatically — no restart needed. This works for multi-step auth (credentials first, then 2FA). +```bash +node /home/tnunamak/code/vana-connect/dist/cli/bin.js connect --json --no-input +``` + +This will: + +- ensure the runtime is installed +- resolve and cache the connector +- try a saved session if one exists +- return structured events and a final outcome such as `needs_input`, `legacy_auth`, `connected_local_only`, or `connected_and_ingested` - ```bash - # Connector emits: {"type":"need-input","message":"Enter credentials","pendingInputPath":"~/.dataconnect/pending-input-steam-playwright.json","responseInputPath":"~/.dataconnect/input-response-steam-playwright.json",...} - # After asking the user, write the response to the path from the message: - echo '{"username":"alice","password":"secret"}' > ~/.dataconnect/input-response-steam-playwright.json - # Connector picks it up within 1 second and continues. - # If it later needs 2FA, another need-input appears — write again. - ``` +If the outcome is `needs_input`, rerun interactively: - **Option B: Pre-supply with `--inputs`.** If you know all inputs upfront, pass them at launch. Fields are consumed as each `requestInput` call is made. +```bash +node /home/tnunamak/code/vana-connect/dist/cli/bin.js connect +``` + +If the user specifically wants to inspect current state before rerunning, use: + +```bash +node /home/tnunamak/code/vana-connect/dist/cli/bin.js status +``` - ```bash - node ~/.dataconnect/run-connector.cjs --inputs '{"username":"x","password":"y"}' - ``` +### 3. Handle outcomes -TOTP codes expire in ~30 seconds — write the response file immediately after receiving a code. Cookies saved in browser profiles persist for days to weeks. +The CLI emits structured JSON events in `--json` mode. -#### Legacy connectors +Key outcomes: -Exit code 3 means the connector uses `showBrowser`/`promptUser` instead of `requestInput`: +- `needs_input` + The connector needs a live login or another manual step. Explain that you'll rerun interactively. +- `legacy_auth` + The connector still depends on `showBrowser` / `promptUser`. Explain that this source still needs a headed/manual session path and may not work in fully headless batch mode yet. +- `connected_local_only` + Data was collected locally but no Personal Server target was available. +- `connected_and_ingested` + Data was collected and synced to the Personal Server. -1. Try without `--inputs` -- if a browser profile exists, login may be skipped. -2. Check for a migrated version on the `main` branch. -3. Write a login script to establish a session, then run the stock connector. +If setup, fetch, or run output is truncated, the CLI may point to a full log file under `~/.dataconnect/logs/`. Use that rather than re-running blindly. -### 5. Validate, present results, and offer to contribute +### 4. Validate, present results, and offer to contribute -On success, immediately run validation — before presenting results to the user: +If you built or modified a connector, immediately run validation — before presenting results to the user: ```bash node scripts/validate.cjs /-playwright.js --check-result ~/.dataconnect/last-result.json @@ -104,7 +110,7 @@ If you built a new connector (not one from the registry), ask the user: If yes, run `node scripts/validate.cjs /-playwright.js --contribute`. If no, move on. -### 6. Suggest what to do with the data +### 5. Suggest what to do with the data After the contribution question is resolved (or if using an existing connector), suggest use cases from `RECIPES.md`: user profile generation, personal knowledge base, data backup, cross-platform synthesis, activity analytics. @@ -121,7 +127,7 @@ The user can't see what you're doing behind the scenes. Keep them informed at ke 3. **After collection**, summarize results in human terms — not file paths: - Good: "Connected! I collected 249 issues, 63 projects, 9 teams, and your profile from Linear." - Bad: "Data saved to ~/.dataconnect/last-result.json" - - Read the result file and build the summary from `exportSummary` and the scoped keys. + - Prefer the CLI outcome plus the result file. Build the summary from `exportSummary` and the scoped keys. 4. **On failure**, explain what went wrong and what the user can do: - Auth failed → "Login didn't work. Can you double-check your credentials?" @@ -134,3 +140,4 @@ The user can't see what you're doing behind the scenes. Keep them informed at ke 3. **One platform at a time** 4. **Check session first** -- try without credentials if a browser profile exists 5. **Read connectors before running them** +6. **Use the CLI as the primary interface** -- only drop to raw scripts when debugging or updating connector internals From 77d424dc1aed293fc20809281d7a107b9985e500 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 12 Mar 2026 18:20:26 -0500 Subject: [PATCH 2/8] chore: remove stale vana-connect fetch helper --- .../vana-connect/scripts/fetch-connector.cjs | 125 ------------------ 1 file changed, 125 deletions(-) delete mode 100644 skills/vana-connect/scripts/fetch-connector.cjs diff --git a/skills/vana-connect/scripts/fetch-connector.cjs b/skills/vana-connect/scripts/fetch-connector.cjs deleted file mode 100644 index de91c7d..0000000 --- a/skills/vana-connect/scripts/fetch-connector.cjs +++ /dev/null @@ -1,125 +0,0 @@ -#!/usr/bin/env node -/** - * fetch-connector.cjs — Download a connector from the registry. - * - * Usage: node scripts/fetch-connector.cjs - * - * Searches the registry for the platform, downloads the connector script - * and metadata to ~/.dataconnect/connectors/. Prints the local path on success. - * - * Exit codes: 0 = found and downloaded, 1 = not found or error. - */ - -const https = require('https'); -const fs = require('fs'); -const path = require('path'); -const os = require('os'); - -const platform = process.argv[2]; -if (!platform) { - console.error('Usage: node scripts/fetch-connector.cjs '); - process.exit(1); -} - -const REGISTRY_URL = 'https://raw.githubusercontent.com/vana-com/data-connectors/main/registry.json'; -const BASE_URL = 'https://raw.githubusercontent.com/vana-com/data-connectors/main'; -const CONNECTORS_DIR = path.join(os.homedir(), '.dataconnect', 'connectors'); - -function fetch(url) { - return new Promise((resolve, reject) => { - https.get(url, { headers: { 'User-Agent': 'dataconnect' } }, (res) => { - if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) { - return fetch(res.headers.location).then(resolve, reject); - } - if (res.statusCode !== 200) { - reject(new Error(`HTTP ${res.statusCode} for ${url}`)); - res.resume(); - return; - } - const chunks = []; - res.on('data', (c) => chunks.push(c)); - res.on('end', () => resolve(Buffer.concat(chunks).toString())); - res.on('error', reject); - }).on('error', reject); - }); -} - -async function main() { - // Fetch registry - let registry; - try { - registry = JSON.parse(await fetch(REGISTRY_URL)); - } catch (e) { - console.error('Failed to fetch registry:', e.message); - process.exit(1); - } - - // Search for platform (case-insensitive, partial match) - const search = platform.toLowerCase(); - const match = (registry.connectors || []).find((c) => { - const name = (c.name || '').toLowerCase(); - const id = (c.id || '').toLowerCase(); - return name === search || id === search || name.includes(search) || id.includes(search); - }); - - if (!match) { - console.log(JSON.stringify({ found: false, platform })); - process.exit(1); - } - - // Download connector files - const scriptPath = match.scriptPath || match.script_path; - const metadataPath = scriptPath.replace(/\.js$/, '.json'); - const company = path.dirname(scriptPath); - - const localDir = path.join(CONNECTORS_DIR, company); - fs.mkdirSync(localDir, { recursive: true }); - - const files = [scriptPath, metadataPath]; - const downloaded = []; - - for (const filePath of files) { - try { - const content = await fetch(`${BASE_URL}/${filePath}`); - const localPath = path.join(CONNECTORS_DIR, filePath); - fs.writeFileSync(localPath, content); - downloaded.push(localPath); - } catch (e) { - // Metadata might not exist, that's OK - if (filePath === metadataPath) continue; - console.error(`Failed to download ${filePath}:`, e.message); - process.exit(1); - } - } - - // Download schemas if referenced in metadata - try { - const metaLocal = path.join(CONNECTORS_DIR, metadataPath); - if (fs.existsSync(metaLocal)) { - const meta = JSON.parse(fs.readFileSync(metaLocal, 'utf-8')); - if (meta.scopes && Array.isArray(meta.scopes)) { - const schemasDir = path.join(localDir, 'schemas'); - fs.mkdirSync(schemasDir, { recursive: true }); - for (const scope of meta.scopes) { - const scopeName = scope.scope || scope.name; - if (!scopeName) continue; - try { - const schemaContent = await fetch(`${BASE_URL}/${company}/schemas/${scopeName}.json`); - fs.writeFileSync(path.join(schemasDir, `${scopeName}.json`), schemaContent); - downloaded.push(path.join(schemasDir, `${scopeName}.json`)); - } catch {} // Schema might not exist yet - } - } - } - } catch {} // Non-critical - - const connectorPath = path.join(CONNECTORS_DIR, scriptPath); - console.log(JSON.stringify({ - found: true, - platform: match.name || platform, - connectorPath, - files: downloaded, - })); -} - -main(); From 8c2ccf6dcbbb588034772b9eb74d02ae5bb5e90a Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Fri, 13 Mar 2026 09:55:58 -0500 Subject: [PATCH 3/8] docs: use published canary cli for vana-connect skill --- skills/vana-connect/SETUP.md | 23 +++++++++++++++-------- skills/vana-connect/SKILL.md | 22 ++++++++++++++-------- 2 files changed, 29 insertions(+), 16 deletions(-) diff --git a/skills/vana-connect/SETUP.md b/skills/vana-connect/SETUP.md index 8668447..00a18d5 100644 --- a/skills/vana-connect/SETUP.md +++ b/skills/vana-connect/SETUP.md @@ -1,19 +1,25 @@ # Connect -- Setup -This setup exists to let the skill use the built local CLI at: +This setup exists to let the skill use the published canary CLI: ```bash -node /home/tnunamak/code/vana-connect/dist/cli/bin.js +pnpm dlx @opendatalabs/connect@canary ``` -Skip runtime setup if `node /home/tnunamak/code/vana-connect/dist/cli/bin.js status --json` reports `"runtime":{"installed":true,...}`. +Skip runtime setup if `pnpm dlx @opendatalabs/connect@canary status --json` reports `"runtime":"installed"` or `"runtime":{"installed":true,...}`. ## Prerequisites - Node.js v18+ - Git -## Build the CLI +## Verify the published CLI + +```bash +pnpm dlx @opendatalabs/connect@canary --help +``` + +## Local development fallback From `/home/tnunamak/code/vana-connect`: @@ -30,10 +36,10 @@ ls /home/tnunamak/code/vana-connect/dist/cli/bin.js ## Install the runtime -Once the CLI is built, use it to provision the runtime: +Use the published CLI to provision the runtime: ```bash -node /home/tnunamak/code/vana-connect/dist/cli/bin.js setup --yes +pnpm dlx @opendatalabs/connect@canary setup --yes ``` Before running, tell the user this downloads a browser engine and some dependencies into `~/.dataconnect/`. This is a one-time step. @@ -41,7 +47,7 @@ Before running, tell the user this downloads a browser engine and some dependenc ## Verify ```bash -node /home/tnunamak/code/vana-connect/dist/cli/bin.js status +pnpm dlx @opendatalabs/connect@canary status ``` You should see `Runtime: installed`. If not, inspect the log path surfaced by the CLI and only fall back to the older script-level flow if the CLI setup path is blocked. @@ -58,7 +64,8 @@ bash skills/vana-connect/scripts/setup.sh | Path | Purpose | |------|---------| -| `/home/tnunamak/code/vana-connect/dist/cli/bin.js` | Local development CLI entrypoint | +| `pnpm dlx @opendatalabs/connect@canary` | Published canary CLI entrypoint | +| `/home/tnunamak/code/vana-connect/dist/cli/bin.js` | Local development fallback | | `~/.dataconnect/playwright-runner/` | Runner process | | `~/.dataconnect/run-connector.cjs` | Batch-mode runner wrapper | | `~/.dataconnect/connectors/` | Connector scripts | diff --git a/skills/vana-connect/SKILL.md b/skills/vana-connect/SKILL.md index a2771dc..287668b 100644 --- a/skills/vana-connect/SKILL.md +++ b/skills/vana-connect/SKILL.md @@ -14,24 +14,30 @@ Connect personal data from web platforms using the `vana` CLI and local browser ## Setup -Use the built local CLI directly during development: +Prefer the published canary CLI: + +```bash +pnpm dlx @opendatalabs/connect@canary +``` + +If the canary CLI is unavailable or the user is explicitly testing local changes, fall back to: ```bash node /home/tnunamak/code/vana-connect/dist/cli/bin.js ``` -If `/home/tnunamak/code/vana-connect/dist/cli/bin.js` does not exist, follow `SETUP.md` in this folder to build it first. +If neither path is available, follow `SETUP.md` in this folder. Before connecting a source, check runtime state with: ```bash -node /home/tnunamak/code/vana-connect/dist/cli/bin.js status --json +pnpm dlx @opendatalabs/connect@canary status --json ``` If the runtime is missing, tell the user: "I need to do a one-time setup first. This downloads a browser engine and some dependencies into `~/.dataconnect/` and usually takes about a minute." Then run: ```bash -node /home/tnunamak/code/vana-connect/dist/cli/bin.js setup --yes +pnpm dlx @opendatalabs/connect@canary setup --yes ``` ## Flow @@ -39,7 +45,7 @@ node /home/tnunamak/code/vana-connect/dist/cli/bin.js setup --yes ### 1. Explore available sources ```bash -node /home/tnunamak/code/vana-connect/dist/cli/bin.js sources --json +pnpm dlx @opendatalabs/connect@canary sources --json ``` This is the source of truth for what the CLI can currently connect. Prefer it over inspecting repo files manually. @@ -53,7 +59,7 @@ If the requested platform is present, use the CLI flow below. Start with the agent-safe probe: ```bash -node /home/tnunamak/code/vana-connect/dist/cli/bin.js connect --json --no-input +pnpm dlx @opendatalabs/connect@canary connect --json --no-input ``` This will: @@ -66,13 +72,13 @@ This will: If the outcome is `needs_input`, rerun interactively: ```bash -node /home/tnunamak/code/vana-connect/dist/cli/bin.js connect +pnpm dlx @opendatalabs/connect@canary connect ``` If the user specifically wants to inspect current state before rerunning, use: ```bash -node /home/tnunamak/code/vana-connect/dist/cli/bin.js status +pnpm dlx @opendatalabs/connect@canary status ``` ### 3. Handle outcomes From 53f53fe1db6152bc262e2fdde82658457ca2ec75 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Fri, 13 Mar 2026 10:32:29 -0500 Subject: [PATCH 4/8] docs: prefer npx canary install for vana-connect skill --- skills/vana-connect/SETUP.md | 12 ++++++------ skills/vana-connect/SKILL.md | 14 +++++++------- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/skills/vana-connect/SETUP.md b/skills/vana-connect/SETUP.md index 00a18d5..eea90bf 100644 --- a/skills/vana-connect/SETUP.md +++ b/skills/vana-connect/SETUP.md @@ -3,10 +3,10 @@ This setup exists to let the skill use the published canary CLI: ```bash -pnpm dlx @opendatalabs/connect@canary +npx -y @opendatalabs/connect@canary ``` -Skip runtime setup if `pnpm dlx @opendatalabs/connect@canary status --json` reports `"runtime":"installed"` or `"runtime":{"installed":true,...}`. +Skip runtime setup if `npx -y @opendatalabs/connect@canary status --json` reports `"runtime":"installed"` or `"runtime":{"installed":true,...}`. ## Prerequisites @@ -16,7 +16,7 @@ Skip runtime setup if `pnpm dlx @opendatalabs/connect@canary status --json` repo ## Verify the published CLI ```bash -pnpm dlx @opendatalabs/connect@canary --help +npx -y @opendatalabs/connect@canary --help ``` ## Local development fallback @@ -39,7 +39,7 @@ ls /home/tnunamak/code/vana-connect/dist/cli/bin.js Use the published CLI to provision the runtime: ```bash -pnpm dlx @opendatalabs/connect@canary setup --yes +npx -y @opendatalabs/connect@canary setup --yes ``` Before running, tell the user this downloads a browser engine and some dependencies into `~/.dataconnect/`. This is a one-time step. @@ -47,7 +47,7 @@ Before running, tell the user this downloads a browser engine and some dependenc ## Verify ```bash -pnpm dlx @opendatalabs/connect@canary status +npx -y @opendatalabs/connect@canary status ``` You should see `Runtime: installed`. If not, inspect the log path surfaced by the CLI and only fall back to the older script-level flow if the CLI setup path is blocked. @@ -64,7 +64,7 @@ bash skills/vana-connect/scripts/setup.sh | Path | Purpose | |------|---------| -| `pnpm dlx @opendatalabs/connect@canary` | Published canary CLI entrypoint | +| `npx -y @opendatalabs/connect@canary` | Published canary CLI entrypoint | | `/home/tnunamak/code/vana-connect/dist/cli/bin.js` | Local development fallback | | `~/.dataconnect/playwright-runner/` | Runner process | | `~/.dataconnect/run-connector.cjs` | Batch-mode runner wrapper | diff --git a/skills/vana-connect/SKILL.md b/skills/vana-connect/SKILL.md index 287668b..4bb5b4e 100644 --- a/skills/vana-connect/SKILL.md +++ b/skills/vana-connect/SKILL.md @@ -17,7 +17,7 @@ Connect personal data from web platforms using the `vana` CLI and local browser Prefer the published canary CLI: ```bash -pnpm dlx @opendatalabs/connect@canary +npx -y @opendatalabs/connect@canary ``` If the canary CLI is unavailable or the user is explicitly testing local changes, fall back to: @@ -31,13 +31,13 @@ If neither path is available, follow `SETUP.md` in this folder. Before connecting a source, check runtime state with: ```bash -pnpm dlx @opendatalabs/connect@canary status --json +npx -y @opendatalabs/connect@canary status --json ``` If the runtime is missing, tell the user: "I need to do a one-time setup first. This downloads a browser engine and some dependencies into `~/.dataconnect/` and usually takes about a minute." Then run: ```bash -pnpm dlx @opendatalabs/connect@canary setup --yes +npx -y @opendatalabs/connect@canary setup --yes ``` ## Flow @@ -45,7 +45,7 @@ pnpm dlx @opendatalabs/connect@canary setup --yes ### 1. Explore available sources ```bash -pnpm dlx @opendatalabs/connect@canary sources --json +npx -y @opendatalabs/connect@canary sources --json ``` This is the source of truth for what the CLI can currently connect. Prefer it over inspecting repo files manually. @@ -59,7 +59,7 @@ If the requested platform is present, use the CLI flow below. Start with the agent-safe probe: ```bash -pnpm dlx @opendatalabs/connect@canary connect --json --no-input +npx -y @opendatalabs/connect@canary connect --json --no-input ``` This will: @@ -72,13 +72,13 @@ This will: If the outcome is `needs_input`, rerun interactively: ```bash -pnpm dlx @opendatalabs/connect@canary connect +npx -y @opendatalabs/connect@canary connect ``` If the user specifically wants to inspect current state before rerunning, use: ```bash -pnpm dlx @opendatalabs/connect@canary status +npx -y @opendatalabs/connect@canary status ``` ### 3. Handle outcomes From da0e826b8afd58e0d840ed4bf0fdbf0fe55683af Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Fri, 13 Mar 2026 12:10:10 -0500 Subject: [PATCH 5/8] docs(skill): prefer installed vana cli --- skills/vana-connect/SETUP.md | 43 ++++++++++++++++++++--- skills/vana-connect/SKILL.md | 68 ++++++++++++++++++++++++++++++++++-- 2 files changed, 105 insertions(+), 6 deletions(-) diff --git a/skills/vana-connect/SETUP.md b/skills/vana-connect/SETUP.md index eea90bf..3410d99 100644 --- a/skills/vana-connect/SETUP.md +++ b/skills/vana-connect/SETUP.md @@ -1,6 +1,26 @@ # Connect -- Setup -This setup exists to let the skill use the published canary CLI: +This setup exists to let the skill use a real installed `vana` CLI when available, with canary npm as the fallback. + +## Preferred path + +If `vana` is already on `PATH`, use it directly: + +```bash +command -v vana +``` + +Then use: + +```bash +vana +``` + +Skip runtime setup if `vana status --json` reports `"runtime":"installed"` or `"runtime":{"installed":true,...}`. + +## Fallback path + +If `vana` is not installed yet, use the published canary package: ```bash npx -y @opendatalabs/connect@canary @@ -19,6 +39,12 @@ Skip runtime setup if `npx -y @opendatalabs/connect@canary status --json` report npx -y @opendatalabs/connect@canary --help ``` +## Verify an installed CLI + +```bash +vana --help +``` + ## Local development fallback From `/home/tnunamak/code/vana-connect`: @@ -36,9 +62,16 @@ ls /home/tnunamak/code/vana-connect/dist/cli/bin.js ## Install the runtime -Use the published CLI to provision the runtime: +Use the installed CLI when possible: ```bash +vana setup --yes +``` + +If `vana` is not installed, use the published canary fallback: + +```bash +npx -y @opendatalabs/connect@canary --help npx -y @opendatalabs/connect@canary setup --yes ``` @@ -47,10 +80,11 @@ Before running, tell the user this downloads a browser engine and some dependenc ## Verify ```bash -npx -y @opendatalabs/connect@canary status +vana status ``` -You should see `Runtime: installed`. If not, inspect the log path surfaced by the CLI and only fall back to the older script-level flow if the CLI setup path is blocked. +You should see `Runtime: installed`. If `vana` is unavailable, run `npx -y @opendatalabs/connect@canary status` instead. +If setup still fails, inspect the log path surfaced by the CLI and only fall back to the older script-level flow if the CLI setup path is blocked. ## Legacy fallback @@ -64,6 +98,7 @@ bash skills/vana-connect/scripts/setup.sh | Path | Purpose | |------|---------| +| `vana` | Preferred installed CLI entrypoint | | `npx -y @opendatalabs/connect@canary` | Published canary CLI entrypoint | | `/home/tnunamak/code/vana-connect/dist/cli/bin.js` | Local development fallback | | `~/.dataconnect/playwright-runner/` | Runner process | diff --git a/skills/vana-connect/SKILL.md b/skills/vana-connect/SKILL.md index 4bb5b4e..52f4198 100644 --- a/skills/vana-connect/SKILL.md +++ b/skills/vana-connect/SKILL.md @@ -14,7 +14,19 @@ Connect personal data from web platforms using the `vana` CLI and local browser ## Setup -Prefer the published canary CLI: +Prefer an installed `vana` binary when it is available: + +```bash +command -v vana +``` + +If that succeeds, use: + +```bash +vana +``` + +If `vana` is not on `PATH`, fall back to the published canary CLI: ```bash npx -y @opendatalabs/connect@canary @@ -28,7 +40,13 @@ node /home/tnunamak/code/vana-connect/dist/cli/bin.js If neither path is available, follow `SETUP.md` in this folder. -Before connecting a source, check runtime state with: +Before connecting a source, check runtime state with the highest-priority available CLI: + +```bash +vana status --json +``` + +If `vana` is unavailable, use: ```bash npx -y @opendatalabs/connect@canary status --json @@ -36,6 +54,12 @@ npx -y @opendatalabs/connect@canary status --json If the runtime is missing, tell the user: "I need to do a one-time setup first. This downloads a browser engine and some dependencies into `~/.dataconnect/` and usually takes about a minute." Then run: +```bash +vana setup --yes +``` + +If `vana` is unavailable, use: + ```bash npx -y @opendatalabs/connect@canary setup --yes ``` @@ -44,6 +68,14 @@ npx -y @opendatalabs/connect@canary setup --yes ### 1. Explore available sources +Use `vana sources --json` if `vana` is installed: + +```bash +vana sources --json +``` + +Otherwise use: + ```bash npx -y @opendatalabs/connect@canary sources --json ``` @@ -58,6 +90,14 @@ If the requested platform is present, use the CLI flow below. Start with the agent-safe probe: +Use `vana connect --json --no-input` if `vana` is installed: + +```bash +vana connect --json --no-input +``` + +Otherwise use: + ```bash npx -y @opendatalabs/connect@canary connect --json --no-input ``` @@ -71,12 +111,28 @@ This will: If the outcome is `needs_input`, rerun interactively: +Use `vana connect ` if `vana` is installed: + +```bash +vana connect +``` + +Otherwise use: + ```bash npx -y @opendatalabs/connect@canary connect ``` If the user specifically wants to inspect current state before rerunning, use: +Use `vana status` if `vana` is installed: + +```bash +vana status +``` + +Otherwise use: + ```bash npx -y @opendatalabs/connect@canary status ``` @@ -147,3 +203,11 @@ The user can't see what you're doing behind the scenes. Keep them informed at ke 4. **Check session first** -- try without credentials if a browser profile exists 5. **Read connectors before running them** 6. **Use the CLI as the primary interface** -- only drop to raw scripts when debugging or updating connector internals + +## CLI fallback order + +Use this order when choosing the CLI entrypoint: + +1. `vana` if it is already installed and on `PATH` +2. `npx -y @opendatalabs/connect@canary` +3. `node /home/tnunamak/code/vana-connect/dist/cli/bin.js` only for local development or debugging From a0ce93df0d93b170354d736a97bb4359365030e6 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Fri, 13 Mar 2026 17:39:42 -0500 Subject: [PATCH 6/8] docs(skill): align vana-connect skill with CLI runtime --- skills/vana-connect/CREATE.md | 4 ++-- skills/vana-connect/RECIPES.md | 2 +- skills/vana-connect/SETUP.md | 10 ++-------- 3 files changed, 5 insertions(+), 11 deletions(-) diff --git a/skills/vana-connect/CREATE.md b/skills/vana-connect/CREATE.md index e12dc8a..e61f959 100644 --- a/skills/vana-connect/CREATE.md +++ b/skills/vana-connect/CREATE.md @@ -7,7 +7,7 @@ Build a data connector for a platform that isn't in the registry yet. - `reference/PAGE-API.md` -- full `page` object API - `reference/PATTERNS.md` -- data extraction approaches and code examples -All `node scripts/...` commands refer to `skills/vana-connect/scripts/` in the data-connectors repo. `run-connector.cjs` is at `~/.dataconnect/run-connector.cjs` (installed by SETUP.md). +All `node scripts/...` commands refer to `skills/vana-connect/scripts/` in the data-connectors repo. Use the `vana` CLI to exercise connectors; only fall back to raw scripts when debugging connector internals. ## Connector Format @@ -167,7 +167,7 @@ Run the connector and validate in one step: ```bash node scripts/validate.cjs /-playwright.js && \ - node ~/.dataconnect/run-connector.cjs /-playwright.js [start-url] && \ + vana connect && \ node scripts/validate.cjs /-playwright.js --check-result ~/.dataconnect/last-result.json ``` diff --git a/skills/vana-connect/RECIPES.md b/skills/vana-connect/RECIPES.md index 60a0b4d..41c5769 100644 --- a/skills/vana-connect/RECIPES.md +++ b/skills/vana-connect/RECIPES.md @@ -130,7 +130,7 @@ console.log([header, ...rows].join('\n')); Run the connector on a schedule (cron, agent heartbeat, etc.) and timestamp each export: ```bash -node run-connector.cjs +vana connect cp ~/.dataconnect/last-result.json ~/backups/-$(date +%Y-%m-%d).json ``` diff --git a/skills/vana-connect/SETUP.md b/skills/vana-connect/SETUP.md index 3410d99..0da07b4 100644 --- a/skills/vana-connect/SETUP.md +++ b/skills/vana-connect/SETUP.md @@ -1,6 +1,6 @@ # Connect -- Setup -This setup exists to let the skill use a real installed `vana` CLI when available, with canary npm as the fallback. +This setup exists to let the skill use a real installed `vana` CLI when available, with the published canary CLI as the fallback. ## Preferred path @@ -28,11 +28,6 @@ npx -y @opendatalabs/connect@canary Skip runtime setup if `npx -y @opendatalabs/connect@canary status --json` reports `"runtime":"installed"` or `"runtime":{"installed":true,...}`. -## Prerequisites - -- Node.js v18+ -- Git - ## Verify the published CLI ```bash @@ -101,8 +96,7 @@ bash skills/vana-connect/scripts/setup.sh | `vana` | Preferred installed CLI entrypoint | | `npx -y @opendatalabs/connect@canary` | Published canary CLI entrypoint | | `/home/tnunamak/code/vana-connect/dist/cli/bin.js` | Local development fallback | -| `~/.dataconnect/playwright-runner/` | Runner process | -| `~/.dataconnect/run-connector.cjs` | Batch-mode runner wrapper | | `~/.dataconnect/connectors/` | Connector scripts | | `~/.dataconnect/browser-profiles/` | Persistent sessions (cookies) | | `~/.dataconnect/last-result.json` | Most recent result | +| `~/.dataconnect/logs/` | Setup and run logs surfaced by the CLI | From cc62f556c88058c453d43f9235f887dda62290d6 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Fri, 13 Mar 2026 18:05:40 -0500 Subject: [PATCH 7/8] docs(skill): prefer installed vana cli --- skills/vana-connect/SETUP.md | 14 +++++++++++++- skills/vana-connect/SKILL.md | 14 ++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/skills/vana-connect/SETUP.md b/skills/vana-connect/SETUP.md index 0da07b4..4303110 100644 --- a/skills/vana-connect/SETUP.md +++ b/skills/vana-connect/SETUP.md @@ -20,7 +20,19 @@ Skip runtime setup if `vana status --json` reports `"runtime":"installed"` or `" ## Fallback path -If `vana` is not installed yet, use the published canary package: +If `vana` is not installed yet, prefer the official installer so the user gets a real installed CLI: + +```bash +curl -fsSL https://raw.githubusercontent.com/vana-com/vana-connect/main/install/install.sh | sh +``` + +Then verify: + +```bash +vana --help +``` + +If the installer path is unavailable or the released CLI does not yet contain the needed behavior, use the published canary package: ```bash npx -y @opendatalabs/connect@canary diff --git a/skills/vana-connect/SKILL.md b/skills/vana-connect/SKILL.md index 52f4198..c9a2395 100644 --- a/skills/vana-connect/SKILL.md +++ b/skills/vana-connect/SKILL.md @@ -28,6 +28,20 @@ vana If `vana` is not on `PATH`, fall back to the published canary CLI: +If the user wants the real installed CLI and not a transient package runner, install `vana` first: + +```bash +curl -fsSL https://raw.githubusercontent.com/vana-com/vana-connect/main/install/install.sh | sh +``` + +Then use: + +```bash +vana +``` + +If the installer path is unavailable, the release channel is too old for the needed CLI behavior, or the user is explicitly testing prerelease changes, fall back to the published canary CLI: + ```bash npx -y @opendatalabs/connect@canary ``` From 97d89539a7e1ce4b343ab09d2b393b33b1d0c95f Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Fri, 13 Mar 2026 18:06:04 -0500 Subject: [PATCH 8/8] docs(skill): clarify installer fallback order --- skills/vana-connect/SKILL.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/skills/vana-connect/SKILL.md b/skills/vana-connect/SKILL.md index c9a2395..fa94239 100644 --- a/skills/vana-connect/SKILL.md +++ b/skills/vana-connect/SKILL.md @@ -26,9 +26,7 @@ If that succeeds, use: vana ``` -If `vana` is not on `PATH`, fall back to the published canary CLI: - -If the user wants the real installed CLI and not a transient package runner, install `vana` first: +If `vana` is not on `PATH`, prefer installing the real CLI before falling back to a transient package runner: ```bash curl -fsSL https://raw.githubusercontent.com/vana-com/vana-connect/main/install/install.sh | sh