diff --git a/.gen.json b/.gen.json index 358d05fa..65a8a375 100644 --- a/.gen.json +++ b/.gen.json @@ -1,3 +1,3 @@ { - "source_commit": "fa7c388d15531622e47f91f4d6341274b5cdc694" + "source_commit": "007006859ac9df636ebfa08d5d25a1d5b59f15e6" } diff --git a/experimental/README.md b/experimental/README.md index 1dc47c2d..1d8d51b1 100644 --- a/experimental/README.md +++ b/experimental/README.md @@ -41,9 +41,6 @@ See the root [README](../README.md) for details on the stable install path. ## Available Skills -### ๐Ÿ“Š Analytics & Dashboards -- **databricks-genie-agents** - Genie Agents: natural-language data Q&A, agent authoring, and tuning (example questions, instructions) - ### ๐Ÿ”ง Data Engineering - **spark-python-data-source** - Python data sources for Spark (custom connectors) @@ -54,7 +51,8 @@ Most of the original ai-dev-kit snapshot now lives in the stable flag). See the root [README](../README.md#available-skills) for the full list. Promoted skills include `databricks-ai-functions`, `databricks-agent-bricks`, `databricks-aibi-dashboards`, `databricks-apps-python`, `databricks-dbsql`, -`databricks-docs`, `databricks-execution-compute`, `databricks-iceberg`, +`databricks-docs`, `databricks-execution-compute`, `databricks-genie-agents`, +`databricks-iceberg`, `databricks-lakeflow-connect`, `databricks-metric-views`, `databricks-ml-training`, `databricks-mlflow-evaluation`, `databricks-python-sdk`, `databricks-spark-structured-streaming`, `databricks-synthetic-data-gen`, diff --git a/experimental/databricks-genie-agents/assets/databricks.png b/experimental/databricks-genie-agents/assets/databricks.png deleted file mode 100644 index 263fe98b..00000000 Binary files a/experimental/databricks-genie-agents/assets/databricks.png and /dev/null differ diff --git a/metaplugin/plugin.meta.json b/metaplugin/plugin.meta.json index dc6b53fc..242dc8e1 100644 --- a/metaplugin/plugin.meta.json +++ b/metaplugin/plugin.meta.json @@ -101,6 +101,7 @@ "databricks-dbsql": { "keyword": "dbsql" }, "databricks-docs": { "keyword": "docs" }, "databricks-execution-compute": { "keyword": "execution-compute" }, + "databricks-genie-agents": { "keyword": "genie-agents" }, "databricks-iceberg": { "keyword": "iceberg" }, "databricks-lakeflow-connect": { "keyword": "lakeflow-connect" }, "databricks-metric-views": { "keyword": "metric-views" }, @@ -177,6 +178,7 @@ { "label": "Vector Search / RAG", "skill": "databricks-vector-search" }, { "label": "Classic-to-serverless migration", "skill": "databricks-serverless-migration" }, { "label": "Data discovery / finding & exploring data / natural-language data Q&A / SQL generation (Genie)", "skill": "databricks-data-discovery" }, + { "label": "Genie Agents (formerly Genie Spaces): building, managing & tuning a curated per-data Genie Agent", "skill": "databricks-genie-agents" }, { "label": "Agent Bricks / Knowledge Assistants / Genie Spaces / Multi-Agent Supervisor", "skill": "databricks-agent-bricks" }, { "label": "AI Functions (ai_query, ai_classify, ai_extract, ai_parse_document)", "skill": "databricks-ai-functions" }, { "label": "AI/BI dashboards", "skill": "databricks-aibi-dashboards" }, diff --git a/skills/databricks-ai-functions/SKILL.md b/skills/databricks-ai-functions/SKILL.md index b83fa923..a44fb586 100644 --- a/skills/databricks-ai-functions/SKILL.md +++ b/skills/databricks-ai-functions/SKILL.md @@ -153,7 +153,7 @@ FROM prepped LATERAL VIEW explode(variant_get(prep, '$.document.contents', 'ARRA Then enable CDF (`ALTER TABLE parsed_chunks SET TBLPROPERTIES (delta.enableChangeDataFeed = true)`) and use the **[databricks-vector-search](../databricks-vector-search/SKILL.md)** skill to build a Delta Sync index: PK `chunk_id`, embedding source `chunk_to_embed`, return `chunk_to_retrieve`. **Beyond batch:** -- **Ask questions over the output** โ€” point a Genie Agent at the resulting Delta table for natural-language querying instead of hand-written SQL; see the **[databricks-genie-agents](../../experimental/databricks-genie-agents/SKILL.md)** skill. +- **Ask questions over the output** โ€” point a Genie Agent at the resulting Delta table for natural-language querying instead of hand-written SQL; see the **[databricks-genie-agents](../databricks-genie-agents/SKILL.md)** skill. - **Low-latency / serving** โ€” to expose this as a real-time, governed endpoint (e.g. register a model to Unity Catalog and serve it), use the **[databricks-model-serving](../databricks-model-serving/SKILL.md)** skill. - **Production incremental ingestion** โ€” for a runnable end-to-end streaming `ai_parse_document` job (checkpoints, `trigger(availableNow=True)`), see [databricks/bundle-examples ยท job_with_ai_parse_document](https://github.com/databricks/bundle-examples/tree/main/contrib/job_with_ai_parse_document). diff --git a/skills/databricks-data-discovery/SKILL.md b/skills/databricks-data-discovery/SKILL.md index 384fd4de..784fa6ab 100644 --- a/skills/databricks-data-discovery/SKILL.md +++ b/skills/databricks-data-discovery/SKILL.md @@ -1,7 +1,7 @@ --- name: databricks-data-discovery description: "Discover, explore, and query Databricks data via Genie โ€” the CLI equivalent of the Genie One MCP. MUST be invoked whenever the user asks to find or locate data ('what tables are in X', 'where does X live', 'which catalog/schema has Y'), answer a natural-language question about the data, or write a SQL query." -compatibility: Requires databricks CLI with the experimental genie command (databricks experimental genie ask) +compatibility: Requires databricks CLI >= v1.9.0 for `databricks genie ask` (older CLIs fall back to the deprecated `databricks experimental genie ask` alias) metadata: version: "0.1.0" parent: databricks-core @@ -12,7 +12,7 @@ parent: databricks-core This skill **routes data work** โ€” decide first: - the request is about *the data* โ€” finding it, asking questions of it, or generating SQL โ†’ delegate to **Genie One**: - `databricks experimental genie ask -s "..."` (see Routing below). + `databricks genie ask -s "..."` (see Routing below). - writing files or anything else โ†’ use your own coding-agent tools. Genie One just needs an authenticated CLI profile (the parent `databricks-core` @@ -58,18 +58,25 @@ and reusing it lets later questions build on everything asked so far ("summarize of the above"). Use a fresh session label only to start a deliberately separate session, or distinct session labels to run several in parallel. +The command is `databricks genie ask` (CLI >= v1.9.0). On an older CLI it lives +under `databricks experimental genie ask` โ€” same flags and behavior; use that +exact fallback if `databricks genie ask` is not found. + ```bash # Always pass a session label, and reuse the SAME one so follow-ups build on each other -databricks experimental genie ask -s trips "How many bookings were there last week?" -databricks experimental genie ask -s trips "Break that down by destination" -databricks experimental genie ask -s trips "Summarize all of the above" +databricks genie ask -s trips "How many bookings were there last week?" +databricks genie ask -s trips "Break that down by destination" +databricks genie ask -s trips "Summarize all of the above" # --include-sql also prints the SQL Genie ran (use it to generate a query, too) -databricks experimental genie ask -s trips "Write SQL for the top 5 destinations by revenue" --include-sql +databricks genie ask -s trips "Write SQL for the top 5 destinations by revenue" --include-sql # --output json gives a parseable result -databricks experimental genie ask -s trips "Top 5 destinations by revenue" --output json +databricks genie ask -s trips "Top 5 destinations by revenue" --output json # โ†’ {"status":"completed","conversation_id":"โ€ฆ","text":"โ€ฆ","tool_calls":[{"name":"execute_sql","sql":"โ€ฆ","title":"โ€ฆ"}]} + +# Older CLI (< v1.9.0) โ€” same command under the deprecated experimental alias: +# databricks experimental genie ask -s trips "How many bookings were there last week?" ``` Genie searches across all the data you can see, runs SQL, and streams a grounded @@ -110,8 +117,9 @@ set up. ## If Genie One isn't available โ€” manual fallback Only fall back if Genie One is genuinely unavailable โ€” first **verify** with -`databricks experimental genie ask --help`; don't assume the command is missing. -When Genie One isn't enabled, the CLI is too old to have `experimental genie ask`, +`databricks genie ask --help` (or `databricks experimental genie ask --help` on a +CLI older than v1.9.0); don't assume the command is missing. When Genie One isn't +enabled, the CLI is too old to have either form of `genie ask`, or Genie can't cover the question, do the discovery yourself with the parent skill's commands โ€” see **[Manual Data Exploration](../databricks-core/manual-data-exploration.md)** (keyword search via `information_schema`, `discover-schema`, and `tools query`). @@ -130,7 +138,7 @@ the same Genie backend. ## Related Skills -- **databricks-genie-agents** (experimental) โ€” build and manage **Genie Agents**: curated +- **databricks-genie-agents** โ€” build and manage **Genie Agents**: curated agents that let you or a group ask questions of specific data (create, configure, import/export). - **databricks-core** (parent) โ€” CLI auth, profiles, and the manual data exploration diff --git a/experimental/databricks-genie-agents/SKILL.md b/skills/databricks-genie-agents/SKILL.md similarity index 90% rename from experimental/databricks-genie-agents/SKILL.md rename to skills/databricks-genie-agents/SKILL.md index c03a82ae..68510c7b 100644 --- a/experimental/databricks-genie-agents/SKILL.md +++ b/skills/databricks-genie-agents/SKILL.md @@ -3,7 +3,7 @@ name: databricks-genie-agents description: "Create, manage, and query Databricks Genie Agents โ€” curated, per-data natural-language agents (formerly Genie Spaces): build, export/import, migrate across workspaces, and ask questions of a *specific* Agent via the Conversation API. For general data questions or finding data across your workspace, use databricks-data-discovery (Genie One) instead." compatibility: Requires databricks CLI (>= v1.0.0) metadata: - version: "0.0.1" + version: "0.1.0" --- # Databricks Genie Agents @@ -101,6 +101,10 @@ databricks genie create-space --json "{ ### Improving a Genie Agent +**Recommendation-first:** when asked to optimize, tune, or fix an Agent (or its queries/tables), start by diagnosing and presenting a recommended change โ€” do not run mutating actions (`update-space`, `ALTER`, `OPTIMIZE`, liquid clustering, warehouse changes) until the user approves. Diagnose with read-only queries only. + +**Wrong filter values** (Genie filters on a value that returns nothing โ€” e.g. asking for `cancelled` when the column stores a different code or casing): fix with prompt matching / synonyms mapping the user's term to the actual categorical value, not a hardcoded text instruction. + When Genie answers are inaccurate or incomplete, improve the agent by updating questions, SQL examples, or instructions: ```bash @@ -129,6 +133,7 @@ The `serialized_space` field is a JSON string containing the full space configur - **Sort order matters:** `data_sources.tables` must be sorted by `identifier`, and each table's `column_configs` must be sorted by `column_name`; `example_question_sqls` and `text_instructions` must be sorted by `id`. (`sample_questions` is silently re-sorted server-side.) - **`text_instructions` accepts at most one item** โ€” the API rejects more than one (`text_instructions must contain at most one item`). Merge all guidance (persona, table guide, investigation flow, answer style) into a single entry. - **Simple ID scheme that satisfies both rules:** prefix per list + monotonic counter, total 32 hex chars โ€” `1โ€ฆ0001`, `1โ€ฆ0002` for `sample_questions`; `2โ€ฆ0001`, `2โ€ฆ0002` for `example_question_sqls`; `3โ€ฆ0001` for `text_instructions`. Authoring order = sort order, no collisions. +- **`benchmarks` is a top-level key** of `serialized_space` (alongside `version`/`config`/`data_sources`/`instructions`), not nested under `instructions`. Each item takes a unique 32-char hex `id`. ### Text Instructions @@ -195,7 +200,7 @@ Use `DATABRICKS_CONFIG_PROFILE=profile_name` to target different workspaces. > **Scope:** use this to query **one specific Genie Agent** โ€” typically to validate an Agent > after creating or editing it, or to lean on its curated business logic and certified queries. > For general natural-language data questions or finding data across your workspace, don't use -> this โ€” route to the **[databricks-data-discovery](../../skills/databricks-data-discovery/SKILL.md)** +> this โ€” route to the **[databricks-data-discovery](../databricks-data-discovery/SKILL.md)** > skill (Genie One) instead. Ask questions of a specific Agent via three CLI primitives: `start-conversation`, `create-message` (follow-ups), and `get-message` (state + SQL + text). `--no-wait` on `start-conversation` / `create-message` returns immediately with `{conversation_id, message_id}`; poll `get-message` until `.status` is `COMPLETED`, `FAILED`, or `CANCELLED`. Intermediate states you'll see: `SUBMITTED`, `FILTERING_CONTEXT`, `ASKING_AI`, `EXECUTING_QUERY`. @@ -239,6 +244,6 @@ On `FAILED`, `get-message` populates `.error.error` with the underlying error st ## Related Skills -- **[databricks-data-discovery](../../skills/databricks-data-discovery/SKILL.md)** - General natural-language data exploration / "ask Genie" (Genie One) across your data; use it when you are not targeting a specific curated Genie Agent -- **[databricks-synthetic-data-gen](../../skills/databricks-synthetic-data-gen/SKILL.md)** - Generate data for Genie tables -- **[databricks-pipelines](../../skills/databricks-pipelines/SKILL.md)** - Build bronze/silver/gold tables +- **[databricks-data-discovery](../databricks-data-discovery/SKILL.md)** - General natural-language data exploration / "ask Genie" (Genie One) across your data; use it when you are not targeting a specific curated Genie Agent +- **[databricks-synthetic-data-gen](../databricks-synthetic-data-gen/SKILL.md)** - Generate data for Genie tables +- **[databricks-pipelines](../databricks-pipelines/SKILL.md)** - Build bronze/silver/gold tables diff --git a/experimental/databricks-genie-agents/agents/openai.yaml b/skills/databricks-genie-agents/agents/openai.yaml similarity index 100% rename from experimental/databricks-genie-agents/agents/openai.yaml rename to skills/databricks-genie-agents/agents/openai.yaml diff --git a/skills/databricks-genie-agents/assets/databricks.png b/skills/databricks-genie-agents/assets/databricks.png new file mode 100644 index 00000000..cf151479 Binary files /dev/null and b/skills/databricks-genie-agents/assets/databricks.png differ diff --git a/experimental/databricks-genie-agents/assets/databricks.svg b/skills/databricks-genie-agents/assets/databricks.svg similarity index 100% rename from experimental/databricks-genie-agents/assets/databricks.svg rename to skills/databricks-genie-agents/assets/databricks.svg