diff --git a/site/sfguides/src/a_guide_to_implementing_a_pipeline_for_dcm/a_guide_to_implementing_a_pipeline_for_dcm.md b/site/sfguides/src/a_guide_to_implementing_a_pipeline_for_dcm/a_guide_to_implementing_a_pipeline_for_dcm.md index b0f9b74eea..b10e9629d5 100644 --- a/site/sfguides/src/a_guide_to_implementing_a_pipeline_for_dcm/a_guide_to_implementing_a_pipeline_for_dcm.md +++ b/site/sfguides/src/a_guide_to_implementing_a_pipeline_for_dcm/a_guide_to_implementing_a_pipeline_for_dcm.md @@ -17,7 +17,6 @@ In this guide we will walkthrough how to use Snowflake Data Change Management fe ### Prerequisites - Familiarity with Snowflake, Github - A snowflake user with [Key Pair](https://docs.snowflake.com/en/user-guide/key-pair-auth), role sysadmin. -- A XSMALL snowflake warehouse. - Setup Snowflake [cli connections](https://docs.snowflake.com/en/developer-guide/snowflake-cli/connecting/configure-connections) ### What You’ll Learn @@ -62,7 +61,7 @@ In this guide we will walkthrough how to use Snowflake Data Change Management fe ```shell openssl genrsa -out snowflake_demo_key 4096 -openssl rsa -in snowflake_demo_key -pubout -out snowflake_demo_key.pub +openssl rsa -in snowflake_demo_key.pem -pubout -out snowflake_demo_key.pub openssl pkcs8 -topk8 -nocrypt -in snowflake_demo_key.pem -out snowflake_demo_key.p8 ``` + You could also use a [online service](https://www.cryptool.org/en/cto/openssl/) to create the key pair. @@ -77,19 +76,33 @@ openssl pkcs8 -topk8 -nocrypt -in snowflake_demo_key.pem -out snowflake_demo_key ```sql USE ROLE ACCOUNTADMIN; -CREATE OR REPLACE USER "DCM_DEMO" RSA_PUBLIC_KEY='RSA_PUBLIC_KEY_HERE' DEFAULT_ROLE=SYSADMIN MUST_CHANGE_PASSWORD=FALSE; +CREATE OR REPLACE USER "DCM_DEMO" RSA_PUBLIC_KEY='RSA_PUBLIC_KEY_HERE' MUST_CHANGE_PASSWORD=FALSE; +GRANT ROLE SYSADMIN TO USER DCM_DEMO; ``` -3) Use role SYSADMIN. +3) Create warehouse and grant -4) Create database and schema to hold the objects that will be managed via DCM. +```sql +CREATE WAREHOUSE XSMALL + WITH WAREHOUSE_SIZE = XSMALL + AUTO_SUSPEND = 30 + AUTO_RESUME = TRUE + COMMENT = 'This is a warehouse for dcm quickstart.'; + + +GRANT USAGE ON WAREHOUSE XSMALL TO ROLE SYSADMIN; +``` + +4) Use role SYSADMIN. + +5) Create database and schema to hold the objects that will be managed via DCM. ```sql create database DCM_PROJECTS; create schema DCM_PROJECTS.DCM_PROJECTS; ``` -5) Create Database, schema, tables for bronze layer. We will be using a subset of the tables/data. +6) Create Database, schema, tables for bronze layer. We will be using a subset of the tables/data. ```sql create database bronze; @@ -146,7 +159,7 @@ CREATE TABLE order_details ( ); ``` -6) Load the data +7) Load the data ```sql INSERT INTO customers VALUES @@ -349,11 +362,11 @@ jobs: --no-interactive --default - name: Create DCM Project if not exists run: | - cd ./data_project && snow dcm create --if-not-exists proj1 + cd ./dcm_project && snow dcm create --if-not-exists proj1 - name: Execute PLAN with config PROD run: | - cd ./data_project && snow dcm plan DCM.DCM.PROJ1 --configuration="prod" + cd ./dcm_project && snow dcm plan DCM.DCM.PROJ1 --configuration="prod" ``` @@ -402,13 +415,13 @@ jobs: --no-interactive --default - name: Create DCM Project if not exists run: | - cd ./data_project && snow dcm create --if-not-exists proj1 + cd ./dcm_project && snow dcm create --if-not-exists proj1 - name: Execute PLAN with config PROD run: | - cd ./data_project && snow dcm plan DCM.DCM.PROJ1 --configuration="prod" + cd ./dcm_project && snow dcm plan DCM.DCM.PROJ1 --configuration="prod" - name: Execute deploy with config prod run: | - cd ./data_project && snow dcm deploy DCM.DCM.PROJ1 --configuration="prod" + cd ./dcm_project && snow dcm deploy DCM.DCM.PROJ1 --configuration="prod" ``` 5) Add sandbox script and requirements.txt @@ -486,7 +499,7 @@ SANITIZED_NAME=$(echo ${BRANCH_NAME//-/_} | tr '[:lower:]' '[:upper:]') pushd $REPO_ROOT snow sql -q "CREATE OR REPLACE STAGE ${SANITIZED_NAME}_FILES" -snow stage copy --recursive ./data_project @${SANITIZED_NAME}_FILES/ +snow stage copy --recursive ./dcm_project @${SANITIZED_NAME}_FILES/ snow sql -q "EXECUTE DCM PROJECT ${SANITIZED_NAME} PLAN USING CONFIGURATION non_prod (target_db => '${SANITIZED_NAME}', target_schema => '${SANITIZED_NAME}') FROM @${SANITIZED_NAME}_FILES/;" popd @@ -502,7 +515,7 @@ SANITIZED_NAME=$(echo ${BRANCH_NAME//-/_} | tr '[:lower:]' '[:upper:]') pushd $REPO_ROOT snow sql -q "CREATE OR REPLACE STAGE ${SANITIZED_NAME}_FILES" -snow stage copy --recursive ./data_project @${SANITIZED_NAME}_FILES/ +snow stage copy --recursive ./dcm_project @${SANITIZED_NAME}_FILES/ snow sql -q "EXECUTE DCM PROJECT ${SANITIZED_NAME} DEPLOY USING CONFIGURATION non_prod (target_db => '${SANITIZED_NAME}', target_schema => '${SANITIZED_NAME}') FROM @${SANITIZED_NAME}_FILES/;" popd @@ -579,12 +592,12 @@ urllib3==2.5.0 wcwidth==0.2.13 ``` -6) Create folder called data_project with manifest.yml +6) Create folder called dcm_project with manifest.yml ```shell cd -mkdir -p data_project/definitions -cd data_project +mkdir -p dcm_project/definitions +cd dcm_project ``` + In your code editor create file manifest.yml. Copy following content into manifest.yml @@ -616,11 +629,11 @@ git push origin main ## Setup DCM to manage the gold layer 1) Create gold database and schema - + Add a sql file into the data_project/definitions folder called gold.sql. + + Add a sql file into the dcm_project/definitions folder called gold.sql. ```shell git checkout -b setup_gold -cd data_project/definitions +cd dcm_project/definitions ``` + In your code editor create file gold.sql. Contents of gold.sql: @@ -659,8 +672,8 @@ git pull cd sandbox ./do.sh create PROJ-001 cd .. -mkdir -p data_project/definitions -cd data_project/definitions +mkdir -p dcm_project/definitions +cd dcm_project/definitions ``` + do.sh creates a git branch named 'PROJ-001' and all changes are made in the context of this branch. It also creates a sandbox environment by cloning the gold layer. @@ -723,8 +736,8 @@ git pull cd sandbox ./do.sh create PROJ-002 cd .. -mkdir -p data_project/definitions -cd data_project/definitions +mkdir -p dcm_project/definitions +cd dcm_project/definitions ``` + In your code editor create file order_fact.sql. Copy following content into order_fact.sql @@ -809,6 +822,7 @@ In this guide we targetted a subset of tables in the northwind database. + You can have the main branch map to the prod snowflake account, and an integration branch can be mapped to the non-prod snowflake account. + The sandbox creation script can be hosted in a CI system like Jenkins, etc + + You can enhance the worklfow by adding Expectations attached to tables in project and using the TEST ALL command to 5) All scripts/shell commands in this quickstart are tested to work on macOS. diff --git a/site/sfguides/src/cortex_ai_demo_framework/assets/architecture_diagram.png b/site/sfguides/src/cortex_ai_demo_framework/assets/architecture_diagram.png new file mode 100644 index 0000000000..2699f62583 Binary files /dev/null and b/site/sfguides/src/cortex_ai_demo_framework/assets/architecture_diagram.png differ diff --git a/site/sfguides/src/cortex_ai_demo_framework/cortex_ai_demo_framework.md b/site/sfguides/src/cortex_ai_demo_framework/cortex_ai_demo_framework.md index 28f1cc34e2..f32f3884ee 100644 --- a/site/sfguides/src/cortex_ai_demo_framework/cortex_ai_demo_framework.md +++ b/site/sfguides/src/cortex_ai_demo_framework/cortex_ai_demo_framework.md @@ -25,6 +25,8 @@ This Quickstart showcases the complete Cortex AI Demo Framework with: - **Production-ready applications** with professional UI/UX +![Architecture Diagram](assets/architecture_diagram.png) + ### What You Will Build - Complete 6-application integrated demo platform - AI-powered synthetic data generation system using Cortex functions @@ -1259,8 +1261,6 @@ Your demo is complete! You can: - Edit YAML to add more steps or visualizations - Share demo with colleagues by sharing the YAML file -**Return to Page 5** to explore other workflows or **continue to Page 12** for cleanup instructions. - ## YAML Wizard @@ -1806,18 +1806,6 @@ Shows complete dataset in table format with sortable columns and CSV export opti - Compare entities side-by-side - Export data for presentations ---- - -### Best Practices - -**Explore systematically**: Start with Overview, then drill into specific tabs -**Use AI Assistant**: Natural language queries are powerful and intuitive -**Compare entities**: VS tab helps identify top performers -**Export insights**: Share findings via CSV export -**Adjust time windows**: Find the right time range for your analysis - ---- - ### What's Next? **For Persona 1 (Full-Stack Developer)**: @@ -1838,8 +1826,6 @@ You now have an interactive analytics dashboard! You can: - Compare products/customers/categories - Export data for presentations -**Return to Page 5** to explore other workflows or **continue to Page 12** for cleanup instructions. - ## Clean Up Resources diff --git a/site/sfguides/src/dbt-projects-on-snowflake/assets/create-schedule.png b/site/sfguides/src/dbt-projects-on-snowflake/assets/create-schedule.png new file mode 100644 index 0000000000..a95fbf6b82 Binary files /dev/null and b/site/sfguides/src/dbt-projects-on-snowflake/assets/create-schedule.png differ diff --git a/site/sfguides/src/dbt-projects-on-snowflake/assets/dbt-create-schedule.png b/site/sfguides/src/dbt-projects-on-snowflake/assets/dbt-create-schedule.png new file mode 100644 index 0000000000..37bb275937 Binary files /dev/null and b/site/sfguides/src/dbt-projects-on-snowflake/assets/dbt-create-schedule.png differ diff --git a/site/sfguides/src/dbt-projects-on-snowflake/assets/dbt_project_details.png b/site/sfguides/src/dbt-projects-on-snowflake/assets/dbt_project_details.png new file mode 100644 index 0000000000..99ee725f89 Binary files /dev/null and b/site/sfguides/src/dbt-projects-on-snowflake/assets/dbt_project_details.png differ diff --git a/site/sfguides/src/dbt-projects-on-snowflake/dbt-projects-on-snowflake.md b/site/sfguides/src/dbt-projects-on-snowflake/dbt-projects-on-snowflake.md index 9994b46fc9..0e7032cd0b 100644 --- a/site/sfguides/src/dbt-projects-on-snowflake/dbt-projects-on-snowflake.md +++ b/site/sfguides/src/dbt-projects-on-snowflake/dbt-projects-on-snowflake.md @@ -35,14 +35,6 @@ In this lab, we will go through everything you need to know to get started with We will be using Tasty Bytes data in this lab. Run the script [here](https://github.com/Snowflake-Labs/getting-started-with-dbt-on-snowflake/blob/main/tasty_bytes_dbt_demo/setup/tasty_bytes_setup.sql) in Snowsight to build the objects and data required for this lab. -Workspaces that you create in Snowflake are created in the personal database associated with the active user. To use Workspaces, you must run the following SQL commands to activate all secondary roles for your user. - -``` sql -ALTER USER my_user SET DEFAULT_SECONDARY_ROLES = ('ALL'); -``` - -Sign out of Snowsight and sign back in. - ## Introduction to Workspaces @@ -96,7 +88,7 @@ Let's now clone an example dbt project we will use in the rest of this lab. 1. Repository URL: `https://github.com/Snowflake-Labs/getting-started-with-dbt-on-snowflake.git` 2. Workspace Name: `Example-dbt-Project` 3. API Integration: `GIT_INTEGRATION`. Note: the [API Integration](https://docs.snowflake.com/en/developer-guide/git/git-setting-up#label-integrating-git-repository-api-integration) has already been configured for you. - 4. Select Public Repository + 4. Select Public Repository. Note: Private repos can be authenticated with personal access tokens and GitHub users can authenticate with [OAuth](https://docs.snowflake.com/en/developer-guide/git/git-setting-up#configure-for-authenticating-with-oauth). 3. Click Create! ![create-dbt-project](assets/create-workspace-git.png) @@ -168,7 +160,7 @@ Let's start by running `dbt deps` to pull in the dbt_utils package. The dbt_util From the dbt toolbar, you get dropdowns for the project, target, and command. Clicking the play button will run the relevant command. You can also click the down arrow to override the arguments. 1. From the toolbar, select dev and deps. -2. Click the dropdown arrow and enter `dbt_access_integration`. This [external access integration](https://docs.snowflake.com/en/sql-reference/sql/create-external-access-integration) has already been configured for you. +2. Click the dropdown arrow and select `dbt_access_integration`. This [external access integration](https://docs.snowflake.com/en/sql-reference/sql/create-external-access-integration) has already been configured for you. 3. Click the Deps button. ![dbt-deps](assets/dbt-deps.png) @@ -228,23 +220,23 @@ Workspaces are fully git backed. To view changes and commit, click changes from ## Orchestration and Monitoring -### Monitor dbt Projects +### Orchestrate with Tasks -You can get an overview of dbt project status from the dbt Projects activity in Snowsight. Navigate to monitoring > dbt Projects to view overall status of dbt Projects and quickly jump to the deployed projects. +Navigate to Catalog > Database Explorer > TASTY_BYTES_DBT_DB > RAW > dbt Projects > DBT_PROJECT to view the project details. From the Run History tab, you can view all runs associated with the project. -![dbt-projects](assets/dbt-projects.png) - -### Orchestrate with Tasks +![project-details](assets/dbt_project_details.png) #### Create Scheduled dbt Tasks Let's create tasks to regularly run and test our dbt project. -1. Click dbt_project in the top right corner of Workspaces -2. Click Create Schedule from the dropdown +1. Navigate to the Project Details tab +2. Click Create Schedule from the Schedules dropdown 3. Enter a name, schedule, and profile, then click create -![create-task](assets/create-task.png) +![create-schedule](assets/dbt-create-schedule.png) + +![create-task](assets/create-schedule.png) #### Complex Tasks and Alerts @@ -263,18 +255,7 @@ CREATE OR REPLACE TASK tasty_bytes_dbt_db.raw.dbt_run_task CREATE OR REPLACE TASK tasty_bytes_dbt_db.raw.dbt_test_task WAREHOUSE=TASTY_BYTES_DBT_WH AFTER tasty_bytes_dbt_db.raw.dbt_run_task - AS - DECLARE - dbt_success BOOLEAN; - dbt_exception STRING; - my_exception EXCEPTION (-20002, 'My exception text'); - BEGIN - EXECUTE DBT PROJECT "TASTY_BYTES_DBT_DB"."RAW"."DBT_PROJECT" args='test --target dev'; - SELECT SUCCESS, EXCEPTION into :dbt_success, :dbt_exception FROM TABLE(result_scan(last_query_id())); - IF (NOT :dbt_success) THEN - raise my_exception; - END IF; - END; + AS EXECUTE DBT PROJECT "TASTY_BYTES_DBT_DB"."RAW"."DBT_PROJECT" args='test --target dev'; -- Run the tasks once ALTER TASK tasty_bytes_dbt_db.raw.dbt_test_task RESUME; @@ -326,6 +307,12 @@ You can view the status or running tasks by going to Monitoring > Task History. ![tasks](assets/tasks.png) +### Monitor dbt Projects + +You can get an overview of dbt project status from the dbt Projects activity in Snowsight. Navigate to Monitoring > dbt Projects to view overall status of dbt Projects and quickly jump to the deployed projects. + +![dbt-projects](assets/dbt-projects.png) + ### Tracing dbt projects in Snowflake integrate with [Tracing and Logging](https://docs.snowflake.com/en/developer-guide/logging-tracing/logging-tracing-overview), allowing you to easily monitor and debug dbt projects. Tracing follows the OpenTelemetry standard and allows you to keep logs within a single platform. diff --git a/site/sfguides/src/getting_started_with_interactive_tables/README.md b/site/sfguides/src/getting_started_with_interactive_tables/README.md new file mode 100644 index 0000000000..90fe795861 --- /dev/null +++ b/site/sfguides/src/getting_started_with_interactive_tables/README.md @@ -0,0 +1,26 @@ +# Getting Started with Snowflake Interactive Tables + +## Overview + +When it comes to near real-time (or sub-second) analytics, the ideal scenario involves achieving consistent, rapid query performance and managing costs effectively, even with large datasets and high user demand. + +Snowflake's new Interactive Warehouses and Tables are designed to deliver on these needs. They provide high-concurrency, low-latency serving layer for near real-time analytics. This allows consistent, sub-second query performance for live dashboards and APIs with great price-for-performance. With this end-to-end solution, you can avoid operational complexities and tool sprawl. + +Here's how interactive warehouses and tables fits in for a typical data analytics pipeline: + +![](assets/architecture.png) + +### What You'll Learn +- The core concepts behind Snowflake's Interactive Warehouses and Tables and how they provide low-latency analytics. +- How to create and configure an Interactive Warehouse using SQL. +- The process of creating an Interactive Table from an existing standard table. +- How to attach a table to an Interactive Warehouse to pre-warm the data cache for faster queries. +- A methodology for benchmarking and comparing the query latency of an interactive setup versus a standard warehouse. + +### What You'll Build + +You will build a complete, functioning interactive data environment in Snowflake, including a dedicated Interactive Warehouse and an Interactive Table populated with data. You will also create a Python-based performance test that executes queries against both your new interactive setup and a standard configuration, culminating in a comparative bar chart that visually proves the latency improvements. + +# Step-By-Step Guide + +For prerequisites, environment setup, step-by-step guide and instructions, please refer to the [QuickStart Guide](template.md). diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/MY_DEMO_DB.BENCHMARK_FDN.HITS2_CSV.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/MY_DEMO_DB.BENCHMARK_FDN.HITS2_CSV.png new file mode 100644 index 0000000000..4952390661 Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/MY_DEMO_DB.BENCHMARK_FDN.HITS2_CSV.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/architecture.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/architecture.png new file mode 100644 index 0000000000..a7f8937101 Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/architecture.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/attach-interactive-table-to-warehouse.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/attach-interactive-table-to-warehouse.png new file mode 100644 index 0000000000..28277444e0 Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/attach-interactive-table-to-warehouse.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/compare-to-standard-warehouse.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/compare-to-standard-warehouse.png new file mode 100644 index 0000000000..bb7d61f8db Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/compare-to-standard-warehouse.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/create-interactive-table.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/create-interactive-table.png new file mode 100644 index 0000000000..cb2c58cac4 Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/create-interactive-table.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/create-turn-on-interactive-warehouse.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/create-turn-on-interactive-warehouse.png new file mode 100644 index 0000000000..cd2a6ba2a3 Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/create-turn-on-interactive-warehouse.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/interactive-tables-and-warehouses.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/interactive-tables-and-warehouses.png new file mode 100644 index 0000000000..a274630f27 Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/interactive-tables-and-warehouses.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/iw_run_exec.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/iw_run_exec.png new file mode 100644 index 0000000000..1ce31f4d0c Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/iw_run_exec.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/py_iw_run.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/py_iw_run.png new file mode 100644 index 0000000000..20d638783f Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/py_iw_run.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/py_run_queries.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/py_run_queries.png new file mode 100644 index 0000000000..d06851f69a Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/py_run_queries.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/py_std_iw_run_exec.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/py_std_iw_run_exec.png new file mode 100644 index 0000000000..d1a659e2ba Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/py_std_iw_run_exec.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/py_std_run.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/py_std_run.png new file mode 100644 index 0000000000..fca7c609ae Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/py_std_run.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/run-queries-with-interactive-warehouse.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/run-queries-with-interactive-warehouse.png new file mode 100644 index 0000000000..a383e19eea Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/run-queries-with-interactive-warehouse.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/assets/use-cases.png b/site/sfguides/src/getting_started_with_interactive_tables/assets/use-cases.png new file mode 100644 index 0000000000..f6f5ad182c Binary files /dev/null and b/site/sfguides/src/getting_started_with_interactive_tables/assets/use-cases.png differ diff --git a/site/sfguides/src/getting_started_with_interactive_tables/getting_started_with_interactive_tables.md b/site/sfguides/src/getting_started_with_interactive_tables/getting_started_with_interactive_tables.md new file mode 100644 index 0000000000..f74547deff --- /dev/null +++ b/site/sfguides/src/getting_started_with_interactive_tables/getting_started_with_interactive_tables.md @@ -0,0 +1,521 @@ +author: Chanin Nantasenamat +id: snowflake-interactive-tables +summary: This guide demonstrates how to set up and use Snowflake's Interactive Warehouses and Tables to achieve sub-second query performance. +categories: featured, getting-started, data-engineering +environments: web +status: Hidden +feedback link: https://github.com/Snowflake-Labs/sfguides/issues +tags: snowflake, interactive-tables, performance-optimization, data-warehousing, analytics + +# Getting Started with Snowflake Interactive Tables + +## Overview +Duration: 5 + +When it comes to near real-time (or sub-second) analytics, the ideal scenario involves achieving consistent, rapid query performance and managing costs effectively, even with large datasets and high user demand. + +Snowflake's new Interactive Warehouses and Tables are designed to deliver on these needs. They provide high-concurrency, low-latency serving layer for near real-time analytics. This allows consistent, sub-second query performance for live dashboards and APIs with great price-for-performance. With this end-to-end solution, you can avoid operational complexities and tool sprawl. + +Here's how interactive warehouses and tables fits in for a typical data analytics pipeline: + +![](assets/architecture.png) + +### What You'll Learn +- The core concepts behind Snowflake's Interactive Warehouses and Tables and how they provide low-latency analytics. +- How to create and configure an Interactive Warehouse using SQL. +- The process of creating an Interactive Table from an existing standard table. +- How to attach a table to an Interactive Warehouse to pre-warm the data cache for faster queries. +- A methodology for benchmarking and comparing the query latency of an interactive setup versus a standard warehouse. + +### What You'll Build + +You will build a complete, functioning interactive data environment in Snowflake, including a dedicated Interactive Warehouse and an Interactive Table populated with data. You will also create a Python-based performance test that executes queries against both your new interactive setup and a standard configuration, culminating in a comparative bar chart that visually proves the latency improvements. + +### Prerequisites +- Access to a [Snowflake account](https://signup.snowflake.com/) +- Basic knowledge of SQL and Python. +- Familiarity with data warehousing and performance concepts. +- A Snowflake role with privileges to create warehouses and tables (*i.e.*, `SYSADMIN` is used in the notebook). + + +## Understand Interactive Warehouses and Interactive Tables +Duration: 5 + +To boost query performance for interactive, sub-second analytics, Snowflake introduces two new, specialized objects that work together: interactive warehouses and interactive tables. + +Think of them as a high-performance pair. Interactive tables are structured for extremely fast data retrieval, and interactive warehouses are the specialized engines required to query them. Using them in tandem is the key to achieving the best possible query performance and lowest latency. + +![](assets/interactive-tables-and-warehouses.png) + +### Interactive Warehouses +An interactive warehouse tunes the Snowflake engine specially for low-latency, interactive workloads. It leverages additional metadata and index information in the underlying interactive tables to accelerate queries. This type of warehouse is optimized to run continuously, serving high volumes of concurrent queries. All interactive warehouses run on the latest generation of hardware. + +### Interactive Tables +You can only query these tables through interactive warehouses. Interactive tables have different methods for data ingestion and support a more limited set of SQL statements and query operators than standard Snowflake tables. + +An interactive table stores additional metadata and index information when created. The additional metadata and index information is compressed and thus does not have a substantial impact on the size of the table. + +### Use cases +Snowflake interactive tables are optimized for fast, simple queries when you require consistent low-latency responses. Interactive warehouses provide the compute resources required to serve these queries efficiently. Together, they enable use cases such as live dashboards, data-powered APIs, and serving high-concurrency workloads. + +![](assets/use-cases.png) + +Furthermore, this pairing of interactive warehouses and tables is ideal for a range of specific, demanding use cases where sub-second query performance is paramount. In industries like ad-tech, IoT, and video analytics, it can power near real-time decisioning on massive event streams. For application development, it enables highly responsive data-powered APIs and in-app user behavior analytics. It's also perfectly suited for internal analytics, providing the speed needed for live dashboards, BI acceleration, and critical observability/APM systems that require high-throughput alerting. + +### Limitations + +The simple queries that work best with interactive tables are usually `SELECT` statements with selective `WHERE` clauses, optionally including a `GROUP BY` clause on a few dimensions. Avoid queries involving joins or large subqueries. The performance of queries that use other features, such as window functions, is highly dependent on the data shapes that you are querying. + +Here are some limitations of interactive warehouses and interactive tables: +- An interactive warehouse is always up and running by design. You can manually suspend the warehouse, but expect significant query latency when you resume the warehouse. +- Snowflake interactive tables don’t support ETL, long-running queries (more than 15 minutes), or data manipulation language (DML) commands such as `UPDATE` and `DELETE`. +- Currently, queries that require JOIN aren’t supported. +- You can't query standard Snowflake tables from an interactive warehouse. To query both standard tables and interactive tables in the same session, run `USE WAREHOUSE` to switch to the appropriate warehouse type depending on the type of table. +- You can't run `CALL` commands to call stored procedures. + + +## Setup +Duration: 10 + +### Data operations + +#### Optional: Create warehouse + +You can use any existing warehouse or create a new one, here we'll create a new warehouse called `WH`: + +```sql +CREATE OR REPLACE WAREHOUSE WH WITH WAREHOUSE_SIZE='X-SMALL'; +``` + +#### Step 1: Create a Database and Schema + +First, we'll start by creating a database called `MY_DEMO_DB` and `BENCHMARK_FDN` and `BENCHMARK_INTERACTIVE` as schemas: + +```sql +CREATE DATABASE IF NOT EXISTS MY_DEMO_DB; +CREATE SCHEMA IF NOT EXISTS MY_DEMO_DB.BENCHMARK_FDN; +CREATE SCHEMA IF NOT EXISTS MY_DEMO_DB.BENCHMARK_INTERACTIVE; +``` + +#### Step 2: Create a new stage +Next, we'll create a stage called `my_csv_stage` where the CSV file will soon be stored: + +```sql +-- Define database and schema to use +USE SCHEMA MY_DEMO_DB.BENCHMARK_FDN; + +-- Create a stage that includes the definition for the CSV file format +CREATE OR REPLACE STAGE my_csv_stage + FILE_FORMAT = ( + TYPE = 'CSV' + SKIP_HEADER = 1 + FIELD_OPTIONALLY_ENCLOSED_BY = '"' + ); +``` + +#### Step 3: Upload CSV to a stage + +1. In the Snowflake UI, navigate to the database you created (`MY_DEMO_DB`). +2. Go to the `my_csv_stage` stage +3. Upload the `synthetic_hits_data.csv` file to this stage. + +#### Step 4: Create the Table and Load Data + +Now that we have the CSV file in the stage, we'll need to create the `HITS2_CSV` table and extract contents from the CSV file into it. + +```sql +-- Use your database and schema +USE SCHEMA MY_DEMO_DB.BENCHMARK_FDN; + +-- Create the table with the correct data types +CREATE OR REPLACE TABLE HITS2_CSV ( + EventDate DATE, + CounterID INT, + ClientIP STRING, + SearchEngineID INT, + SearchPhrase STRING, + ResolutionWidth INT, + Title STRING, + IsRefresh INT, + DontCountHits INT +); + +-- Copy the data from your stage into the table +-- Make sure to replace 'my_csv_stage' with your stage name +COPY INTO HITS2_CSV FROM @my_csv_stage/synthetic_hits_data.csv + FILE_FORMAT = (TYPE = 'CSV' SKIP_HEADER = 1); +``` + +#### Step 5: Query the data + +Finally, we'll now retrieve contents from the table by performing a simple query with the `SELECT` statement: + +```sql +-- Query +SELECT * FROM MY_DEMO_DB.BENCHMARK_FDN.HITS2_CSV; +``` + +This essentially retrieves data from the `MY_DEMO_DB` database, `BENCHMARK_FDN` schema and `HITS2_CSV` table: + +![](assets/MY_DEMO_DB.BENCHMARK_FDN.HITS2_CSV.png) + + +## Performance Demo of Snowflake's Interactive Warehouses/Tables +Duration: 15 + +To proceed with carrying out this performance comparison of interactive warehouses/tables with standard ones, you can download notebook file [Getting_Started_with_Interactive_Tables.ipynb](https://github.com/Snowflake-Labs/snowflake-demo-notebooks/blob/main/Interactive_Tables/Getting_Started_with_Interactive_Tables.ipynb) provided in the repo. + +### Load libraries and define custom functions + +We'll start by loading the prerequisite libraries and define helper functions that will be used for the benchmark. + +```python +import snowflake.connector as snow +import matplotlib.pyplot as plt +import numpy as np +import pandas as pd +import time +import datetime +import random +import statistics +import tabulate +from concurrent.futures import ThreadPoolExecutor, as_completed + +conn_kwargs={} + +def execute_and_print(query): + cursor.execute(query) + print(tabulate.tabulate(cursor.fetchall())) + +def run_and_measure(count, mode): + + if mode =="std": + query = """ + SELECT SearchEngineID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM + BENCHMARK_FDN.HITS2_CSV + WHERE SearchPhrase <> '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10; + """ + warehouse_query ="USE WAREHOUSE wh" + else: + query = """ + SELECT SearchEngineID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM + BENCHMARK_INTERACTIVE.CUSTOMERS + WHERE SearchPhrase <> '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10; + """ + warehouse_query ="USE WAREHOUSE interactive_demo_b" + timings = [] + with snow.connect(**conn_kwargs) as conn: + with conn.cursor() as cur: + cur.execute(warehouse_query) + cursor.execute('ALTER SESSION SET USE_CACHED_RESULT = FALSE;') + for i in range(count+1): + t0 = time.time() + cur.execute(query).fetchall() + time_taken = time.time() - t0 + timings.append(time_taken) + + return timings[1:] + +def plot_data(data, title, time_taken, color='blue'): + # Separate titles and counts + titles = [item[0] for item in data] + counts = [item[1] for item in data] + + # Plot bar chart + + plt.figure(figsize=(12, 4)) + plt.bar(titles, counts, color=color) + plt.xticks(rotation=45, ha='right') + plt.ylabel("Counts") + plt.xlabel("Title") + plt.title(title) + plt.text(6, 100, f'Time taken: {time_taken:.4f} seconds', ha='right',va='top', fontdict={'size': 16}) + #plt.tight_layout() + plt.show() + +# Separate titles and counts +#titles = ['Run 1', 'Run 2', 'Run 3', 'Run 4', 'Run 5', 'Run 6', 'Run 7', 'Run 8'] +counts_std = [0.1,0.15, 0.09, 0.12, 0.11, 0.13, 0.10, 0.14] +counts_iw = [0.05, 0.08, 0.07, 0.06, 0.09, 0.08, 0.07, 0.06] +``` + +### Setting up connection to a Snowflake deployment and verifying versions + +Here, we'll connect to Snowflake and verify the version and confirm that key interactive features are enabled, before setting the active database and role for the session. + +```python +config = { } +cursor = snow.connect(**config).cursor() +execute_and_print('select current_version();') +execute_and_print("show parameters like 'ENABLE_INTERACTIVE_WAREHOUSES' for account;") +execute_and_print("show parameters like 'ENABLE_INTERACTIVE_TABLE_DDL' for account;") +execute_and_print("show parameters like 'SHOW_INCLUDE_INTERACTIVE_TABLES' for account;") +query = """ USE DATABASE MY_DEMO_DB; """ +execute_and_print(query) + +query = """ USE ROLE SYSADMIN; """ +execute_and_print(query) +``` + +This should yield an output similar to the following: + +``` +------------------------------- +9.26.50 b20250903184849fccf20d9 +------------------------------- +----------------------------- ---- ----- ------- -------------------------------------------------------------------- ------- ------ ------------------------------------ ----------------------------- ------ -------------- ---------------------------------- +ENABLE_INTERACTIVE_WAREHOUSES true false ACCOUNT Enables interactive warehouses with simplified configuration options BOOLEAN 268037 01bed41c-0000-d4fd-0000-00041e60df12 2025-09-04 08:56:54.919 -0700 294796 qc-17689534226 cmcelhanney ClickNoMore validation +----------------------------- ---- ----- ------- -------------------------------------------------------------------- ------- ------ ------------------------------------ ----------------------------- ------ -------------- ---------------------------------- +---------------------------- ---- ----- ------- ----------------------------------------------------------------------------------------------------------- ------- ------ ------------------------------------ ----------------------------- ------ -------------- ---------------------------------- ------------------------- +ENABLE_INTERACTIVE_TABLE_DDL true false ACCOUNT If true, enables the DDL for Interactive Table, which allows users to create and manage interactive tables. BOOLEAN 268037 01bed41c-0000-d4fd-0000-00041e60df12 2025-09-04 08:56:54.924 -0700 294796 qc-17689534226 cmcelhanney ClickNoMore validation FEATURE_INTERACTIVE_TABLE +---------------------------- ---- ----- ------- ----------------------------------------------------------------------------------------------------------- ------- ------ ------------------------------------ ----------------------------- ------ -------------- ---------------------------------- ------------------------- +------------------------------- ---- ----- ------- ----------------------------------------------------------------------------------------- ------- ------ ------------------------------------ ----------------------------- ------ -------------- ---------------------------------- ------------------------- +SHOW_INCLUDE_INTERACTIVE_TABLES true false ACCOUNT If true, include interactive tables in SHOW TABLES/OBJECTS with the is_interactive column BOOLEAN 268037 01bed41c-0000-d4fd-0000-00041e60df12 2025-09-04 08:56:54.923 -0700 294796 qc-17689534226 cmcelhanney ClickNoMore validation FEATURE_INTERACTIVE_TABLE +------------------------------- ---- ----- ------- ----------------------------------------------------------------------------------------- ------- ------ ------------------------------------ ----------------------------- ------ -------------- ---------------------------------- ------------------------- +-------------------------------- +Statement executed successfully. +-------------------------------- +-------------------------------- +Statement executed successfully. +-------------------------------- +``` + +### Create an interactive warehouse + +![](assets/create-turn-on-interactive-warehouse.png) + +Next, let's create our `interactive_demo_b` warehouse and immediately turn it on: + +```python +query = """ +CREATE or REPLACE INTERACTIVE WAREHOUSE interactive_demo_b + WAREHOUSE_SIZE = 'XSMALL' + MIN_CLUSTER_COUNT = 1 + MAX_CLUSTER_COUNT = 1 + COMMENT = 'Interactive warehouse demo'; +""" +execute_and_print(query) +query = """ +ALTER WAREHOUSE INTERACTIVE_DEMO_B RESUME; +""" +execute_and_print(query) +``` + +This should yield the following output: + +``` +-------------------------------------------------------------- +INTERACTIVE WAREHOUSE INTERACTIVE_DEMO_B successfully created. +-------------------------------------------------------------- +-------------------------------- +Statement executed successfully. +-------------------------------- +``` + +### Create an interactive table + +![](assets/create-interactive-table.png) + +Now, we'll use the `WH` warehouse to efficiently create our new interactive `CUSTOMERS` table by copying all the data from the original standard table: + +```python +print("Switch to demo database") +print(cursor.execute("USE DATABASE MY_DEMO_DB").fetchall()) + +print("Use a standard warehouse for creating the interactive table's data") +print(cursor.execute("USE WAREHOUSE WH").fetchall()) + +query = """ +CREATE OR REPLACE INTERACTIVE TABLE +MY_DEMO_DB.BENCHMARK_INTERACTIVE.CUSTOMERS CLUSTER BY (ClientIP) +AS + SELECT * FROM MY_DEMO_DB.BENCHMARK_FDN.HITS2_CSV + +""" +execute_and_print(query) +``` + +This gives the following output: +``` +Switch to demo database +[('Statement executed successfully.',)] +Use a standard warehouse for creating the interactive table's data +[('Statement executed successfully.',)] +------------------------------------- +Table CUSTOMERS successfully created. +------------------------------------- +``` + +### Attach interactive table to a warehouse + +![](assets/attach-interactive-table-to-warehouse.png) + +Next, we'll attach our interactive table to the warehouse, which pre-warms the data cache for optimal query performance: + +```python +query = """ +USE DATABASE MY_DEMO_DB; +""" +execute_and_print(query) + +query = """ +ALTER WAREHOUSE interactive_demo_b ADD TABLES(BENCHMARK_INTERACTIVE.CUSTOMERS); +""" +execute_and_print(query) +``` + +Running the above statement should yield the following: +``` +-------------------------------- +Statement executed successfully. +-------------------------------- +-------------------------------- +Statement executed successfully. +-------------------------------- +``` + +### Run queries with interactive warehouse + +![](assets/run-queries-with-interactive-warehouse.png) + +Now, we'll run our first performance test on the interactive setup by executing a page-view query, timing its execution, and then plotting the results. + +We'll start by activating the interactive warehouse and disabling the result cache: + +```python +print("Use a standard warehouse for creating the interactive table's data") +cursor.execute("USE WAREHOUSE interactive_demo_b") +cursor.execute('USE DATABASE MY_DEMO_DB;') +cursor.execute('ALTER SESSION SET USE_CACHED_RESULT = FALSE;') +``` + +![](assets/py_iw_run.png) + +Next, we'll run a query to find the top 10 most viewed pages for July 2013, measures how long it takes, and then plots the results and execution time: + +```python +query = """ +SELECT Title, COUNT(*) AS PageViews +FROM BENCHMARK_INTERACTIVE.CUSTOMERS +WHERE CounterID = 62 + AND EventDate >= '2013-07-01' + AND EventDate <= '2013-07-31' + AND DontCountHits = 0 + AND IsRefresh = 0 + AND Title <> '' + AND REGEXP_LIKE(Title, '^[\\x00-\\x7F]+$') + AND LENGTH(Title) < 20 +GROUP BY Title +ORDER BY PageViews DESC +LIMIT 10; +""" + +start_time = time.time() +result = cursor.execute(query).fetchall() +end_time = time.time() +time_taken = end_time - start_time + +plot_data(result, "Page visit analysis (Interactive)", time_taken) +``` + +This gives the following plot: + +![](assets/iw_run_exec.png) + +### Compare to a standard warehouse + +![](assets/compare-to-standard-warehouse.png) + +To establish a performance baseline, we'll run an identical page-view query on a standard warehouse to measure and plot its results for comparison. + +We'll start by preparing the session for a performance benchmark by selecting a standard `XSMALL` warehouse, disabling the result cache, and setting the active database: + +```python +print("Use a standard warehouse for creating the interactive table's data") +cursor.execute("USE WAREHOUSE WH") +cursor.execute('ALTER SESSION SET USE_CACHED_RESULT = FALSE;') +cursor.execute('USE DATABASE MY_DEMO_DB;') +``` + +![](assets/py_std_run.png) + +Here, we'll run a top 10 page views analysis by executing the query, measuring its performance, and immediately plotting the results and execution time: + +```python +query = """ +SELECT Title, COUNT(*) AS PageViews +FROM BENCHMARK_FDN.HITS2_CSV +WHERE CounterID = 62 + AND EventDate >= '2013-07-01' + AND EventDate <= '2013-07-31' + AND DontCountHits = 0 + AND IsRefresh = 0 + AND Title <> '' + AND REGEXP_LIKE(Title, '^[\\x00-\\x7F]+$') + AND LENGTH(Title) < 20 +GROUP BY Title +ORDER BY PageViews DESC +LIMIT 10; +""" + +start_time = time.time() +result = cursor.execute(query).fetchall() +end_time = time.time() +time_taken = end_time - start_time + +plot_data(result, "Page visit analysis (Interactive)", time_taken, 'green') +``` + +![](assets/py_std_iw_run_exec.png) + +### Run some queries concurrently + +To directly compare performance, we'll benchmark both the interactive and standard warehouses over several runs and then plot their latencies side-by-side in a grouped bar chart: + +```python +runs = 4 + +counts_iw = run_and_measure(runs,"iw") +print(counts_iw) + +counts_std = run_and_measure(runs,"std") +print(counts_std) + +titles = [f"R{i}" for i in range(1, len(counts_iw)+1)] + +x = np.arange(len(titles)) # the label locations +width = 0.35 # bar width + +fig, ax = plt.subplots(figsize=(8, 5)) +ax.bar(x - width/2, counts_std, width, label="Standard", color="green") +ax.bar(x + width/2, counts_iw, width, label="Interactive", color="blue") + +ax.set_ylabel("Latency") +ax.set_xlabel("Query run") +ax.set_title("Standard vs Interactive warehouse") +ax.set_xticks(x) +ax.set_xticklabels(titles) +ax.legend( + loc='upper center', + bbox_to_anchor=(0.5, -0.15), + ncol=2 +) +plt.show() +``` + +![](assets/py_run_queries.png) + +## Conclusion And Resources +Duration: 5 + +In this guide, we explored how to address the challenge of low-latency, near real-time analytics using Snowflake's interactive warehouses and tables. We walked through the complete setup process, from creating the necessary database objects and loading data to configuring and attaching an interactive table to an interactive warehouse. The subsequent performance benchmark clearly demonstrated the substantial latency improvements these specialized features provide over standard configurations, especially under concurrent query loads. This confirms their value as a powerful solution for demanding use cases like live dashboards and high-throughput data APIs, where sub-second performance is critical. + +### What You Learned +- Interactive warehouses and tables work together as a specialized pair to deliver low-latency analytics for use cases like live dashboards and APIs. +- How to create, configure, and attach interactive warehouses and tables using SQL to prepare a high-performance analytics environment. +- How to benchmark and visually demonstrate the performance gains of interactive setups over standard ones using Python, proving their effectiveness for high-concurrency workloads. + +### Related Resources + +Documentation: +- [Snowflake interactive tables and interactive warehouses](https://docs.snowflake.com/LIMITEDACCESS/interactive) diff --git a/site/sfguides/src/resource_optimization_usage_monitoring/resource_optimization_usage_monitoring.md b/site/sfguides/src/resource_optimization_usage_monitoring/resource_optimization_usage_monitoring.md index 86a086133f..f8cb363be3 100644 --- a/site/sfguides/src/resource_optimization_usage_monitoring/resource_optimization_usage_monitoring.md +++ b/site/sfguides/src/resource_optimization_usage_monitoring/resource_optimization_usage_monitoring.md @@ -100,7 +100,7 @@ SELECT START_TIME ``` #### Screenshot ![alt-text-here](assets/averagehourbyhourconsumption.png) -####SQL (by hour) +#### SQL (by hour) ```sql SELECT DATE_PART('HOUR', START_TIME) AS START_HOUR ,WAREHOUSE_NAME