Skip to content
Open
17,055 changes: 0 additions & 17,055 deletions site/package-lock.json

This file was deleted.

67 changes: 0 additions & 67 deletions site/package.json

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -7,21 +7,23 @@ status: Published
feedback link: https://github.com/Snowflake-Labs/sfguides/issues
tags: Getting Started, Kafka, Data Streaming, Open Catalog, Iceberg

# Cost-Effective Data Streaming in Apache Iceberg™ Format and Snowflake Open Catalog Integration with StreamNative
# Cost-Effective Data Streaming in Apache Iceberg™ Format and Snowflake Open Catalog Integration with StreamNative BYOC on AWS
<!-- ------------------------ -->
## Overview
Duration: 5

StreamNative is partnering with Snowflake to provide you with cost-effective real-time data streaming to Snowflake. This means providing you with best-in-class data ingestion methods tailored to meet your specific needs. This guide focuses on creating a **Streaming Augmented Lakehouse** using **StreamNative's Ursa Engine** with built-in support for **Apache Iceberg&trade; and Snowflake Open Catalog**. Apache Kafka&reg; messages published to the StreamNative Ursa Cluster will be stored in object storage in Iceberg format. Without copying over the data through connectors, you can directly access data from Snowflake Open Catalog and start analyzing the data. To learn more about cost savings when using the StreamNative Ursa Engine to ingest data to Snowflake, visit this [link](https://streamnative.io/blog/leaderless-architecture-and-lakehouse-native-storage-for-reducing-kafka-cost).

This guide focuses on a Bring Your Own Cloud (BYOC) StreamNative Ursa Cluster deployment on AWS. A separate Snowflake Quickstart is available for [deploying the StreamNative BYOC Ursa Cluster on GCP](https://quickstarts.snowflake.com/guide/streamnative_snowflake_open_catalog_integration_gcp/).

StreamNative also supports ingesting data into Snowflake using **Snowpipe or Snowpipe Streaming** with **Kafka or Pulsar Connectors** that will not be discussed in this tutorial. For more information on using Connectors to ingest data into Snowflake, follow this [link](https://courses.streamnative.io/courses/streamnative-snowflake-streaming-augmented-lakehouse-and-connectors/lessons/introduction-to-uniconn-kafka-and-pulsar-io-connectors/).

### What You Will Build
- A Streaming Augmented Lakehouse powered by a Kafka-compatible StreamNative BYOC Ursa Cluster integrated with Snowflake Open Catalog.
- A Streaming Augmented Lakehouse powered by a Kafka-compatible StreamNative BYOC Ursa Cluster deployed on AWS integrated with Snowflake Open Catalog.

### What You Will Learn
- How to create a Snowflake Open Catalog
- How to deploy a StreamNative BYOC Ursa Cluster integrated with Snowflake Open Catalog
- How to deploy a StreamNative BYOC Ursa Cluster on AWS integrated with Snowflake Open Catalog
- How to publish Kafka messages to the StreamNative Ursa Cluster using Kafka Java Client
- How to query tables visible in Snowflake Open Catalog in Snowflake AI Data Cloud

Expand Down Expand Up @@ -80,7 +82,7 @@ Click the **Account URL** and sign into your Open Catalog Account with the User

![Snowflake Open Catalog Account Page](assets/click-account-url.png)

If you need the **Account URL** of your Snowflake Open Catalog Account in the future, navigate to **Admin → Accounts → … → Manage URLs** of your Snowflake Open Catalog Account. This page is available in your Snowflake AI Data Cloud Account.
If you need the **Account URL** of your Snowflake Open Catalog Account in the future, navigate to **Admin → Accounts → … → Manage URLs** of your Snowflake Open Catalog Account. This page is available in your Snowflake AI Data Cloud Account. The Locator column, in combination with the Region, can be used to construct the Account Locator URL. The Account Locator URL will be needed when configuring the StreamNative BYOC Ursa cluster.

![Snowflake Open Catalog Account URL](assets/manage-snowflake-account-urls.png)

Expand Down Expand Up @@ -393,7 +395,10 @@ s3://aws-usw2-test-rni68-tiered-storage-snc/o-naa2l-c-vo06zqe-ursa/compaction
To integrate with Snowflake Open Catalog, Enable Catalog Integration and select Snowflake Open Catalog.

* **Warehouse:** catalog created in Snowflake Open Catalog
* **URI:** Account URL when creating Snowflake Open Catalog. Append ***'/polaris/api/catalog'*** to the URI. See screenshot below.
* **URI:** Account Locator URL when creating Snowflake Open Catalog. Append ***'/polaris/api/catalog'*** to the URI. The StreamNative UI requires you use the Account Locator URL instead of the Account URL. The Account Locator URL will be in the following format.
```markdown
https://<your-account-locator>.<catalog-region>.snowflakecomputing.com/polaris/api/catalog
```
* **Select Authentication Type/OAuth2:** create a new secret in StreamNative using Snowflake Open Catalog Service Connection "CLIENT ID:SECRET"

![Enable Snowflake Open Catalog Integration](assets/enable-snowflake-open-catalog.png)
Expand Down Expand Up @@ -576,7 +581,7 @@ You will need to create a new catalog integration for each tenant.namespace.

### Create an externally managed table

Please refer to the [Snowflake documentation here](https://docs.streamnative.io/docs/integrate-with-snowflake-open-catalog) for the latest code samples.
Please refer to the [Snowflake documentation here](https://docs.snowflake.com/en/user-guide/tables-iceberg-open-catalog-query) for the latest code samples.

The video includes the following details from our example:
* A Snowflake Open Catalog warehouse.schema.table (e.g. streamnative.public.default.kafkaschematopic) is mapped to a Snowflake database.schema.table (e.g. training.public.kafkaschematopic)
Expand Down Expand Up @@ -607,6 +612,7 @@ select COUNT(*) FROM TRAINING.PUBLIC.kafkaschematopic
## Conclusion and Resources
Duration: 5

### Conclusion
Congratulations on creating a Streaming Augmented Lakehouse powered by StreamNative's Ursa Engine with built-in support for Iceberg and Snowflake Open Catalog. [Contact StreamNative](https://streamnative.io/contact) to learn more.

### What You Learned
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading