feat(ingest/snowflake): add Streamlit apps ingestion support #15046
+11,057
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add support for ingesting Snowflake Streamlit apps as Dashboard entities - Streamlit apps are modeled as dashboards with Streamlit subtype - Use Dashboard SDK for simplified metadata generation - Apply snowflake_identifier() for consistent URN casing with datasets - Build browse paths manually to avoid duplicate platform instance entries - Add comprehensive unit and integration tests with golden files - Support streamlit_pattern filtering configuration This enables DataHub to discover and catalog Streamlit applications deployed in Snowflake, providing visibility into interactive data apps.
github-actions
bot
added
ingestion
datahub-cyborg
bot
added
the
needs-review
- Register 'streamlit' platform in bootstrap (v4 → v5) with logo - Add Streamlit constants and logos to frontend (ingest V1 and V2) - Ingest apps with platform="streamlit", parent containers, and Snowsight URLs - Support filtering via streamlit_pattern configuration
Bundle ReportChanges will increase total bundle size by 5.5kB (0.02%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: datahub-react-web-esmAssets Changed:
Files in
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py
Outdated
Show resolved
Hide resolved
datahub-cyborg
bot
added
pending-submitter-response
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py
Show resolved
Hide resolved
| "url_id": "abc123def456_1", | ||
| "comment": "Comment for Streamlit App 1" | ||
| }, | ||
| "externalUrl": "https://app.snowflake.com/ap-south-1.aws/abc12345/#/streamlit-apps/TEST_DB.TEST_SCHEMA.STREAMLIT_APP_1", |
There was a problem hiding this comment.
probably it's an issue with the test data... how is the externalUrl does not include the url_id ?
There was a problem hiding this comment.
url_id is the unique id for the streamlit app. Based on the investigation, it's not related to access url to Streamlit page. Thank you for checking.
…oard identifiers Streamlit apps are identified using their url_id field instead of name to ensure stable and unique dashboard identifiers. Updated documentation to include required permissions for Streamlit app ingestion.
datahub-cyborg
bot
added
needs-review
datahub-cyborg
bot
added
merge-pending-ci
This commit reverts data platform version as it's not breacking change.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This enables DataHub to discover and catalog Streamlit applications deployed in Snowflake, providing visibility into interactive data apps.