Skip to content

Feature Request: Comprehensive Metadata Discovery for Data Development Tools #77

Description

@mullinsms

Summary

The ADBC Snowflake driver currently supports discovery of databases, schemas, tables, views, columns, and key constraints. However, Snowflake has a rich object model that data development tools need to surface — functions, procedures, stages, tasks, warehouses, and many more. This issue requests expanding metadata discovery to cover the full breadth of Snowflake objects.

We build a collaborative data operations platform and currently use the Snowflake JDBC driver supplemented by SHOW and DESCRIBE commands to extract metadata. We are evaluating the ADBC driver as a modern Arrow-native replacement but cannot adopt it without coverage for the object categories listed below.

Each section describes what we need, why it matters, what metadata properties are required, and how the data can be obtained from Snowflake.


Table of Contents

  1. Functions (Scalar & Table-Valued)
  2. Function & Procedure Parameters
  3. System / Built-in Functions
  4. Stored Procedures
  5. Sequences
  6. Stages (Internal & External)
  7. File Formats
  8. Tasks
  9. Streams
  10. Pipes
  11. Warehouses
  12. Users & Roles
  13. Shares (Inbound & Outbound)
  14. Storage Integrations
  15. View Definitions (SQL Text)
  16. Extended Table Types
  17. Check Constraints
  18. Table Statistics

1. Functions (Scalar & Table-Valued)

Motivation

User-defined functions are first-class schema objects in Snowflake. Data development tools need to enumerate them alongside tables and views for autocomplete, documentation, and catalog browsing. Snowflake supports function overloading (same name, different signatures), which requires careful handling.

Required Metadata

Property Description Source Column
Database Containing database catalog_name
Schema Containing schema schema_name
Function name Function identifier name from SHOW / split from arguments
Is table function Scalar vs table-valued is_table_function
Language SQL, JavaScript, Python, Java, Scala language
Return type Return data type (scalar) or result columns (table) Parsed from arguments / DESCRIBE output
Volatility VOLATILE, IMMUTABLE, STABLE is_memoizable and DESCRIBE output
Comment User-provided description description
Created time Creation timestamp created_on
Is secure Whether the function is secure is_secure
Is external Whether it's an external function is_external_function
Arguments signature Full type signature for overload resolution arguments

How to Obtain

-- List all user functions in a schema
SHOW USER FUNCTIONS IN SCHEMA "db"."schema";

-- List all user functions in a schema (single function)
SHOW USER FUNCTIONS LIKE 'func_name' IN SCHEMA "db"."schema";

-- Get detailed metadata including parameter names, body, language
DESCRIBE FUNCTION "db"."schema"."func_name"(VARCHAR, NUMBER);

SHOW USER FUNCTIONS returns: created_on, name, schema_name, is_builtin, is_aggregate, is_ansi, min_num_arguments, max_num_arguments, arguments, description, catalog_name, is_table_function, valid_for_clustering, is_secure, is_external_function, language, is_memoizable.

The arguments column contains the full signature, e.g. MY_FUNC(VARCHAR, NUMBER) RETURN VARCHAR. This must be parsed to separate input types from return type and to construct the signature needed for DESCRIBE FUNCTION.

DESCRIBE FUNCTION returns rows with property and value columns including: signature, returns, language, body, null handling, volatility, is_external, external_access_integrations, secrets, imports, handler, packages.

Overload Handling

Multiple functions can share the same name with different parameter lists. The driver should:

  1. Group results from SHOW USER FUNCTIONS by name
  2. Expose each overload as a separate entry (or as children of a parent function)
  3. Include the full type signature so callers can distinguish overloads

2. Function & Procedure Parameters

Motivation

Parameter metadata enables signature help, call template generation, and documentation. Without parameter names and types, tools can only show the function name.

Required Metadata

Property Description
Parameter name e.g., input_date, threshold
Ordinal position 1-based position
Data type Snowflake type name
Parameter mode IN, OUT, INOUT (procedures only)
Default value Default expression if optional
Comment Parameter description if available

For table functions, the result column schema:

Property Description
Column name Result column name
Data type Result column type
Ordinal position Position in result set

How to Obtain

-- Returns rows including 'signature' and 'returns' properties
DESCRIBE FUNCTION "db"."schema"."func_name"(VARCHAR, NUMBER);
DESCRIBE PROCEDURE "db"."schema"."proc_name"(VARCHAR);

The signature property contains parameter names and types: (input_date VARCHAR, threshold NUMBER).
The returns property contains the return type (scalar) or TABLE(col1 TYPE, col2 TYPE) (table function).


3. System / Built-in Functions

Motivation

SQL editors provide autocomplete and inline documentation for built-in functions like COALESCE, DATE_TRUNC, ARRAY_AGG, etc. A hardcoded list drifts out of date across Snowflake releases.

Required Metadata

Property Description Source Column
Function name e.g., ABS, CONCAT Parsed from arguments
Category Aggregate, string, date, etc. Not directly available
Description What the function does description
Arguments signature Input types arguments
Min/max arguments Arity range min_num_arguments, max_num_arguments
Is aggregate Whether it's an aggregate function is_aggregate

How to Obtain

-- Returns all built-in functions (no schema qualifier)
SHOW FUNCTIONS;

Returns the same columns as SHOW USER FUNCTIONS but for system-provided functions. Filter where is_builtin = 'Y'.


4. Stored Procedures

Motivation

Stored procedures are a core part of Snowflake's programmability model. They support SQL, JavaScript, Python, Java, and Scala. Like functions, they support overloading.

Required Metadata

Property Description Source Column
Database Containing database catalog_name
Schema Containing schema schema_name
Procedure name Procedure identifier Parsed from arguments
Language SQL, JavaScript, Python, Java, Scala language (from DESCRIBE)
Return type Return data type Parsed from arguments
Comment Description description
Created time Creation timestamp created_on
Arguments signature Full type signature arguments
Execute as OWNER or CALLER execute as

How to Obtain

SHOW PROCEDURES IN SCHEMA "db"."schema";
SHOW PROCEDURES LIKE 'proc_name' IN SCHEMA "db"."schema";
DESCRIBE PROCEDURE "db"."schema"."proc_name"(VARCHAR, NUMBER);

SHOW PROCEDURES returns: created_on, name, schema_name, is_builtin, is_aggregate, is_ansi, min_num_arguments, max_num_arguments, arguments, description, catalog_name, is_table_function, valid_for_clustering, is_secure.

DESCRIBE PROCEDURE returns: signature, returns, language, execute as, body, null handling, volatility, external_access_integrations, secrets, imports, handler, packages.


5. Sequences

Motivation

Sequences are used for generating unique identifiers. Tools need to display them in the schema browser and show their current state.

Required Metadata

Property Description Source Column
Database Containing database database_name
Schema Containing schema schema_name
Sequence name Sequence identifier name
Next value Next value to be issued next_value
Interval Increment step interval
Comment Description comment
Owner Owning role owner
Created time Creation timestamp created_on

How to Obtain

SHOW SEQUENCES IN SCHEMA "db"."schema";

Returns: created_on, name, schema_name, database_name, next_value, interval, owner, owner_role_type, comment.


6. Stages (Internal & External)

Motivation

Stages are the primary mechanism for loading and unloading data in Snowflake. Tools that support data pipeline authoring or data loading workflows need to enumerate stages and understand their configuration.

Required Metadata — Internal Stages

Property Description Source Column
Stage name Stage identifier name
Region Storage region region
Comment Description comment
Owner Owning role owner
Type INTERNAL type
Created time Creation timestamp created_on

Required Metadata — External Stages

Property Description Source Column
Stage name Stage identifier name
URL External location URI url
Cloud AWS, Azure, GCP cloud
Region Storage region region
Storage integration Associated integration name storage_integration
Has credentials Whether credentials are configured has_credentials
Has encryption key Whether encryption is configured has_encryption_key
Comment Description comment
Owner Owning role owner
Type EXTERNAL type
Created time Creation timestamp created_on

How to Obtain

SHOW STAGES IN SCHEMA "db"."schema";

Returns: created_on, name, database_name, schema_name, url, has_credentials, has_encryption_key, owner, comment, region, type (INTERNAL/EXTERNAL), cloud, storage_integration, owner_role_type, directory_enabled.

Differentiate internal vs external by the type column.


7. File Formats

Motivation

File formats define how staged data is parsed during COPY operations. They are referenced by name in COPY INTO statements and need to be browsable.

Required Metadata

Property Description Source Column
Format name File format identifier name
Format type CSV, JSON, PARQUET, AVRO, ORC, XML type
Comment Description comment
Owner Owning role owner
Created time Creation timestamp created_on

How to Obtain

SHOW FILE FORMATS IN SCHEMA "db"."schema";

Returns: created_on, name, database_name, schema_name, type, owner, comment, owner_role_type.


8. Tasks

Motivation

Tasks are Snowflake's native scheduling mechanism for SQL and stored procedure execution. Tools need to display task definitions, schedules, states, and dependency chains (DAGs).

Required Metadata

Property Description Source Column
Task name Task identifier name
Schedule Cron or interval expression schedule
State STARTED, SUSPENDED, etc. state
Definition The SQL statement executed definition
Condition WHEN expression (optional) condition
Predecessors Parent tasks in DAG predecessors
Warehouse Execution warehouse warehouse
Comment Description comment
Owner Owning role owner
Created time Creation timestamp created_on

How to Obtain

SHOW TASKS IN SCHEMA "db"."schema";

Returns: created_on, name, id, database_name, schema_name, owner, comment, warehouse, schedule, predecessors, state, definition, condition, allow_overlapping_execution, error_integration, last_committed_on, last_suspended_on, owner_role_type, config, budget, last_suspended_reason.


9. Streams

Motivation

Streams track DML changes (inserts, updates, deletes) on tables, providing change data capture (CDC) functionality. They are central to Snowflake's continuous data pipeline patterns and are referenced in task definitions.

Required Metadata

Property Description Source Column
Stream name Stream identifier name
Source object Table, view, or directory table being tracked table_name
Source type Type of source object source_type
Stream type STANDARD, APPEND_ONLY, INSERT_ONLY type
Mode DEFAULT, APPEND_ONLY, INSERT_ONLY mode
Is stale Whether the stream has fallen behind stale
Stale after Timestamp when stream becomes stale stale_after
Comment Description comment
Owner Owning role owner
Created time Creation timestamp created_on
Database Containing database database_name
Schema Containing schema schema_name
Base tables Underlying base tables base_tables

How to Obtain

SHOW STREAMS IN SCHEMA "db"."schema";

Returns: created_on, name, database_name, schema_name, owner, comment, table_name, source_type, base_tables, type, stale, mode, stale_after, invalid_reason, owner_role_type.


10. Pipes

Motivation

Pipes provide continuous, serverless data ingestion from stages into tables (Snowpipe). Tools need to display pipe configurations and monitor their status.

Required Metadata

Property Description Source Column
Pipe name Pipe identifier name
Definition The COPY INTO statement definition
Is auto-ingest Whether auto-ingest is enabled notification_channel presence
Notification channel SQS/SNS/Event Grid endpoint notification_channel
Comment Description comment
Owner Owning role owner
Created time Creation timestamp created_on
Integration Notification integration name integration
Pattern File pattern filter Embedded in definition
Database Containing database database_name
Schema Containing schema schema_name

How to Obtain

SHOW PIPES IN SCHEMA "db"."schema";

Returns: created_on, name, database_name, schema_name, definition, owner, notification_channel, comment, integration, pattern, error_integration, owner_role_type, invalid_reason, budget.


11. Warehouses

Motivation

Warehouses are Snowflake's compute resources. Tools that manage connections or display environment context need to enumerate available warehouses and their state.

Required Metadata

Property Description Source Column
Warehouse name Warehouse identifier name
Type STANDARD, SNOWPARK_OPTIMIZED type
State STARTED, SUSPENDED, RESIZING state
Size XS, S, M, L, XL, etc. size
Auto-suspend (seconds) Idle timeout before suspend auto_suspend
Auto-resume Whether auto-resume is enabled auto_resume
Comment Description comment
Owner Owning role owner
Running queries Active query count running
Queued queries Queued query count queued
Is default Whether it's the session default is_default
Created time Creation timestamp created_on
Resumed time Last resume timestamp resumed_on
Updated time Last config change updated_on
Resource monitor Associated resource monitor resource_monitor

How to Obtain

SHOW WAREHOUSES;

Returns: name, state, type, size, min_cluster_count, max_cluster_count, started_clusters, running, queued, is_default, is_current, auto_suspend, auto_resume, available, provisioning, quiescing, other, created_on, resumed_on, updated_on, owner, comment, enable_query_acceleration, query_acceleration_max_scale_factor, resource_monitor, owner_role_type, budget.


12. Users & Roles

Motivation

Tools that manage access, display connection context, or support administrative workflows need to discover users and roles.

Required Metadata — Users

Property Description Source Column
User name User identifier name
Email User email address email
Default role Default role at login default_role
Default warehouse Default warehouse at login default_warehouse
Comment Description comment
Owner Owning role owner
Disabled Whether the user is disabled disabled
Locked until Lock expiration timestamp locked_until_time
Created time Creation timestamp created_on

Required Metadata — Roles

Property Description Source Column
Role name Role identifier name
Comment Description comment
Owner Owning role owner
Is default Whether it's the user's default role is_default
Created time Creation timestamp created_on
Granted roles Number of granted child roles granted_roles
Granted to roles Number of parent roles granted_to_roles

How to Obtain

SHOW USERS;
SHOW ROLES;
SELECT CURRENT_AVAILABLE_ROLES();  -- roles accessible to current user

13. Shares (Inbound & Outbound)

Motivation

Snowflake Secure Data Sharing is a core platform feature. Tools need to display both outbound shares (data this account provides) and inbound shares (data consumed from other accounts).

Required Metadata

Property Description Source Column
Share name Share identifier name
Kind OUTBOUND or INBOUND kind
Database name Database associated with the share database_name
Comment Description comment
Owner Owning role (outbound only) owner
Owner account Account that owns the share owner_account
To (consumers) List of consumer accounts (outbound) to
Created time Creation timestamp created_on

How to Obtain

SHOW SHARES;
SELECT current_organization_name() || '.' || current_account_name() AS current_account_name;

The kind column distinguishes OUTBOUND (this account shares data) from INBOUND (this account consumes shared data).


14. Storage Integrations

Motivation

Storage integrations configure secure access to external cloud storage (S3, GCS, Azure Blob). They are referenced by external stages and external tables.

Required Metadata

Property Description Source Column
Integration name Integration identifier name
Type EXTERNAL_STAGE type
Enabled Whether the integration is active enabled
Comment Description comment
Created time Creation timestamp created_on

How to Obtain

SHOW STORAGE INTEGRATIONS;

Returns: name, type, category, enabled, comment, created_on.


15. View Definitions (SQL Text)

Motivation

Users inspect, copy, and refactor view definitions. View SQL is essential for understanding data lineage, migration between environments, and documentation.

Required Metadata

Property Description
View database Containing database
View schema Containing schema
View name View identifier
View definition The SELECT statement defining the view
Is secure Whether the view is secure (definition hidden from non-owners)
Is materialized Whether it's a materialized view

How to Obtain

-- Via INFORMATION_SCHEMA
SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, VIEW_DEFINITION,
       CHECK_OPTION, IS_UPDATABLE, IS_SECURE, IS_MATERIALIZED
FROM "db".INFORMATION_SCHEMA.VIEWS
WHERE TABLE_SCHEMA = 'schema_name';

-- Via DDL generation
SELECT GET_DDL('VIEW', '"db"."schema"."view_name"');

Note: Secure view definitions are only visible to the view owner role.


16. Extended Table Types

Motivation

The driver currently returns only "TABLE" and "VIEW" from GetTableTypes(). Snowflake has many specialized table types that tools need to distinguish for correct display, icon rendering, and context-appropriate operations.

Required Types

Table Type Description How to Identify
EXTERNAL TABLE Tables backed by external cloud storage files SHOW EXTERNAL TABLES IN SCHEMA
MATERIALIZED VIEW Precomputed views with automatic refresh SHOW MATERIALIZED VIEWS IN SCHEMA
DYNAMIC TABLE Declarative tables that auto-refresh from a query SHOW DYNAMIC TABLES IN SCHEMA
ICEBERG TABLE Apache Iceberg-format tables (managed or unmanaged) SHOW ICEBERG TABLES IN SCHEMA
HYBRID TABLE HTAP tables with row-level locking for transactional workloads SHOW HYBRID TABLES IN SCHEMA
EVENT TABLE Tables for logging and telemetry events SHOW EVENT TABLES IN SCHEMA
TEMPORARY TABLE Session-scoped temporary tables is_temporary column in SHOW TABLES
TRANSIENT TABLE Tables without Fail-safe (reduced storage cost) retention_time = 0 in SHOW TABLES

Required Metadata by Type

External Tables

Property Description Source Column
Location External storage path location
File format type CSV, JSON, Parquet, etc. file_format_type
Cloud AWS, Azure, GCP cloud
Region Storage region region
Auto-refresh Whether auto-refresh is enabled auto_refresh
Notification channel Notification endpoint for auto-refresh notification_channel
Is invalid Whether the external table is in error state invalid
Invalid reason Why the table is invalid invalid_reason
Last refreshed Last refresh timestamp last_refreshed_on
Comment Description comment
Owner Owning role owner
SHOW EXTERNAL TABLES IN SCHEMA "db"."schema";
DESCRIBE EXTERNAL TABLE "db"."schema"."table" TYPE = COLUMNS;
DESCRIBE EXTERNAL TABLE "db"."schema"."table" TYPE = STAGE;

Materialized Views

Property Description Source Column
Name View identifier name
Definition The defining SQL query text
Is secure Whether the view is secure is_secure
Cluster by Clustering columns cluster_by
Rows Approximate row count rows
Bytes Storage size bytes
Refreshed on Last refresh timestamp refreshed_on
Behind by How far behind source data behind_by
Comment Description comment
Owner Owning role owner
SHOW MATERIALIZED VIEWS IN SCHEMA "db"."schema";

Dynamic Tables

Property Description Source Column
Name Table identifier name
Target lag Desired freshness (e.g., '1 minute', 'DOWNSTREAM') target_lag
Refresh mode AUTO, FULL, INCREMENTAL refresh_mode
Scheduling state RUNNING, SUSPENDED scheduling_state
Definition The defining SQL query text
Warehouse Execution warehouse warehouse
Data timestamp Freshness of the data data_timestamp
Last suspended on Last suspension timestamp last_suspended_on
Is iceberg Whether backed by Iceberg format is_iceberg
Comment Description comment
Owner Owning role owner
SHOW DYNAMIC TABLES IN SCHEMA "db"."schema";

Iceberg Tables

Property Description Source Column
Name Table identifier name
Is managed Snowflake-managed vs externally managed is_default
External volume External volume name external_volume_name
Catalog integration Catalog integration for external Iceberg catalog_integration_name
Catalog namespace Iceberg catalog namespace catalog_namespace
Catalog table name Name in external catalog catalog_table_name
Base location Storage base location base_location
Metadata location Path to current metadata file metadata_location
Comment Description comment
Owner Owning role owner
SHOW ICEBERG TABLES IN SCHEMA "db"."schema";

Hybrid Tables

Property Description Source Column
Name Table identifier name
Comment Description comment
Owner Owning role owner
Rows Row count rows
Bytes Storage size bytes
SHOW HYBRID TABLES IN SCHEMA "db"."schema";

Event Tables

Property Description Source Column
Name Table identifier name
Comment Description comment
Owner Owning role owner
SHOW EVENT TABLES IN SCHEMA "db"."schema";

17. Check Constraints

Motivation

Check constraints enforce data quality rules at the table level. Tools need to display them alongside primary/foreign/unique keys.

Required Metadata

Property Description
Constraint name Constraint identifier
Constraint type CHECK
Expression The boolean expression (e.g., price > 0)
Column names Columns referenced by the expression

Current State

The GetObjects schema already includes table_constraints with support for a CHECK type, but it is not populated. Snowflake supports check constraints and lists them in INFORMATION_SCHEMA.TABLE_CONSTRAINTS.

How to Obtain

SELECT CONSTRAINT_NAME, CONSTRAINT_TYPE, TABLE_NAME, CHECK_CLAUSE
FROM "db".INFORMATION_SCHEMA.TABLE_CONSTRAINTS tc
LEFT JOIN "db".INFORMATION_SCHEMA.CHECK_CONSTRAINTS cc
  ON tc.CONSTRAINT_NAME = cc.CONSTRAINT_NAME
WHERE tc.CONSTRAINT_TYPE = 'CHECK'
  AND tc.TABLE_SCHEMA = 'schema_name';

Note: An extension to the ADBC CONSTRAINT_SCHEMA may be needed to carry the constraint_expression text, as the spec only defines constraint_column_names.


18. Table Statistics

Motivation

Row counts and storage sizes are fundamental metadata for data tools — displayed in catalog browsers, used for query planning, and needed for capacity monitoring.

Required Metadata

Statistic ADBC Key Description
Row count StatisticRowCountKey (6) Number of rows
Size in bytes Custom key Total storage size
Active bytes Custom key Active storage (before Time Travel)
Time travel bytes Custom key Time Travel storage
Failsafe bytes Custom key Fail-safe storage
Clustering depth Custom key Average clustering depth
Retention time (days) Custom key Time Travel retention period

How to Obtain

-- From INFORMATION_SCHEMA (fast, cached, may be stale up to ~30 mins)
SELECT TABLE_NAME, ROW_COUNT, BYTES, RETENTION_TIME,
       CLUSTERING_KEY, AUTO_CLUSTERING_ON
FROM "db".INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA = 'schema_name';

-- From SHOW TABLES (returns live-ish counts)
SHOW TABLES IN SCHEMA "db"."schema";
-- Includes: rows, bytes, retention_time, automatic_clustering, cluster_by

ADBC Spec Alignment

The ADBC standard defines GetStatistics() (since v1.1.0) with a result schema including statistic_key, statistic_value, and statistic_is_approximate. Row count and size would map naturally to this API.


Summary Priority Table

# Category Object Count Snowflake API Impact
1 User Functions (scalar + table) Per schema SHOW USER FUNCTIONS + DESCRIBE FUNCTION High — core SQL development
2 Function/Procedure Parameters Per function DESCRIBE FUNCTION/PROCEDURE High — signature help
3 System Functions Global SHOW FUNCTIONS High — autocomplete
4 Stored Procedures Per schema SHOW PROCEDURES + DESCRIBE PROCEDURE High — core SQL development
5 Sequences Per schema SHOW SEQUENCES Medium
6 Stages Per schema SHOW STAGES Medium — data loading
7 File Formats Per schema SHOW FILE FORMATS Medium — data loading
8 Tasks Per schema SHOW TASKS Medium — pipeline tools
9 Streams Per schema SHOW STREAMS Medium — CDC pipelines
10 Pipes Per schema SHOW PIPES Medium — continuous ingest
11 Warehouses Account-level SHOW WAREHOUSES Medium — admin tools
12 Users & Roles Account-level SHOW USERS / SHOW ROLES Low — admin tools
13 Shares Account-level SHOW SHARES Low — data sharing
14 Storage Integrations Account-level SHOW STORAGE INTEGRATIONS Low — infra tools
15 View Definitions Per view INFORMATION_SCHEMA.VIEWS / GET_DDL() High — SQL development
16 Extended Table Types Per schema Various SHOW commands High — correct object display
17 Check Constraints Per table INFORMATION_SCHEMA.TABLE_CONSTRAINTS Medium
18 Table Statistics Per table INFORMATION_SCHEMA.TABLES / SHOW TABLES Medium

We're happy to provide further details on any of these categories.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions