Skip to content

Commit aefa809

Browse files
committed
snowflake: mark custom schema evolution as advanced
1 parent 8f3805a commit aefa809

File tree

2 files changed

+2
-12
lines changed

2 files changed

+2
-12
lines changed

docs/modules/components/pages/outputs/snowflake_streaming.adoc

-10
Original file line numberDiff line numberDiff line change
@@ -53,16 +53,6 @@ output:
5353
CREATE TABLE IF NOT EXISTS mytable (amount NUMBER);
5454
schema_evolution:
5555
enabled: false # No default (required)
56-
new_column_type_mapping: |-
57-
root = match this.value.type() {
58-
this == "string" => "STRING"
59-
this == "bytes" => "BINARY"
60-
this == "number" => "DOUBLE"
61-
this == "bool" => "BOOLEAN"
62-
this == "timestamp" => "TIMESTAMP"
63-
_ => "VARIANT"
64-
}
65-
processors: [] # No default (optional)
6656
batching:
6757
count: 0
6858
byte_size: 0

internal/impl/snowflake/output_snowflake_streaming.go

+2-2
Original file line numberDiff line numberDiff line change
@@ -118,11 +118,11 @@ ALTER TABLE t1 ADD COLUMN a2 NUMBER;
118118
service.NewBloblangField(ssoFieldSchemaEvolutionNewColumnTypeMapping).Description(`
119119
The mapping function from Redpanda Connect type to column type in Snowflake. Overriding this can allow for customization of the datatype if there is specific information that you know about the data types in use. This mapping should result in the `+"`root`"+` variable being assigned a string with the data type for the new column in Snowflake.
120120
121-
The input to this mapping is either the output of `+"`processors`"+` if specified, otherwise it is an object with the value and the name of the new column, the original message and table being written too. The metadata is unchanged from the original message that caused the schema to change. For example: `+"`"+`{"value": 42.3, "name":"new_data_field", "message": {"existing_data_field": 42, "new_data_field": "foo"}, "db": MY_DATABASE", "schema": "MY_SCHEMA", "table": "MY_TABLE"}`).Default(defaultSchemaEvolutionNewColumnMapping),
121+
The input to this mapping is either the output of `+"`processors`"+` if specified, otherwise it is an object with the value and the name of the new column, the original message and table being written too. The metadata is unchanged from the original message that caused the schema to change. For example: `+"`"+`{"value": 42.3, "name":"new_data_field", "message": {"existing_data_field": 42, "new_data_field": "foo"}, "db": MY_DATABASE", "schema": "MY_SCHEMA", "table": "MY_TABLE"}`).Default(defaultSchemaEvolutionNewColumnMapping).Advanced(),
122122
service.NewProcessorListField(ssoFieldSchemaEvolutionProcessors).Description(`
123123
A series of processors to execute when new columns are added to the table. Specifying this can support running side effects when the schema evolves or enriching the message with additional message to guide the schema changes. For example, one could read the schema the message was produced with from the schema registry and use that to decide which type the new column in Snowflake should be.
124124
125-
The input to these processors is an object with the value and the name of the new column, the original message and table being written too. The metadata is unchanged from the original message that caused the schema to change. For example: `+"`"+`{"value": 42.3, "name":"new_data_field", "message": {"existing_data_field": 42, "new_data_field": "foo"}, "db": MY_DATABASE", "schema": "MY_SCHEMA", "table": "MY_TABLE"}`+"`").Optional(),
125+
The input to these processors is an object with the value and the name of the new column, the original message and table being written too. The metadata is unchanged from the original message that caused the schema to change. For example: `+"`"+`{"value": 42.3, "name":"new_data_field", "message": {"existing_data_field": 42, "new_data_field": "foo"}, "db": MY_DATABASE", "schema": "MY_SCHEMA", "table": "MY_TABLE"}`+"`").Optional().Advanced(),
126126
).Description(`Options to control schema evolution within the pipeline as new columns are added to the pipeline.`).Optional(),
127127
service.NewIntField(ssoFieldBuildParallelism).Description("The maximum amount of parallelism to use when building the output for Snowflake. The metric to watch to see if you need to change this is `snowflake_build_output_latency_ns`.").Optional().Advanced().Deprecated(),
128128
service.NewObjectField(ssoFieldBuildOpts,

0 commit comments

Comments
 (0)