IBM watsonx.data destination connector: improve performance with partitioning, metadata cleanup, and increased retries #606
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e partitioning, periodic metadata cleanup, and increased connection retries
Paul-Cornell
changed the title
Paul-Cornell
requested review from
MKhalusova,
mpolomdeepsense,
tarunnv-uio and
ajay23-uns
| "id" varchar, | ||
| "record_id" varchar, | ||
| "parent_id" varchar | ||
| ) | ||
| WITH ( | ||
| delete_mode = 'copy-on-write', | ||
| format = 'PARQUET', | ||
| format_version = '2' | ||
| format_version = '2', | ||
| partitioning = ARRAY['record_id'] |
There was a problem hiding this comment.
@potter-potter We decided not to use partitioning right? We can probably remove this line.
But still I think we should include a section about partitioning, it could simply just say what partitioning is used for and that it can slightly increase the performance (3-4 sentences). And also add a link to docs about partitioning. and to Presto docs.
Comment on lines
194
to
+196
| format = 'PARQUET', | ||
| format_version = '2' | ||
| format_version = '2', | ||
| partitioning = ARRAY['record_id'] |
There was a problem hiding this comment.
We should probably state somewhere that this SQL command is using Presto SQL syntax.
https://prestodb.io/docs/current/connector/iceberg.html
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See the updated
CREATE TABLEstatement, the new Python script, and additions to Max Connection Retries and Max Retries, in https://unstructured-53-ibm-watsonxdata-2025-04-22.mintlify.app/ui/destinations/ibm-watsonxdata