-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhance Schema adapter to accommodate evolving struct #15295
Open
kosiew
wants to merge
55
commits into
apache:main
Choose a base branch
from
kosiew:schema-adapter
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,044
−13
Open
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit
Hold shift + click to select a range
c8236ed
feat: implement NestedStructSchemaAdapter for handling schema evoluti…
kosiew afbe1ed
feat: enhance NestedStructSchemaAdapter with schema mapping capabilities
kosiew c774cab
test: add schema mapping test for NestedStructSchemaAdapter
kosiew 5f5cd45
feat: implement NestedStructSchemaAdapterFactory for handling nested …
kosiew 6065bc1
test: add unit test for NestedStructSchemaAdapterFactory to validate …
kosiew 410f8d7
test: refactor test_create_appropriate_adapter for clarity and effici…
kosiew 50cf134
feat: enhance create_appropriate_adapter to support nested schema tra…
kosiew 3f52617
refactor: simplify create_appropriate_adapter logic for nested schema…
kosiew ad74d3a
refactor: remove redundant default adapter test in nested schema adapter
kosiew 134dace
feat: enhance NestedStructSchemaAdapter to support additional table s…
kosiew aa89671
refactor: simplify test_nested_struct_evolution
kosiew f361311
refactor: streamline schema creation in nested schema adapter tests
kosiew a914a6b
Fix clippy errors
kosiew d8eb3eb
test: add async test for schema evolution with compaction in NestedSt…
kosiew 1735b45
refactor: add missing imports and clean up test code in nested_schema…
kosiew 72aee85
Rollback to before adding test_datafusion_schema_evolution_with_compa…
kosiew 772fbce
feat: add nested_struct.rs to test nested schema evolution test with …
kosiew 20af2c0
chore: remove nested_struct.rs example file to streamline repository …
kosiew 3c0844c
feat: Add nested_struct.rs async function for schema evolution with c…
kosiew ad09e60
feat: Enhance logging in nested_struct.rs for better traceability 📜✨
kosiew 61f1f6e
created helper functions
kosiew 16a47d3
map batch1 to schema2
kosiew 7b7183e
feat: Enhance NestedStructSchemaAdapter with custom schema mapping fo…
kosiew 84ab195
feat: Add debug print statements to map_batch for tracing execution f…
kosiew 51dacc5
fix: Refactor nested schema mapping for improved error handling and c…
kosiew aa5128a
refactor: Remove debug print statements for cleaner code execution 🧹✨
kosiew 839bf61
nested_struct - plug adapter into ListingTableConfig
kosiew 2e99158
feat: Add optional schema adapter factory to ListingTableConfig for e…
kosiew fe7ff84
feat: Add optional schema adapter factory to FileScanConfig for enhan…
kosiew 3689140
feat: Enhance ListingTableConfig to support schema adapter factory fo…
kosiew 76fbc6f
struct NestedStructSchemaMapping - remove table_schema, file_schema
kosiew f2d6b60
refactor: Remove nested_struct.rs example for schema evolution and co…
kosiew 6b7fed9
style: Fix comment tests in ListingOptions documentation 📜✨
kosiew 2cef654
Merge branch 'main' into test-merge
kosiew 565ad5c
SchemaMapping remove table_schema, nested_schema_adapter remove map_p…
kosiew 778da1e
docs: Update comments for schema_adapter_factory in ListingTableConfi…
kosiew f066e59
refactor: Extract schema adapter preservation logic into a helper fun…
kosiew 4cc5f77
refactor: Extract schema adapter application logic into a dedicated f…
kosiew b6a828c
docs: Enhance adapt_fields documentation with performance considerati…
kosiew 41fb40c
docs: Add detailed documentation for RecordBatch mapping in NestedStr…
kosiew 3133cd7
refactor: Add missing import for FileSource in ListingTable implement…
kosiew 5ad6287
refactor: Update license documentation comments for NestedSchemaAdapt…
kosiew 8fa34da
refactor: Remove unused file_scan_exec.rs to clean up the codebase 🗑️✨
kosiew d229dd3
refactor: Remove unused file_scan_config.rs to streamline the codebas…
kosiew ff41c43
Moved the adapt_column method from NestedStructSchemaMapping to a sta…
kosiew 2df74b6
Fix Clippy errors
kosiew bb4a5de
docs: Correct the struct names in documentation for NestedStructSchem…
kosiew a8cce59
Merge branch 'main' into schema-adapter
kosiew f547355
fix: remove unnecessary clone in create_physical_plan call for Listin…
kosiew fa7c17f
refactor: rename preserve_schema_adapter_factory to preserve_conf_sch…
kosiew e9c93d6
refactor: rename create_appropriate_adapter to create_adapter for cla…
kosiew 64a4e3f
feature gate parquet
kosiew dd9f66d
Trigger CI
kosiew ca511df
refactor: mod tests, add user_infos
kosiew 54590f4
feat: expose nested schema adapter and source for improved data handl…
kosiew File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Fix clippy errors
commit a914a6bc9c50f5d57756533c340d08db0709651f
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you also add another field called
user_infos
which is a List<user_info>? This is another common evolving struct use case that's good to support.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added user_infos to the tests