Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(datasets): Refactored the ManagedTableDataset by Separating Out Common Table Logic #827

Merged
merged 55 commits into from
Oct 10, 2024
Merged

feat(datasets): Refactored the ManagedTableDataset by Separating Out Common Table Logic #827

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
fd0de72
added a base table implementation
MinuraPunchihewa Aug 26, 2024
0392aba
added a base table dataset implementation
MinuraPunchihewa Aug 26, 2024
38c470f
renamed the module with the base classes
MinuraPunchihewa Aug 26, 2024
d424829
refactored ManagedTable using BaseTable
MinuraPunchihewa Aug 26, 2024
7091986
refactored ManagedTableDataset using BaseTableDataset
MinuraPunchihewa Aug 26, 2024
081d5d1
removed primary_key attr from BaseTable
MinuraPunchihewa Aug 26, 2024
e2cd580
updated the format attrs of ManagedTable
MinuraPunchihewa Aug 26, 2024
33814be
implemented the _load() method of ManagedTableDataset
MinuraPunchihewa Aug 26, 2024
6f91dc8
removed unnecessary imports
MinuraPunchihewa Aug 26, 2024
410cf4a
reorganized the attrs of BaseTable
MinuraPunchihewa Aug 27, 2024
1bf0b8b
implemented create_table() in ManagedTableDataset
MinuraPunchihewa Aug 27, 2024
ee7e073
added the version attr to BaseTableDataset and updated load()
MinuraPunchihewa Aug 27, 2024
d87f4ed
updated the base and managed datasets with all attrs
MinuraPunchihewa Aug 29, 2024
a4feeff
updated the supported formats
MinuraPunchihewa Aug 29, 2024
0db5639
added external table and external table dataset implementations
MinuraPunchihewa Aug 29, 2024
e19eba4
added a val func to check for format when using upsert mode
MinuraPunchihewa Aug 29, 2024
2c46978
imported the ExternalTableDataset into the main pkg
MinuraPunchihewa Sep 3, 2024
2777ea1
improved the docstrings in the code
MinuraPunchihewa Sep 3, 2024
6e1b17e
added format to the _describe()
MinuraPunchihewa Sep 3, 2024
039f9e0
updated the save methods to incorporate partition columns
MinuraPunchihewa Sep 4, 2024
fb87133
reverted the default write_mode back to None
MinuraPunchihewa Sep 4, 2024
b7a5c33
extended the _validate_write_mode() func to include formats
MinuraPunchihewa Sep 4, 2024
577dd91
updated the save() logic to work with single or multiple partition cols
MinuraPunchihewa Sep 8, 2024
6e729e3
updated the docstrings for the datasets with missing attrs
MinuraPunchihewa Sep 8, 2024
cbb8514
introduced a location attr for creating ext tables
MinuraPunchihewa Sep 8, 2024
4a2dc6e
updated the save funcs to incorporate the locations attr
MinuraPunchihewa Sep 8, 2024
1734f44
moved the func to check if table exists to BaseTable
MinuraPunchihewa Sep 8, 2024
b03bee8
added a val func to check if location is provided if table does not e…
MinuraPunchihewa Sep 8, 2024
dc3550e
moved the val func for checking if write_mode supported to ExternalTable
MinuraPunchihewa Sep 8, 2024
78eee4d
removed the func for adding options to writer for better readability
MinuraPunchihewa Sep 21, 2024
09a8cb2
added a validation check for overwrites on ext tables
MinuraPunchihewa Sep 21, 2024
7e493c2
implemented the _save_overwrite() func for ext tables
MinuraPunchihewa Sep 21, 2024
6343226
removed mentions of a default write mode
MinuraPunchihewa Sep 21, 2024
ccc60c4
improved the docstrings
MinuraPunchihewa Sep 21, 2024
99202ee
fixed lint issues
MinuraPunchihewa Sep 21, 2024
31f385a
fixed a couple of bugs in the Table classes
MinuraPunchihewa Sep 22, 2024
5016dee
updated the _save_overwrite() logic for ext tables
MinuraPunchihewa Sep 22, 2024
77afc50
renamed the val funcs of the ext tables
MinuraPunchihewa Sep 24, 2024
e7ba0e3
updated _save_overwrite() of ext tables to handle no existing tables
MinuraPunchihewa Sep 24, 2024
d6ba426
fixed bug in supporting string partition cols
MinuraPunchihewa Sep 25, 2024
73dc1eb
removed the external table dataset
MinuraPunchihewa Oct 1, 2024
fef688f
removed irrelevant attrs from describe() for managed tables
MinuraPunchihewa Oct 1, 2024
8ee9ea8
preserved order of args
MinuraPunchihewa Oct 1, 2024
4cd7c0f
updated the tests for base table dataset and managed table dataset
MinuraPunchihewa Oct 4, 2024
9982817
initialized the base table in the base table dataset
MinuraPunchihewa Oct 4, 2024
5a05a34
fixed lint issues
MinuraPunchihewa Oct 4, 2024
35bbe3b
Merge branch 'main' into feature/databricks_external_dataset
MinuraPunchihewa Oct 4, 2024
c5d68ea
fixed an incorrect type hint
MinuraPunchihewa Oct 4, 2024
e651008
Merge branch 'main' into feature/databricks_external_dataset
MinuraPunchihewa Oct 7, 2024
b6dc5b9
removed redundant check on save()
MinuraPunchihewa Oct 7, 2024
c5ea9d9
added the missing unit tests
MinuraPunchihewa Oct 7, 2024
0558d96
fixed the tests for saving external tables
MinuraPunchihewa Oct 7, 2024
145a2cf
Merge branch 'main' into feature/databricks_external_dataset
ankatiyar Oct 10, 2024
d598d79
lint
ankatiyar Oct 10, 2024
e34715a
Merge main
ankatiyar Oct 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading