feat(datasets): Refactored the ManagedTableDataset by Separating Out …

…Common Table Logic (#827)

* added a base table implementation

Signed-off-by: Minura Punchihewa <[email protected]>

* added a base table dataset implementation

Signed-off-by: Minura Punchihewa <[email protected]>

* renamed the module with the base classes

Signed-off-by: Minura Punchihewa <[email protected]>

* refactored ManagedTable using BaseTable

Signed-off-by: Minura Punchihewa <[email protected]>

* refactored ManagedTableDataset using BaseTableDataset

Signed-off-by: Minura Punchihewa <[email protected]>

* removed primary_key attr from BaseTable

Signed-off-by: Minura Punchihewa <[email protected]>

* updated the format attrs of ManagedTable

Signed-off-by: Minura Punchihewa <[email protected]>

* implemented the _load() method of ManagedTableDataset

Signed-off-by: Minura Punchihewa <[email protected]>

* removed unnecessary imports

Signed-off-by: Minura Punchihewa <[email protected]>

* reorganized the attrs of BaseTable

Signed-off-by: Minura Punchihewa <[email protected]>

* implemented create_table() in ManagedTableDataset

Signed-off-by: Minura Punchihewa <[email protected]>

* added the version attr to BaseTableDataset and updated load()

Signed-off-by: Minura Punchihewa <[email protected]>

* updated the base and managed datasets with all attrs

Signed-off-by: Minura Punchihewa <[email protected]>

* updated the supported formats

Signed-off-by: Minura Punchihewa <[email protected]>

* added external table and external table dataset implementations

Signed-off-by: Minura Punchihewa <[email protected]>

* added a val func to check for format when using upsert mode

Signed-off-by: Minura Punchihewa <[email protected]>

* imported the ExternalTableDataset into the main pkg

Signed-off-by: Minura Punchihewa <[email protected]>

* improved the docstrings in the code

Signed-off-by: Minura Punchihewa <[email protected]>

* added format to the _describe()

Signed-off-by: Minura Punchihewa <[email protected]>

* updated the save methods to incorporate partition columns

Signed-off-by: Minura Punchihewa <[email protected]>

* reverted the default write_mode back to None

Signed-off-by: Minura Punchihewa <[email protected]>

* extended the _validate_write_mode() func to include formats

Signed-off-by: Minura Punchihewa <[email protected]>

* updated the save() logic to work with single or multiple partition cols

Signed-off-by: Minura Punchihewa <[email protected]>

* updated the docstrings for the datasets with missing attrs

Signed-off-by: Minura Punchihewa <[email protected]>

* introduced a location attr for creating ext tables

Signed-off-by: Minura Punchihewa <[email protected]>

* updated the save funcs to incorporate the locations attr

Signed-off-by: Minura Punchihewa <[email protected]>

* moved the func to check if table exists to BaseTable

Signed-off-by: Minura Punchihewa <[email protected]>

* added a val func to check if location is provided if table does not exist

Signed-off-by: Minura Punchihewa <[email protected]>

* moved the val func for checking if write_mode supported to ExternalTable

Signed-off-by: Minura Punchihewa <[email protected]>

* removed the func for adding options to writer for better readability

Signed-off-by: Minura Punchihewa <[email protected]>

* added a validation check for overwrites on ext tables

Signed-off-by: Minura Punchihewa <[email protected]>

* implemented the _save_overwrite() func for ext tables

Signed-off-by: Minura Punchihewa <[email protected]>

* removed mentions of a default write mode

Signed-off-by: Minura Punchihewa <[email protected]>

* improved the docstrings

Signed-off-by: Minura Punchihewa <[email protected]>

* fixed lint issues

Signed-off-by: Minura Punchihewa <[email protected]>

* fixed a couple of bugs in the Table classes

Signed-off-by: Minura Punchihewa <[email protected]>

* updated the _save_overwrite() logic for ext tables

Signed-off-by: Minura Punchihewa <[email protected]>

* renamed the val funcs of the ext tables

Signed-off-by: Minura Punchihewa <[email protected]>

* updated _save_overwrite() of ext tables to handle no existing tables

Signed-off-by: Minura Punchihewa <[email protected]>

* fixed bug in supporting string partition cols

Signed-off-by: Minura Punchihewa <[email protected]>

* removed the external table dataset

Signed-off-by: Minura Punchihewa <[email protected]>

* removed irrelevant attrs from describe() for managed tables

Signed-off-by: Minura Punchihewa <[email protected]>

* preserved order of args

Signed-off-by: Minura Punchihewa <[email protected]>

* updated the tests for base table dataset and managed table dataset

Signed-off-by: Minura Punchihewa <[email protected]>

* initialized the base table in the base table dataset

Signed-off-by: Minura Punchihewa <[email protected]>

* fixed lint issues

Signed-off-by: Minura Punchihewa <[email protected]>

* fixed an incorrect type hint

Signed-off-by: Minura Punchihewa <[email protected]>

* removed redundant check on save()

Signed-off-by: Minura Punchihewa <[email protected]>

* added the missing unit tests

Signed-off-by: Minura Punchihewa <[email protected]>

* fixed the tests for saving external tables

Signed-off-by: Minura Punchihewa <[email protected]>

* lint

Signed-off-by: Ankita Katiyar <[email protected]>

---------

Signed-off-by: Minura Punchihewa <[email protected]>
Signed-off-by: Minura Punchihewa <[email protected]>
Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: Ankita Katiyar <[email protected]>
Co-authored-by: Ankita Katiyar <[email protected]>
Co-authored-by: Ankita Katiyar <[email protected]>