refactor: input and output abstractions (WIP) #715
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| description="Number of errors while writing to files", | ||
| ) | ||
| logger.error(f"Error writing pandas dataframe to files: {str(e)}") | ||
| raise |
There was a problem hiding this comment.
Bug: Missing Attributes in Output Class
The write_dataframe method in the Output base class, now a concrete implementation, attempts to use attributes like chunk_part and metrics, and calls methods such as path_gen, _flush_buffer, and _upload_file. These members are not defined in the Output base class, which causes AttributeErrors when the method executes.
| # Get the generated file path and rename to final location | ||
| result_dict = result.to_pydict() | ||
| generated_file = result_dict["path"][0] | ||
| os.rename(generated_file, consolidated_file_path) |
There was a problem hiding this comment.
Bug: Consolidation Fails on Empty Daft Output
The consolidation logic in _consolidate_current_folder assumes daft_df.write_parquet always returns a dictionary with a "path" key containing a non-empty list. If Daft's output structure changes or is empty, accessing result_dict["path"][0] could cause a KeyError or IndexError.
Additional Locations (1)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changelog
Additional context (e.g. screenshots, logs, links)
Checklist
Copyleft License Compliance