[GLUTEN-12206][FLINK]Support filesystem writer bucket state snapshot and restore#12472
[GLUTEN-12206][FLINK]Support filesystem writer bucket state snapshot and restore#12472KevinyhZou wants to merge 1 commit into
Conversation
4415f05 to
885795c
Compare
lgbo-ustc
left a comment
There was a problem hiding this comment.
Code taste / maintainability suggestion (non-blocking):\n\n logs full restored checkpoint records at INFO. These records may contain many pending file paths / partition values, so this can be noisy and expose unnecessary path details. Also is always true because the field is initialized to an empty array.\n\nSuggestion: log only restored flag + record count at INFO, and either remove contents or move them to DEBUG.\n\nExample:\n\n\n\nI also ran under ; it passed.
lgbo-ustc
left a comment
There was a problem hiding this comment.
Correction: the previous review body was mangled by shell Markdown expansion. Please use this version instead.
Code taste / maintainability suggestion (non-blocking):
GlutenStreamingFileWriterOperator.initializeNativeState() logs full restored checkpoint records at INFO. These records may contain many pending file paths / partition values, so this can be noisy and expose unnecessary path details. Also restoredCheckpointRecords != null is always true because the field is initialized to an empty array.
Suggestion: log only restored flag + record count at INFO, and either remove contents or move them to DEBUG.
Example:
LOG.info(
"Restore native file writer state for operator {}, restored {}, records {}",
getDescription(),
context.isRestored(),
restoredCheckpointRecords.length);I also ran ../build/mvn -pl runtime,ut -DskipTests spotless:check under gluten-flink/; it passed.
What changes are proposed in this pull request?
support filesystem writer bucket state snapshot and restore
How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?
codex
some code in this pr copy from: #12318, we should wait that pr to be merged first.
Related issue: #12206