Skip to content

[GLUTEN-12206][FLINK]Support filesystem writer bucket state snapshot and restore#12472

Open
KevinyhZou wants to merge 1 commit into
apache:mainfrom
KevinyhZou:support_fs_bucket_state3
Open

[GLUTEN-12206][FLINK]Support filesystem writer bucket state snapshot and restore#12472
KevinyhZou wants to merge 1 commit into
apache:mainfrom
KevinyhZou:support_fs_bucket_state3

Conversation

@KevinyhZou

@KevinyhZou KevinyhZou commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

support filesystem writer bucket state snapshot and restore

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

codex

some code in this pr copy from: #12318, we should wait that pr to be merged first.

Related issue: #12206

@KevinyhZou KevinyhZou marked this pull request as draft July 8, 2026 03:13
@github-actions github-actions Bot added the FLINK label Jul 8, 2026
@KevinyhZou KevinyhZou marked this pull request as ready for review July 13, 2026 02:12
@KevinyhZou KevinyhZou force-pushed the support_fs_bucket_state3 branch from 4415f05 to 885795c Compare July 13, 2026 10:10
@KevinyhZou KevinyhZou requested a review from lgbo-ustc July 13, 2026 10:11

@lgbo-ustc lgbo-ustc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code taste / maintainability suggestion (non-blocking):\n\n logs full restored checkpoint records at INFO. These records may contain many pending file paths / partition values, so this can be noisy and expose unnecessary path details. Also is always true because the field is initialized to an empty array.\n\nSuggestion: log only restored flag + record count at INFO, and either remove contents or move them to DEBUG.\n\nExample:\n\n\n\nI also ran under ; it passed.

@lgbo-ustc lgbo-ustc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction: the previous review body was mangled by shell Markdown expansion. Please use this version instead.

Code taste / maintainability suggestion (non-blocking):

GlutenStreamingFileWriterOperator.initializeNativeState() logs full restored checkpoint records at INFO. These records may contain many pending file paths / partition values, so this can be noisy and expose unnecessary path details. Also restoredCheckpointRecords != null is always true because the field is initialized to an empty array.

Suggestion: log only restored flag + record count at INFO, and either remove contents or move them to DEBUG.

Example:

LOG.info(
    "Restore native file writer state for operator {}, restored {}, records {}",
    getDescription(),
    context.isRestored(),
    restoredCheckpointRecords.length);

I also ran ../build/mvn -pl runtime,ut -DskipTests spotless:check under gluten-flink/; it passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants