feat: Support bucket state in filesystem writer#69
Open
KevinyhZou wants to merge 4 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
为 FileSystem Writer 补齐与 Flink
StreamingFileSink/ Bucket 对齐的 checkpoint snapshot & restore 能力,使 failover / 重启后能恢复 writer 的 bucket 状态(尤其是partCounter与未提交 pending 文件),避免 part 文件号回退导致覆盖已提交文件,并保证 barrier 之后的写出不会被错误提交到更早的 checkpoint。Changes
Connector / Operator API
DataSink上扩展snapshot()/restoreState()接口,用于按 checkpoint 封印写出并恢复 sink 状态。TableWriter将 stateful 算子的snapshotState/restoreState/commit转发到底层DataSink。FileSystemDataSink
snapshot(checkpointId):关闭当前 writers,把 pending writer 归入对应 checkpoint,并序列化 bucket state(JSON)。restoreState(checkpointRecords):从 checkpoint 记录恢复:maxPartCounter(对齐 FlinkpartCounterState)pendingFilespartCounter的 state 格式(restore 时取 max)FsFileNameGenerator:仅 checkpoint 全局maxPartCounter;各 bucket 内存中的 partCounter 从该全局值初始化,防止 restore 后覆盖已提交 part 文件。commit(checkpointId):只提交属于该(及更早)checkpoint 的 pending 文件;barrier 后新写入不进入已 snapshot 的 checkpoint。Tests
FileSystemConnectorTest覆盖:partCounter延续Notes
Checkpoint state 格式与 Flink StreamingFileSink 对齐:checkpoint 全局
partCounter+pendingFiles;partition commit policy(如_SUCCESS/ metastore)仍由 Flink 侧基于commit()返回的路径处理。