Repository navigation
Description
Version: clickhouse-backup 2.8.1 (altinity/clickhouse-backup:2.8.1). The code is the same on master (95313ed).
Setup: S3, compression_format: tar, download_by_part: true (default), use_resumable_state: true, ClickHouse 25.12.
What happens
We downloaded an incremental backup, mainnet-20261019T063856Z-incremental, whose required_backup is mainnet-20261015T061441Z-full. We stopped the download (stopped its container) while the downloads of 4 required parts from the full backup were still in progress, streaming or waiting for a retry, then started the same download again. It resumed and finished without an error, but those 4 parts were incomplete:
20261001_29939_40045_8 files=29 size=3.3G checksums.txt=NO columns.txt=NO
20261001_42394_44582_7 files=11 size=276M checksums.txt=NO columns.txt=NO
20261001_40046_42393_7 files=254 size=1.4G checksums.txt=yes columns.txt=yes (complete part, for comparison)
restore --data then failed:
FTL cmd/clickhouse-backup/main.go:1364 > , error=RestoreData: restoreData: one of restoreDataRegular go-routine return error: restoreDataRegularByParts: can't attach data parts for table '<db>._final_validator_dashboard_roi_daily': AttachDataParts: attach part: code: 226, message: Marks file '/var/lib/clickhouse/store/66b/66b4cda7-.../detached/attaching_20260101_15_15_49/data.cmrk4' doesn't exist: Part contains files: minmax_t.idx (8 bytes), checksums.txt (646 bytes), partition.dat (2 bytes), primary.cidx (91507 bytes), columns.txt (178 bytes), default_compression_codec.txt (10 bytes), columns_substreams.txt (239 bytes), count.txt (9 bytes), metadata_version.txt (1 bytes), roi_dividend.bin (607819608 bytes): Part is empty
The 4 broken parts were exactly the 4 whose DownloadCompressedStream hadn't finished when the first run stopped. All other parts attached correctly.
Why
downloadDiffParts downloads a required part into the required backup's local directory (existsPath = backup/<required>/shadow/<db>/<table>/<disk>/<part>) and then hard-links it into the incremental backup (newPath). When existsPath already exists, it hard-links it without checking it, unless the resumable state already lists it as processed:
} else {
existsAlreadyProcessed := false
if b.resume {
if existsAlreadyProcessed, statErr = b.resumableState.IsAlreadyProcessedBool(existsPath); statErr != nil {
return 0, errors.Wrap(statErr, "resumableState.IsAlreadyProcessedBool")
}
}
if !b.resume || !existsAlreadyProcessed {
if statErr = b.makePartHardlinks(existsPath, newPath); statErr != nil {
return 0, errors.Wrap(statErr, "can't to add exists part")
}
}
}A part not yet listed as processed is exactly the case of an interrupted run. DownloadCompressedStream extracts straight into the final part directory and leaves what it wrote when it fails or is cancelled, so the half-extracted directory passes os.Stat. Read from the code, the same applies without --resume: after delete local <incremental>, a new download <incremental> reuses the leftover directory of the required backup too. Only delete local <required backup> (listed as broken metadata.json not found) before downloading again avoids it.
Reproduce
Deterministic, derived from the code path above (we haven't run this exact sequence):
create_remote full, insert more rows,create_remote --diff-from-remote=full incr, thendelete localboth.- Fake an interrupted extraction of one part that
incrmarksrequired: create/var/lib/clickhouse/backup/full/shadow/<db>/<table>/default/<part>/with a single file in it. clickhouse-backup download incrfinishes without an error.clickhouse-backup restore incrfails to attach that part.
The way we hit it: kill download incr while the log shows DownloadCompressedStream full/shadow/... for a large part, then run download incr again with use_resumable_state: true.
Code (v2.8.1)
pkg/backup/download.go#L1868-L1872:existsPathand itsos.Stat.pkg/backup/download.go#L1965-L1977: the branch that hard-links an existingexistsPathwithout checking it.pkg/backup/download.go#L2238makePartHardlinks: links whatever files are there.pkg/storage/general.go#L547-L593DownloadCompressedStream: extracts into the final directory and doesn't remove partial files on error.
Proposed fix
- Extract atomically:
DownloadCompressedStreamextracts into a temporary directory beside the target (for example<localPath>/.tmp_<random>/) and renames the part directory into place only afterExtractreturned without an error; on error it removes the temporary directory. An existing part directory is then always complete, for required and own parts alike. - Don't trust a bare
os.StatindownloadDiffParts: whenexistsPathexists but isn't listed as processed, remove it and download the part like a missing one. Where the table metadata carries the part'ssize(properly calculate free space before download for --hardlink-exists-files #1268), comparing it with the size on disk is a cheap extra check before hard-linking.
Related: #1384 (resume on incremental backups), #1178 (clean_local_broken).
Activity
RyoKusnadi commentedon Nov 3, 2026
@Slach I opened #1665 for this. Could you please review it when you have a moment? Thank you!