Skip to content

download of an incremental backup hard-links required parts that an interrupted run left half-extracted; restore then fails with Marks file ... doesn't exist #1633

Open

Description

@guybrush

Version: clickhouse-backup 2.8.1 (altinity/clickhouse-backup:2.8.1). The code is the same on master (95313ed).

Setup: S3, compression_format: tar, download_by_part: true (default), use_resumable_state: true, ClickHouse 25.12.

What happens

We downloaded an incremental backup, mainnet-20261019T063856Z-incremental, whose required_backup is mainnet-20261015T061441Z-full. We stopped the download (stopped its container) while the downloads of 4 required parts from the full backup were still in progress, streaming or waiting for a retry, then started the same download again. It resumed and finished without an error, but those 4 parts were incomplete:

20261001_29939_40045_8   files=29   size=3.3G  checksums.txt=NO   columns.txt=NO
20261001_42394_44582_7   files=11   size=276M  checksums.txt=NO   columns.txt=NO
20261001_40046_42393_7   files=254  size=1.4G  checksums.txt=yes  columns.txt=yes   (complete part, for comparison)

restore --data then failed:

FTL cmd/clickhouse-backup/main.go:1364 > , error=RestoreData: restoreData: one of restoreDataRegular go-routine return error: restoreDataRegularByParts: can't attach data parts for table '<db>._final_validator_dashboard_roi_daily': AttachDataParts: attach part: code: 226, message: Marks file '/var/lib/clickhouse/store/66b/66b4cda7-.../detached/attaching_20260101_15_15_49/data.cmrk4' doesn't exist: Part contains files: minmax_t.idx (8 bytes), checksums.txt (646 bytes), partition.dat (2 bytes), primary.cidx (91507 bytes), columns.txt (178 bytes), default_compression_codec.txt (10 bytes), columns_substreams.txt (239 bytes), count.txt (9 bytes), metadata_version.txt (1 bytes), roi_dividend.bin (607819608 bytes): Part is empty

The 4 broken parts were exactly the 4 whose DownloadCompressedStream hadn't finished when the first run stopped. All other parts attached correctly.

Why

downloadDiffParts downloads a required part into the required backup's local directory (existsPath = backup/<required>/shadow/<db>/<table>/<disk>/<part>) and then hard-links it into the incremental backup (newPath). When existsPath already exists, it hard-links it without checking it, unless the resumable state already lists it as processed:

} else {
	existsAlreadyProcessed := false
	if b.resume {
		if existsAlreadyProcessed, statErr = b.resumableState.IsAlreadyProcessedBool(existsPath); statErr != nil {
			return 0, errors.Wrap(statErr, "resumableState.IsAlreadyProcessedBool")
		}
	}
	if !b.resume || !existsAlreadyProcessed {
		if statErr = b.makePartHardlinks(existsPath, newPath); statErr != nil {
			return 0, errors.Wrap(statErr, "can't to add exists part")
		}
	}
}

A part not yet listed as processed is exactly the case of an interrupted run. DownloadCompressedStream extracts straight into the final part directory and leaves what it wrote when it fails or is cancelled, so the half-extracted directory passes os.Stat. Read from the code, the same applies without --resume: after delete local <incremental>, a new download <incremental> reuses the leftover directory of the required backup too. Only delete local <required backup> (listed as broken metadata.json not found) before downloading again avoids it.

Reproduce

Deterministic, derived from the code path above (we haven't run this exact sequence):

  1. create_remote full, insert more rows, create_remote --diff-from-remote=full incr, then delete local both.
  2. Fake an interrupted extraction of one part that incr marks required: create /var/lib/clickhouse/backup/full/shadow/<db>/<table>/default/<part>/ with a single file in it.
  3. clickhouse-backup download incr finishes without an error.
  4. clickhouse-backup restore incr fails to attach that part.

The way we hit it: kill download incr while the log shows DownloadCompressedStream full/shadow/... for a large part, then run download incr again with use_resumable_state: true.

Code (v2.8.1)

Proposed fix

  1. Extract atomically: DownloadCompressedStream extracts into a temporary directory beside the target (for example <localPath>/.tmp_<random>/) and renames the part directory into place only after Extract returned without an error; on error it removes the temporary directory. An existing part directory is then always complete, for required and own parts alike.
  2. Don't trust a bare os.Stat in downloadDiffParts: when existsPath exists but isn't listed as processed, remove it and download the part like a missing one. Where the table metadata carries the part's size (properly calculate free space before download for --hardlink-exists-files #1268), comparing it with the size on disk is a cheap extra check before hard-linking.

Related: #1384 (resume on incremental backups), #1178 (clean_local_broken).

Activity

  1. added this to the 2.8.3 milestone on Oct 22, 2026
  2. modified the milestones: 2.8.3, 2.8.4 on Oct 30, 2026
  3. RyoKusnadi commented on Nov 3, 2026

    @RyoKusnadi

    @Slach I opened #1665 for this. Could you please review it when you have a moment? Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions