Skip to content

Auto-detect ClickHouse Cloud backup layout in list remote and restore_remote - #1576

Merged
Slach merged 3 commits into
masterfrom
issue-1574
Sep 24, 2026
Merged

Slach merged 3 commits into
masterfrom
issue-1574

Conversation

@Slach

@Slach Slach commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1574

Problem

A backup created by native BACKUP ... TO S3 / AzureBlobStorage (ClickHouse Cloud) under s3->path / azblob->path has <name>/.backup but no metadata.json:

  • list remote showed it as broken (can't stat metadata.json);
  • clean_remote_broken deleted it and backups_to_keep_remote retention counted and could delete it;
  • restore_remote <name> failed, the user had to know about restore_cloud <path>/<name>.

Changes

  • BackupList (slow and fast path): only when metadata.json is missing, StatFile(<name>/.backup) detects the native layout. The manifest is streamed with xml.Decoder.Token() (constant memory, the manifest has one <file> per logical file and no totals) to get <timestamp> and the stored data size, computed with the size_of_entries formula of BackupImpl::writeBackupMetadata (deduplicated and use_base bytes excluded). The entry gets data_format: cloud and is cached. A read/parse error only logs a warning and keeps the entry non-broken (not cached), so clean_remote_broken never deletes it. A cached cloud entry is re-checked with a HEAD request on metadata.json, so an embedded create_remote listed between BACKUP and the metadata.json upload is not stuck as cloud.
  • restore_remote (CLI and POST /backup/restore_remote/{name}) switches to RestoreCloud with prefix <resolved path>/<name>. It maps --tables, --partitions, --skip-empty-tables, --rm/--drop and general.restore_schema_on_cluster, and supports --dry-run. It works only with remote_storage: s3|azblob. Options restore_cloud can't apply (mappings, --schema, --data, --rbac*, --configs*, --named-collections*, --resume, --streaming, …) fail explicitly.
  • download refuses cloud-layout backups with a hint.
  • RemoveOldBackupsRemote retention skips cloud-layout backups, since clickhouse-backup did not create them. clean_remote_broken skips them because they are no longer broken. An explicit delete remote still works.
  • restore_remote help description, ReadMe and Manual.

Incremental cloud backups: base chain from <base_backup>

<base_backup> in .backup is BackupInfo::toString(), i.e. formatWithSecretsOneLine, so it contains the credentials (S3('url','key','secret'), S3('url', extra_credentials(role_arn=...)), AzureBlobStorage('<conn string with AccountKey>','container','path')). RESTORE uses it when SETTINGS base_backup is not given (BackupImpl::readBackupMetadata).

  • storage.ParseCloudBackupLocation extracts the engine and the location, and collects the credentials for redaction. KeyIn(bucket|container) handles S3 path-style and virtual-hosted URLs, GCS, MinIO, s3://, and Azure container/blob_path.
  • list remote: if the base lives in the same bucket/container under the same path, the backup gets RequiredBackup and shows +<base>, and the existing delete remote guard protects the base. A base stored elsewhere is shown as base=<url> in the description, without credentials.
  • restore_cloud / restore_remote: the base chain is resolved from each .backup header, which is streamed only until <contents>, and each link is verified by base_backup_uuid. DDL blobs with use_base are looked up along the chain. RESTORE gets SETTINGS base_backup = S3(<resolved url>, <configured creds>) / AzureBlobStorage(...), so the base is read with our credentials and not with the ones embedded in the manifest. Deeper bases are still read by ClickHouse from their own .backup. --base-prefix stays as an override of the first base, for backups copied to another bucket/prefix, and is still honored for manifests without <base_backup>.

Tests

  • Unit: TestParseCloudBackupLocation, TestCloudBaseBackupName, TestBackupListCloudBaseBackup, TestReadCloudBackupHeader, TestResolveCloudBaseChain (chain, override, uuid mismatch, other bucket, missing base, no secrets in errors), TestFetchCloudBlobBaseChain. Also TestParseCloudBackupSummary covers the FirstFileName/checksum generators, dedup, use_base/base_size and deduplicate_files=0. TestBackupListDetectsCloudBackup covers regular, embedded (.backup + metadata.json), cloud and broken entries, the fast path and the stale cloud cache entry. TestCloudUnsupportedRestoreRemoteOptions covers the unsupported-option list.
  • E2E TestRestoreRemoteCloudAutoDetect, against a real ClickHouse Cloud BACKUP TO S3 next to a regular create_remote in the same s3.path:
    • backups_to_keep_remote: 1 retention keeps the cloud backup;
    • list remote shows cloud with a size, not broken;
    • clean_remote_broken keeps it;
    • download fails;
    • --restore-table-mapping fails;
    • restore_remote restores 10000 rows into ReplicatedMergeTree;
    • --rm --partitions=0,1 restores 5000 rows;
    • POST /backup/restore_remote succeeds; then an incremental Cloud BACKUP ... SETTINGS base_backup into the same path shows +<base> in list remote, delete remote <base> is refused, and both restore_remote <incremental> and restore_cloud --drop <incremental prefix> (without --base-prefix) restore 15000 rows.
  • GOFLAGS= make test passes. RUN_TESTS='TestRestoreRemoteCloudAutoDetect' ./test/integration/run.sh passes, and TestRestoreCloudS3, TestRestoreCloudGCS and TestRestoreCloudAzblob pass as regression checks.

Note, pre-existing and not changed here: ClickHouse.LogQuery logs every statement as is (INFO with clickhouse.log_sql_queries: true, DEBUG otherwise), so RESTORE/BACKUP ... S3('url','key','secret') credentials reach the log through it. restore_cloud's own log lines are redacted.

🤖 Generated with Claude Code

Slach and others added 3 commits September 24, 2026 11:41
…_remote

A backup written by native `BACKUP ... TO S3/AzureBlobStorage` (ClickHouse
Cloud) under s3->path / azblob->path has `.backup` but no metadata.json,
so `list remote` showed it as broken, `clean_remote_broken` and
backups_to_keep_remote retention could delete it, and `restore_remote`
failed.

BackupList now detects this layout only when metadata.json is missing:
the `.backup` manifest is streamed token by token (it has no totals and
can be hundreds of MB) to get the timestamp and the stored data size
(size_of_entries formula of BackupImpl::writeBackupMetadata), the entry
gets data_format `cloud` and is cached; a cached cloud entry is dropped
once metadata.json appears (embedded create_remote writes .backup first).

restore_remote switches to restore_cloud for such backups (s3 / azblob
only, unsupported options fail explicitly), download refuses them and
retention skips them.

Fixes #1574

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`<base_backup>` of an incremental native BACKUP holds the base location
with credentials (BackupInfo::toString uses formatWithSecretsOneLine),
and RESTORE reads the base from it unless SETTINGS base_backup is set.

- list remote: a cloud backup whose base lives under the same remote
  path gets RequiredBackup (`+<base>`, the delete guard protects the
  base), a base stored elsewhere is shown as `base=<url>` without
  credentials
- restore_cloud / restore_remote: the base chain is resolved from each
  `.backup` header (streamed until <contents>) and verified by
  base_backup_uuid, DDL blobs with use_base are looked up along the
  chain, RESTORE gets SETTINGS base_backup with the configured
  credentials; --base-prefix stays as an override of the first base
  for backups copied to another bucket/prefix

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 35968870322

Coverage increased (+1.7%) to 69.458%

Details

  • Coverage increased (+1.7%) from the base build.
  • Patch coverage: 65 uncovered changes across 5 files (359 of 424 lines covered, 84.67%).
  • 31 coverage regressions across 6 files.

Uncovered Changes

File Changed Covered %
pkg/backup/restore_cloud.go 134 106 79.1%
pkg/storage/cloud_backup_location.go 118 101 85.59%
pkg/storage/cloud_backup.go 120 108 90.0%
pkg/backup/restore_remote.go 23 19 82.61%
pkg/storage/general.go 20 16 80.0%
Total (7 files) 424 359 84.67%

Coverage Regressions

31 previously-covered lines in 6 files lost coverage.

File Lines Losing Coverage Coverage
pkg/storage/gcs.go 15 54.46%
pkg/clickhouse/clickhouse.go 8 80.63%
pkg/backup/freezes.go 3 72.95%
pkg/backup/upload.go 2 72.85%
pkg/storage/structs.go 2 81.82%
pkg/status/status.go 1 90.91%

Coverage Stats

Coverage Status
Relevant Lines: 27837
Covered Lines: 19335
Line Coverage: 69.46%
Coverage Strength: 34974.28 hits per line

💛 - Coveralls

@Slach
Slach merged commit 156d289 into master Sep 24, 2026
60 of 90 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add auto-detection of backup layout (cloud/regular) for list remote + restore_remote (switch to restore_cloud)

2 participants