Repository navigation
Auto-detect ClickHouse Cloud backup layout in list remote and restore_remote - #1576
Merged
Merged
Conversation
…_remote A backup written by native `BACKUP ... TO S3/AzureBlobStorage` (ClickHouse Cloud) under s3->path / azblob->path has `.backup` but no metadata.json, so `list remote` showed it as broken, `clean_remote_broken` and backups_to_keep_remote retention could delete it, and `restore_remote` failed. BackupList now detects this layout only when metadata.json is missing: the `.backup` manifest is streamed token by token (it has no totals and can be hundreds of MB) to get the timestamp and the stored data size (size_of_entries formula of BackupImpl::writeBackupMetadata), the entry gets data_format `cloud` and is cached; a cached cloud entry is dropped once metadata.json appears (embedded create_remote writes .backup first). restore_remote switches to restore_cloud for such backups (s3 / azblob only, unsupported options fail explicitly), download refuses them and retention skips them. Fixes #1574 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`<base_backup>` of an incremental native BACKUP holds the base location with credentials (BackupInfo::toString uses formatWithSecretsOneLine), and RESTORE reads the base from it unless SETTINGS base_backup is set. - list remote: a cloud backup whose base lives under the same remote path gets RequiredBackup (`+<base>`, the delete guard protects the base), a base stored elsewhere is shown as `base=<url>` without credentials - restore_cloud / restore_remote: the base chain is resolved from each `.backup` header (streamed until <contents>) and verified by base_backup_uuid, DDL blobs with use_base are looked up along the chain, RESTORE gets SETTINGS base_backup with the configured credentials; --base-prefix stays as an override of the first base for backups copied to another bucket/prefix Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Coverage Report for CI Build 35968870322Coverage increased (+1.7%) to 69.458%Details
Uncovered Changes
Coverage Regressions31 previously-covered lines in 6 files lost coverage.
Coverage Stats
💛 - Coveralls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1574
Problem
A backup created by native
BACKUP ... TO S3 / AzureBlobStorage(ClickHouse Cloud) unders3->path/azblob->pathhas<name>/.backupbut nometadata.json:list remoteshowed it asbroken (can't stat metadata.json);clean_remote_brokendeleted it andbackups_to_keep_remoteretention counted and could delete it;restore_remote <name>failed, the user had to know aboutrestore_cloud <path>/<name>.Changes
BackupList(slow and fast path): only whenmetadata.jsonis missing,StatFile(<name>/.backup)detects the native layout. The manifest is streamed withxml.Decoder.Token()(constant memory, the manifest has one<file>per logical file and no totals) to get<timestamp>and the stored data size, computed with thesize_of_entriesformula ofBackupImpl::writeBackupMetadata(deduplicated anduse_basebytes excluded). The entry getsdata_format: cloudand is cached. A read/parse error only logs a warning and keeps the entry non-broken (not cached), soclean_remote_brokennever deletes it. A cached cloud entry is re-checked with a HEAD request onmetadata.json, so an embeddedcreate_remotelisted betweenBACKUPand themetadata.jsonupload is not stuck ascloud.restore_remote(CLI andPOST /backup/restore_remote/{name}) switches toRestoreCloudwith prefix<resolved path>/<name>. It maps--tables,--partitions,--skip-empty-tables,--rm/--dropandgeneral.restore_schema_on_cluster, and supports--dry-run. It works only withremote_storage: s3|azblob. Options restore_cloud can't apply (mappings,--schema,--data,--rbac*,--configs*,--named-collections*,--resume,--streaming, …) fail explicitly.downloadrefuses cloud-layout backups with a hint.RemoveOldBackupsRemoteretention skips cloud-layout backups, since clickhouse-backup did not create them.clean_remote_brokenskips them because they are no longer broken. An explicitdelete remotestill works.restore_remotehelp description, ReadMe and Manual.Incremental cloud backups: base chain from
<base_backup><base_backup>in.backupisBackupInfo::toString(), i.e.formatWithSecretsOneLine, so it contains the credentials (S3('url','key','secret'),S3('url', extra_credentials(role_arn=...)),AzureBlobStorage('<conn string with AccountKey>','container','path')).RESTOREuses it whenSETTINGS base_backupis not given (BackupImpl::readBackupMetadata).storage.ParseCloudBackupLocationextracts the engine and the location, and collects the credentials for redaction.KeyIn(bucket|container)handles S3 path-style and virtual-hosted URLs, GCS, MinIO,s3://, and Azure container/blob_path.list remote: if the base lives in the same bucket/container under the samepath, the backup getsRequiredBackupand shows+<base>, and the existingdelete remoteguard protects the base. A base stored elsewhere is shown asbase=<url>in the description, without credentials.restore_cloud/restore_remote: the base chain is resolved from each.backupheader, which is streamed only until<contents>, and each link is verified bybase_backup_uuid. DDL blobs withuse_baseare looked up along the chain.RESTOREgetsSETTINGS base_backup = S3(<resolved url>, <configured creds>)/AzureBlobStorage(...), so the base is read with our credentials and not with the ones embedded in the manifest. Deeper bases are still read by ClickHouse from their own.backup.--base-prefixstays as an override of the first base, for backups copied to another bucket/prefix, and is still honored for manifests without<base_backup>.Tests
TestParseCloudBackupLocation,TestCloudBaseBackupName,TestBackupListCloudBaseBackup,TestReadCloudBackupHeader,TestResolveCloudBaseChain(chain, override, uuid mismatch, other bucket, missing base, no secrets in errors),TestFetchCloudBlobBaseChain. AlsoTestParseCloudBackupSummarycovers the FirstFileName/checksum generators, dedup,use_base/base_sizeanddeduplicate_files=0.TestBackupListDetectsCloudBackupcovers regular, embedded (.backup+metadata.json), cloud and broken entries, the fast path and the stale cloud cache entry.TestCloudUnsupportedRestoreRemoteOptionscovers the unsupported-option list.TestRestoreRemoteCloudAutoDetect, against a real ClickHouse CloudBACKUP TO S3next to a regularcreate_remotein the sames3.path:backups_to_keep_remote: 1retention keeps the cloud backup;list remoteshowscloudwith a size, not broken;clean_remote_brokenkeeps it;downloadfails;--restore-table-mappingfails;restore_remoterestores 10000 rows into ReplicatedMergeTree;--rm --partitions=0,1restores 5000 rows;POST /backup/restore_remotesucceeds; then an incremental CloudBACKUP ... SETTINGS base_backupinto the same path shows+<base>inlist remote,delete remote <base>is refused, and bothrestore_remote <incremental>andrestore_cloud --drop <incremental prefix>(without--base-prefix) restore 15000 rows.GOFLAGS= make testpasses.RUN_TESTS='TestRestoreRemoteCloudAutoDetect' ./test/integration/run.shpasses, andTestRestoreCloudS3,TestRestoreCloudGCSandTestRestoreCloudAzblobpass as regression checks.Note, pre-existing and not changed here:
ClickHouse.LogQuerylogs every statement as is (INFO withclickhouse.log_sql_queries: true, DEBUG otherwise), soRESTORE/BACKUP ... S3('url','key','secret')credentials reach the log through it. restore_cloud's own log lines are redacted.🤖 Generated with Claude Code