Skip to content

Commit 3a0692a

Browse files
authored
Merge pull request #1571 from Altinity/issue-928
Orchestrate embedded BACKUP/RESTORE ON CLUSTER across all nodes (#928)
2 parents 0ae1c95 + 09f5545 commit 3a0692a

23 files changed

Lines changed: 2676 additions & 1241 deletions

‎ChangeLog.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
11
# v2.8.2
2+
NEW FEATURES
3+
- `use_embedded_backup_restore_cluster` now orchestrates the whole cluster instead of only adding `ON CLUSTER` to the BACKUP / RESTORE SQL: `create`, `create_remote`, `restore`, `restore_remote` on the initiator node send the same command with the new `--embedded-on-cluster-worker` flag to every other node of the cluster through its `system.backup_actions` table (`clickhouse-backup server` with `api.create_integration_tables: true` is required on every node) and wait for them, a worker skips the BACKUP / RESTORE SQL and only describes, uploads, downloads and fixes its own `shards/N/replicas/M/` part of the shared destination, per table `metadata/<db>/<table>.json` moved under `shards/N/replicas/M/` next to the `.sql` of each node (such backups are tagged `embedded,cluster=<name>`, older backups without the tag keep the flat layout), `.backup`, `metadata.json`, rbac, configs and named collections stay on the initiator, `delete local` is propagated to the other nodes as `delete local --embedded-on-cluster-worker`, so `delete` accepts the flag too, a single node cluster behaves as before, fix [#928](https://github.com/Altinity/clickhouse-backup/issues/928)
4+
25
BUG FIXES
36
- `list` and `tables` text output is now written to stdout as a single write, so a log line on stderr can no longer land in the middle of a backup row when both streams are merged (`docker exec`, `2>&1`), the same fix `json` / `yaml` / `csv` / `tsv` formats got in v2.8.1
47
- `restore` / `restore_remote` with `--restore-database-mapping` / `--restore-table-mapping` of tables on object disks (`s3`, `azure_blob_storage`, including `cache` and `encrypted` wrappers) no longer leaves the restored copy sharing object storage keys with the source table, so `DROP TABLE|DATABASE ... SYNC` of the copy no longer deletes the objects of the source (`Code: 499 ... The specified key does not exist (S3_ERROR)` on the source afterwards): the key rewrite was skipped when the disk endpoint key prefix contained the backup name (e.g. backup `regression` on the disk `https://s3/bucket/regression-test/live/`), and the metadata files were linked into `detached` / the table directory before the keys were rewritten, so an interrupted restore left the copy pointing to the source keys; the rewritten key now also carries a hash of the destination table (`<dir>_<backup_name>_<crc32(db.table)>/<object>`), so several copies of one backup restored next to each other never share objects, the source key of the copy is always derived from the backup, so the same local backup can be restored several times, and metadata files are written via temp file + rename, so a repeated restore never modifies the parts of an already restored table through their shared hardlink, fix [#1568](https://github.com/Altinity/clickhouse-backup/issues/1568)

‎Manual.md‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ OPTIONS:
5555
--skip-check-parts-columns Skip check system.parts_columns to allow backup inconsistent column types for data parts
5656
--skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
5757
--resume use_embedded_backup_restore: true, --resumable use_embedded_backup_restore: true Will resume upload for object disk data, hard links on local disk still continue to recreate, not work when use_embedded_backup_restore: true
58+
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
5859
--dry-run Show tables count and data size which would be created, without creating
5960
--help, -h show help
6061
@@ -96,6 +97,7 @@ OPTIONS:
9697
--skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
9798
--delete, --delete-source, --delete-local explicitly delete local backup during upload
9899
--streaming Upload each table right after its freeze and delete its local copy, keeps only a small local footprint, https://github.com/Altinity/clickhouse-backup/issues/780
100+
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
99101
--dry-run Show tables count and data size which would be created and uploaded, without creating and uploading
100102
--help, -h show help
101103
@@ -129,6 +131,7 @@ OPTIONS:
129131
--skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
130132
--resume, --resumable Save intermediate upload state and resume upload if backup exists on remote storage, ignored with 'remote_storage: custom' or 'use_embedded_backup_restore: true'
131133
--delete, --delete-source, --delete-local explicitly delete local backup during upload
134+
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
132135
--dry-run Show tables count and data size which would be uploaded, without uploading
133136
--help, -h show help
134137
@@ -177,6 +180,7 @@ OPTIONS:
177180
--hardlink-exists-files Create hardlinks for existing files instead of downloading
178181
--disk-limit int Refuse download when usage of any local disk would exceed this percent (1-100) after download, overrides general->download_disk_limit, 0 means use config value, https://github.com/Altinity/clickhouse-backup/issues/1458 (default: 0)
179182
--allow-missing-files Skip data part files which are missing on remote storage (404/NoSuchKey) with an error log and drop them from local table metadata instead of failing, salvage mode for partially corrupted backups, overrides general->allow_missing_files_on_download, https://github.com/Altinity/clickhouse-backup/issues/1456
183+
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
180184
--dry-run Show tables count and data size which would be downloaded, without downloading
181185
--help, -h show help
182186
@@ -252,6 +256,7 @@ OPTIONS:
252256
--skip-empty-tables Skip restoring tables that have no data (empty tables with only schema)
253257
--rebind-replica-path-if-exists Override clickhouse.rebind_replica_path_if_exists, rebind a restored ReplicatedMergeTree to default_replica_path when the original ZK path still has leftover state but our replica entry is absent
254258
--drop-replica-if-exists Override clickhouse.drop_replica_if_exists, execute SYSTEM DROP REPLICA ... FROM ZKPATH ... and keep the original replication path when our own replica entry still exists in ZooKeeper but no local table uses it
259+
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
255260
--dry-run Show tables count and data size which would be restored, without restoring
256261
--help, -h show help
257262
@@ -298,6 +303,7 @@ OPTIONS:
298303
--streaming Restore each table right after its download and delete its local copy, keeps only a small local footprint, https://github.com/Altinity/clickhouse-backup/issues/780
299304
--rebind-replica-path-if-exists Override clickhouse.rebind_replica_path_if_exists, rebind a restored ReplicatedMergeTree to default_replica_path when the original ZK path still has leftover state but our replica entry is absent
300305
--drop-replica-if-exists Override clickhouse.drop_replica_if_exists, execute SYSTEM DROP REPLICA ... FROM ZKPATH ... and keep the original replication path when our own replica entry still exists in ZooKeeper but no local table uses it
306+
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
301307
--dry-run Show tables count and data size which would be downloaded and restored, without downloading and restoring
302308
--help, -h show help
303309
@@ -359,9 +365,10 @@ USAGE:
359365
clickhouse-backup delete [--force] <local|remote> <backup_name>
360366
361367
OPTIONS:
362-
--force, -f Delete the backup even when other backups depend on it via required_backup, breaks the incremental backups chain, also skips general.rebase_during_delete
363-
--dry-run Show tables count and data size which would be deleted, without deleting
364-
--help, -h show help
368+
--force, -f Delete the backup even when other backups depend on it via required_backup, breaks the incremental backups chain, also skips general.rebase_during_delete
369+
--embedded-on-cluster-worker delete local Delete only the local copy of this node, don't propagate delete local to the other nodes of `use_embedded_backup_restore_cluster`, used by the initiator node
370+
--dry-run Show tables count and data size which would be deleted, without deleting
371+
--help, -h show help
365372
366373
GLOBAL OPTIONS:
367374
--config string, -c string Config 'FILE' name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG]

0 commit comments

Comments
 (0)