Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions ChangeLog.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
# v2.8.2
NEW FEATURES
- `use_embedded_backup_restore_cluster` now orchestrates the whole cluster instead of only adding `ON CLUSTER` to the BACKUP / RESTORE SQL: `create`, `create_remote`, `restore`, `restore_remote` on the initiator node send the same command with the new `--embedded-on-cluster-worker` flag to every other node of the cluster through its `system.backup_actions` table (`clickhouse-backup server` with `api.create_integration_tables: true` is required on every node) and wait for them, a worker skips the BACKUP / RESTORE SQL and only describes, uploads, downloads and fixes its own `shards/N/replicas/M/` part of the shared destination, per table `metadata/<db>/<table>.json` moved under `shards/N/replicas/M/` next to the `.sql` of each node (such backups are tagged `embedded,cluster=<name>`, older backups without the tag keep the flat layout), `.backup`, `metadata.json`, rbac, configs and named collections stay on the initiator, `delete local` is propagated to the other nodes as `delete local --embedded-on-cluster-worker`, so `delete` accepts the flag too, a single node cluster behaves as before, fix [#928](https://github.com/Altinity/clickhouse-backup/issues/928)

BUG FIXES
- `list` and `tables` text output is now written to stdout as a single write, so a log line on stderr can no longer land in the middle of a backup row when both streams are merged (`docker exec`, `2>&1`), the same fix `json` / `yaml` / `csv` / `tsv` formats got in v2.8.1
- `restore` / `restore_remote` with `--restore-database-mapping` / `--restore-table-mapping` of tables on object disks (`s3`, `azure_blob_storage`, including `cache` and `encrypted` wrappers) no longer leaves the restored copy sharing object storage keys with the source table, so `DROP TABLE|DATABASE ... SYNC` of the copy no longer deletes the objects of the source (`Code: 499 ... The specified key does not exist (S3_ERROR)` on the source afterwards): the key rewrite was skipped when the disk endpoint key prefix contained the backup name (e.g. backup `regression` on the disk `https://s3/bucket/regression-test/live/`), and the metadata files were linked into `detached` / the table directory before the keys were rewritten, so an interrupted restore left the copy pointing to the source keys; the rewritten key now also carries a hash of the destination table (`<dir>_<backup_name>_<crc32(db.table)>/<object>`), so several copies of one backup restored next to each other never share objects, the source key of the copy is always derived from the backup, so the same local backup can be restored several times, and metadata files are written via temp file + rename, so a repeated restore never modifies the parts of an already restored table through their shared hardlink, fix [#1568](https://github.com/Altinity/clickhouse-backup/issues/1568)
Expand Down
13 changes: 10 additions & 3 deletions Manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ OPTIONS:
--skip-check-parts-columns Skip check system.parts_columns to allow backup inconsistent column types for data parts
--skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
--resume use_embedded_backup_restore: true, --resumable use_embedded_backup_restore: true Will resume upload for object disk data, hard links on local disk still continue to recreate, not work when use_embedded_backup_restore: true
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
--dry-run Show tables count and data size which would be created, without creating
--help, -h show help

Expand Down Expand Up @@ -96,6 +97,7 @@ OPTIONS:
--skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
--delete, --delete-source, --delete-local explicitly delete local backup during upload
--streaming Upload each table right after its freeze and delete its local copy, keeps only a small local footprint, https://github.com/Altinity/clickhouse-backup/issues/780
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
--dry-run Show tables count and data size which would be created and uploaded, without creating and uploading
--help, -h show help

Expand Down Expand Up @@ -129,6 +131,7 @@ OPTIONS:
--skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
--resume, --resumable Save intermediate upload state and resume upload if backup exists on remote storage, ignored with 'remote_storage: custom' or 'use_embedded_backup_restore: true'
--delete, --delete-source, --delete-local explicitly delete local backup during upload
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
--dry-run Show tables count and data size which would be uploaded, without uploading
--help, -h show help

Expand Down Expand Up @@ -177,6 +180,7 @@ OPTIONS:
--hardlink-exists-files Create hardlinks for existing files instead of downloading
--disk-limit int Refuse download when usage of any local disk would exceed this percent (1-100) after download, overrides general->download_disk_limit, 0 means use config value, https://github.com/Altinity/clickhouse-backup/issues/1458 (default: 0)
--allow-missing-files Skip data part files which are missing on remote storage (404/NoSuchKey) with an error log and drop them from local table metadata instead of failing, salvage mode for partially corrupted backups, overrides general->allow_missing_files_on_download, https://github.com/Altinity/clickhouse-backup/issues/1456
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
--dry-run Show tables count and data size which would be downloaded, without downloading
--help, -h show help

Expand Down Expand Up @@ -252,6 +256,7 @@ OPTIONS:
--skip-empty-tables Skip restoring tables that have no data (empty tables with only schema)
--rebind-replica-path-if-exists Override clickhouse.rebind_replica_path_if_exists, rebind a restored ReplicatedMergeTree to default_replica_path when the original ZK path still has leftover state but our replica entry is absent
--drop-replica-if-exists Override clickhouse.drop_replica_if_exists, execute SYSTEM DROP REPLICA ... FROM ZKPATH ... and keep the original replication path when our own replica entry still exists in ZooKeeper but no local table uses it
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
--dry-run Show tables count and data size which would be restored, without restoring
--help, -h show help

Expand Down Expand Up @@ -298,6 +303,7 @@ OPTIONS:
--streaming Restore each table right after its download and delete its local copy, keeps only a small local footprint, https://github.com/Altinity/clickhouse-backup/issues/780
--rebind-replica-path-if-exists Override clickhouse.rebind_replica_path_if_exists, rebind a restored ReplicatedMergeTree to default_replica_path when the original ZK path still has leftover state but our replica entry is absent
--drop-replica-if-exists Override clickhouse.drop_replica_if_exists, execute SYSTEM DROP REPLICA ... FROM ZKPATH ... and keep the original replication path when our own replica entry still exists in ZooKeeper but no local table uses it
--embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster
--dry-run Show tables count and data size which would be downloaded and restored, without downloading and restoring
--help, -h show help

Expand Down Expand Up @@ -359,9 +365,10 @@ USAGE:
clickhouse-backup delete [--force] <local|remote> <backup_name>

OPTIONS:
--force, -f Delete the backup even when other backups depend on it via required_backup, breaks the incremental backups chain, also skips general.rebase_during_delete
--dry-run Show tables count and data size which would be deleted, without deleting
--help, -h show help
--force, -f Delete the backup even when other backups depend on it via required_backup, breaks the incremental backups chain, also skips general.rebase_during_delete
--embedded-on-cluster-worker delete local Delete only the local copy of this node, don't propagate delete local to the other nodes of `use_embedded_backup_restore_cluster`, used by the initiator node
--dry-run Show tables count and data size which would be deleted, without deleting
--help, -h show help

GLOBAL OPTIONS:
--config string, -c string Config 'FILE' name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG]
Expand Down
Loading