diff --git a/ChangeLog.md b/ChangeLog.md index 98babe3af..9b37f9fce 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,4 +1,7 @@ # v2.8.2 +NEW FEATURES +- `use_embedded_backup_restore_cluster` now orchestrates the whole cluster instead of only adding `ON CLUSTER` to the BACKUP / RESTORE SQL: `create`, `create_remote`, `restore`, `restore_remote` on the initiator node send the same command with the new `--embedded-on-cluster-worker` flag to every other node of the cluster through its `system.backup_actions` table (`clickhouse-backup server` with `api.create_integration_tables: true` is required on every node) and wait for them, a worker skips the BACKUP / RESTORE SQL and only describes, uploads, downloads and fixes its own `shards/N/replicas/M/` part of the shared destination, per table `metadata//.json` moved under `shards/N/replicas/M/` next to the `.sql` of each node (such backups are tagged `embedded,cluster=`, older backups without the tag keep the flat layout), `.backup`, `metadata.json`, rbac, configs and named collections stay on the initiator, `delete local` is propagated to the other nodes as `delete local --embedded-on-cluster-worker`, so `delete` accepts the flag too, a single node cluster behaves as before, fix [#928](https://github.com/Altinity/clickhouse-backup/issues/928) + BUG FIXES - `list` and `tables` text output is now written to stdout as a single write, so a log line on stderr can no longer land in the middle of a backup row when both streams are merged (`docker exec`, `2>&1`), the same fix `json` / `yaml` / `csv` / `tsv` formats got in v2.8.1 - `restore` / `restore_remote` with `--restore-database-mapping` / `--restore-table-mapping` of tables on object disks (`s3`, `azure_blob_storage`, including `cache` and `encrypted` wrappers) no longer leaves the restored copy sharing object storage keys with the source table, so `DROP TABLE|DATABASE ... SYNC` of the copy no longer deletes the objects of the source (`Code: 499 ... The specified key does not exist (S3_ERROR)` on the source afterwards): the key rewrite was skipped when the disk endpoint key prefix contained the backup name (e.g. backup `regression` on the disk `https://s3/bucket/regression-test/live/`), and the metadata files were linked into `detached` / the table directory before the keys were rewritten, so an interrupted restore left the copy pointing to the source keys; the rewritten key now also carries a hash of the destination table (`__/`), so several copies of one backup restored next to each other never share objects, the source key of the copy is always derived from the backup, so the same local backup can be restored several times, and metadata files are written via temp file + rename, so a repeated restore never modifies the parts of an already restored table through their shared hardlink, fix [#1568](https://github.com/Altinity/clickhouse-backup/issues/1568) diff --git a/Manual.md b/Manual.md index 0389d6943..cd19a84bf 100644 --- a/Manual.md +++ b/Manual.md @@ -55,6 +55,7 @@ OPTIONS: --skip-check-parts-columns Skip check system.parts_columns to allow backup inconsistent column types for data parts --skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax --resume use_embedded_backup_restore: true, --resumable use_embedded_backup_restore: true Will resume upload for object disk data, hard links on local disk still continue to recreate, not work when use_embedded_backup_restore: true + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be created, without creating --help, -h show help @@ -96,6 +97,7 @@ OPTIONS: --skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax --delete, --delete-source, --delete-local explicitly delete local backup during upload --streaming Upload each table right after its freeze and delete its local copy, keeps only a small local footprint, https://github.com/Altinity/clickhouse-backup/issues/780 + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be created and uploaded, without creating and uploading --help, -h show help @@ -129,6 +131,7 @@ OPTIONS: --skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax --resume, --resumable Save intermediate upload state and resume upload if backup exists on remote storage, ignored with 'remote_storage: custom' or 'use_embedded_backup_restore: true' --delete, --delete-source, --delete-local explicitly delete local backup during upload + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be uploaded, without uploading --help, -h show help @@ -177,6 +180,7 @@ OPTIONS: --hardlink-exists-files Create hardlinks for existing files instead of downloading --disk-limit int Refuse download when usage of any local disk would exceed this percent (1-100) after download, overrides general->download_disk_limit, 0 means use config value, https://github.com/Altinity/clickhouse-backup/issues/1458 (default: 0) --allow-missing-files Skip data part files which are missing on remote storage (404/NoSuchKey) with an error log and drop them from local table metadata instead of failing, salvage mode for partially corrupted backups, overrides general->allow_missing_files_on_download, https://github.com/Altinity/clickhouse-backup/issues/1456 + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be downloaded, without downloading --help, -h show help @@ -252,6 +256,7 @@ OPTIONS: --skip-empty-tables Skip restoring tables that have no data (empty tables with only schema) --rebind-replica-path-if-exists Override clickhouse.rebind_replica_path_if_exists, rebind a restored ReplicatedMergeTree to default_replica_path when the original ZK path still has leftover state but our replica entry is absent --drop-replica-if-exists Override clickhouse.drop_replica_if_exists, execute SYSTEM DROP REPLICA ... FROM ZKPATH ... and keep the original replication path when our own replica entry still exists in ZooKeeper but no local table uses it + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be restored, without restoring --help, -h show help @@ -298,6 +303,7 @@ OPTIONS: --streaming Restore each table right after its download and delete its local copy, keeps only a small local footprint, https://github.com/Altinity/clickhouse-backup/issues/780 --rebind-replica-path-if-exists Override clickhouse.rebind_replica_path_if_exists, rebind a restored ReplicatedMergeTree to default_replica_path when the original ZK path still has leftover state but our replica entry is absent --drop-replica-if-exists Override clickhouse.drop_replica_if_exists, execute SYSTEM DROP REPLICA ... FROM ZKPATH ... and keep the original replication path when our own replica entry still exists in ZooKeeper but no local table uses it + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be downloaded and restored, without downloading and restoring --help, -h show help @@ -359,9 +365,10 @@ USAGE: clickhouse-backup delete [--force] OPTIONS: - --force, -f Delete the backup even when other backups depend on it via required_backup, breaks the incremental backups chain, also skips general.rebase_during_delete - --dry-run Show tables count and data size which would be deleted, without deleting - --help, -h show help + --force, -f Delete the backup even when other backups depend on it via required_backup, breaks the incremental backups chain, also skips general.rebase_during_delete + --embedded-on-cluster-worker delete local Delete only the local copy of this node, don't propagate delete local to the other nodes of `use_embedded_backup_restore_cluster`, used by the initiator node + --dry-run Show tables count and data size which would be deleted, without deleting + --help, -h show help GLOBAL OPTIONS: --config string, -c string Config 'FILE' name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] diff --git a/ReadMe.md b/ReadMe.md index 90ad2ff60..07f1ac06e 100644 --- a/ReadMe.md +++ b/ReadMe.md @@ -304,7 +304,7 @@ clickhouse: # WARNING: MUST be false during a concurrent multi-replica (HA) restore, a leftover replica entry is indistinguishable from an entry of a live sibling replica. Can be overridden per invocation with the `--drop-replica-if-exists` flag of `restore`/`restore_remote`. drop_replica_if_exists: false # CLICKHOUSE_DROP_REPLICA_IF_EXISTS use_embedded_backup_restore: false # CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE, use BACKUP / RESTORE SQL statements instead of regular SQL queries to use features of modern ClickHouse server versions - use_embedded_backup_restore_cluster: "" # CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE_CLUSTER, add ON CLUSTER clause to BACKUP / RESTORE SQL statements when `use_embedded_backup_restore: true`, value is cluster name from system.clusters, e.g. "{cluster}" + use_embedded_backup_restore_cluster: "" # CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE_CLUSTER, add ON CLUSTER clause to BACKUP / RESTORE SQL statements when `use_embedded_backup_restore: true`, value is cluster name from system.clusters, e.g. "{cluster}", every node writes its own `shards/N/replicas/M/` part into the SAME shared destination (S3 / GCS / AzureBlob or a shared mount, node local disks don't work), `create`, `create_remote`, `restore`, `restore_remote` on one node drive the other nodes through their `system.backup_actions` tables, so every node needs a running `clickhouse-backup server` with `api.create_integration_tables: true` and the same `clickhouse.username` / `clickhouse.password` / `clickhouse.secure` (system.clusters has no secure column, the initiator connects to other nodes with remote() or remoteSecure() according to its own `clickhouse.secure`), the same `remote_storage` `path` on all nodes, the initiator writes `.backup`, `metadata.json` (tagged `embedded,cluster=`), rbac, configs, named collections, each node writes `shards/N/replicas/M/metadata//
.json`, `delete local ` on one node is propagated to the other nodes, `delete remote` is not (shared destination) embedded_backup_disk: "" # CLICKHOUSE_EMBEDDED_BACKUP_DISK - disk from system.disks which will use when `use_embedded_backup_restore: true` backup_mutations: true # CLICKHOUSE_BACKUP_MUTATIONS, allow backup mutations from system.mutations WHERE is_done=0 and apply it during restore restore_as_attach: false # CLICKHOUSE_RESTORE_AS_ATTACH, allow restore tables which have inconsistent data parts structure and mutations in progress @@ -935,6 +935,7 @@ OPTIONS: --skip-check-parts-columns Skip check system.parts_columns to allow backup inconsistent column types for data parts --skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax --resume use_embedded_backup_restore: true, --resumable use_embedded_backup_restore: true Will resume upload for object disk data, hard links on local disk still continue to recreate, not work when use_embedded_backup_restore: true + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be created, without creating --help, -h show help @@ -976,6 +977,7 @@ OPTIONS: --skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax --delete, --delete-source, --delete-local explicitly delete local backup during upload --streaming Upload each table right after its freeze and delete its local copy, keeps only a small local footprint, https://github.com/Altinity/clickhouse-backup/issues/780 + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be created and uploaded, without creating and uploading --help, -h show help @@ -1009,6 +1011,7 @@ OPTIONS: --skip-projections db_pattern.table_pattern:projections_pattern [ --skip-projections db_pattern.table_pattern:projections_pattern ] Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax --resume, --resumable Save intermediate upload state and resume upload if backup exists on remote storage, ignored with 'remote_storage: custom' or 'use_embedded_backup_restore: true' --delete, --delete-source, --delete-local explicitly delete local backup during upload + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be uploaded, without uploading --help, -h show help @@ -1057,6 +1060,7 @@ OPTIONS: --hardlink-exists-files Create hardlinks for existing files instead of downloading --disk-limit int Refuse download when usage of any local disk would exceed this percent (1-100) after download, overrides general->download_disk_limit, 0 means use config value, https://github.com/Altinity/clickhouse-backup/issues/1458 (default: 0) --allow-missing-files Skip data part files which are missing on remote storage (404/NoSuchKey) with an error log and drop them from local table metadata instead of failing, salvage mode for partially corrupted backups, overrides general->allow_missing_files_on_download, https://github.com/Altinity/clickhouse-backup/issues/1456 + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be downloaded, without downloading --help, -h show help @@ -1132,6 +1136,7 @@ OPTIONS: --skip-empty-tables Skip restoring tables that have no data (empty tables with only schema) --rebind-replica-path-if-exists Override clickhouse.rebind_replica_path_if_exists, rebind a restored ReplicatedMergeTree to default_replica_path when the original ZK path still has leftover state but our replica entry is absent --drop-replica-if-exists Override clickhouse.drop_replica_if_exists, execute SYSTEM DROP REPLICA ... FROM ZKPATH ... and keep the original replication path when our own replica entry still exists in ZooKeeper but no local table uses it + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be restored, without restoring --help, -h show help @@ -1178,6 +1183,7 @@ OPTIONS: --streaming Restore each table right after its download and delete its local copy, keeps only a small local footprint, https://github.com/Altinity/clickhouse-backup/issues/780 --rebind-replica-path-if-exists Override clickhouse.rebind_replica_path_if_exists, rebind a restored ReplicatedMergeTree to default_replica_path when the original ZK path still has leftover state but our replica entry is absent --drop-replica-if-exists Override clickhouse.drop_replica_if_exists, execute SYSTEM DROP REPLICA ... FROM ZKPATH ... and keep the original replication path when our own replica entry still exists in ZooKeeper but no local table uses it + --embedded-on-cluster-worker Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster --dry-run Show tables count and data size which would be downloaded and restored, without downloading and restoring --help, -h show help @@ -1239,9 +1245,10 @@ USAGE: clickhouse-backup delete [--force] OPTIONS: - --force, -f Delete the backup even when other backups depend on it via required_backup, breaks the incremental backups chain, also skips general.rebase_during_delete - --dry-run Show tables count and data size which would be deleted, without deleting - --help, -h show help + --force, -f Delete the backup even when other backups depend on it via required_backup, breaks the incremental backups chain, also skips general.rebase_during_delete + --embedded-on-cluster-worker delete local Delete only the local copy of this node, don't propagate delete local to the other nodes of `use_embedded_backup_restore_cluster`, used by the initiator node + --dry-run Show tables count and data size which would be deleted, without deleting + --help, -h show help GLOBAL OPTIONS: --config string, -c string Config 'FILE' name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] diff --git a/cmd/clickhouse-backup/main.go b/cmd/clickhouse-backup/main.go index 502f364a5..90f4ee71b 100644 --- a/cmd/clickhouse-backup/main.go +++ b/cmd/clickhouse-backup/main.go @@ -163,6 +163,7 @@ func newRootCommand() *cli.Command { Action: func(ctx context.Context, c *cli.Command) error { b := backup.NewBackuper(config.GetConfigFromCli(c)) b.DryRun = c.Bool("dry-run") + b.EmbeddedOnClusterWorker = c.Bool("embedded-on-cluster-worker") return withDryRunResult(c, b, b.CreateBackup(c.Args().First(), c.String("diff-from-remote"), c.String("t"), c.StringSlice("partitions"), c.Bool("s"), c.Bool("rbac"), c.Bool("rbac-only"), c.Bool("configs"), c.Bool("configs-only"), c.Bool("named-collections"), c.Bool("named-collections-only"), c.Bool("skip-check-parts-columns"), c.StringSlice("skip-projections"), c.Bool("resume"), version, commandIdFromCli(c))) }, Flags: []cli.Flag{ @@ -243,6 +244,10 @@ func newRootCommand() *cli.Command { Hidden: false, Usage: "Will resume upload for object disk data, hard links on local disk still continue to recreate, not work when `use_embedded_backup_restore: true`", }, + &cli.BoolFlag{ + Name: "embedded-on-cluster-worker", + Usage: "Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster", + }, &cli.BoolFlag{ Name: "dry-run", Usage: "Show tables count and data size which would be created, without creating", @@ -257,6 +262,7 @@ func newRootCommand() *cli.Command { Action: func(ctx context.Context, c *cli.Command) error { b := backup.NewBackuper(config.GetConfigFromCli(c)) b.DryRun = c.Bool("dry-run") + b.EmbeddedOnClusterWorker = c.Bool("embedded-on-cluster-worker") return withDryRunResult(c, b, b.CreateToRemote(c.Args().First(), c.Bool("delete-source"), c.String("diff-from"), c.String("diff-from-remote"), c.String("tables"), c.StringSlice("partitions"), c.StringSlice("skip-projections"), c.Bool("schema"), c.Bool("rbac"), c.Bool("rbac-only"), c.Bool("configs"), c.Bool("configs-only"), c.Bool("named-collections"), c.Bool("named-collections-only"), c.Bool("skip-check-parts-columns"), c.Bool("resume"), c.Bool("streaming"), version, commandIdFromCli(c))) }, Flags: []cli.Flag{ @@ -353,6 +359,10 @@ func newRootCommand() *cli.Command { Hidden: false, Usage: "Upload each table right after its freeze and delete its local copy, keeps only a small local footprint, https://github.com/Altinity/clickhouse-backup/issues/780", }, + &cli.BoolFlag{ + Name: "embedded-on-cluster-worker", + Usage: "Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster", + }, &cli.BoolFlag{ Name: "dry-run", Usage: "Show tables count and data size which would be created and uploaded, without creating and uploading", @@ -366,6 +376,7 @@ func newRootCommand() *cli.Command { Action: func(ctx context.Context, c *cli.Command) error { b := backup.NewBackuper(config.GetConfigFromCli(c)) b.DryRun = c.Bool("dry-run") + b.EmbeddedOnClusterWorker = c.Bool("embedded-on-cluster-worker") return withDryRunResult(c, b, b.Upload(c.Args().First(), c.Bool("delete-source"), c.String("diff-from"), c.String("diff-from-remote"), c.String("t"), c.StringSlice("partitions"), c.StringSlice("skip-projections"), c.Bool("schema"), c.Bool("rbac-only"), c.Bool("configs-only"), c.Bool("named-collections-only"), c.Bool("resume"), version, commandIdFromCli(c))) }, Flags: []cli.Flag{ @@ -437,6 +448,10 @@ func newRootCommand() *cli.Command { Hidden: false, Usage: "explicitly delete local backup during upload", }, + &cli.BoolFlag{ + Name: "embedded-on-cluster-worker", + Usage: "Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster", + }, &cli.BoolFlag{ Name: "dry-run", Usage: "Show tables count and data size which would be uploaded, without uploading", @@ -468,6 +483,7 @@ func newRootCommand() *cli.Command { Action: func(ctx context.Context, c *cli.Command) error { b := backup.NewBackuper(config.GetConfigFromCli(c)) b.DryRun = c.Bool("dry-run") + b.EmbeddedOnClusterWorker = c.Bool("embedded-on-cluster-worker") b.DiskLimit = c.Int("disk-limit") return withDryRunResult(c, b, b.Download(c.Args().First(), c.String("t"), c.StringSlice("partitions"), c.Bool("schema"), c.Bool("rbac-only"), c.Bool("configs-only"), c.Bool("named-collections-only"), c.Bool("resume"), c.Bool("hardlink-exists-files"), version, commandIdFromCli(c))) }, @@ -534,6 +550,10 @@ func newRootCommand() *cli.Command { Hidden: false, Usage: "Skip data part files which are missing on remote storage (404/NoSuchKey) with an error log and drop them from local table metadata instead of failing, salvage mode for partially corrupted backups, overrides general->allow_missing_files_on_download, https://github.com/Altinity/clickhouse-backup/issues/1456", }, + &cli.BoolFlag{ + Name: "embedded-on-cluster-worker", + Usage: "Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster", + }, &cli.BoolFlag{ Name: "dry-run", Usage: "Show tables count and data size which would be downloaded, without downloading", @@ -586,6 +606,7 @@ func newRootCommand() *cli.Command { Action: func(ctx context.Context, c *cli.Command) error { b := backup.NewBackuper(config.GetConfigFromCli(c)) b.DryRun = c.Bool("dry-run") + b.EmbeddedOnClusterWorker = c.Bool("embedded-on-cluster-worker") return withDryRunResult(c, b, b.Restore(c.Args().First(), c.String("tables"), c.StringSlice("restore-database-mapping"), c.StringSlice("restore-table-mapping"), c.StringSlice("partitions"), c.StringSlice("skip-projections"), c.Bool("schema"), c.Bool("data"), c.Bool("drop"), c.Bool("ignore-dependencies"), c.Bool("rbac"), c.Bool("rbac-only"), c.Bool("configs"), c.Bool("configs-only"), c.Bool("named-collections"), c.Bool("named-collections-only"), c.Bool("resume"), c.Bool("restore-schema-as-attach"), c.Bool("replicated-copy-to-detached"), c.Bool("skip-empty-tables"), version, commandIdFromCli(c))) }, Flags: []cli.Flag{ @@ -711,6 +732,10 @@ func newRootCommand() *cli.Command { Hidden: false, Usage: "Override clickhouse.drop_replica_if_exists, execute SYSTEM DROP REPLICA ... FROM ZKPATH ... and keep the original replication path when our own replica entry still exists in ZooKeeper but no local table uses it", }, + &cli.BoolFlag{ + Name: "embedded-on-cluster-worker", + Usage: "Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster", + }, &cli.BoolFlag{ Name: "dry-run", Usage: "Show tables count and data size which would be restored, without restoring", @@ -724,6 +749,7 @@ func newRootCommand() *cli.Command { Action: func(ctx context.Context, c *cli.Command) error { b := backup.NewBackuper(config.GetConfigFromCli(c)) b.DryRun = c.Bool("dry-run") + b.EmbeddedOnClusterWorker = c.Bool("embedded-on-cluster-worker") b.DiskLimit = c.Int("disk-limit") return withDryRunResult(c, b, b.RestoreFromRemote(c.Args().First(), c.String("tables"), c.StringSlice("restore-database-mapping"), c.StringSlice("restore-table-mapping"), c.StringSlice("partitions"), c.StringSlice("skip-projections"), c.Bool("schema"), c.Bool("d"), c.Bool("rm"), c.Bool("i"), c.Bool("rbac"), c.Bool("rbac-only"), c.Bool("configs"), c.Bool("configs-only"), c.Bool("named-collections"), c.Bool("named-collections-only"), c.Bool("resume"), c.Bool("restore-schema-as-attach"), c.Bool("replicated-copy-to-detached"), c.Bool("skip-empty-tables"), c.Bool("hardlink-exists-files"), c.Bool("streaming"), version, commandIdFromCli(c))) }, @@ -865,6 +891,10 @@ func newRootCommand() *cli.Command { Hidden: false, Usage: "Override clickhouse.drop_replica_if_exists, execute SYSTEM DROP REPLICA ... FROM ZKPATH ... and keep the original replication path when our own replica entry still exists in ZooKeeper but no local table uses it", }, + &cli.BoolFlag{ + Name: "embedded-on-cluster-worker", + Usage: "Run only this node's part of an embedded BACKUP/RESTORE ON CLUSTER (skip BACKUP/RESTORE SQL, handle own shards/N/replicas/M metadata), the initiator node passes it to other nodes through system.backup_actions, requires use_embedded_backup_restore: true and use_embedded_backup_restore_cluster", + }, &cli.BoolFlag{ Name: "dry-run", Usage: "Show tables count and data size which would be downloaded and restored, without downloading and restoring", @@ -1016,6 +1046,7 @@ func newRootCommand() *cli.Command { cli.ShowCommandHelpAndExit(ctx, c.Root(), c.Name, 1) } b.DryRun = c.Bool("dry-run") + b.EmbeddedOnClusterWorker = c.Bool("embedded-on-cluster-worker") return withDryRunResult(c, b, b.Delete(c.Args().Get(0), c.Args().Get(1), c.Bool("force"), commandIdFromCli(c))) }, Flags: []cli.Flag{ @@ -1025,6 +1056,10 @@ func newRootCommand() *cli.Command { Hidden: false, Usage: "Delete the backup even when other backups depend on it via required_backup, breaks the incremental backups chain, also skips general.rebase_during_delete", }, + &cli.BoolFlag{ + Name: "embedded-on-cluster-worker", + Usage: "Delete only the local copy of this node, don't propagate `delete local` to the other nodes of `use_embedded_backup_restore_cluster`, used by the initiator node", + }, &cli.BoolFlag{ Name: "dry-run", Usage: "Show tables count and data size which would be deleted, without deleting", diff --git a/pkg/backup/backuper.go b/pkg/backup/backuper.go index 78d996e55..4bd7ebc6b 100644 --- a/pkg/backup/backuper.go +++ b/pkg/backup/backuper.go @@ -49,13 +49,22 @@ type Backuper struct { DefaultDataPath string EmbeddedBackupDataPath string embeddedClusterPrefix string // "shards/{shard_num}/replicas/{replica_num}" when UseEmbeddedBackupRestoreCluster is set, otherwise "" - isEmbedded bool - resume bool - resumableState *resumable.State + // embeddedClusterName - macros expanded use_embedded_backup_restore_cluster, stored as `cluster=` tag of the backup + embeddedClusterName string + // embeddedClusterFilesPrefix - prefix of this node's own clickhouse-backup files on remote storage (per table .json, shadow data), + // equals embeddedClusterPrefix for backups tagged `cluster=`, "" for older flat layout backups, see issues/928 + embeddedClusterFilesPrefix string + isEmbedded bool + resume bool + resumableState *resumable.State // DryRun report what the command would do instead of doing it, see issues/1012 DryRun bool // DryRunResult holds the report produced when DryRun is set, so REST API handlers can read it after the command returns DryRunResult *DryRunReport + // EmbeddedOnClusterWorker - run only the node local part of an embedded BACKUP/RESTORE ON CLUSTER, see issues/928 + EmbeddedOnClusterWorker bool + // skipEmbeddedClusterFanOut - create_remote/restore_remote drive the worker nodes themselves, the inner create/restore must not + skipEmbeddedClusterFanOut bool // DiskLimit - max allowed local disk usage in percent after download, 0 disables the check, see issues/1458 DiskLimit int // freezes - shadow uuids of the running `create`, persisted in /freezes.tmp, see issues/1563 @@ -199,6 +208,7 @@ func (b *Backuper) getLocalBackupDataPathForTable(backupName string, disk string func (b *Backuper) resolveEmbeddedClusterShardReplica(ctx context.Context) error { if b.cfg.ClickHouse.UseEmbeddedBackupRestoreCluster == "" { b.embeddedClusterPrefix = "" + b.embeddedClusterName = "" return nil } clusterName, err := b.ch.ApplyMacros(ctx, b.cfg.ClickHouse.UseEmbeddedBackupRestoreCluster) @@ -218,6 +228,7 @@ func (b *Backuper) resolveEmbeddedClusterShardReplica(ctx context.Context) error return errors.Errorf("no local replica found in system.clusters for cluster '%s'", clusterName) } b.embeddedClusterPrefix = path.Join("shards", fmt.Sprintf("%d", result[0].ShardNum), "replicas", fmt.Sprintf("%d", result[0].ReplicaNum)) + b.embeddedClusterName = clusterName log.Debug().Msgf("resolved embedded cluster prefix: %s", b.embeddedClusterPrefix) return nil } diff --git a/pkg/backup/create.go b/pkg/backup/create.go index d6214732d..c34c2e664 100644 --- a/pkg/backup/create.go +++ b/pkg/backup/create.go @@ -103,6 +103,13 @@ func (b *Backuper) CreateBackup(backupName, diffFromRemote, tablePattern string, if b.cfg.General.NamedCollectionsBackupAlways { createNamedCollections = true } + if err := b.validateEmbeddedOnClusterWorker(); err != nil { + return err + } + if b.EmbeddedOnClusterWorker { + // rbac, configs and named collections are backed up by the initiator node only, https://github.com/Altinity/clickhouse-backup/issues/928 + createRBAC, createConfigs, createNamedCollections = false, false, false + } b.adjustResumeFlag(resume) p, err := b.createPrologue(ctx, tablePattern, partitions, schemaOnly, rbacOnly, configsOnly, namedCollectionsOnly) @@ -145,6 +152,13 @@ func (b *Backuper) CreateBackup(backupName, diffFromRemote, tablePattern string, } return errors.Wrap(err, "createBackup failed") } + // BACKUP ... ON CLUSTER already wrote the shards/N/replicas/M data of every node, now each worker node describes its + // own part with clickhouse-backup metadata, https://github.com/Altinity/clickhouse-backup/issues/928 + if b.embeddedClusterInitiator() && !b.skipEmbeddedClusterFanOut && !rbacOnly && !configsOnly && !namedCollectionsOnly { + if err := b.runEmbeddedClusterWorkers(ctx, embeddedClusterCreateWorkerCommand("create", backupName, diffFromRemote, tablePattern, partitions, schemaOnly, skipCheckPartsColumns, false)); err != nil { + return errors.Wrap(err, "embedded ON CLUSTER workers") + } + } // fix https://github.com/Altinity/clickhouse-backup/issues/1345 clean only shadow UUIDs created by this backup if cleanShadowErr := b.cleanOwnFreezes(disks); cleanShadowErr != nil { @@ -632,11 +646,18 @@ func (b *Backuper) createBackupEmbedded(ctx context.Context, backupName, baseBac return errors.Wrap(err, "b.generateEmbeddedBackupSQL") } backupResult := make([]clickhouse.SystemBackups, 0) - if err := b.ch.SelectContext(ctx, &backupResult, backupSQL); err != nil { - return errors.Wrap(err, "backup error") - } - if len(backupResult) != 1 || (backupResult[0].Status != "BACKUP_COMPLETE" && backupResult[0].Status != "BACKUP_CREATED") { - return errors.Errorf("backup return wrong results: %+v", backupResult) + if b.EmbeddedOnClusterWorker { + // the initiator already executed BACKUP ... ON CLUSTER, this node only describes its own shards/N/replicas/M part, + // the empty result routes the size calculation to system.parts below, https://github.com/Altinity/clickhouse-backup/issues/928 + log.Info().Msgf("--%s: skip BACKUP SQL, describe %s only", embeddedOnClusterWorkerFlag, b.embeddedClusterPrefix) + backupResult = append(backupResult, clickhouse.SystemBackups{}) + } else { + if err := b.ch.SelectContext(ctx, &backupResult, backupSQL); err != nil { + return errors.Wrap(err, "backup error") + } + if len(backupResult) != 1 || (backupResult[0].Status != "BACKUP_COMPLETE" && backupResult[0].Status != "BACKUP_CREATED") { + return errors.Errorf("backup return wrong results: %+v", backupResult) + } } if schemaOnly { @@ -710,7 +731,8 @@ func (b *Backuper) createBackupEmbedded(ctx context.Context, backupName, baseBac return errors.Wrap(err, "getPartsFromEmbeddedBackup") } if schemaOnly || doBackupData { - metadataSize, err := b.createTableMetadata(path.Join(backupPath, "metadata"), metadata.TableMetadata{ + // per table .json next to the .sql of this node, so every node of an ON CLUSTER backup has its own set + metadataSize, err := b.createTableMetadata(path.Join(backupPath, b.embeddedClusterPrefix, "metadata"), metadata.TableMetadata{ Table: table.Name, Database: table.Database, UUID: table.UUID, @@ -730,7 +752,11 @@ func (b *Backuper) createBackupEmbedded(ctx context.Context, backupName, baseBac } // end if l > 0 } backupMetaFile := path.Join(backupPath, "metadata.json") - if err := b.createBackupMetadata(ctx, backupMetaFile, backupName, baseBackup, backupVersion, "embedded", diskMap, diskTypes, disks, backupDataSize[0].Size, 0, backupMetadataSize, backupRBACSize, backupConfigSize, backupNamedCollectionsSize, tablesTitle, allDatabases, allFunctions); err != nil { + // a shared `embedded_backup_disk` is one directory for all nodes, the initiator wrote metadata.json before the fan-out + // and a worker must not overwrite it, in URL mode the path is node local and the worker writes its own + if _, statErr := os.Stat(backupMetaFile); b.EmbeddedOnClusterWorker && statErr == nil { + log.Info().Msgf("--%s: %s already written by the initiator, skip", embeddedOnClusterWorkerFlag, backupMetaFile) + } else if err := b.createBackupMetadata(ctx, backupMetaFile, backupName, baseBackup, backupVersion, b.embeddedBackupTags(), diskMap, diskTypes, disks, backupDataSize[0].Size, 0, backupMetadataSize, backupRBACSize, backupConfigSize, backupNamedCollectionsSize, tablesTitle, allDatabases, allFunctions); err != nil { return errors.Wrap(err, "b.createBackupMetadata") } diff --git a/pkg/backup/create_remote.go b/pkg/backup/create_remote.go index 134d30678..4bf898589 100644 --- a/pkg/backup/create_remote.go +++ b/pkg/backup/create_remote.go @@ -41,6 +41,8 @@ func (b *Backuper) CreateToRemote(backupName string, deleteSource bool, diffFrom return b.createToRemoteStreaming(ctx, backupName, diffFrom, diffFromRemote, tablePattern, partitions, skipProjections, schemaOnly, backupRBAC, rbacOnly, backupConfigs, configsOnly, namedCollections, namedCollectionsOnly, skipCheckPartsColumns, resume, version) } } + // the worker nodes run `create_remote --embedded-on-cluster-worker` below, CreateBackup must not start `create` on them + b.skipEmbeddedClusterFanOut = true if createErr := b.CreateBackup(backupName, diffFromRemote, tablePattern, partitions, schemaOnly, backupRBAC, rbacOnly, backupConfigs, configsOnly, namedCollections, namedCollectionsOnly, skipCheckPartsColumns, skipProjections, resume, version, commandId); createErr != nil { return createErr } @@ -52,9 +54,22 @@ func (b *Backuper) CreateToRemote(backupName string, deleteSource bool, diffFrom } return nil } + // BACKUP ... ON CLUSTER already wrote the shards/N/replicas/M data of every node, each worker describes and uploads + // its own part in parallel with our upload, .backup and metadata.json are uploaded by this node only, + // https://github.com/Altinity/clickhouse-backup/issues/928 + var workerHosts []embeddedClusterHost + workerCommand := embeddedClusterCreateWorkerCommand("create_remote", backupName, diffFromRemote, tablePattern, partitions, schemaOnly, skipCheckPartsColumns, deleteSource) + if b.embeddedClusterInitiator() && !rbacOnly && !configsOnly && !namedCollectionsOnly { + if workerHosts, err = b.startEmbeddedClusterWorkers(ctx, workerCommand); err != nil { + return errors.Wrap(err, "embedded ON CLUSTER workers") + } + } if uploadErr := b.Upload(backupName, deleteSource, diffFrom, diffFromRemote, tablePattern, partitions, skipProjections, schemaOnly, rbacOnly, configsOnly, namedCollectionsOnly, resume, version, commandId); uploadErr != nil { return uploadErr } + if err = b.waitEmbeddedClusterWorkers(ctx, workerHosts, workerCommand); err != nil { + return errors.Wrap(err, "embedded ON CLUSTER workers") + } return nil } diff --git a/pkg/backup/delete.go b/pkg/backup/delete.go index 27f9f7d5b..03f771c56 100644 --- a/pkg/backup/delete.go +++ b/pkg/backup/delete.go @@ -150,7 +150,7 @@ func (b *Backuper) Delete(backupType, backupName string, force bool, commandId i switch backupType { case "local": - return b.RemoveBackupLocal(ctx, backupName, nil, force) + return b.RemoveBackupLocalOnCluster(ctx, backupName, force) case "remote": return b.RemoveBackupRemote(ctx, backupName, force) default: @@ -158,6 +158,41 @@ func (b *Backuper) Delete(backupType, backupName string, force bool, commandId i } } +// RemoveBackupLocalOnCluster removes the local backup and, on the initiator node of an embedded ON CLUSTER backup, the local +// copies of the other nodes too. `delete remote` is not propagated because the remote destination is shared and one call +// removes everything, https://github.com/Altinity/clickhouse-backup/issues/928 +func (b *Backuper) RemoveBackupLocalOnCluster(ctx context.Context, backupName string, force bool) error { + clusterName, err := b.localBackupEmbeddedCluster(ctx, backupName) + if err != nil { + return err + } + if err = b.RemoveBackupLocal(ctx, backupName, nil, force); err != nil { + return err + } + if clusterName != "" && b.embeddedClusterInitiator() { + return b.deleteLocalOnEmbeddedClusterWorkers(ctx, backupName) + } + return nil +} + +// localBackupEmbeddedCluster returns the `cluster=` tag of a local backup, "" when the backup is not found or not tagged +func (b *Backuper) localBackupEmbeddedCluster(ctx context.Context, backupName string) (string, error) { + if err := b.ch.Connect(); err != nil { + return "", errors.Wrap(err, "can't connect to clickhouse") + } + defer b.ch.Close() + localBackups, _, err := b.GetLocalBackups(ctx, nil) + if err != nil { + return "", errors.Wrap(err, "b.GetLocalBackups") + } + for _, localBackup := range localBackups { + if localBackup.BackupName == utils.CleanBackupNameRE.ReplaceAllString(backupName, "") { + return localBackup.EmbeddedCluster(), nil + } + } + return "", nil +} + // backupChainLink - `required_backup` link of one backup, allows to check incremental chains // dependencies for both local (LocalBackup) and remote (storage.Backup) backup lists type backupChainLink struct { @@ -332,7 +367,8 @@ func (b *Backuper) cleanEmbeddedAndObjectDiskLocalIfSameRemoteNotPresent(ctx con if err != nil { return errors.Wrap(err, "b.skipIfTheSameRemoteBackupPresent") } - if !skip && (hasObjectDisks || (b.isEmbedded && b.cfg.ClickHouse.EmbeddedBackupDisk == "")) { + // a worker node of an embedded ON CLUSTER backup never owns the shared remote destination, https://github.com/Altinity/clickhouse-backup/issues/928 + if !skip && !b.EmbeddedOnClusterWorker && (hasObjectDisks || (b.isEmbedded && b.cfg.ClickHouse.EmbeddedBackupDisk == "")) { startTime := time.Now() if deletedKeys, deleteErr := b.cleanBackupObjectDisks(ctx, backupName); deleteErr != nil { log.Warn().Msgf("b.cleanBackupObjectDisks return error: %v", deleteErr) diff --git a/pkg/backup/download.go b/pkg/backup/download.go index e2d1beb8a..e699636fc 100644 --- a/pkg/backup/download.go +++ b/pkg/backup/download.go @@ -85,6 +85,9 @@ func (b *Backuper) Download(backupName string, tablePattern string, partitions [ if b.DiskLimit < 0 || b.DiskLimit > 100 { return errors.Errorf("--disk-limit shall be between 1 and 100 percent, got %d", b.DiskLimit) } + if err := b.validateEmbeddedOnClusterWorker(); err != nil { + return err + } b.adjustResumeFlag(resume) if backupName == "" { @@ -282,6 +285,7 @@ func (b *Backuper) downloadTablesMetadata(ctx context.Context, backupName, comma if err = b.resolveEmbeddedClusterShardReplica(ctx); err != nil { return nil, errors.Wrap(err, "resolveEmbeddedClusterShardReplica") } + b.applyEmbeddedClusterLayout(&remoteBackup.BackupMetadata) } localBackupDir := path.Join(b.DefaultDataPath, "backup", backupName) if b.isEmbedded { @@ -373,12 +377,18 @@ func (b *Backuper) downloadTablesMetadata(ctx context.Context, backupName, comma // downloadEpilogue downloads rbac/configs/named collections and the embedded .backup file, saves local // backup-level metadata.json, chowns backup disks and cleans partially downloaded required backup func (b *Backuper) downloadEpilogue(ctx context.Context, backupName string, remoteBackup storage.Backup, tablesForDownload []metadata.TableTitle, disks []clickhouse.Disk, dataSize, metadataSize uint64, doDownloadData, rbacOnly, configsOnly, namedCollectionsOnly bool, backupVersion string, startDownload time.Time) error { - rbacSize, configSize, namedCollectionsSize, err := b.downloadBackupRelatedData(ctx, remoteBackup, rbacOnly, configsOnly, namedCollectionsOnly) - if err != nil { - return err + var rbacSize, configSize, namedCollectionsSize uint64 + var err error + // rbac, configs, named collections and .backup are downloaded by the initiator node of an embedded ON CLUSTER + // restore only, a worker downloads its own shards/N/replicas/M part, https://github.com/Altinity/clickhouse-backup/issues/928 + if !b.EmbeddedOnClusterWorker { + rbacSize, configSize, namedCollectionsSize, err = b.downloadBackupRelatedData(ctx, remoteBackup, rbacOnly, configsOnly, namedCollectionsOnly) + if err != nil { + return err + } } - if doDownloadData && b.isEmbedded && b.cfg.ClickHouse.EmbeddedBackupDisk != "" && tablesForDownload != nil && len(tablesForDownload) > 0 { + if doDownloadData && !b.EmbeddedOnClusterWorker && b.isEmbedded && b.cfg.ClickHouse.EmbeddedBackupDisk != "" && tablesForDownload != nil && len(tablesForDownload) > 0 { localClickHouseBackupFile := path.Join(b.EmbeddedBackupDataPath, backupName, ".backup") remoteClickHouseBackupFile := path.Join(backupName, ".backup") localEmbeddedMetadataSize := int64(0) @@ -634,14 +644,15 @@ func (b *Backuper) downloadTableMetadata(ctx context.Context, backupName string, start := time.Now() size := uint64(0) metadataFiles := map[string]string{} - remoteMedataPrefix := path.Join(backupName, "metadata", common.TablePathEncode(tableTitle.Database), common.TablePathEncode(tableTitle.Table)) - metadataFiles[fmt.Sprintf("%s.json", remoteMedataPrefix)] = path.Join(b.DefaultDataPath, "backup", backupName, "metadata", common.TablePathEncode(tableTitle.Database), fmt.Sprintf("%s.json", common.TablePathEncode(tableTitle.Table))) + localBackupDir := path.Join(b.DefaultDataPath, "backup", backupName) partitionsIdMap := make(map[metadata.TableTitle]common.EmptyMap) if b.isEmbedded && b.cfg.ClickHouse.EmbeddedBackupDisk != "" { + localBackupDir = path.Join(b.EmbeddedBackupDataPath, backupName) remoteSqlPrefix := path.Join(backupName, b.embeddedClusterPrefix, "metadata", common.TablePathEncode(tableTitle.Database), common.TablePathEncode(tableTitle.Table)) metadataFiles[fmt.Sprintf("%s.sql", remoteSqlPrefix)] = path.Join(b.EmbeddedBackupDataPath, backupName, b.embeddedClusterPrefix, "metadata", common.TablePathEncode(tableTitle.Database), fmt.Sprintf("%s.sql", common.TablePathEncode(tableTitle.Table))) - metadataFiles[fmt.Sprintf("%s.json", remoteMedataPrefix)] = path.Join(b.EmbeddedBackupDataPath, backupName, "metadata", common.TablePathEncode(tableTitle.Database), fmt.Sprintf("%s.json", common.TablePathEncode(tableTitle.Table))) } + // per table .json lives next to the .sql of this node for embedded ON CLUSTER backups, see issues/928 + metadataFiles[remoteTableMetadataJSON(backupName, b.embeddedClusterFilesPrefix, tableTitle.Database, tableTitle.Table)] = path.Join(localBackupDir, b.embeddedClusterFilesPrefix, "metadata", common.TablePathEncode(tableTitle.Database), fmt.Sprintf("%s.json", common.TablePathEncode(tableTitle.Table))) var tableMetadata metadata.TableMetadata for remoteMetadataFile, localMetadataFile := range metadataFiles { if resume { @@ -995,7 +1006,7 @@ func (b *Backuper) downloadTableData(ctx context.Context, remoteBackup metadata. capturedParts := table.Parts[capturedDisk] tableLocalDir := b.getLocalBackupDataPathForTable(remoteBackup.BackupName, capturedDisk, dbAndTableDir) downloadOffset[disk] += 1 - tableRemoteFile := path.Join(remoteBackup.BackupName, "shadow", common.TablePathEncode(table.Database), common.TablePathEncode(table.Table), archiveFile) + tableRemoteFile := path.Join(remoteBackup.BackupName, b.embeddedClusterFilesPrefix, "shadow", common.TablePathEncode(table.Database), common.TablePathEncode(table.Table), archiveFile) dataGroup.Go(func() error { log.Debug().Msgf("start download %s", tableRemoteFile) if b.resume { @@ -1096,7 +1107,7 @@ func (b *Backuper) downloadTableData(ctx context.Context, remoteBackup metadata. if remoteBackup.IsPlainDisk(disk) { continue } - tableRemotePath := path.Join(remoteBackup.BackupName, "shadow", dbAndTableDir, disk) + tableRemotePath := path.Join(remoteBackup.BackupName, b.embeddedClusterFilesPrefix, "shadow", dbAndTableDir, disk) diskPath, diskExists := b.DiskToPathMap[disk] tableLocalPath := path.Join(diskPath, "backup", remoteBackup.BackupName, "shadow", dbAndTableDir, disk) if b.isEmbedded { diff --git a/pkg/backup/embedded_cluster.go b/pkg/backup/embedded_cluster.go new file mode 100644 index 000000000..51f0b0498 --- /dev/null +++ b/pkg/backup/embedded_cluster.go @@ -0,0 +1,335 @@ +package backup + +import ( + "context" + "fmt" + "os" + "path" + "strings" + "time" + + "github.com/pkg/errors" + "github.com/rs/zerolog/log" + + "github.com/Altinity/clickhouse-backup/v2/pkg/common" + "github.com/Altinity/clickhouse-backup/v2/pkg/metadata" + "github.com/Altinity/clickhouse-backup/v2/pkg/status" +) + +// Embedded BACKUP/RESTORE ... ON CLUSTER orchestration, https://github.com/Altinity/clickhouse-backup/issues/928 +// +// ClickHouse executes BACKUP/RESTORE ON CLUSTER on every node of the cluster, each node writes/reads its own +// `shards/N/replicas/M/{metadata,data}` part of the same shared destination and only the initiator writes `.backup`. +// clickhouse-backup metadata (`metadata.json`, per table `.json`) and the `.sql` fixes before RESTORE are node local, +// so the node which runs the command (initiator) drives every other node (worker) through its `system.backup_actions` +// table with the same command plus `--embedded-on-cluster-worker`, which skips the BACKUP/RESTORE SQL and touches only +// the own `shards/N/replicas/M` prefix. + +// embeddedOnClusterWorkerFlag is the CLI flag which turns create/upload/download/restore into the per node part of an +// embedded BACKUP/RESTORE ON CLUSTER +const embeddedOnClusterWorkerFlag = "embedded-on-cluster-worker" + +// embeddedClusterPollInterval is the delay between two system.backup_actions polls of the worker nodes +const embeddedClusterPollInterval = 5 * time.Second + +// embeddedClusterHost is one non-local replica of `use_embedded_backup_restore_cluster` +type embeddedClusterHost struct { + HostName string `ch:"host_name"` + Port uint16 `ch:"port"` + ShardNum uint32 `ch:"shard_num"` + ReplicaNum uint32 `ch:"replica_num"` +} + +func (h embeddedClusterHost) String() string { + return fmt.Sprintf("%s:%d (shard %d, replica %d)", h.HostName, h.Port, h.ShardNum, h.ReplicaNum) +} + +// remoteBackupActions renders the remote()/remoteSecure() table function which reads system.backup_actions of the host, +// system.clusters has no `secure` column, so every node is expected to use the same transport as `clickhouse.secure` +func (h embeddedClusterHost) remoteBackupActions(user, password string, secure bool) string { + tableFunction := "remote" + if secure { + tableFunction = "remoteSecure" + } + return fmt.Sprintf("%s(%s, 'system', 'backup_actions', %s, %s)", tableFunction, chQuote(fmt.Sprintf("%s:%d", h.HostName, h.Port)), chQuote(user), chQuote(password)) +} + +// chQuote renders s as a ClickHouse single quoted string literal +func chQuote(s string) string { + return "'" + strings.NewReplacer(`\`, `\\`, `'`, `\'`).Replace(s) + "'" +} + +// shlexQuote wraps v in double quotes, so github.com/google/shlex in the API server returns it as one argument +func shlexQuote(v string) string { + return `"` + strings.NewReplacer(`\`, `\\`, `"`, `\"`).Replace(v) + `"` +} + +// workerCommand accumulates the CLI arguments of the command a worker node executes via system.backup_actions +type workerCommand struct { + args []string +} + +func newWorkerCommand(command string) *workerCommand { + return &workerCommand{args: []string{command}} +} + +func (w *workerCommand) flag(name string, set bool) *workerCommand { + if set { + w.args = append(w.args, "--"+name) + } + return w +} + +func (w *workerCommand) value(name, v string) *workerCommand { + if v != "" { + w.args = append(w.args, fmt.Sprintf("--%s=%s", name, shlexQuote(v))) + } + return w +} + +func (w *workerCommand) values(name string, vs []string) *workerCommand { + for _, v := range vs { + w.value(name, v) + } + return w +} + +// String terminates the command with the worker flag and the backup name +func (w *workerCommand) String(backupName string) string { + args := append(append([]string{}, w.args...), "--"+embeddedOnClusterWorkerFlag, shlexQuote(backupName)) + return strings.Join(args, " ") +} + +// embeddedClusterCreateWorkerCommand renders `create`/`create_remote` for a worker node, rbac/configs/named collections +// stay on the initiator +func embeddedClusterCreateWorkerCommand(command, backupName, diffFromRemote, tablePattern string, partitions []string, schemaOnly, skipCheckPartsColumns, deleteSource bool) string { + return newWorkerCommand(command). + value("tables", tablePattern). + values("partitions", partitions). + value("diff-from-remote", diffFromRemote). + flag("schema", schemaOnly). + flag("skip-check-parts-columns", skipCheckPartsColumns). + flag("delete-source", deleteSource). + String(backupName) +} + +// embeddedClusterRestoreWorkerCommand renders `restore`/`restore_remote` for a worker node, rbac/configs/named +// collections stay on the initiator +func embeddedClusterRestoreWorkerCommand(command, backupName, tablePattern string, databaseMapping, tableMapping, partitions []string, schemaOnly, dataOnly, dropExists, ignoreDependencies, schemaAsAttach, skipEmptyTables bool) string { + return newWorkerCommand(command). + value("tables", tablePattern). + values("restore-database-mapping", databaseMapping). + values("restore-table-mapping", tableMapping). + values("partitions", partitions). + flag("schema", schemaOnly). + flag("data", dataOnly). + flag("rm", dropExists). + flag("ignore-dependencies", ignoreDependencies). + flag("restore-schema-as-attach", schemaAsAttach). + flag("skip-empty-tables", skipEmptyTables). + String(backupName) +} + +// embeddedMetadataDir returns //metadata when the cluster prefixed layout exists, +// otherwise the legacy flat /metadata, so backups made before ON CLUSTER support still work +func embeddedMetadataDir(backupDir, clusterPrefix string, exists func(string) bool) string { + if clusterPrefix != "" && exists(path.Join(backupDir, clusterPrefix, "metadata")) { + return path.Join(backupDir, clusterPrefix, "metadata") + } + return path.Join(backupDir, "metadata") +} + +// remoteTableMetadataJSON is the remote path of a per table .json, under the cluster prefix for embedded ON CLUSTER +// backups, the legacy flat path when clusterPrefix is empty +func remoteTableMetadataJSON(backupName, clusterPrefix, database, table string) string { + return path.Join(backupName, clusterPrefix, "metadata", common.TablePathEncode(database), fmt.Sprintf("%s.json", common.TablePathEncode(table))) +} + +func dirExists(p string) bool { + info, err := os.Stat(p) + return err == nil && info.IsDir() +} + +// localMetadataDir is the per table metadata directory of a local backup, the cluster prefixed layout wins when present +func (b *Backuper) localMetadataDir(backupDir string) string { + return embeddedMetadataDir(backupDir, b.embeddedClusterPrefix, dirExists) +} + +// validateEmbeddedOnClusterWorker rejects --embedded-on-cluster-worker outside of embedded ON CLUSTER mode +func (b *Backuper) validateEmbeddedOnClusterWorker() error { + if b.EmbeddedOnClusterWorker && (!b.cfg.ClickHouse.UseEmbeddedBackupRestore || b.cfg.ClickHouse.UseEmbeddedBackupRestoreCluster == "") { + return errors.Errorf("--%s requires `use_embedded_backup_restore: true` and non-empty `use_embedded_backup_restore_cluster` in the `clickhouse` config section", embeddedOnClusterWorkerFlag) + } + return nil +} + +// embeddedClusterInitiator reports whether this run drives the worker nodes: embedded ON CLUSTER mode, not a worker +// itself and not a dry-run +func (b *Backuper) embeddedClusterInitiator() bool { + return b.cfg.ClickHouse.UseEmbeddedBackupRestore && b.cfg.ClickHouse.UseEmbeddedBackupRestoreCluster != "" && !b.EmbeddedOnClusterWorker && !b.DryRun +} + +// embeddedClusterWorkerSkipsDDL reports whether the node local DDL (CREATE DATABASE, DROP TABLE, UDF) is left to the +// initiator, which executes it ON CLUSTER for every node when `restore_schema_on_cluster` is set +func (b *Backuper) embeddedClusterWorkerSkipsDDL() bool { + return b.EmbeddedOnClusterWorker && b.cfg.General.RestoreSchemaOnCluster != "" +} + +// getEmbeddedClusterRemoteHosts lists the other nodes of `use_embedded_backup_restore_cluster`, empty for a single node cluster +func (b *Backuper) getEmbeddedClusterRemoteHosts(ctx context.Context) ([]embeddedClusterHost, error) { + clusterName, err := b.ch.ApplyMacros(ctx, b.cfg.ClickHouse.UseEmbeddedBackupRestoreCluster) + if err != nil { + return nil, errors.Wrap(err, "ApplyMacros for use_embedded_backup_restore_cluster") + } + hosts := make([]embeddedClusterHost, 0) + query := fmt.Sprintf("SELECT host_name, port, shard_num, replica_num FROM system.clusters WHERE cluster=%s AND NOT is_local ORDER BY shard_num, replica_num", chQuote(clusterName)) + if err = b.ch.SelectContext(ctx, &hosts, query); err != nil { + return nil, errors.Wrapf(err, "can't list remote hosts of cluster '%s'", clusterName) + } + return hosts, nil +} + +// ensureClickHouseConnected connects b.ch when the previous command already closed it, the returned function undoes it +func (b *Backuper) ensureClickHouseConnected() (func(), error) { + if b.ch.IsOpen { + return func() {}, nil + } + if err := b.ch.Connect(); err != nil { + return nil, errors.Wrap(err, "can't connect to clickhouse") + } + return b.ch.Close, nil +} + +// startEmbeddedClusterWorkers enqueues command on every remote node of the cluster through its system.backup_actions +// table, each node executes it with its own `clickhouse-backup server`, returns the nodes for waitEmbeddedClusterWorkers +func (b *Backuper) startEmbeddedClusterWorkers(ctx context.Context, command string) ([]embeddedClusterHost, error) { + closeConnection, err := b.ensureClickHouseConnected() + if err != nil { + return nil, err + } + defer closeConnection() + hosts, err := b.getEmbeddedClusterRemoteHosts(ctx) + if err != nil { + return nil, err + } + for _, host := range hosts { + insertSQL := fmt.Sprintf("INSERT INTO FUNCTION %s (command) VALUES (%s)", host.remoteBackupActions(b.cfg.ClickHouse.Username, b.cfg.ClickHouse.Password, b.cfg.ClickHouse.Secure), chQuote(command)) + if err = b.ch.QueryContext(ctx, insertSQL); err != nil { + return nil, errors.Wrapf(err, "can't start `%s` on %s, every node of cluster '%s' needs a running `clickhouse-backup server` with `api.create_integration_tables: true`", command, host, b.cfg.ClickHouse.UseEmbeddedBackupRestoreCluster) + } + log.Info().Msgf("embedded ON CLUSTER worker started `%s` on %s", command, host) + } + return hosts, nil +} + +// waitEmbeddedClusterWorkers polls system.backup_actions of every host until command leaves the "in progress" status, +// fails when any worker ends with an error +func (b *Backuper) waitEmbeddedClusterWorkers(ctx context.Context, hosts []embeddedClusterHost, command string) error { + if len(hosts) == 0 { + return nil + } + closeConnection, err := b.ensureClickHouseConnected() + if err != nil { + return err + } + defer closeConnection() + type actionStatus struct { + Status string `ch:"status"` + Error string `ch:"error"` + } + pending := hosts + pollErrors := 0 + ticker := time.NewTicker(embeddedClusterPollInterval) + defer ticker.Stop() + for len(pending) > 0 { + stillPending := make([]embeddedClusterHost, 0, len(pending)) + for _, host := range pending { + rows := make([]actionStatus, 0) + query := fmt.Sprintf("SELECT status, error FROM %s WHERE command = %s ORDER BY start DESC LIMIT 1", host.remoteBackupActions(b.cfg.ClickHouse.Username, b.cfg.ClickHouse.Password, b.cfg.ClickHouse.Secure), chQuote(command)) + if err = b.ch.SelectContext(ctx, &rows, query); err != nil { + pollErrors++ + if pollErrors > b.cfg.General.RetriesOnFailure { + return errors.Wrapf(err, "can't read status of `%s` from %s", command, host) + } + log.Warn().Msgf("can't read status of `%s` from %s, will retry: %v", command, host, err) + stillPending = append(stillPending, host) + continue + } + pollErrors = 0 + if len(rows) == 0 { + return errors.Errorf("`%s` is not registered in system.backup_actions on %s", command, host) + } + switch rows[0].Status { + case status.InProgressStatus: + stillPending = append(stillPending, host) + case status.SuccessStatus: + log.Info().Msgf("embedded ON CLUSTER worker finished `%s` on %s", command, host) + default: + return errors.Errorf("`%s` on %s finished with status '%s': %s", command, host, rows[0].Status, rows[0].Error) + } + } + pending = stillPending + if len(pending) == 0 { + break + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-ticker.C: + } + } + return nil +} + +// runEmbeddedClusterWorkers starts command on every remote node of the cluster and waits until all of them finish +func (b *Backuper) runEmbeddedClusterWorkers(ctx context.Context, command string) error { + hosts, err := b.startEmbeddedClusterWorkers(ctx, command) + if err != nil { + return err + } + return b.waitEmbeddedClusterWorkers(ctx, hosts, command) +} + +// applyEmbeddedClusterLayout picks the remote layout of this node's clickhouse-backup files from the backup tags: +// backups tagged `cluster=` keep per table .json and shadow data under shards/N/replicas/M, older ones use the flat layout +func (b *Backuper) applyEmbeddedClusterLayout(backupMetadata *metadata.BackupMetadata) { + b.embeddedClusterFilesPrefix = "" + if backupMetadata.EmbeddedCluster() != "" { + b.embeddedClusterFilesPrefix = b.embeddedClusterPrefix + } +} + +// embeddedBackupTags renders the Tags of a new embedded backup, `cluster=` marks the ON CLUSTER layout +func (b *Backuper) embeddedBackupTags() string { + if b.embeddedClusterName == "" { + return "embedded" + } + return "embedded,cluster=" + b.embeddedClusterName +} + +// deleteLocalOnEmbeddedClusterWorkers runs `delete local ` on every remote node through system.backup_actions, +// the API delete handler is synchronous, so the INSERT returns after the node finished, a node without this backup is fine +func (b *Backuper) deleteLocalOnEmbeddedClusterWorkers(ctx context.Context, backupName string) error { + closeConnection, err := b.ensureClickHouseConnected() + if err != nil { + return err + } + defer closeConnection() + hosts, err := b.getEmbeddedClusterRemoteHosts(ctx) + if err != nil { + return err + } + command := "delete local --" + embeddedOnClusterWorkerFlag + " " + shlexQuote(backupName) + for _, host := range hosts { + insertSQL := fmt.Sprintf("INSERT INTO FUNCTION %s (command) VALUES (%s)", host.remoteBackupActions(b.cfg.ClickHouse.Username, b.cfg.ClickHouse.Password, b.cfg.ClickHouse.Secure), chQuote(command)) + if err = b.ch.QueryContext(ctx, insertSQL); err != nil { + if strings.Contains(err.Error(), "is not found on local storage") { + log.Info().Msgf("embedded ON CLUSTER `%s` on %s: nothing to delete", command, host) + continue + } + return errors.Wrapf(err, "can't run `%s` on %s", command, host) + } + log.Info().Msgf("embedded ON CLUSTER `%s` done on %s", command, host) + } + return nil +} diff --git a/pkg/backup/embedded_cluster_test.go b/pkg/backup/embedded_cluster_test.go new file mode 100644 index 000000000..b10badf96 --- /dev/null +++ b/pkg/backup/embedded_cluster_test.go @@ -0,0 +1,112 @@ +package backup + +import ( + "path" + "testing" + + "github.com/google/shlex" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/Altinity/clickhouse-backup/v2/pkg/config" + "github.com/Altinity/clickhouse-backup/v2/pkg/metadata" +) + +func TestEmbeddedClusterCreateWorkerCommand(t *testing.T) { + command := embeddedClusterCreateWorkerCommand("create_remote", "backup 1", "base", "db.t*,db2.x", []string{"2024-01", "(1,'a b')"}, true, true, true) + assert.Equal(t, `create_remote --tables="db.t*,db2.x" --partitions="2024-01" --partitions="(1,'a b')" --diff-from-remote="base" --schema --skip-check-parts-columns --delete-source --embedded-on-cluster-worker "backup 1"`, command) + // the API server parses the command back with shlex, quoted values must survive as single arguments + args, err := shlex.Split(command) + require.NoError(t, err) + assert.Equal(t, []string{"create_remote", "--tables=db.t*,db2.x", "--partitions=2024-01", "--partitions=(1,'a b')", "--diff-from-remote=base", "--schema", "--skip-check-parts-columns", "--delete-source", "--embedded-on-cluster-worker", "backup 1"}, args) + + // empty values and unset flags are omitted + assert.Equal(t, `create --embedded-on-cluster-worker "b"`, embeddedClusterCreateWorkerCommand("create", "b", "", "", nil, false, false, false)) +} + +func TestEmbeddedClusterRestoreWorkerCommand(t *testing.T) { + command := embeddedClusterRestoreWorkerCommand("restore", "b", "db.*", []string{"db:db2"}, []string{"t:t2"}, []string{"p1"}, false, true, true, true, true, true) + assert.Equal(t, `restore --tables="db.*" --restore-database-mapping="db:db2" --restore-table-mapping="t:t2" --partitions="p1" --data --rm --ignore-dependencies --restore-schema-as-attach --skip-empty-tables --embedded-on-cluster-worker "b"`, command) + assert.Equal(t, `restore_remote --schema --embedded-on-cluster-worker "b"`, embeddedClusterRestoreWorkerCommand("restore_remote", "b", "", nil, nil, nil, true, false, false, false, false, false)) +} + +func TestShlexQuote(t *testing.T) { + for _, v := range []string{`plain`, `with space`, `with "quotes"`, `back\slash`, `mixed "a\b" c`} { + args, err := shlex.Split("cmd " + shlexQuote(v)) + require.NoError(t, err, v) + assert.Equal(t, []string{"cmd", v}, args, v) + } +} + +func TestChQuote(t *testing.T) { + assert.Equal(t, `'plain'`, chQuote("plain")) + assert.Equal(t, `'it\'s'`, chQuote("it's")) + assert.Equal(t, `'back\\slash\'q'`, chQuote(`back\slash'q`)) +} + +func TestEmbeddedClusterHostRemoteBackupActions(t *testing.T) { + host := embeddedClusterHost{HostName: "clickhouse-2", Port: 9000, ShardNum: 1, ReplicaNum: 2} + assert.Equal(t, `remote('clickhouse-2:9000', 'system', 'backup_actions', 'default', 'p\'w')`, host.remoteBackupActions("default", "p'w", false)) + // system.clusters has no `secure` column, the transport follows clickhouse.secure of the initiator + host.Port = 9440 + assert.Equal(t, `remoteSecure('clickhouse-2:9440', 'system', 'backup_actions', 'backup', '')`, host.remoteBackupActions("backup", "", true)) + assert.Equal(t, "clickhouse-2:9440 (shard 1, replica 2)", host.String()) +} + +func TestEmbeddedMetadataDir(t *testing.T) { + existing := map[string]bool{path.Join("/var/lib/backups/b1", "shards/1/replicas/2", "metadata"): true} + exists := func(p string) bool { return existing[p] } + // cluster prefixed layout wins when present + assert.Equal(t, "/var/lib/backups/b1/shards/1/replicas/2/metadata", embeddedMetadataDir("/var/lib/backups/b1", "shards/1/replicas/2", exists)) + // legacy flat layout of backups made before ON CLUSTER support + assert.Equal(t, "/var/lib/backups/b2/metadata", embeddedMetadataDir("/var/lib/backups/b2", "shards/1/replicas/2", exists)) + // no cluster mode, never probes + assert.Equal(t, "/var/lib/backups/b1/metadata", embeddedMetadataDir("/var/lib/backups/b1", "", func(string) bool { t.Fatal("must not probe"); return false })) +} + +func TestRemoteTableMetadataJSON(t *testing.T) { + assert.Equal(t, "b1/shards/1/replicas/2/metadata/db/t%20x.json", remoteTableMetadataJSON("b1", "shards/1/replicas/2", "db", "t x")) + assert.Equal(t, "b1/metadata/db/t.json", remoteTableMetadataJSON("b1", "", "db", "t")) +} + +func TestEmbeddedOnClusterWorkerValidation(t *testing.T) { + cfg := &config.Config{} + b := NewBackuper(cfg) + b.ch.BreakConnectOnError = true + // the flag is a no-op when unset, whatever the config is + require.NoError(t, b.validateEmbeddedOnClusterWorker()) + assert.False(t, b.embeddedClusterInitiator()) + + b.EmbeddedOnClusterWorker = true + require.Error(t, b.validateEmbeddedOnClusterWorker()) + cfg.ClickHouse.UseEmbeddedBackupRestore = true + require.Error(t, b.validateEmbeddedOnClusterWorker()) + cfg.ClickHouse.UseEmbeddedBackupRestoreCluster = "{cluster}" + require.NoError(t, b.validateEmbeddedOnClusterWorker()) + // a worker never drives other nodes + assert.False(t, b.embeddedClusterInitiator()) + assert.False(t, b.embeddedClusterWorkerSkipsDDL()) + cfg.General.RestoreSchemaOnCluster = "{cluster}" + assert.True(t, b.embeddedClusterWorkerSkipsDDL()) + + b.EmbeddedOnClusterWorker = false + assert.True(t, b.embeddedClusterInitiator()) + assert.False(t, b.embeddedClusterWorkerSkipsDDL()) + b.DryRun = true + assert.False(t, b.embeddedClusterInitiator()) +} + +func TestEmbeddedClusterTagsAndLayout(t *testing.T) { + b := NewBackuper(&config.Config{}) + b.ch.BreakConnectOnError = true + assert.Equal(t, "embedded", b.embeddedBackupTags()) + b.embeddedClusterName = "sharded_cluster" + b.embeddedClusterPrefix = "shards/1/replicas/2" + assert.Equal(t, "embedded,cluster=sharded_cluster", b.embeddedBackupTags()) + + // the remote layout of per table .json and shadow data follows the backup tags, not the current config + b.applyEmbeddedClusterLayout(&metadata.BackupMetadata{Tags: "embedded"}) + assert.Equal(t, "", b.embeddedClusterFilesPrefix) + b.applyEmbeddedClusterLayout(&metadata.BackupMetadata{Tags: b.embeddedBackupTags()}) + assert.Equal(t, "shards/1/replicas/2", b.embeddedClusterFilesPrefix) +} diff --git a/pkg/backup/restore.go b/pkg/backup/restore.go index f5feca52d..57a3e921b 100644 --- a/pkg/backup/restore.go +++ b/pkg/backup/restore.go @@ -76,11 +76,29 @@ func (b *Backuper) Restore(backupName, tablePattern string, databaseMapping, tab doRestoreData := (!schemaOnly && !rbacOnly && !configsOnly) || dataOnly + if err := b.validateEmbeddedOnClusterWorker(); err != nil { + return err + } + if b.EmbeddedOnClusterWorker { + // rbac, configs and named collections are restored by the initiator node only, https://github.com/Altinity/clickhouse-backup/issues/928 + if rbacOnly || configsOnly || namedCollectionsOnly { + return errors.Errorf("--%s restores tables only, --rbac-only, --configs-only and --named-collections-only are executed by the initiator node", embeddedOnClusterWorkerFlag) + } + restoreRBAC, restoreConfigs, restoreNamedCollections = false, false, false + } if err := b.ch.Connect(); err != nil { return errors.Wrap(err, "can't connect to clickhouse") } defer b.ch.Close() + // every node of RESTORE ... ON CLUSTER reads its own shards/N/replicas/M/metadata/*.sql, so the workers fix them + // before the initiator issues the RESTORE SQL, https://github.com/Altinity/clickhouse-backup/issues/928 + if b.embeddedClusterInitiator() && !b.skipEmbeddedClusterFanOut && !rbacOnly && !configsOnly && !namedCollectionsOnly { + if err := b.runEmbeddedClusterWorkers(ctx, embeddedClusterRestoreWorkerCommand("restore", backupName, tablePattern, databaseMapping, tableMapping, partitions, schemaOnly, dataOnly, dropExists, ignoreDependencies, schemaAsAttach, skipEmptyTables)); err != nil { + return errors.Wrap(err, "embedded ON CLUSTER workers") + } + } + prologue, err := b.restorePrologue(ctx, backupName, tablePattern, partitions, skipProjections, schemaOnly, dataOnly, dropExists, ignoreDependencies, restoreRBAC, rbacOnly, restoreConfigs, configsOnly, restoreNamedCollections, namedCollectionsOnly, resume, schemaAsAttach, skipEmptyTables, doRestoreData, false) if err != nil { return err @@ -111,7 +129,7 @@ func (b *Backuper) Restore(backupName, tablePattern string, databaseMapping, tab } } // do not create UDF when use --data, --rbac-only, --configs-only flags, https://github.com/Altinity/clickhouse-backup/issues/697 - if schemaOnly || (schemaOnly == dataOnly) { + if (schemaOnly || (schemaOnly == dataOnly)) && !b.embeddedClusterWorkerSkipsDDL() { if funcErr := b.restoreFunctions(ctx, backupMetadata); funcErr != nil { return errors.Wrap(funcErr, "restoreFunctions") } @@ -282,6 +300,7 @@ func (b *Backuper) restorePrologue(ctx context.Context, backupName, tablePattern if err = b.resolveEmbeddedClusterShardReplica(ctx); err != nil { return nil, errors.Wrap(err, "resolveEmbeddedClusterShardReplica") } + b.applyEmbeddedClusterLayout(&backupMetadata) } // report what would be restored before the first side effect, no CREATE/DROP DATABASE, no RBAC, @@ -292,7 +311,7 @@ func (b *Backuper) restorePrologue(ctx context.Context, backupName, tablePattern return result, b.dryRunRestore(ctx, backupName, backupMetadata, disks, tablePattern, partitions, schemaOnly, dataOnly, dropExists, rbacOnly, configsOnly, namedCollectionsOnly, restoreRBAC, restoreConfigs, restoreNamedCollections, skipEmptyTables) } - if schemaOnly || doRestoreData { + if (schemaOnly || doRestoreData) && !b.embeddedClusterWorkerSkipsDDL() { for _, database := range backupMetadata.Databases { targetDB := database.Name if !IsInformationSchema(targetDB) { @@ -389,9 +408,9 @@ func (b *Backuper) restorePrologue(ctx context.Context, backupName, tablePattern if tablePattern == "" { tablePattern = "*" } - metadataPath := path.Join(b.DefaultDataPath, "backup", backupName, "metadata") + metadataPath := b.localMetadataDir(path.Join(b.DefaultDataPath, "backup", backupName)) if b.isEmbedded && b.cfg.ClickHouse.EmbeddedBackupDisk != "" { - metadataPath = path.Join(b.EmbeddedBackupDataPath, backupName, "metadata") + metadataPath = b.localMetadataDir(path.Join(b.EmbeddedBackupDataPath, backupName)) } tablesForRestore, partitionsNames, err = b.getTablesForRestoreLocal(ctx, backupName, metadataPath, tablePattern, dropExists, partitions) @@ -441,7 +460,7 @@ func (b *Backuper) restorePrologue(ctx context.Context, backupName, tablePattern // Safety check: prevent accidental data loss when restore_schema_on_cluster is set // via config but RESTORE_SCHEMA_ON_CLUSTER env var is empty, and --rm/--drop is not provided. // https://github.com/Altinity/clickhouse-backup/issues/1325 - if !dropExists && !b.resume && b.cfg.General.RestoreSchemaOnCluster != "" && os.Getenv("RESTORE_SCHEMA_ON_CLUSTER") == "" { + if !dropExists && !b.resume && b.cfg.General.RestoreSchemaOnCluster != "" && os.Getenv("RESTORE_SCHEMA_ON_CLUSTER") == "" && !b.EmbeddedOnClusterWorker { if err = b.checkClusterTablesHaveDataBeforeDrop(ctx, tablesForRestore, version); err != nil { return nil, errors.Wrap(err, "checkClusterTablesHaveDataBeforeDrop") } @@ -1719,8 +1738,10 @@ func (b *Backuper) dropExistPartitions(ctx context.Context, tablesForRestore Lis func (b *Backuper) RestoreSchema(ctx context.Context, backupName string, backupMetadata metadata.BackupMetadata, disks []clickhouse.Disk, tablesForRestore ListOfTables, ignoreDependencies bool, version int, schemaAsAttach bool) error { startRestoreSchema := time.Now() databaseEnginesForRestore := b.prepareDatabaseEnginesMap(backupMetadata.Databases) - if dropErr := b.dropExistsTables(tablesForRestore, databaseEnginesForRestore, ignoreDependencies, version, schemaAsAttach); dropErr != nil { - return errors.Wrap(dropErr, "dropExistsTables") + if !b.embeddedClusterWorkerSkipsDDL() { + if dropErr := b.dropExistsTables(tablesForRestore, databaseEnginesForRestore, ignoreDependencies, version, schemaAsAttach); dropErr != nil { + return errors.Wrap(dropErr, "dropExistsTables") + } } var restoreErr error if b.isEmbedded { @@ -1762,6 +1783,11 @@ func (b *Backuper) restoreSchemaEmbedded(ctx context.Context, backupName string, if err != nil { return errors.Wrap(err, "fixEmbeddedMetadata") } + if b.EmbeddedOnClusterWorker { + // the initiator issues RESTORE ... ON CLUSTER for every node once all workers fixed their metadata + log.Info().Msgf("--%s: %s metadata fixed, skip RESTORE SQL", embeddedOnClusterWorkerFlag, b.embeddedClusterPrefix) + return nil + } return b.restoreEmbedded(ctx, backupName, true, false, version, tablesForRestore, nil) } @@ -2356,6 +2382,10 @@ func (b *Backuper) waitForObjectStorageCleanup(ctx context.Context, disks []clic } func (b *Backuper) restoreDataEmbedded(ctx context.Context, backupName string, dataOnly bool, version int, tablesForRestore ListOfTables, partitionsNameList map[metadata.TableTitle][]string) error { + if b.EmbeddedOnClusterWorker { + log.Info().Msgf("--%s: skip RESTORE SQL, the initiator restores data ON CLUSTER", embeddedOnClusterWorkerFlag) + return nil + } return b.restoreEmbedded(ctx, backupName, false, dataOnly, version, tablesForRestore, partitionsNameList) } diff --git a/pkg/backup/restore_remote.go b/pkg/backup/restore_remote.go index 2e04ea10b..02ced8f92 100644 --- a/pkg/backup/restore_remote.go +++ b/pkg/backup/restore_remote.go @@ -27,6 +27,21 @@ import ( func (b *Backuper) RestoreFromRemote(backupName, tablePattern string, databaseMapping, tableMapping, partitions, skipProjections []string, schemaOnly, dataOnly, dropExists, ignoreDependencies, restoreRBAC, rbacOnly, restoreConfigs, configsOnly, restoreNamedCollections, namedCollectionsOnly, resume, schemaAsAttach, replicatedCopyToDetached, skipEmptyTables, hardlinkExistsFiles, streaming bool, version string, commandId int) error { // don't need to create pid separately because we combine Download+Restore defer pidlock.RemovePidFile(backupName) + // the worker nodes run `restore_remote --embedded-on-cluster-worker` below, Restore must not start `restore` on them + b.skipEmbeddedClusterFanOut = true + // every node of RESTORE ... ON CLUSTER reads its own shards/N/replicas/M part, so the workers download and fix it + // before the initiator issues the RESTORE SQL, https://github.com/Altinity/clickhouse-backup/issues/928 + if b.embeddedClusterInitiator() && !streaming && !rbacOnly && !configsOnly && !namedCollectionsOnly { + ctx, cancel, ctxErr := status.Current.GetContextWithCancel(commandId) + if ctxErr != nil { + return pkgerrors.Wrap(ctxErr, "RestoreFromRemote GetContextWithCancel") + } + workerErr := b.runEmbeddedClusterWorkers(ctx, embeddedClusterRestoreWorkerCommand("restore_remote", backupName, tablePattern, databaseMapping, tableMapping, partitions, schemaOnly, dataOnly, dropExists, ignoreDependencies, schemaAsAttach, skipEmptyTables)) + cancel() + if workerErr != nil { + return pkgerrors.Wrap(workerErr, "embedded ON CLUSTER workers") + } + } // dry-run has no side effects, so the download report is re-labeled the same way as without --streaming if streaming && !b.DryRun { err := b.restoreFromRemoteStreaming(backupName, tablePattern, databaseMapping, tableMapping, partitions, skipProjections, schemaOnly, dataOnly, dropExists, ignoreDependencies, restoreRBAC, rbacOnly, restoreConfigs, configsOnly, restoreNamedCollections, namedCollectionsOnly, resume, schemaAsAttach, replicatedCopyToDetached, skipEmptyTables, hardlinkExistsFiles, version, commandId) diff --git a/pkg/backup/upload.go b/pkg/backup/upload.go index a3f29cd7f..7301ccc20 100644 --- a/pkg/backup/upload.go +++ b/pkg/backup/upload.go @@ -65,6 +65,9 @@ func (b *Backuper) Upload(backupName string, deleteSource bool, diffFrom, diffFr if err = b.validateUploadParams(ctx, backupName, diffFrom, diffFromRemote); err != nil { return errors.Wrap(err, "validateUploadParams") } + if err = b.validateEmbeddedOnClusterWorker(); err != nil { + return err + } if b.cfg.General.RemoteStorage == "custom" { if b.DryRun { return errors.New("--dry-run is not supported for `remote_storage: custom`, the upload command is executed by an external script") @@ -209,6 +212,11 @@ func (b *Backuper) uploadPrologue(ctx context.Context, backupName, diffFrom, dif backupExistsOnRemote := false for i := range remoteBackups { if backupName == remoteBackups[i].BackupName { + // the nodes of an embedded ON CLUSTER backup share one remote backup folder: a worker joins it, and the initiator + // can see it as broken (no metadata.json yet) while the workers upload in parallel, https://github.com/Altinity/clickhouse-backup/issues/928 + if b.EmbeddedOnClusterWorker || (remoteBackups[i].Broken != "" && b.embeddedClusterInitiator()) { + continue + } backupExistsOnRemote = true if !b.resume { return nil, errors.Errorf("'%s' already exists on remote storage", backupName) @@ -227,6 +235,7 @@ func (b *Backuper) uploadPrologue(ctx context.Context, backupName, diffFrom, dif if err = b.resolveEmbeddedClusterShardReplica(ctx); err != nil { return nil, errors.Wrap(err, "resolveEmbeddedClusterShardReplica") } + b.applyEmbeddedClusterLayout(backupMetadata) } // will ignore partitions cause can't manipulate .backup if b.isEmbedded { @@ -354,6 +363,26 @@ func (b *Backuper) uploadOneTable(ctx context.Context, backupName string, delete // then applies remote/local retention and optionally removes the local source backup func (b *Backuper) uploadEpilogue(ctx context.Context, backupName string, deleteSource bool, tablesForUpload ListOfTables, backupMetadata *metadata.BackupMetadata, disks []clickhouse.Disk, compressedDataSize, metadataSize int64, doUploadData, schemaOnly, rbacOnly, configsOnly, namedCollectionsOnly bool, backupVersion string, startUpload time.Time) error { var err error + if b.EmbeddedOnClusterWorker { + // rbac, configs, named collections, .backup, metadata.json, the manifest and the remote retention belong to the + // initiator node of an embedded ON CLUSTER backup, https://github.com/Altinity/clickhouse-backup/issues/928 + log.Info().Fields(map[string]interface{}{ + "backup": backupName, + "operation": "upload", + "duration": utils.HumanizeDuration(time.Since(startUpload)), + "upload_size": utils.FormatBytes(uint64(compressedDataSize) + uint64(metadataSize)), + "version": backupVersion, + }).Msgf("done --%s", embeddedOnClusterWorkerFlag) + if err = b.RemoveOldBackupsLocal(ctx, false, nil); err != nil { + return errors.Wrap(err, "can't remove old local backups") + } + if b.cfg.General.BackupsToKeepLocal >= 0 && deleteSource { + if err = b.RemoveBackupLocal(ctx, backupName, disks, true); err != nil { + return errors.Wrap(err, "can't explicitly delete local source backup") + } + } + return nil + } // upload rbac for backup, if not configsOnly if rbacOnly || configsOnly == rbacOnly == namedCollectionsOnly == false { if backupMetadata.RBACSize, err = b.uploadRBACData(ctx, backupName); err != nil { @@ -535,9 +564,9 @@ func (b *Backuper) uploadSingleBackupFile(ctx context.Context, localFile, remote } func (b *Backuper) prepareTableListToUpload(ctx context.Context, backupName string, tablePattern string, partitions []string) (tablesForUpload ListOfTables, err error) { - metadataPath := path.Join(b.DefaultDataPath, "backup", backupName, "metadata") + metadataPath := b.localMetadataDir(path.Join(b.DefaultDataPath, "backup", backupName)) if b.isEmbedded && b.cfg.ClickHouse.EmbeddedBackupDisk != "" { - metadataPath = path.Join(b.EmbeddedBackupDataPath, backupName, "metadata") + metadataPath = b.localMetadataDir(path.Join(b.EmbeddedBackupDataPath, backupName)) } tablesForUpload, _, err = b.getTableListByPatternLocal(ctx, metadataPath, tablePattern, false, partitions) if err != nil { @@ -767,7 +796,7 @@ func (b *Backuper) uploadTableData(ctx context.Context, backupName string, delet partSuffix := splitPart.Prefix partFiles := splitPart.Files splitPartsOffset[diskName] += 1 - baseRemoteDataPath := path.Join(backupName, "shadow", common.TablePathEncode(table.Database), common.TablePathEncode(table.Table)) + baseRemoteDataPath := path.Join(backupName, b.embeddedClusterFilesPrefix, "shadow", common.TablePathEncode(table.Database), common.TablePathEncode(table.Table)) if b.cfg.GetCompressionFormat() == "none" { remotePath := path.Join(baseRemoteDataPath, diskName) remotePathFull := path.Join(remotePath, partSuffix) @@ -897,7 +926,7 @@ func (b *Backuper) uploadTableMetadataRegular(ctx context.Context, backupName st if err != nil { return 0, errors.Wrap(err, "can't marshal json") } - remoteTableMetaFile := path.Join(backupName, "metadata", common.TablePathEncode(tableMetadata.Database), fmt.Sprintf("%s.json", common.TablePathEncode(tableMetadata.Table))) + remoteTableMetaFile := remoteTableMetadataJSON(backupName, b.embeddedClusterFilesPrefix, tableMetadata.Database, tableMetadata.Table) if b.resume { isProcessed, processedSize, resumeErr := b.resumableState.IsAlreadyProcessed(remoteTableMetaFile) if resumeErr != nil { diff --git a/pkg/metadata/backup_metadata.go b/pkg/metadata/backup_metadata.go index f82eec8a1..b090fcdb9 100644 --- a/pkg/metadata/backup_metadata.go +++ b/pkg/metadata/backup_metadata.go @@ -37,6 +37,17 @@ type BackupMetadata struct { // IsPlainDisk returns true when the disk had metadata_type=plain or plain_rewritable at backup time, // such disks have no local metadata files inside the backup, the data objects live only in object_disk_path +// EmbeddedCluster returns the cluster name of an embedded BACKUP ... ON CLUSTER backup from the `cluster=` tag, +// "" for regular and single node embedded backups, https://github.com/Altinity/clickhouse-backup/issues/928 +func (b *BackupMetadata) EmbeddedCluster() string { + for _, tag := range strings.Split(b.Tags, ",") { + if strings.HasPrefix(tag, "cluster=") { + return strings.TrimPrefix(tag, "cluster=") + } + } + return "" +} + func (b *BackupMetadata) IsPlainDisk(diskName string) bool { metadataType, exists := b.DiskMetadataTypes[diskName] return exists && (metadataType == "plain" || metadataType == "plain_rewritable") diff --git a/pkg/metadata/backup_metadata_test.go b/pkg/metadata/backup_metadata_test.go new file mode 100644 index 000000000..3c0e12c01 --- /dev/null +++ b/pkg/metadata/backup_metadata_test.go @@ -0,0 +1,15 @@ +package metadata + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestEmbeddedCluster(t *testing.T) { + assert.Equal(t, "", (&BackupMetadata{Tags: "regular"}).EmbeddedCluster()) + assert.Equal(t, "", (&BackupMetadata{Tags: "embedded"}).EmbeddedCluster()) + assert.Equal(t, "", (&BackupMetadata{}).EmbeddedCluster()) + assert.Equal(t, "my_cluster", (&BackupMetadata{Tags: "embedded,cluster=my_cluster"}).EmbeddedCluster()) + assert.Equal(t, "c1", (&BackupMetadata{Tags: "cluster=c1,embedded"}).EmbeddedCluster()) +} diff --git a/pkg/server/server.go b/pkg/server/server.go index 5aac03ddb..c95762aaf 100644 --- a/pkg/server/server.go +++ b/pkg/server/server.go @@ -1129,6 +1129,11 @@ func (api *APIServer) httpCreateHandler(w http.ResponseWriter, r *http.Request) fullCommand += " --resume" } + embeddedOnClusterWorker := false + if _, exist := api.getQueryParameter(query, "embedded-on-cluster-worker"); exist { + embeddedOnClusterWorker = true + fullCommand += " --embedded-on-cluster-worker" + } if dryRun { fullCommand += " --dry-run" } @@ -1149,6 +1154,7 @@ func (api *APIServer) httpCreateHandler(w http.ResponseWriter, r *http.Request) if dryRun { commandId, _ := status.Current.Start(fullCommand) b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker b.DryRun = true err = b.CreateBackup(backupName, diffFromRemote, tablePattern, partitionsToBackup, schemaOnly, createRBAC, rbacOnly, createConfigs, configsOnly, createNamedCollections, namedCollectionsOnly, checkPartsColumns, skipProjections, resume, api.clickhouseBackupVersion, commandId) status.Current.SetResult(commandId, b.DryRunResult.JSONString()) @@ -1161,6 +1167,7 @@ func (api *APIServer) httpCreateHandler(w http.ResponseWriter, r *http.Request) go func() { err, _ := api.metrics.ExecuteWithMetrics("create", 0, func() error { b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker return b.CreateBackup(backupName, diffFromRemote, tablePattern, partitionsToBackup, schemaOnly, createRBAC, rbacOnly, createConfigs, configsOnly, createNamedCollections, namedCollectionsOnly, checkPartsColumns, skipProjections, resume, api.clickhouseBackupVersion, commandId) }) if err != nil { @@ -1287,6 +1294,11 @@ func (api *APIServer) httpCreateRemoteHandler(w http.ResponseWriter, r *http.Req streaming = true fullCommand += " --streaming" } + embeddedOnClusterWorker := false + if _, exist := api.getQueryParameter(query, "embedded-on-cluster-worker"); exist { + embeddedOnClusterWorker = true + fullCommand += " --embedded-on-cluster-worker" + } if dryRun { fullCommand += " --dry-run" } @@ -1306,6 +1318,7 @@ func (api *APIServer) httpCreateRemoteHandler(w http.ResponseWriter, r *http.Req if dryRun { commandId, _ := status.Current.Start(fullCommand) b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker b.DryRun = true err = b.CreateToRemote(backupName, deleteSource, diffFrom, diffFromRemote, tablePattern, partitionsToBackup, skipProjections, schemaOnly, backupRBAC, rbacOnly, backupConfigs, configsOnly, backupNamedCollections, namedCollectionsOnly, skipCheckPartsColumns, resume, streaming, api.clickhouseBackupVersion, commandId) status.Current.SetResult(commandId, b.DryRunResult.JSONString()) @@ -1318,6 +1331,7 @@ func (api *APIServer) httpCreateRemoteHandler(w http.ResponseWriter, r *http.Req go func() { err, _ := api.metrics.ExecuteWithMetrics("create_remote", 0, func() error { b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker return b.CreateToRemote(backupName, deleteSource, diffFrom, diffFromRemote, tablePattern, partitionsToBackup, skipProjections, schemaOnly, backupRBAC, rbacOnly, backupConfigs, configsOnly, backupNamedCollections, namedCollectionsOnly, skipCheckPartsColumns, resume, streaming, api.clickhouseBackupVersion, commandId) }) if err != nil { @@ -1643,6 +1657,11 @@ func (api *APIServer) httpUploadHandler(w http.ResponseWriter, r *http.Request) if resume { fullCommand += " --resume" } + embeddedOnClusterWorker := false + if _, exist := api.getQueryParameter(query, "embedded-on-cluster-worker"); exist { + embeddedOnClusterWorker = true + fullCommand += " --embedded-on-cluster-worker" + } if dryRun { fullCommand += " --dry-run" } @@ -1660,6 +1679,7 @@ func (api *APIServer) httpUploadHandler(w http.ResponseWriter, r *http.Request) if dryRun { commandId, _ := status.Current.Start(fullCommand) b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker b.DryRun = true err = b.Upload(name, deleteSource, diffFrom, diffFromRemote, tablePattern, partitionsToBackup, skipProjections, schemaOnly, rbacOnly, configsOnly, namedCollectionsOnly, resume, api.clickhouseBackupVersion, commandId) status.Current.SetResult(commandId, b.DryRunResult.JSONString()) @@ -1672,6 +1692,7 @@ func (api *APIServer) httpUploadHandler(w http.ResponseWriter, r *http.Request) go func() { err, _ := api.metrics.ExecuteWithMetrics("upload", 0, func() error { b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker return b.Upload(name, deleteSource, diffFrom, diffFromRemote, tablePattern, partitionsToBackup, skipProjections, schemaOnly, rbacOnly, configsOnly, namedCollectionsOnly, resume, api.clickhouseBackupVersion, commandId) }) if err != nil { @@ -2145,6 +2166,11 @@ func (api *APIServer) httpRestoreHandler(w http.ResponseWriter, r *http.Request) } } + embeddedOnClusterWorker := false + if _, exist := api.getQueryParameter(query, "embedded-on-cluster-worker"); exist { + embeddedOnClusterWorker = true + fullCommand += " --embedded-on-cluster-worker" + } if dryRun { fullCommand += " --dry-run" } @@ -2163,6 +2189,7 @@ func (api *APIServer) httpRestoreHandler(w http.ResponseWriter, r *http.Request) if dryRun { commandId, _ := status.Current.Start(fullCommand) b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker b.DryRun = true err = b.Restore(name, tablePattern, databaseMappingToRestore, tableMappingToRestore, partitionsToBackup, skipProjections, schemaOnly, dataOnly, dropExists, ignoreDependencies, restoreRBAC, rbacOnly, restoreConfigs, configsOnly, restoreNamedCollections, namedCollectionsOnly, resume, restoreSchemaAsAttach, replicatedCopyToDetached, skipEmptyTables, api.clickhouseBackupVersion, commandId) status.Current.SetResult(commandId, b.DryRunResult.JSONString()) @@ -2175,6 +2202,7 @@ func (api *APIServer) httpRestoreHandler(w http.ResponseWriter, r *http.Request) go func() { err, _ := api.metrics.ExecuteWithMetrics("restore", 0, func() error { b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker return b.Restore(name, tablePattern, databaseMappingToRestore, tableMappingToRestore, partitionsToBackup, skipProjections, schemaOnly, dataOnly, dropExists, ignoreDependencies, restoreRBAC, rbacOnly, restoreConfigs, configsOnly, restoreNamedCollections, namedCollectionsOnly, resume, restoreSchemaAsAttach, replicatedCopyToDetached, skipEmptyTables, api.clickhouseBackupVersion, commandId) }) if metricsErr := api.UpdateBackupMetrics(context.Background(), true); metricsErr != nil { @@ -2446,6 +2474,11 @@ func (api *APIServer) httpRestoreRemoteHandler(w http.ResponseWriter, r *http.Re } } + embeddedOnClusterWorker := false + if _, exist := api.getQueryParameter(query, "embedded-on-cluster-worker"); exist { + embeddedOnClusterWorker = true + fullCommand += " --embedded-on-cluster-worker" + } if dryRun { fullCommand += " --dry-run" } @@ -2464,6 +2497,7 @@ func (api *APIServer) httpRestoreRemoteHandler(w http.ResponseWriter, r *http.Re if dryRun { commandId, _ := status.Current.Start(fullCommand) b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker b.DryRun = true b.DiskLimit = diskLimit err = b.RestoreFromRemote(name, tablePattern, databaseMappingToRestore, tableMappingToRestore, partitionsToBackup, skipProjections, schemaOnly, dataOnly, dropExists, ignoreDependencies, restoreRBAC, rbacOnly, restoreConfigs, configsOnly, restoreNamedCollections, namedCollectionsOnly, resume, restoreSchemaAsAttach, replicatedCopyToDetached, skipEmptyTables, hardlinkExistsFiles, streaming, api.clickhouseBackupVersion, commandId) @@ -2477,6 +2511,7 @@ func (api *APIServer) httpRestoreRemoteHandler(w http.ResponseWriter, r *http.Re go func() { err, _ := api.metrics.ExecuteWithMetrics("restore_remote", 0, func() error { b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker b.DiskLimit = diskLimit return b.RestoreFromRemote(name, tablePattern, databaseMappingToRestore, tableMappingToRestore, partitionsToBackup, skipProjections, schemaOnly, dataOnly, dropExists, ignoreDependencies, restoreRBAC, rbacOnly, restoreConfigs, configsOnly, restoreNamedCollections, namedCollectionsOnly, resume, restoreSchemaAsAttach, replicatedCopyToDetached, skipEmptyTables, hardlinkExistsFiles, streaming, api.clickhouseBackupVersion, commandId) }) @@ -2585,6 +2620,11 @@ func (api *APIServer) httpDownloadHandler(w http.ResponseWriter, r *http.Request diskLimit = parsedDiskLimit fullCommand += fmt.Sprintf(" --disk-limit=%d", diskLimit) } + embeddedOnClusterWorker := false + if _, exist := api.getQueryParameter(query, "embedded-on-cluster-worker"); exist { + embeddedOnClusterWorker = true + fullCommand += " --embedded-on-cluster-worker" + } if dryRun { fullCommand += " --dry-run" } @@ -2602,6 +2642,7 @@ func (api *APIServer) httpDownloadHandler(w http.ResponseWriter, r *http.Request if dryRun { commandId, _ := status.Current.Start(fullCommand) b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker b.DryRun = true b.DiskLimit = diskLimit err = b.Download(name, tablePattern, partitionsToBackup, schemaOnly, rbacOnly, configsOnly, namedCollectionsOnly, resume, hardlinkExistsFiles, api.clickhouseBackupVersion, commandId) @@ -2615,6 +2656,7 @@ func (api *APIServer) httpDownloadHandler(w http.ResponseWriter, r *http.Request go func() { err, _ := api.metrics.ExecuteWithMetrics("download", 0, func() error { b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker b.DiskLimit = diskLimit return b.Download(name, tablePattern, partitionsToBackup, schemaOnly, rbacOnly, configsOnly, namedCollectionsOnly, resume, hardlinkExistsFiles, api.clickhouseBackupVersion, commandId) }) @@ -2660,6 +2702,10 @@ func (api *APIServer) httpDeleteHandler(w http.ResponseWriter, r *http.Request) if force { fullCommand += " --force" } + _, embeddedOnClusterWorker := api.getQueryParameter(r.URL.Query(), "embedded-on-cluster-worker") + if embeddedOnClusterWorker { + fullCommand += " --embedded-on-cluster-worker" + } // dry-run returns the report in the response body, without metrics update if dryRun { fullCommand += " --dry-run" @@ -2681,10 +2727,11 @@ func (api *APIServer) httpDeleteHandler(w http.ResponseWriter, r *http.Request) } commandId, ctx := status.Current.Start(fullCommand) b := backup.NewBackuper(cfg) + b.EmbeddedOnClusterWorker = embeddedOnClusterWorker switch vars["where"] { case "local": err, _ = api.metrics.ExecuteWithMetrics("delete", 0, func() error { - return b.RemoveBackupLocal(ctx, vars["name"], nil, force) + return b.RemoveBackupLocalOnCluster(ctx, vars["name"], force) }) case "remote": err, _ = api.metrics.ExecuteWithMetrics("delete", 0, func() error { diff --git a/test/integration/utils.go b/test/integration/utils.go index 24e27c20f..80cf45b66 100644 --- a/test/integration/utils.go +++ b/test/integration/utils.go @@ -1484,9 +1484,10 @@ func testBackupSpecifiedPartitions(t *testing.T, r *require.Assertions, env *Tes if strings.HasPrefix(remoteStorageType, "EMBEDDED") { fullBackupDir = "/var/lib/clickhouse/disks/backups" + strings.ToLower(strings.TrimPrefix(remoteStorageType, "EMBEDDED")) + "/" + fullBackupName + "/shards/1/replicas/1/data/" + dbName + "/t?" } - // embedded storage without embedded disks doesn't contain `shadow` and contain only `metadata` + // embedded storage without embedded disks doesn't contain `shadow` and contain only `metadata`, + // per table .json lives under the node's shards/{shard_num}/replicas/{replica_num}/ prefix, see issues/928 if strings.HasPrefix(remoteStorageType, "EMBEDDED") && strings.HasSuffix(remoteStorageType, "_URL") { - fullBackupDir = "/var/lib/clickhouse/backup/" + fullBackupName + "/metadata/" + dbName + "/t?.json" + fullBackupDir = "/var/lib/clickhouse/backup/" + fullBackupName + "/shards/1/replicas/1/metadata/" + dbName + "/t?.json" } out, err = env.DockerExecOut("clickhouse-backup", "bash", "-c", "ls -la "+fullBackupDir+" | wc -l") r.NoError(err) @@ -1555,7 +1556,7 @@ func testBackupSpecifiedPartitions(t *testing.T, r *require.Assertions, env *Tes } // embedded storage without embedded disks doesn't contain `shadow` and contain only `metadata` if strings.HasPrefix(remoteStorageType, "EMBEDDED") && strings.HasSuffix(remoteStorageType, "_URL") { - fullBackupDir = "/var/lib/clickhouse/backup/" + fullBackupName + "/metadata/" + dbName + "/t?.json" + fullBackupDir = "/var/lib/clickhouse/backup/" + fullBackupName + "/shards/1/replicas/1/metadata/" + dbName + "/t?.json" expectedLines = "2" } out, err = env.DockerExecOut("clickhouse-backup", "bash", "-c", "ls -la "+fullBackupDir+"| wc -l") @@ -1586,7 +1587,7 @@ func testBackupSpecifiedPartitions(t *testing.T, r *require.Assertions, env *Tes } //embedded backup without a disk has only local metadata if strings.HasPrefix(remoteStorageType, "EMBEDDED") && strings.HasSuffix(remoteStorageType, "_URL") { - partitionBackupDir = "/var/lib/clickhouse/backup/" + partitionBackupName + "/metadata/" + dbName + "/t?.json" + partitionBackupDir = "/var/lib/clickhouse/backup/" + partitionBackupName + "/shards/1/replicas/1/metadata/" + dbName + "/t?.json" expectedLines = "1" } out, err = env.DockerExecOut("clickhouse-backup", "bash", "-c", "ls -la "+partitionBackupDir+"| wc -l") @@ -1631,7 +1632,7 @@ func testBackupSpecifiedPartitions(t *testing.T, r *require.Assertions, env *Tes } //embedded backup without a disk has only local metadata if strings.HasPrefix(remoteStorageType, "EMBEDDED") && strings.HasSuffix(remoteStorageType, "_URL") { - partitionBackupDir = "/var/lib/clickhouse/backup/" + partitionBackupName + "/metadata/" + dbName + "/t?.json" + partitionBackupDir = "/var/lib/clickhouse/backup/" + partitionBackupName + "/shards/1/replicas/1/metadata/" + dbName + "/t?.json" expectedLines = "1" } out, err = env.DockerExecOut("clickhouse-backup", "bash", "-c", "ls -la "+partitionBackupDir+" | wc -l") diff --git a/test/testflows/clickhouse_backup/regression.py b/test/testflows/clickhouse_backup/regression.py index 185234e54..8b3d9270f 100755 --- a/test/testflows/clickhouse_backup/regression.py +++ b/test/testflows/clickhouse_backup/regression.py @@ -67,7 +67,7 @@ def regression(self, local, stress=False, fips=True, fips_godebug="only"): """ nodes = { "clickhouse": ("clickhouse1", "clickhouse2"), - "clickhouse_backup": ("clickhouse_backup",), + "clickhouse_backup": ("clickhouse_backup", "clickhouse_backup2"), "clickhouse_backup_fips": ("clickhouse_backup_fips",), "kafka": ("kafka",), "mysql": ("mysql",), @@ -117,6 +117,11 @@ def regression(self, local, stress=False, fips=True, fips_godebug="only"): self.context.backup_api_port = cluster.get_mapped_port("clickhouse_backup", 7171) + # Second backup server bound to clickhouse2, used by the embedded + # BACKUP/RESTORE ON CLUSTER suite (tests/embedded_on_cluster.py). + self.context.backup2 = self.context.cluster.node("clickhouse_backup2") + self.context.backup2_api_port = cluster.get_mapped_port("clickhouse_backup2", 7171) + # FIPS backup container is optional: only present when the FIPS-compatible # binary was built (``make build-race-fips-docker``). FIPS scenarios skip # gracefully when this is None (see tests/fips_140_3.py). @@ -145,6 +150,7 @@ def regression(self, local, stress=False, fips=True, fips_godebug="only"): Scenario(run=load("clickhouse_backup.tests.generic", "generic")) Scenario(run=load("clickhouse_backup.tests.views", "views")) Scenario(run=load("clickhouse_backup.tests.config_rbac", "config_rbac")) + Scenario(run=load("clickhouse_backup.tests.embedded_on_cluster", "embedded_on_cluster")) finally: shutil.rmtree(config_dir, ignore_errors=True) diff --git a/test/testflows/clickhouse_backup/requirements/requirements.md b/test/testflows/clickhouse_backup/requirements/requirements.md index a64b56cf9..d4f371455 100644 --- a/test/testflows/clickhouse_backup/requirements/requirements.md +++ b/test/testflows/clickhouse_backup/requirements/requirements.md @@ -1,1034 +1,1098 @@ -# QA-SRS013 ClickHouse Backup Utility -# Software Requirements Specification - - -## Table of Contents - -* 1 [Revision History](#revision-history) -* 2 [Introduction](#introduction) -* 3 [Terminology](#terminology) - * 3.1 [SRS](#srs) - * 3.2 [Supported engine families](#supported-engine-families) -* 4 [Requirements](#requirements) - * 4.1 [Generic](#generic) - * 4.1.1 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.AllDataTypes](#rqsrs-013clickhousebackuputilitygenericalldatatypes) - * 4.1.2 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.RestorePartiallyDropped](#rqsrs-013clickhousebackuputilitygenericrestorepartiallydropped) - * 4.1.3 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.BackupDuringMutation](#rqsrs-013clickhousebackuputilitygenericbackupduringmutation) - * 4.1.4 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.AllTables](#rqsrs-013clickhousebackuputilitygenericalltables) - * 4.1.5 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables](#rqsrs-013clickhousebackuputilitygenericspecifictables) - * 4.1.6 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables.Performance](#rqsrs-013clickhousebackuputilitygenericspecifictablesperformance) - * 4.1.7 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables.ManyColumns](#rqsrs-013clickhousebackuputilitygenericspecifictablesmanycolumns) - * 4.1.8 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.ReplicatedTable](#rqsrs-013clickhousebackuputilitygenericreplicatedtable) - * 4.1.9 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.MultipleBackups.EfficientStorageByUsingHardLinks](#rqsrs-013clickhousebackuputilitygenericmultiplebackupsefficientstoragebyusinghardlinks) - * 4.1.10 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.CompressionLevel](#rqsrs-013clickhousebackuputilitygenericcompressionlevel) - * 4.1.11 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.ClickHouseVersions](#rqsrs-013clickhousebackuputilitygenericclickhouseversions) - * 4.1.12 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.TieredStorage](#rqsrs-013clickhousebackuputilitygenerictieredstorage) - * 4.1.13 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.EncryptedStorage](#rqsrs-013clickhousebackuputilitygenericencryptedstorage) - * 4.2 [Views](#views) - * 4.2.1 [RQ.SRS-013.ClickHouse.BackupUtility.Views.View](#rqsrs-013clickhousebackuputilityviewsview) - * 4.2.2 [RQ.SRS-013.ClickHouse.BackupUtility.Views.MaterializedView](#rqsrs-013clickhousebackuputilityviewsmaterializedview) - * 4.2.3 [RQ.SRS-013.ClickHouse.BackupUtility.Views.LiveView](#rqsrs-013clickhousebackuputilityviewsliveview) - * 4.2.4 [RQ.SRS-013.ClickHouse.BackupUtility.Views.WindowView](#rqsrs-013clickhousebackuputilityviewswindowview) - * 4.2.5 [RQ.SRS-013.ClickHouse.BackupUtility.Views.NestedViews](#rqsrs-013clickhousebackuputilityviewsnestedviews) - * 4.3 [Table Engines](#table-engines) - * 4.3.1 [RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.MergeTree](#rqsrs-013clickhousebackuputilitytableenginesmergetree) - * 4.3.2 [RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.OtherEngines](#rqsrs-013clickhousebackuputilitytableenginesotherengines) - * 4.3.3 [RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.OtherEngines.Kafka](#rqsrs-013clickhousebackuputilitytableenginesotherengineskafka) - * 4.4 [Database Engines](#database-engines) - * 4.4.1 [RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.Atomic](#rqsrs-013clickhousebackuputilitydatabaseenginesatomic) - * 4.4.2 [RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.MaterializedMySQL](#rqsrs-013clickhousebackuputilitydatabaseenginesmaterializedmysql) - * 4.4.3 [RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.MaterializedPostgreSQL](#rqsrs-013clickhousebackuputilitydatabaseenginesmaterializedpostgresql) - * 4.5 [Cloud Storage](#cloud-storage) - * 4.5.1 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage](#rqsrs-013clickhousebackuputilitycloudstorage) - * 4.5.2 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.IncrementalBackups](#rqsrs-013clickhousebackuputilitycloudstorageincrementalbackups) - * 4.5.3 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.CompressionFormats](#rqsrs-013clickhousebackuputilitycloudstoragecompressionformats) - * 4.5.4 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.S3.Minio](#rqsrs-013clickhousebackuputilitycloudstorages3minio) - * 4.5.5 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.S3.AWS](#rqsrs-013clickhousebackuputilitycloudstorages3aws) - * 4.5.6 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.GCS](#rqsrs-013clickhousebackuputilitycloudstoragegcs) - * 4.5.7 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.Azure](#rqsrs-013clickhousebackuputilitycloudstorageazure) - * 4.5.8 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.TencentCOS](#rqsrs-013clickhousebackuputilitycloudstoragetencentcos) - * 4.5.9 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.FTP](#rqsrs-013clickhousebackuputilitycloudstorageftp) - * 4.5.10 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.SFTP](#rqsrs-013clickhousebackuputilitycloudstoragesftp) - * 4.5.11 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.GreaterThan5TB](#rqsrs-013clickhousebackuputilitycloudstoragegreaterthan5tb) - * 4.6 [Backup and restore configs](#backup-and-restore-configs) - * 4.6.1 [RQ.SRS-013.ClickHouse.BackupUtility.Configs.Backup](#rqsrs-013clickhousebackuputilityconfigsbackup) - * 4.6.2 [RQ.SRS-013.ClickHouse.BackupUtility.Configs.Restore](#rqsrs-013clickhousebackuputilityconfigsrestore) - * 4.7 [Backup and restore RBAC](#backup-and-restore-rbac) - * 4.7.1 [RQ.SRS-013.ClickHouse.BackupUtility.RBAC.Backup](#rqsrs-013clickhousebackuputilityrbacbackup) - * 4.7.2 [RQ.SRS-013.ClickHouse.BackupUtility.RBAC.Restore](#rqsrs-013clickhousebackuputilityrbacrestore) - * 4.8 [Command Line Interface](#command-line-interface) - * 4.8.1 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Usage](#rqsrs-013clickhousebackuputilitycliusage) - * 4.8.2 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Tables](#rqsrs-013clickhousebackuputilityclitables) - * 4.8.3 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Create](#rqsrs-013clickhousebackuputilityclicreate) - * 4.8.4 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.CreateRemote](#rqsrs-013clickhousebackuputilityclicreateremote) - * 4.8.5 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.MatchExpression](#rqsrs-013clickhousebackuputilityclimatchexpression) - * 4.8.6 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Upload](#rqsrs-013clickhousebackuputilitycliupload) - * 4.8.7 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.List](#rqsrs-013clickhousebackuputilityclilist) - * 4.8.8 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Download](#rqsrs-013clickhousebackuputilityclidownload) - * 4.8.9 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Restore](#rqsrs-013clickhousebackuputilityclirestore) - * 4.8.10 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.RestoreRemote](#rqsrs-013clickhousebackuputilityclirestoreremote) - * 4.8.11 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Delete](#rqsrs-013clickhousebackuputilityclidelete) - * 4.8.12 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.DefaultConfig](#rqsrs-013clickhousebackuputilityclidefaultconfig) - * 4.8.13 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Server](#rqsrs-013clickhousebackuputilitycliserver) - * 4.8.14 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Help](#rqsrs-013clickhousebackuputilityclihelp) - * 4.9 [Configuration](#configuration) - * 4.9.1 [RQ.SRS-013.ClickHouse.BackupUtility.Configuration](#rqsrs-013clickhousebackuputilityconfiguration) - * 4.10 [REST API Server](#rest-api-server) - * 4.10.1 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server](#rqsrs-013clickhousebackuputilityrestapiserver) - * 4.10.2 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.AsyncPostQuery](#rqsrs-013clickhousebackuputilityrestapiasyncpostquery) - * 4.10.3 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.OutputFormat.Default](#rqsrs-013clickhousebackuputilityrestapiserveroutputformatdefault) - * 4.10.4 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetTables](#rqsrs-013clickhousebackuputilityrestapiservergettables) - * 4.10.5 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostCreate](#rqsrs-013clickhousebackuputilityrestapiserverpostcreate) - * 4.10.6 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostCreate.Remote](#rqsrs-013clickhousebackuputilityrestapiserverpostcreateremote) - * 4.10.7 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostUpload](#rqsrs-013clickhousebackuputilityrestapiserverpostupload) - * 4.10.8 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList](#rqsrs-013clickhousebackuputilityrestapiservergetlist) - * 4.10.9 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList.Local](#rqsrs-013clickhousebackuputilityrestapiservergetlistlocal) - * 4.10.10 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList.Remote](#rqsrs-013clickhousebackuputilityrestapiservergetlistremote) - * 4.10.11 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDownload](#rqsrs-013clickhousebackuputilityrestapiserverpostdownload) - * 4.10.12 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostRestore](#rqsrs-013clickhousebackuputilityrestapiserverpostrestore) - * 4.10.13 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostRestore.Remote](#rqsrs-013clickhousebackuputilityrestapiserverpostrestoreremote) - * 4.10.14 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDelete.Remote](#rqsrs-013clickhousebackuputilityrestapiserverpostdeleteremote) - * 4.10.15 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDelete.Local](#rqsrs-013clickhousebackuputilityrestapiserverpostdeletelocal) - * 4.10.16 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetStatus](#rqsrs-013clickhousebackuputilityrestapiservergetstatus) - * 4.10.17 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetActions](#rqsrs-013clickhousebackuputilityrestapiservergetactions) - * 4.10.18 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostActions](#rqsrs-013clickhousebackuputilityrestapiserverpostactions) -* 5 [References](#references) - -## Revision History - -This document is stored in an electronic form using [Git] source control management software -hosted in a [Github Repository]. -All the updates are tracked using the [Revision History]. - -## Introduction - -[ClickHouse] does not support any convenient method to make backup and restore databases or tables. -BACKUP/RESTORE command in non production ready status. -This [SRS] covers the requirements for the [clickhouse-backup] utility that supports -easy [ClickHouse] backup and restore tool with cloud storages support. - -## Terminology - -### SRS - -Software Requirements Specification - -### Supported engine families - -Currently, [clickhouse-backup] utility only supports following table engines: - -* [MergeTree family](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/) -* [MergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/mergetree/) -* [ReplacingMergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/replacingmergetree/) -* [SummingMergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/summingmergetree/) -* [CollapsingMergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/collapsingmergetree/) -* [VersionedCollapsingMergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/versionedcollapsingmergetree/) -* [GraphiteMergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/graphitemergetree/) -* [Replicated MergeTree Tables](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/replication/) - -## Requirements - -### Generic - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.AllDataTypes -version: 1.0 - -The [clickhouse-backup] utility SHALL support backing up and restoring tables containing any existing data type. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.RestorePartiallyDropped -version: 1.0 - -The [clickhouse-backup] utility SHALL support restoring table that has lost some piece of data. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.BackupDuringMutation -version: 1.0 - -The [clickhouse-backup] utility SHALL be able to create a backup if there is an ongoing mutation. Restoring from such backups SHALL also be possible. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.AllTables -version: 1.0 - -The [clickhouse-backup] utility SHALL support easy backup and restore of all [ClickHouse] tables that SHALL result in table having identical state as at the time of backup. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables -version: 1.0 - -The [clickhouse-backup] utility SHALL support easy backup and restore of specific [ClickHouse] tables that SHALL result in table having identical state as at the time of backup. - - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables.Performance -version: 1.0 - -The [clickhouse-backup] utility SHALL create backups for a particular table with high performance due to avoiding scanning whole `system.tables` when set `table` query string parameter or `--tables` CLI parameter. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables.ManyColumns -version: 1.0 - -The [clickhouse-backup] utility SHALL support creating backups and restoring from them for a table with high number of columns. - - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.ReplicatedTable -version: 1.0 - -The [clickhouse-backup] utility SHALL support easy backup and restore of [ClickHouse] replicated tables from the backup -that SHALL result in table having identical state as at the time of backup. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.MultipleBackups.EfficientStorageByUsingHardLinks -version: 1.0 - -The [clickhouse-backup] utility SHALL support efficient storage of multiple backups on the file system by -manipulating hardlinks used by [ClickHouse] parts. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.CompressionLevel -version: 1.0 - -The [clickhouse-backup] utility SHALL support uploading to/downloading from remote storage with specified compression level. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.ClickHouseVersions -version: 1.0 - -The [clickhouse-backup] utility SHALL support [ClickHouse] versions above `1.1.54390`. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.TieredStorage -version: 1.0 - -The [clickhouse-backup] utility SHALL support `tiered storage` that was added in ClickHouse `19.15.2.2`. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.EncryptedStorage -version: 1.0 - -The [clickhouse-backup] utility SHALL support creating backups from [encrypted storages](https://clickhouse.tech/docs/en/operations/storing-data/#encrypted-virtual-file-system) and restoring them. - -### Views - -#### RQ.SRS-013.ClickHouse.BackupUtility.Views.View -version: 1.0 - -The [clickhouse-backup] utility SHALL support proper handling of `View` over `MergeTree` table engines. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Views.MaterializedView -version: 1.0 - -The [clickhouse-backup] utility SHALL support proper handling of `MaterializedView` over `MergeTree` table engines. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Views.LiveView -version: 1.0 - -The [clickhouse-backup] utility SHALL support proper handling of `LiveView` over `MergeTree` table engines. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Views.WindowView -version: 1.0 - -The [clickhouse-backup] utility SHALL support proper handling of `WindowView` over `MergeTree` table engines. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Views.NestedViews -version: 1.0 - -The [clickhouse-backup] utility SHALL support proper handling of nested views, e.g. `View` over `View`. - -### Table Engines - -#### RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.MergeTree -version: 1.0 - -The [clickhouse-backup] utility SHALL support only `MergeTree` family table engines. - -#### RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.OtherEngines -version: 1.0 - -The [clickhouse-backup] utility workflow SHALL NOT be broken by presence of tables with engines other that listed above. - -#### RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.OtherEngines.Kafka -version: 1.0 - -The [clickhouse-backup] utility SHALL only support creating table schema backup and restore for `Kafka` table engine. - -### Database Engines - -#### RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.Atomic -version: 1.0 - -The [clickhouse-backup] utility SHALL support `MergeTree` family table engines with `Atomic` database engine. - -#### RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.MaterializedMySQL -version: 1.0 - -The [clickhouse-backup] utility SHALL support `MergeTree` family table engines with `MaterializedMySQL` database engine. - -#### RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.MaterializedPostgreSQL -version: 1.0 - -The [clickhouse-backup] utility SHALL support `MergeTree` family table engines with `MaterializedPostgreSQL` database engine. - -### Cloud Storage - -#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage -version: 1.0 - -The [clickhouse-backup] utility SHALL support creating and uploading backups on cloud storage. - -#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.IncrementalBackups -version: 1.0 - -The [clickhouse-backup] utility SHALL support uploading an incremental backup to cloud storage. -This means a data chunk will be uploaded only in case it is not contained in original backup. -Incremental backup can be based on a full backup or on another incremental backup. - -The [clickhouse-backup] utility SHALL also support incremental backups after -one or more mutations have been performed on a table using the `ALTER` statement. -For example, - -* `ALTER TABLE
DELETE` -* `ALTER TABLE
UPDATE` -* `ALTER TABLE
ADD COLUMN` -* `ALTER TABLE
CLEAR COLUMN` -* `ALTER TABLE
DROP COLUMN` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.CompressionFormats -version: 1.0 - -The [clickhouse-backup] utility SHALL support storing backup on a cloud storage in following compression formats: `tar`, `bzip2`, `gzip`, `sz`, `xz`, `br/brotli`, `zstd`. In case of other value, error shall be thrown and no backup uploaded. - -#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.S3.Minio -version: 1.0 - -The [clickhouse-backup] utility SHALL support storing backups on a Minio S3 storage. It also SHALL be possible to restore from such backups. - -#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.S3.AWS -version: 1.0 - -The [clickhouse-backup] utility SHALL support storing backups on [AWS](https://aws.amazon.com/products/storage/) cloud servers. It also SHALL be possible to restore from such backups. - -#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.GCS -version: 1.0 - -The [clickhouse-backup] utility SHALL support storing backups on [GCS](https://cloud.google.com/storage) cloud servers. It also SHALL be possible to restore from such backups. - -#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.Azure -version: 1.0 - -The [clickhouse-backup] utility SHALL support storing backups on [Azure](https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction) cloud servers. It also SHALL be possible to restore from such backups. - -#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.TencentCOS -version: 1.0 - -The [clickhouse-backup] utility SHALL support storing backups on [Tencent COS](https://intl.cloud.tencent.com/product/cos) cloud servers. It also SHALL be possible to restore from such backups. - -#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.FTP -version: 1.0 - -The [clickhouse-backup] utility SHALL support storing backups on a remote FTP server. It also SHALL be possible to restore from such backup. - -#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.SFTP -version: 1.0 - -The [clickhouse-backup] utility SHALL support storing backups on a remote SFTP (also known as Secure FTP) server. It also SHALL be possible to restore from such backup. - -#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.GreaterThan5TB -version: 1.0 - -The [clickhouse-backup] utility SHALL support uploading backups greater than `5TB` using partitioning by 5TB chunks. -More info can be found [here](https://stackoverflow.com/questions/54012602/is-there-a-file-size-limit-for-amazon-s3#:~:text=While%20the%20maximum%20file%20size,any%20files%20larger%20than%20100MB) - -### Backup and restore configs - -#### RQ.SRS-013.ClickHouse.BackupUtility.Configs.Backup -version: 1.0 - -The [clickhouse-backup] utility SHALL support backing up ClickHouse configs. - -By configs, we mean the following files: - * `/etc/clickhouse-server/config.xml` - * `/etc/clickhouse-server/config.d/*.xml` - * `/etc/clickhouse-server/users.xml` - * `/etc/clickhouse-server/users.d/*.xml` - -Listed above are default config files. However, these values may be overriden on clickhouse-server launch. - -#### RQ.SRS-013.ClickHouse.BackupUtility.Configs.Restore -version: 1.0 - -The [clickhouse-backup] utility SHALL support restoring ClickHouse configs from backup. This should be done automatically in case configs backup is present. If configs are found in backup, ClickHouse server SHALL restart automatically. - -By configs, we mean the following files: - * `/etc/clickhouse-server/config.xml` - * `/etc/clickhouse-server/config.d/*.xml` - * `/etc/clickhouse-server/users.xml` - * `/etc/clickhouse-server/users.d/*.xml` - -Listed above are default config files. However, these values may be overriden on clickhouse-server launch. - -### Backup and restore RBAC - -#### RQ.SRS-013.ClickHouse.BackupUtility.RBAC.Backup -version: 1.0 - -The [clickhouse-backup] utility SHALL support backing up ClickHouse RBAC objects. By RBAC objects we mean the following: - * `USER` - * `ROLE` - * `ROW POLICY` - * `SETTINGS PROFILE` - * `QUOTA` - -#### RQ.SRS-013.ClickHouse.BackupUtility.RBAC.Restore -version: 1.0 - -The [clickhouse-backup] utility SHALL support restoring ClickHouse RBAC objects from backup. This should be done automatically in case RBAC objects backup is present. If RBAC objects are found in backup, ClickHouse server SHALL restart automatically. By RBAC objects we mean the following: - * `USER` - * `ROLE` - * `ROW POLICY` - * `SETTINGS PROFILE` - * `QUOTA` - -### Command Line Interface - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Usage -version: 1.0 - -The [clickhouse-backup] utility SHALL have CLI with the following usage - -```bash -USAGE: - clickhouse-backup [-t, --tables=.
] - -DESCRIPTION: - Run as 'root' or 'clickhouse' user - -COMMANDS: - tables Print list of tables - create Create new backup - create_remote Create and upload - upload Upload backup to remote storage - list Print list of backups - download Download backup from remote storage - restore Create schema and restore data from backup - restore_remote Download and restore - delete Delete specific backup - default-config Print default config - server Run API server - help, h Shows a list of commands or help for one command - -GLOBAL OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] - --help, -h show help - --version, -v print the version -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Tables -version: 1.0 - -The [clickhouse-backup] utility SHALL support `tables` option to list of tables that are present in the system. - -``` -USAGE: - clickhouse-backup tables - -OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] - -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Create -version: 1.0 - -The [clickhouse-backup] utility SHALL support `create` option to create new backup. - -* `-t, --table, --tables` option SHALL specify table(s) for backup. The option SHALL - support match expression syntax and SHALL support more than one expression separated by - a comma. - -``` -USAGE: - clickhouse-backup create [-t, --tables=.
] - -DESCRIPTION: - Create new backup - -OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] - --table value, --tables value, -t value -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.CreateRemote -version: 1.0 - -The [clickhouse-backup] utility SHALL support `create_remote` option to create new backup. - -* `-t, --table, --tables` option SHALL specify table(s) for backup. The option SHALL - support match expression syntax and SHALL support more than one expression separated by - a comma. - -``` -USAGE: - clickhouse-backup create_remote [-t, --tables=.
] [--diff-from=] [--delete] - -DESCRIPTION: - Create and upload - -OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] - --table value, --tables value, -t value - --diff-from value - --schema, -s Schemas only - -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.MatchExpression -version: 1.0 - -The [clickhouse-backup] utility SHALL support match expression syntax for specifying `skip_tables` option in configuration and for use with `--table` flag. -Syntax must be based on -the https://golang.org/pkg/path/filepath/#Match that SHALL have the following -pattern syntax: - -``` -pattern: - { term } -term: - '*' matches any sequence of non-Separator characters - '?' matches any single non-Separator character - '[' [ '^' ] { character-range } ']' - character class (must be non-empty) - c matches character c (c != '*', '?', '\\', '[') - '\\' c matches character c - -character-range: - c matches character c (c != '\\', '-', ']') - '\\' c matches character c - lo '-' hi matches character c for lo <= c <= hi -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Upload -version: 1.0 - -The [clickhouse-backup] utility SHALL support `upload` option to upload backup to remote storage. This command SHALL automatically upload ClickHouse configs and RBAC objects backups if they are present (in case backup was created with `--configs` and/or `--rbac`). - -* `--diff-from` option SHALL specify backup to be used for difference - -``` -USAGE: - clickhouse-backup upload [--diff-from=] - -OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] - --diff-from value -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.List -version: 1.0 - -The [clickhouse-backup] utility SHALL support `list` option to print the list of backups. - -* type `all|local|remote` option SHALL specify backup type -* version `latest|penult` option SHALL specify backup for difference with `latest` selecting the latest backup - and `penult` selecting the backup right before the latest. - -``` -USAGE: - clickhouse-backup list [all|local|remote] [latest|penult] - -OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Download -version: 1.0 - -The [clickhouse-backup] utility SHALL support `download` option to download backup from remote storage. This command SHALL automatically download ClickHouse configs and RBAC objects backups if they are present (in case backup was created with `--configs` and/or `--rbac`). - -* `backup_name` SHALL specify the name of the backup - -``` -USAGE: - clickhouse-backup download - -OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Restore -version: 1.0 - -The [clickhouse-backup] utility SHALL support `restore` option to create schema and restore data from backup. - -* `--table, --tables, -t` SHALL specify table(s) name -* `-s, --schema` SHALL specify to restore schema only -* `-d, --data` SHALL specify to restore data only -* `--rm, --drop` SHALL specify to drop table before restore - -``` -USAGE: - clickhouse-backup restore [--schema] [--data] [-t, --tables=.
] - -OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] - --table value, --tables value, -t value - --schema, -s Restore schema only - --data, -d Restore data only - --rm, --drop Drop table before restore -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.RestoreRemote -version: 1.0 - -The [clickhouse-backup] utility SHALL support `restore_remote` option to create schema and restore data from remote backup. - -* `--table, --tables, -t` SHALL specify table(s) name -* `-s, --schema` SHALL specify to restore schema only -* `-d, --data` SHALL specify to restore data only -* `--rm, --drop` SHALL specify to drop table before restore - -``` -USAGE: - clickhouse-backup create_remote [-t, --tables=.
] [--diff-from=] [--delete] - -DESCRIPTION: - Create and upload - -OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] - --table value, --tables value, -t value - --diff-from value - --schema, -s Schemas only -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Delete -version: 1.0 - -The [clickhouse-backup] utility SHALL support `delete` option to delete specific backup. - -* type `local|remote` SHALL specify backup type -* `backup_name` SHALL specify the name of the backup - -``` -USAGE: - clickhouse-backup delete - -OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.DefaultConfig -version: 1.0 - -The [clickhouse-backup] utility SHALL support `default-config` option to print default config. - -``` -USAGE: - clickhouse-backup default-config [command options] [arguments...] - -OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Server -version: 1.0 - -The [clickhouse-backup] utility SHALL support `server` option to run REST API server. - -``` -USAGE: - clickhouse-backup server [command options] [arguments...] - -OPTIONS: - --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Help -version: 1.0 - -The [clickhouse-backup] utility SHALL support `help` option to show the list of commands or help for one command. - -### Configuration - -#### RQ.SRS-013.ClickHouse.BackupUtility.Configuration -version: 1.0 - -The [clickhouse-backup] utility SHALL support the following options in the configuration file. -All options SHALL be overridable via environment variables. - -```bash -general: - remote_storage: none # REMOTE_STORAGE - max_file_size: 107374182400 # MAX_FILE_SIZE - backups_to_keep_local: 0 # BACKUPS_TO_KEEP_LOCAL - backups_to_keep_remote: 0 # BACKUPS_TO_KEEP_REMOTE - log_level: info # LOG_LEVEL - allow_empty_backups: false # ALLOW_EMPTY_BACKUPS -clickhouse: - username: default # CLICKHOUSE_USERNAME - password: "" # CLICKHOUSE_PASSWORD - host: localhost # CLICKHOUSE_HOST - port: 9000 # CLICKHOUSE_PORT - disk_mapping: {} # CLICKHOUSE_DISK_MAPPING - skip_tables: # CLICKHOUSE_SKIP_TABLES - - system.* - timeout: 5m # CLICKHOUSE_TIMEOUT - freeze_by_part: false # CLICKHOUSE_FREEZE_BY_PART - secure: false # CLICKHOUSE_SECURE - skip_verify: false # CLICKHOUSE_SKIP_VERIFY - sync_replicated_tables: true # CLICKHOUSE_SYNC_REPLICATED_TABLES - log_sql_queries: true # CLICKHOUSE_LOG_SQL_QUERIES - - config_dir: "/etc/clickhouse-server" # CLICKHOUSE_CONFIG_DIR - restart_command: "systemctl restart clickhouse-server" # CLICKHOUSE_RESTART_COMMAND - -azblob: - endpoint_suffix: "core.windows.net" # AZBLOB_ENDPOINT_SUFFIX - account_name: "" # AZBLOB_ACCOUNT_NAME - account_key: "" # AZBLOB_ACCOUNT_KEY - sas: "" # AZBLOB_SAS - container: "" # AZBLOB_CONTAINER - path: "" # AZBLOB_PATH - compression_level: 1 # AZBLOB_COMPRESSION_LEVEL - compression_format: tar # AZBLOB_COMPRESSION_FORMAT - sse_key: "" # AZBLOB_SSE_KEY -s3: - access_key: "" # S3_ACCESS_KEY - secret_key: "" # S3_SECRET_KEY - bucket: "" # S3_BUCKET - endpoint: "" # S3_ENDPOINT - region: us-east-1 # S3_REGION - acl: private # S3_ACL - force_path_style: false # S3_FORCE_PATH_STYLE - path: "" # S3_PATH - disable_ssl: false # S3_DISABLE_SSL - compression_level: 1 # S3_COMPRESSION_LEVEL - # supports 'tar', 'gzip', 'zstd', 'brotli' - compression_format: tar # S3_COMPRESSION_FORMAT - # empty (default), AES256, or aws:kms - sse: AES256 # S3_SSE - disable_cert_verification: false # S3_DISABLE_CERT_VERIFICATION - storage_class: STANDARD # S3_STORAGE_CLASS - debug: false # S3_DEBUG -gcs: - credentials_file: "" # GCS_CREDENTIALS_FILE - credentials_json: "" # GCS_CREDENTIALS_JSON - credentials_json_encoded: "" # GCS_CREDENTIALS_JSON_ENCODED - bucket: "" # GCS_BUCKET - path: "" # GCS_PATH - compression_level: 1 # GCS_COMPRESSION_LEVEL - compression_format: tar # GCS_COMPRESSION_FORMAT -cos: - url: "" # COS_URL - timeout: 2m # COS_TIMEOUT - secret_id: "" # COS_SECRET_ID - secret_key: "" # COS_SECRET_KEY - path: "" # COS_PATH - compression_format: tar # COS_COMPRESSION_FORMAT - compression_level: 1 # COS_COMPRESSION_LEVEL -api: - listen: "localhost:7171" # API_LISTEN - enable_metrics: true # API_ENABLE_METRICS - enable_pprof: false # API_ENABLE_PPROF - username: "" # API_USERNAME - password: "" # API_PASSWORD - secure: false # API_SECURE - certificate_file: "" # API_CERTIFICATE_FILE - private_key_file: "" # API_PRIVATE_KEY_FILE - create_integration_tables: false # API_CREATE_INTEGRATION_TABLES -ftp: - address: "" # FTP_ADDRESS - timeout: 2m # FTP_TIMEOUT - username: "" # FTP_USERNAME - password: "" # FTP_PASSWORD - tls: false # FTP_TLS - path: "" # FTP_PATH - compression_format: tar # FTP_COMPRESSION_FORMAT - compression_level: 1 # FTP_COMPRESSION_LEVEL - debug: false # FTP_DEBUG -sftp: - address: "" # SFTP_ADDRESS - username: "" # SFTP_USERNAME - password: "" # SFTP_PASSWORD - key: "" # SFTP_KEY - path: "" # SFTP_PATH - compression_format: tar # SFTP_COMPRESSION_FORMAT - compression_level: 1 # SFTP_COMPRESSION_LEVEL -``` - -### REST API Server - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server -version: 1.0 - -The [clickhouse-backup] utility SHALL support `clickhouse-backup server` -command to start REST API server. - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.AsyncPostQuery -version: 1.0 - -The [clickhouse-backup] utility SHALL return `code 423` if a POST request is received while previous operation is not finished yet. The request SHALL NOT be executed in such case. - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.OutputFormat.Default -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL return results -in a format compatible with [ClickHouse]'s `JSONEachRow` format -for all the endpoints. - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetTables -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -GET /backup/tables -``` - -to print the list of tables. For example, `curl -s localhost:7171/backup/tables | jq`. - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostCreate -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -POST /backup/create -``` - -to create new backup. For example, `curl -s localhost:7171/backup/create -X POST | jq`. - -Also, the following optional arguments SHALL be supported: -* Optional query argument `table` SHALL work the same as the `--table ` CLI argument. -* Optional query argument `name` SHALL work the same as specifying a backup name with the CLI. -* Optional query argument `schema` SHALL work the same as the `--schema` CLI argument (backup schema only). -* Optional query argument `rbac` SHALL work the same as the `--rbac` CLI argument (backup RBAC only). -* Optional query argument `configs` SHALL work the same as the `--configs` CLI argument (backup configs only). - -Optional arguments may be defined as following: `curl -s 'localhost:7171/backup/create?table=default.billing&name=billing_test' -X POST`. - -This operation SHALL be async, so the API will return once the operation has been started. - -Response contents shall have the following structure: -``` -{"status":"acknowledged","operation":"create","backup_name":""} -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostCreate.Remote -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -POST /backup/actions -H "Content-Type: application/json" -d '{"command":"create_remote "}' -``` - -to create new remote backup. - -This operation SHALL be async, so the API will return once the operation has been started. - - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostUpload -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -POST /backup/upload -``` - -to upload backup to remote storage. This command SHALL automatically upload ClickHouse configs and RBAC objects backups if they are present (in case backup was created with `--configs` and/or `--rbac`). - -For example, `curl -s localhost:7171/backup/upload/ -X POST | jq`. - -* optional query argument `diff-from` SHALL work the same as the `--diff-from` CLI argument. - -This operation SHALL be async, so the API will return once the operation has been started. - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -GET /backup/list -``` - -to print list of backups. For example, `curl -s localhost:7171/backup/list | jq`. - -The `size` field SHALL not be populated for local backups. - -Response SHALL have the following format: -``` -{'name': 'backup_1', 'created': '', 'size': , 'location': '', 'required': '', 'desc': ''} -{'name': 'backup_2', 'created': '', 'size': , 'location': '', 'required': '', 'desc': ''} -... -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList.Local -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -GET /backup/list/local -``` - -to print list of Remote backups. For example, `curl -s localhost:7171/backup/list/local | jq`. - -The `size` field SHALL not be populated. - -Response SHALL have the following format: -``` -{'name': 'backup_1', 'created': '', 'size': , 'location': 'local', 'required': '', 'desc': ''} -{'name': 'backup_2', 'created': '', 'size': , 'location': 'local', 'required': '', 'desc': ''} -... -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList.Remote -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -GET /backup/list/remote -``` - -to print list of remote backups. For example, `curl -s localhost:7171/backup/list/remote | jq`. - -Response SHALL have the following format: -``` -{'name': 'backup_1', 'created': '', 'size': , 'location': 'remote', 'required': '', 'desc': ''} -{'name': 'backup_2', 'created': '', 'size': , 'location': 'remote', 'required': '', 'desc': ''} -... -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDownload -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -POST /backup/download -``` - -to download backup from remote storage. This command SHALL automatically download ClickHouse configs and RBAC objects backups if they are present (in case backup was created with `--configs` and/or `--rbac`). - -For example, `curl -s localhost:7171/backup/download/ -X POST | jq`. - -This operation SHALL be async, so the API will return once the operation has been started. - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostRestore -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -POST /backup/restore -``` - -to create schema and restore data from backup. - -For example, `curl -s localhost:7171/backup/restore/ -X POST | jq`. - -* optional query argument `table` SHALL work the same as the `--table` value CLI argument -* optional query argument `schema` SHALL work the same as the `--schema` CLI argument (restore schema only) -* Optional query argument `data` SHALL work the same as the `--data` CLI argument (restore data only). -* Optional query argument `rbac` SHALL work the same as the `--rbac` CLI argument (restore RBAC only). -* Optional query argument `configs` SHALL work the same as the `--configs` CLI argument (restore configs only). - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostRestore.Remote -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -POST /backup/actions -H "Content-Type: application/json" -d '{"command":"restore_remote "}' -``` - -to restore from an existing remote backup. - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDelete.Remote -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -POST /backup/delete/remote -``` - -to delete specific remote backup. - -For example, `curl -s localhost:7171/backup/delete/remote/ -X POST | jq`. - -Response contents shall have the following structure: -``` -{"status":"success","operation":"delete","backup_name":"","location":"remote"} - -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDelete.Local -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -POST /backup/delete/local -``` - -to delete specific local backup. - - -For example, `curl -s localhost:7171/backup/delete/local/ -X POST | jq`. - -Response contents shall have the following structure: -``` -{"status":"success","operation":"delete","backup_name":"","location":"local"} -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetStatus -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -GET /backup/status -``` - -to display list of current async operations. -For example, `curl -s localhost:7171/backup/status | jq`. - -Response contents shall have the following structure: -``` -{'command': '', 'status': 'in progress', 'start': '2021-10-26 12:11:34'} -``` - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetActions -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -GET /backup/actions -``` - -to display a list of current async operations. - -#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostActions -version: 1.0 - -The [clickhouse-backup] utility REST API server SHALL support - -``` -POST /backup/actions -``` - -to create a new action with parameters being the same as for the CLI commands. -All CLI actions such as `create`, `upload` etc. SHALL be supported. - -## References - -* **clickhouse-backup**: https://github.com/Altinity/clickhouse-backup -* **ClickHouse:** https://clickhouse.tech -* **GitHub Repository:** https://github.com/Altinity/clickhouse-backup/blob/master/test/testflows/clickhouse_backup/requirements/requirements.md -* **Revision History:** https://github.com/Altinity/clickhouse-backup/commits/master/test/testflows/clickhouse_backup/requirements/requirements.md -* **Git:** https://git-scm.com/ - -[SRS]: #srs -[clickhouse-backup]: https://github.com/Altinity/clickhouse-backup -[ClickHouse]: https://clickhouse.tech -[GitHub Repository]: https://github.com/Altinity/clickhouse-backup/blob/master/test/testflows/clickhouse_backup/requirements/requirements.md -[Revision History]: https://github.com/Altinity/clickhouse-backup/commits/master/test/testflows/clickhouse_backup/requirements/requirements.md -[Git]: https://git-scm.com/ -[GitHub]: https://github.com/ +# QA-SRS013 ClickHouse Backup Utility +# Software Requirements Specification + + +## Table of Contents + +* 1 [Revision History](#revision-history) +* 2 [Introduction](#introduction) +* 3 [Terminology](#terminology) + * 3.1 [SRS](#srs) + * 3.2 [Supported engine families](#supported-engine-families) +* 4 [Requirements](#requirements) + * 4.1 [Generic](#generic) + * 4.1.1 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.AllDataTypes](#rqsrs-013clickhousebackuputilitygenericalldatatypes) + * 4.1.2 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.RestorePartiallyDropped](#rqsrs-013clickhousebackuputilitygenericrestorepartiallydropped) + * 4.1.3 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.BackupDuringMutation](#rqsrs-013clickhousebackuputilitygenericbackupduringmutation) + * 4.1.4 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.AllTables](#rqsrs-013clickhousebackuputilitygenericalltables) + * 4.1.5 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables](#rqsrs-013clickhousebackuputilitygenericspecifictables) + * 4.1.6 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables.Performance](#rqsrs-013clickhousebackuputilitygenericspecifictablesperformance) + * 4.1.7 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables.ManyColumns](#rqsrs-013clickhousebackuputilitygenericspecifictablesmanycolumns) + * 4.1.8 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.ReplicatedTable](#rqsrs-013clickhousebackuputilitygenericreplicatedtable) + * 4.1.9 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.MultipleBackups.EfficientStorageByUsingHardLinks](#rqsrs-013clickhousebackuputilitygenericmultiplebackupsefficientstoragebyusinghardlinks) + * 4.1.10 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.CompressionLevel](#rqsrs-013clickhousebackuputilitygenericcompressionlevel) + * 4.1.11 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.ClickHouseVersions](#rqsrs-013clickhousebackuputilitygenericclickhouseversions) + * 4.1.12 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.TieredStorage](#rqsrs-013clickhousebackuputilitygenerictieredstorage) + * 4.1.13 [RQ.SRS-013.ClickHouse.BackupUtility.Generic.EncryptedStorage](#rqsrs-013clickhousebackuputilitygenericencryptedstorage) + * 4.2 [Views](#views) + * 4.2.1 [RQ.SRS-013.ClickHouse.BackupUtility.Views.View](#rqsrs-013clickhousebackuputilityviewsview) + * 4.2.2 [RQ.SRS-013.ClickHouse.BackupUtility.Views.MaterializedView](#rqsrs-013clickhousebackuputilityviewsmaterializedview) + * 4.2.3 [RQ.SRS-013.ClickHouse.BackupUtility.Views.LiveView](#rqsrs-013clickhousebackuputilityviewsliveview) + * 4.2.4 [RQ.SRS-013.ClickHouse.BackupUtility.Views.WindowView](#rqsrs-013clickhousebackuputilityviewswindowview) + * 4.2.5 [RQ.SRS-013.ClickHouse.BackupUtility.Views.NestedViews](#rqsrs-013clickhousebackuputilityviewsnestedviews) + * 4.3 [Table Engines](#table-engines) + * 4.3.1 [RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.MergeTree](#rqsrs-013clickhousebackuputilitytableenginesmergetree) + * 4.3.2 [RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.OtherEngines](#rqsrs-013clickhousebackuputilitytableenginesotherengines) + * 4.3.3 [RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.OtherEngines.Kafka](#rqsrs-013clickhousebackuputilitytableenginesotherengineskafka) + * 4.4 [Database Engines](#database-engines) + * 4.4.1 [RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.Atomic](#rqsrs-013clickhousebackuputilitydatabaseenginesatomic) + * 4.4.2 [RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.MaterializedMySQL](#rqsrs-013clickhousebackuputilitydatabaseenginesmaterializedmysql) + * 4.4.3 [RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.MaterializedPostgreSQL](#rqsrs-013clickhousebackuputilitydatabaseenginesmaterializedpostgresql) + * 4.5 [Cloud Storage](#cloud-storage) + * 4.5.1 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage](#rqsrs-013clickhousebackuputilitycloudstorage) + * 4.5.2 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.IncrementalBackups](#rqsrs-013clickhousebackuputilitycloudstorageincrementalbackups) + * 4.5.3 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.CompressionFormats](#rqsrs-013clickhousebackuputilitycloudstoragecompressionformats) + * 4.5.4 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.S3.Minio](#rqsrs-013clickhousebackuputilitycloudstorages3minio) + * 4.5.5 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.S3.AWS](#rqsrs-013clickhousebackuputilitycloudstorages3aws) + * 4.5.6 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.GCS](#rqsrs-013clickhousebackuputilitycloudstoragegcs) + * 4.5.7 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.Azure](#rqsrs-013clickhousebackuputilitycloudstorageazure) + * 4.5.8 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.TencentCOS](#rqsrs-013clickhousebackuputilitycloudstoragetencentcos) + * 4.5.9 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.FTP](#rqsrs-013clickhousebackuputilitycloudstorageftp) + * 4.5.10 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.SFTP](#rqsrs-013clickhousebackuputilitycloudstoragesftp) + * 4.5.11 [RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.GreaterThan5TB](#rqsrs-013clickhousebackuputilitycloudstoragegreaterthan5tb) + * 4.6 [Backup and restore configs](#backup-and-restore-configs) + * 4.6.1 [RQ.SRS-013.ClickHouse.BackupUtility.Configs.Backup](#rqsrs-013clickhousebackuputilityconfigsbackup) + * 4.6.2 [RQ.SRS-013.ClickHouse.BackupUtility.Configs.Restore](#rqsrs-013clickhousebackuputilityconfigsrestore) + * 4.7 [Backup and restore RBAC](#backup-and-restore-rbac) + * 4.7.1 [RQ.SRS-013.ClickHouse.BackupUtility.RBAC.Backup](#rqsrs-013clickhousebackuputilityrbacbackup) + * 4.7.2 [RQ.SRS-013.ClickHouse.BackupUtility.RBAC.Restore](#rqsrs-013clickhousebackuputilityrbacrestore) + * 4.8 [Command Line Interface](#command-line-interface) + * 4.8.1 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Usage](#rqsrs-013clickhousebackuputilitycliusage) + * 4.8.2 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Tables](#rqsrs-013clickhousebackuputilityclitables) + * 4.8.3 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Create](#rqsrs-013clickhousebackuputilityclicreate) + * 4.8.4 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.CreateRemote](#rqsrs-013clickhousebackuputilityclicreateremote) + * 4.8.5 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.MatchExpression](#rqsrs-013clickhousebackuputilityclimatchexpression) + * 4.8.6 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Upload](#rqsrs-013clickhousebackuputilitycliupload) + * 4.8.7 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.List](#rqsrs-013clickhousebackuputilityclilist) + * 4.8.8 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Download](#rqsrs-013clickhousebackuputilityclidownload) + * 4.8.9 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Restore](#rqsrs-013clickhousebackuputilityclirestore) + * 4.8.10 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.RestoreRemote](#rqsrs-013clickhousebackuputilityclirestoreremote) + * 4.8.11 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Delete](#rqsrs-013clickhousebackuputilityclidelete) + * 4.8.12 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.DefaultConfig](#rqsrs-013clickhousebackuputilityclidefaultconfig) + * 4.8.13 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Server](#rqsrs-013clickhousebackuputilitycliserver) + * 4.8.14 [RQ.SRS-013.ClickHouse.BackupUtility.CLI.Help](#rqsrs-013clickhousebackuputilityclihelp) + * 4.9 [Configuration](#configuration) + * 4.9.1 [RQ.SRS-013.ClickHouse.BackupUtility.Configuration](#rqsrs-013clickhousebackuputilityconfiguration) + * 4.10 [REST API Server](#rest-api-server) + * 4.10.1 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server](#rqsrs-013clickhousebackuputilityrestapiserver) + * 4.10.2 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.AsyncPostQuery](#rqsrs-013clickhousebackuputilityrestapiasyncpostquery) + * 4.10.3 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.OutputFormat.Default](#rqsrs-013clickhousebackuputilityrestapiserveroutputformatdefault) + * 4.10.4 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetTables](#rqsrs-013clickhousebackuputilityrestapiservergettables) + * 4.10.5 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostCreate](#rqsrs-013clickhousebackuputilityrestapiserverpostcreate) + * 4.10.6 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostCreate.Remote](#rqsrs-013clickhousebackuputilityrestapiserverpostcreateremote) + * 4.10.7 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostUpload](#rqsrs-013clickhousebackuputilityrestapiserverpostupload) + * 4.10.8 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList](#rqsrs-013clickhousebackuputilityrestapiservergetlist) + * 4.10.9 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList.Local](#rqsrs-013clickhousebackuputilityrestapiservergetlistlocal) + * 4.10.10 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList.Remote](#rqsrs-013clickhousebackuputilityrestapiservergetlistremote) + * 4.10.11 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDownload](#rqsrs-013clickhousebackuputilityrestapiserverpostdownload) + * 4.10.12 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostRestore](#rqsrs-013clickhousebackuputilityrestapiserverpostrestore) + * 4.10.13 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostRestore.Remote](#rqsrs-013clickhousebackuputilityrestapiserverpostrestoreremote) + * 4.10.14 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDelete.Remote](#rqsrs-013clickhousebackuputilityrestapiserverpostdeleteremote) + * 4.10.15 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDelete.Local](#rqsrs-013clickhousebackuputilityrestapiserverpostdeletelocal) + * 4.10.16 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetStatus](#rqsrs-013clickhousebackuputilityrestapiservergetstatus) + * 4.10.17 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetActions](#rqsrs-013clickhousebackuputilityrestapiservergetactions) + * 4.10.18 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostActions](#rqsrs-013clickhousebackuputilityrestapiserverpostactions) + * 4.11 [Embedded Backup](#embedded-backup) + * 4.11.1 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster](#rqsrs-013clickhousebackuputilityembeddedbackuponcluster) + * 4.11.2 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Worker](#rqsrs-013clickhousebackuputilityembeddedbackuponclusterworker) + * 4.11.3 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Requirements](#rqsrs-013clickhousebackuputilityembeddedbackuponclusterrequirements) + * 4.11.4 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.RestoreOrder](#rqsrs-013clickhousebackuputilityembeddedbackuponclusterrestoreorder) + * 4.11.5 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.MetadataLayout](#rqsrs-013clickhousebackuputilityembeddedbackuponclustermetadatalayout) + * 4.11.6 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.DeleteLocal](#rqsrs-013clickhousebackuputilityembeddedbackuponclusterdeletelocal) +* 5 [References](#references) + +## Revision History + +This document is stored in an electronic form using [Git] source control management software +hosted in a [Github Repository]. +All the updates are tracked using the [Revision History]. + +## Introduction + +[ClickHouse] does not support any convenient method to make backup and restore databases or tables. +BACKUP/RESTORE command in non production ready status. +This [SRS] covers the requirements for the [clickhouse-backup] utility that supports +easy [ClickHouse] backup and restore tool with cloud storages support. + +## Terminology + +### SRS + +Software Requirements Specification + +### Supported engine families + +Currently, [clickhouse-backup] utility only supports following table engines: + +* [MergeTree family](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/) +* [MergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/mergetree/) +* [ReplacingMergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/replacingmergetree/) +* [SummingMergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/summingmergetree/) +* [CollapsingMergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/collapsingmergetree/) +* [VersionedCollapsingMergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/versionedcollapsingmergetree/) +* [GraphiteMergeTree](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/graphitemergetree/) +* [Replicated MergeTree Tables](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/replication/) + +## Requirements + +### Generic + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.AllDataTypes +version: 1.0 + +The [clickhouse-backup] utility SHALL support backing up and restoring tables containing any existing data type. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.RestorePartiallyDropped +version: 1.0 + +The [clickhouse-backup] utility SHALL support restoring table that has lost some piece of data. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.BackupDuringMutation +version: 1.0 + +The [clickhouse-backup] utility SHALL be able to create a backup if there is an ongoing mutation. Restoring from such backups SHALL also be possible. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.AllTables +version: 1.0 + +The [clickhouse-backup] utility SHALL support easy backup and restore of all [ClickHouse] tables that SHALL result in table having identical state as at the time of backup. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables +version: 1.0 + +The [clickhouse-backup] utility SHALL support easy backup and restore of specific [ClickHouse] tables that SHALL result in table having identical state as at the time of backup. + + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables.Performance +version: 1.0 + +The [clickhouse-backup] utility SHALL create backups for a particular table with high performance due to avoiding scanning whole `system.tables` when set `table` query string parameter or `--tables` CLI parameter. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables.ManyColumns +version: 1.0 + +The [clickhouse-backup] utility SHALL support creating backups and restoring from them for a table with high number of columns. + + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.ReplicatedTable +version: 1.0 + +The [clickhouse-backup] utility SHALL support easy backup and restore of [ClickHouse] replicated tables from the backup +that SHALL result in table having identical state as at the time of backup. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.MultipleBackups.EfficientStorageByUsingHardLinks +version: 1.0 + +The [clickhouse-backup] utility SHALL support efficient storage of multiple backups on the file system by +manipulating hardlinks used by [ClickHouse] parts. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.CompressionLevel +version: 1.0 + +The [clickhouse-backup] utility SHALL support uploading to/downloading from remote storage with specified compression level. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.ClickHouseVersions +version: 1.0 + +The [clickhouse-backup] utility SHALL support [ClickHouse] versions above `1.1.54390`. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.TieredStorage +version: 1.0 + +The [clickhouse-backup] utility SHALL support `tiered storage` that was added in ClickHouse `19.15.2.2`. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Generic.EncryptedStorage +version: 1.0 + +The [clickhouse-backup] utility SHALL support creating backups from [encrypted storages](https://clickhouse.tech/docs/en/operations/storing-data/#encrypted-virtual-file-system) and restoring them. + +### Views + +#### RQ.SRS-013.ClickHouse.BackupUtility.Views.View +version: 1.0 + +The [clickhouse-backup] utility SHALL support proper handling of `View` over `MergeTree` table engines. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Views.MaterializedView +version: 1.0 + +The [clickhouse-backup] utility SHALL support proper handling of `MaterializedView` over `MergeTree` table engines. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Views.LiveView +version: 1.0 + +The [clickhouse-backup] utility SHALL support proper handling of `LiveView` over `MergeTree` table engines. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Views.WindowView +version: 1.0 + +The [clickhouse-backup] utility SHALL support proper handling of `WindowView` over `MergeTree` table engines. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Views.NestedViews +version: 1.0 + +The [clickhouse-backup] utility SHALL support proper handling of nested views, e.g. `View` over `View`. + +### Table Engines + +#### RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.MergeTree +version: 1.0 + +The [clickhouse-backup] utility SHALL support only `MergeTree` family table engines. + +#### RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.OtherEngines +version: 1.0 + +The [clickhouse-backup] utility workflow SHALL NOT be broken by presence of tables with engines other that listed above. + +#### RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.OtherEngines.Kafka +version: 1.0 + +The [clickhouse-backup] utility SHALL only support creating table schema backup and restore for `Kafka` table engine. + +### Database Engines + +#### RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.Atomic +version: 1.0 + +The [clickhouse-backup] utility SHALL support `MergeTree` family table engines with `Atomic` database engine. + +#### RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.MaterializedMySQL +version: 1.0 + +The [clickhouse-backup] utility SHALL support `MergeTree` family table engines with `MaterializedMySQL` database engine. + +#### RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.MaterializedPostgreSQL +version: 1.0 + +The [clickhouse-backup] utility SHALL support `MergeTree` family table engines with `MaterializedPostgreSQL` database engine. + +### Cloud Storage + +#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage +version: 1.0 + +The [clickhouse-backup] utility SHALL support creating and uploading backups on cloud storage. + +#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.IncrementalBackups +version: 1.0 + +The [clickhouse-backup] utility SHALL support uploading an incremental backup to cloud storage. +This means a data chunk will be uploaded only in case it is not contained in original backup. +Incremental backup can be based on a full backup or on another incremental backup. + +The [clickhouse-backup] utility SHALL also support incremental backups after +one or more mutations have been performed on a table using the `ALTER` statement. +For example, + +* `ALTER TABLE
DELETE` +* `ALTER TABLE
UPDATE` +* `ALTER TABLE
ADD COLUMN` +* `ALTER TABLE
CLEAR COLUMN` +* `ALTER TABLE
DROP COLUMN` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.CompressionFormats +version: 1.0 + +The [clickhouse-backup] utility SHALL support storing backup on a cloud storage in following compression formats: `tar`, `bzip2`, `gzip`, `sz`, `xz`, `br/brotli`, `zstd`. In case of other value, error shall be thrown and no backup uploaded. + +#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.S3.Minio +version: 1.0 + +The [clickhouse-backup] utility SHALL support storing backups on a Minio S3 storage. It also SHALL be possible to restore from such backups. + +#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.S3.AWS +version: 1.0 + +The [clickhouse-backup] utility SHALL support storing backups on [AWS](https://aws.amazon.com/products/storage/) cloud servers. It also SHALL be possible to restore from such backups. + +#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.GCS +version: 1.0 + +The [clickhouse-backup] utility SHALL support storing backups on [GCS](https://cloud.google.com/storage) cloud servers. It also SHALL be possible to restore from such backups. + +#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.Azure +version: 1.0 + +The [clickhouse-backup] utility SHALL support storing backups on [Azure](https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction) cloud servers. It also SHALL be possible to restore from such backups. + +#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.TencentCOS +version: 1.0 + +The [clickhouse-backup] utility SHALL support storing backups on [Tencent COS](https://intl.cloud.tencent.com/product/cos) cloud servers. It also SHALL be possible to restore from such backups. + +#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.FTP +version: 1.0 + +The [clickhouse-backup] utility SHALL support storing backups on a remote FTP server. It also SHALL be possible to restore from such backup. + +#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.SFTP +version: 1.0 + +The [clickhouse-backup] utility SHALL support storing backups on a remote SFTP (also known as Secure FTP) server. It also SHALL be possible to restore from such backup. + +#### RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.GreaterThan5TB +version: 1.0 + +The [clickhouse-backup] utility SHALL support uploading backups greater than `5TB` using partitioning by 5TB chunks. +More info can be found [here](https://stackoverflow.com/questions/54012602/is-there-a-file-size-limit-for-amazon-s3#:~:text=While%20the%20maximum%20file%20size,any%20files%20larger%20than%20100MB) + +### Backup and restore configs + +#### RQ.SRS-013.ClickHouse.BackupUtility.Configs.Backup +version: 1.0 + +The [clickhouse-backup] utility SHALL support backing up ClickHouse configs. + +By configs, we mean the following files: + * `/etc/clickhouse-server/config.xml` + * `/etc/clickhouse-server/config.d/*.xml` + * `/etc/clickhouse-server/users.xml` + * `/etc/clickhouse-server/users.d/*.xml` + +Listed above are default config files. However, these values may be overriden on clickhouse-server launch. + +#### RQ.SRS-013.ClickHouse.BackupUtility.Configs.Restore +version: 1.0 + +The [clickhouse-backup] utility SHALL support restoring ClickHouse configs from backup. This should be done automatically in case configs backup is present. If configs are found in backup, ClickHouse server SHALL restart automatically. + +By configs, we mean the following files: + * `/etc/clickhouse-server/config.xml` + * `/etc/clickhouse-server/config.d/*.xml` + * `/etc/clickhouse-server/users.xml` + * `/etc/clickhouse-server/users.d/*.xml` + +Listed above are default config files. However, these values may be overriden on clickhouse-server launch. + +### Backup and restore RBAC + +#### RQ.SRS-013.ClickHouse.BackupUtility.RBAC.Backup +version: 1.0 + +The [clickhouse-backup] utility SHALL support backing up ClickHouse RBAC objects. By RBAC objects we mean the following: + * `USER` + * `ROLE` + * `ROW POLICY` + * `SETTINGS PROFILE` + * `QUOTA` + +#### RQ.SRS-013.ClickHouse.BackupUtility.RBAC.Restore +version: 1.0 + +The [clickhouse-backup] utility SHALL support restoring ClickHouse RBAC objects from backup. This should be done automatically in case RBAC objects backup is present. If RBAC objects are found in backup, ClickHouse server SHALL restart automatically. By RBAC objects we mean the following: + * `USER` + * `ROLE` + * `ROW POLICY` + * `SETTINGS PROFILE` + * `QUOTA` + +### Command Line Interface + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Usage +version: 1.0 + +The [clickhouse-backup] utility SHALL have CLI with the following usage + +```bash +USAGE: + clickhouse-backup [-t, --tables=.
] + +DESCRIPTION: + Run as 'root' or 'clickhouse' user + +COMMANDS: + tables Print list of tables + create Create new backup + create_remote Create and upload + upload Upload backup to remote storage + list Print list of backups + download Download backup from remote storage + restore Create schema and restore data from backup + restore_remote Download and restore + delete Delete specific backup + default-config Print default config + server Run API server + help, h Shows a list of commands or help for one command + +GLOBAL OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] + --help, -h show help + --version, -v print the version +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Tables +version: 1.0 + +The [clickhouse-backup] utility SHALL support `tables` option to list of tables that are present in the system. + +``` +USAGE: + clickhouse-backup tables + +OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] + +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Create +version: 1.0 + +The [clickhouse-backup] utility SHALL support `create` option to create new backup. + +* `-t, --table, --tables` option SHALL specify table(s) for backup. The option SHALL + support match expression syntax and SHALL support more than one expression separated by + a comma. + +``` +USAGE: + clickhouse-backup create [-t, --tables=.
] + +DESCRIPTION: + Create new backup + +OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] + --table value, --tables value, -t value +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.CreateRemote +version: 1.0 + +The [clickhouse-backup] utility SHALL support `create_remote` option to create new backup. + +* `-t, --table, --tables` option SHALL specify table(s) for backup. The option SHALL + support match expression syntax and SHALL support more than one expression separated by + a comma. + +``` +USAGE: + clickhouse-backup create_remote [-t, --tables=.
] [--diff-from=] [--delete] + +DESCRIPTION: + Create and upload + +OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] + --table value, --tables value, -t value + --diff-from value + --schema, -s Schemas only + +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.MatchExpression +version: 1.0 + +The [clickhouse-backup] utility SHALL support match expression syntax for specifying `skip_tables` option in configuration and for use with `--table` flag. +Syntax must be based on +the https://golang.org/pkg/path/filepath/#Match that SHALL have the following +pattern syntax: + +``` +pattern: + { term } +term: + '*' matches any sequence of non-Separator characters + '?' matches any single non-Separator character + '[' [ '^' ] { character-range } ']' + character class (must be non-empty) + c matches character c (c != '*', '?', '\\', '[') + '\\' c matches character c + +character-range: + c matches character c (c != '\\', '-', ']') + '\\' c matches character c + lo '-' hi matches character c for lo <= c <= hi +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Upload +version: 1.0 + +The [clickhouse-backup] utility SHALL support `upload` option to upload backup to remote storage. This command SHALL automatically upload ClickHouse configs and RBAC objects backups if they are present (in case backup was created with `--configs` and/or `--rbac`). + +* `--diff-from` option SHALL specify backup to be used for difference + +``` +USAGE: + clickhouse-backup upload [--diff-from=] + +OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] + --diff-from value +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.List +version: 1.0 + +The [clickhouse-backup] utility SHALL support `list` option to print the list of backups. + +* type `all|local|remote` option SHALL specify backup type +* version `latest|penult` option SHALL specify backup for difference with `latest` selecting the latest backup + and `penult` selecting the backup right before the latest. + +``` +USAGE: + clickhouse-backup list [all|local|remote] [latest|penult] + +OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Download +version: 1.0 + +The [clickhouse-backup] utility SHALL support `download` option to download backup from remote storage. This command SHALL automatically download ClickHouse configs and RBAC objects backups if they are present (in case backup was created with `--configs` and/or `--rbac`). + +* `backup_name` SHALL specify the name of the backup + +``` +USAGE: + clickhouse-backup download + +OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Restore +version: 1.0 + +The [clickhouse-backup] utility SHALL support `restore` option to create schema and restore data from backup. + +* `--table, --tables, -t` SHALL specify table(s) name +* `-s, --schema` SHALL specify to restore schema only +* `-d, --data` SHALL specify to restore data only +* `--rm, --drop` SHALL specify to drop table before restore + +``` +USAGE: + clickhouse-backup restore [--schema] [--data] [-t, --tables=.
] + +OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] + --table value, --tables value, -t value + --schema, -s Restore schema only + --data, -d Restore data only + --rm, --drop Drop table before restore +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.RestoreRemote +version: 1.0 + +The [clickhouse-backup] utility SHALL support `restore_remote` option to create schema and restore data from remote backup. + +* `--table, --tables, -t` SHALL specify table(s) name +* `-s, --schema` SHALL specify to restore schema only +* `-d, --data` SHALL specify to restore data only +* `--rm, --drop` SHALL specify to drop table before restore + +``` +USAGE: + clickhouse-backup create_remote [-t, --tables=.
] [--diff-from=] [--delete] + +DESCRIPTION: + Create and upload + +OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] + --table value, --tables value, -t value + --diff-from value + --schema, -s Schemas only +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Delete +version: 1.0 + +The [clickhouse-backup] utility SHALL support `delete` option to delete specific backup. + +* type `local|remote` SHALL specify backup type +* `backup_name` SHALL specify the name of the backup + +``` +USAGE: + clickhouse-backup delete + +OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.DefaultConfig +version: 1.0 + +The [clickhouse-backup] utility SHALL support `default-config` option to print default config. + +``` +USAGE: + clickhouse-backup default-config [command options] [arguments...] + +OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Server +version: 1.0 + +The [clickhouse-backup] utility SHALL support `server` option to run REST API server. + +``` +USAGE: + clickhouse-backup server [command options] [arguments...] + +OPTIONS: + --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG] +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.CLI.Help +version: 1.0 + +The [clickhouse-backup] utility SHALL support `help` option to show the list of commands or help for one command. + +### Configuration + +#### RQ.SRS-013.ClickHouse.BackupUtility.Configuration +version: 1.0 + +The [clickhouse-backup] utility SHALL support the following options in the configuration file. +All options SHALL be overridable via environment variables. + +```bash +general: + remote_storage: none # REMOTE_STORAGE + max_file_size: 107374182400 # MAX_FILE_SIZE + backups_to_keep_local: 0 # BACKUPS_TO_KEEP_LOCAL + backups_to_keep_remote: 0 # BACKUPS_TO_KEEP_REMOTE + log_level: info # LOG_LEVEL + allow_empty_backups: false # ALLOW_EMPTY_BACKUPS +clickhouse: + username: default # CLICKHOUSE_USERNAME + password: "" # CLICKHOUSE_PASSWORD + host: localhost # CLICKHOUSE_HOST + port: 9000 # CLICKHOUSE_PORT + disk_mapping: {} # CLICKHOUSE_DISK_MAPPING + skip_tables: # CLICKHOUSE_SKIP_TABLES + - system.* + timeout: 5m # CLICKHOUSE_TIMEOUT + freeze_by_part: false # CLICKHOUSE_FREEZE_BY_PART + secure: false # CLICKHOUSE_SECURE + skip_verify: false # CLICKHOUSE_SKIP_VERIFY + sync_replicated_tables: true # CLICKHOUSE_SYNC_REPLICATED_TABLES + log_sql_queries: true # CLICKHOUSE_LOG_SQL_QUERIES + + config_dir: "/etc/clickhouse-server" # CLICKHOUSE_CONFIG_DIR + restart_command: "systemctl restart clickhouse-server" # CLICKHOUSE_RESTART_COMMAND + +azblob: + endpoint_suffix: "core.windows.net" # AZBLOB_ENDPOINT_SUFFIX + account_name: "" # AZBLOB_ACCOUNT_NAME + account_key: "" # AZBLOB_ACCOUNT_KEY + sas: "" # AZBLOB_SAS + container: "" # AZBLOB_CONTAINER + path: "" # AZBLOB_PATH + compression_level: 1 # AZBLOB_COMPRESSION_LEVEL + compression_format: tar # AZBLOB_COMPRESSION_FORMAT + sse_key: "" # AZBLOB_SSE_KEY +s3: + access_key: "" # S3_ACCESS_KEY + secret_key: "" # S3_SECRET_KEY + bucket: "" # S3_BUCKET + endpoint: "" # S3_ENDPOINT + region: us-east-1 # S3_REGION + acl: private # S3_ACL + force_path_style: false # S3_FORCE_PATH_STYLE + path: "" # S3_PATH + disable_ssl: false # S3_DISABLE_SSL + compression_level: 1 # S3_COMPRESSION_LEVEL + # supports 'tar', 'gzip', 'zstd', 'brotli' + compression_format: tar # S3_COMPRESSION_FORMAT + # empty (default), AES256, or aws:kms + sse: AES256 # S3_SSE + disable_cert_verification: false # S3_DISABLE_CERT_VERIFICATION + storage_class: STANDARD # S3_STORAGE_CLASS + debug: false # S3_DEBUG +gcs: + credentials_file: "" # GCS_CREDENTIALS_FILE + credentials_json: "" # GCS_CREDENTIALS_JSON + credentials_json_encoded: "" # GCS_CREDENTIALS_JSON_ENCODED + bucket: "" # GCS_BUCKET + path: "" # GCS_PATH + compression_level: 1 # GCS_COMPRESSION_LEVEL + compression_format: tar # GCS_COMPRESSION_FORMAT +cos: + url: "" # COS_URL + timeout: 2m # COS_TIMEOUT + secret_id: "" # COS_SECRET_ID + secret_key: "" # COS_SECRET_KEY + path: "" # COS_PATH + compression_format: tar # COS_COMPRESSION_FORMAT + compression_level: 1 # COS_COMPRESSION_LEVEL +api: + listen: "localhost:7171" # API_LISTEN + enable_metrics: true # API_ENABLE_METRICS + enable_pprof: false # API_ENABLE_PPROF + username: "" # API_USERNAME + password: "" # API_PASSWORD + secure: false # API_SECURE + certificate_file: "" # API_CERTIFICATE_FILE + private_key_file: "" # API_PRIVATE_KEY_FILE + create_integration_tables: false # API_CREATE_INTEGRATION_TABLES +ftp: + address: "" # FTP_ADDRESS + timeout: 2m # FTP_TIMEOUT + username: "" # FTP_USERNAME + password: "" # FTP_PASSWORD + tls: false # FTP_TLS + path: "" # FTP_PATH + compression_format: tar # FTP_COMPRESSION_FORMAT + compression_level: 1 # FTP_COMPRESSION_LEVEL + debug: false # FTP_DEBUG +sftp: + address: "" # SFTP_ADDRESS + username: "" # SFTP_USERNAME + password: "" # SFTP_PASSWORD + key: "" # SFTP_KEY + path: "" # SFTP_PATH + compression_format: tar # SFTP_COMPRESSION_FORMAT + compression_level: 1 # SFTP_COMPRESSION_LEVEL +``` + +### REST API Server + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server +version: 1.0 + +The [clickhouse-backup] utility SHALL support `clickhouse-backup server` +command to start REST API server. + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.AsyncPostQuery +version: 1.0 + +The [clickhouse-backup] utility SHALL return `code 423` if a POST request is received while previous operation is not finished yet. The request SHALL NOT be executed in such case. + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.OutputFormat.Default +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL return results +in a format compatible with [ClickHouse]'s `JSONEachRow` format +for all the endpoints. + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetTables +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +GET /backup/tables +``` + +to print the list of tables. For example, `curl -s localhost:7171/backup/tables | jq`. + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostCreate +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +POST /backup/create +``` + +to create new backup. For example, `curl -s localhost:7171/backup/create -X POST | jq`. + +Also, the following optional arguments SHALL be supported: +* Optional query argument `table` SHALL work the same as the `--table ` CLI argument. +* Optional query argument `name` SHALL work the same as specifying a backup name with the CLI. +* Optional query argument `schema` SHALL work the same as the `--schema` CLI argument (backup schema only). +* Optional query argument `rbac` SHALL work the same as the `--rbac` CLI argument (backup RBAC only). +* Optional query argument `configs` SHALL work the same as the `--configs` CLI argument (backup configs only). + +Optional arguments may be defined as following: `curl -s 'localhost:7171/backup/create?table=default.billing&name=billing_test' -X POST`. + +This operation SHALL be async, so the API will return once the operation has been started. + +Response contents shall have the following structure: +``` +{"status":"acknowledged","operation":"create","backup_name":""} +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostCreate.Remote +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +POST /backup/actions -H "Content-Type: application/json" -d '{"command":"create_remote "}' +``` + +to create new remote backup. + +This operation SHALL be async, so the API will return once the operation has been started. + + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostUpload +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +POST /backup/upload +``` + +to upload backup to remote storage. This command SHALL automatically upload ClickHouse configs and RBAC objects backups if they are present (in case backup was created with `--configs` and/or `--rbac`). + +For example, `curl -s localhost:7171/backup/upload/ -X POST | jq`. + +* optional query argument `diff-from` SHALL work the same as the `--diff-from` CLI argument. + +This operation SHALL be async, so the API will return once the operation has been started. + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +GET /backup/list +``` + +to print list of backups. For example, `curl -s localhost:7171/backup/list | jq`. + +The `size` field SHALL not be populated for local backups. + +Response SHALL have the following format: +``` +{'name': 'backup_1', 'created': '', 'size': , 'location': '', 'required': '', 'desc': ''} +{'name': 'backup_2', 'created': '', 'size': , 'location': '', 'required': '', 'desc': ''} +... +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList.Local +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +GET /backup/list/local +``` + +to print list of Remote backups. For example, `curl -s localhost:7171/backup/list/local | jq`. + +The `size` field SHALL not be populated. + +Response SHALL have the following format: +``` +{'name': 'backup_1', 'created': '', 'size': , 'location': 'local', 'required': '', 'desc': ''} +{'name': 'backup_2', 'created': '', 'size': , 'location': 'local', 'required': '', 'desc': ''} +... +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList.Remote +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +GET /backup/list/remote +``` + +to print list of remote backups. For example, `curl -s localhost:7171/backup/list/remote | jq`. + +Response SHALL have the following format: +``` +{'name': 'backup_1', 'created': '', 'size': , 'location': 'remote', 'required': '', 'desc': ''} +{'name': 'backup_2', 'created': '', 'size': , 'location': 'remote', 'required': '', 'desc': ''} +... +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDownload +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +POST /backup/download +``` + +to download backup from remote storage. This command SHALL automatically download ClickHouse configs and RBAC objects backups if they are present (in case backup was created with `--configs` and/or `--rbac`). + +For example, `curl -s localhost:7171/backup/download/ -X POST | jq`. + +This operation SHALL be async, so the API will return once the operation has been started. + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostRestore +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +POST /backup/restore +``` + +to create schema and restore data from backup. + +For example, `curl -s localhost:7171/backup/restore/ -X POST | jq`. + +* optional query argument `table` SHALL work the same as the `--table` value CLI argument +* optional query argument `schema` SHALL work the same as the `--schema` CLI argument (restore schema only) +* Optional query argument `data` SHALL work the same as the `--data` CLI argument (restore data only). +* Optional query argument `rbac` SHALL work the same as the `--rbac` CLI argument (restore RBAC only). +* Optional query argument `configs` SHALL work the same as the `--configs` CLI argument (restore configs only). + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostRestore.Remote +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +POST /backup/actions -H "Content-Type: application/json" -d '{"command":"restore_remote "}' +``` + +to restore from an existing remote backup. + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDelete.Remote +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +POST /backup/delete/remote +``` + +to delete specific remote backup. + +For example, `curl -s localhost:7171/backup/delete/remote/ -X POST | jq`. + +Response contents shall have the following structure: +``` +{"status":"success","operation":"delete","backup_name":"","location":"remote"} + +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDelete.Local +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +POST /backup/delete/local +``` + +to delete specific local backup. + + +For example, `curl -s localhost:7171/backup/delete/local/ -X POST | jq`. + +Response contents shall have the following structure: +``` +{"status":"success","operation":"delete","backup_name":"","location":"local"} +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetStatus +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +GET /backup/status +``` + +to display list of current async operations. +For example, `curl -s localhost:7171/backup/status | jq`. + +Response contents shall have the following structure: +``` +{'command': '', 'status': 'in progress', 'start': '2021-10-26 12:11:34'} +``` + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetActions +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +GET /backup/actions +``` + +to display a list of current async operations. + +#### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostActions +version: 1.0 + +The [clickhouse-backup] utility REST API server SHALL support + +``` +POST /backup/actions +``` + +to create a new action with parameters being the same as for the CLI commands. +All CLI actions such as `create`, `upload` etc. SHALL be supported. + +### Embedded Backup + +The [clickhouse-backup] utility SHALL support the native [ClickHouse] `BACKUP` / `RESTORE` SQL commands +(`clickhouse.use_embedded_backup_restore: true`) and SHALL be able to orchestrate them across a whole cluster. + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster +version: 1.0 + +The [clickhouse-backup] utility SHALL support `clickhouse.use_embedded_backup_restore_cluster: ""` +(macros such as `{cluster}` SHALL be allowed). When it is set together with `clickhouse.use_embedded_backup_restore: true` +and `clickhouse.embedded_backup_disk: ""`, the node where `create_remote ` or `restore_remote ` is issued +(the initiator) SHALL run `BACKUP ... ON CLUSTER TO S3(...)` (resp. `RESTORE ... ON CLUSTER FROM S3(...)`) +so that the data of every shard and replica of the cluster is written to (resp. read from) the shared remote storage +configured by `general.remote_storage` in a single operation. + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Worker +version: 1.0 + +The [clickhouse-backup] utility SHALL support the `--embedded-on-cluster-worker` flag for the `create`, `upload`, `download`, +`restore`, `create_remote` and `restore_remote` commands. For every non-local host of the cluster listed in `system.clusters` +the initiator SHALL execute +`INSERT INTO FUNCTION remote(':', 'system', 'backup_actions', '', '') (command) VALUES ('create_remote --embedded-on-cluster-worker ')` +(resp. `restore_remote --embedded-on-cluster-worker `), SHALL wait for those worker operations to finish by polling +the same remote `system.backup_actions` table, and SHALL report the operation as failed when any worker fails or cannot be reached. + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Requirements +version: 1.0 + +Every [ClickHouse] node of the cluster SHALL run its own `clickhouse-backup server` with `api.create_integration_tables: true` +(which creates the `system.backup_actions`, `system.backup_list` and `system.backup_version` `ENGINE=URL` tables pointing at +`api.integration_tables_host:`), and all servers SHALL point at the same remote storage destination +(same bucket and same `path`). + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.RestoreOrder +version: 1.0 + +During `restore_remote` the initiator SHALL make the workers process the backup metadata (download and prepare the +per-table metadata of their own shard and replica) before the `RESTORE ... ON CLUSTER` SQL command is executed, +so that the data of every node of the cluster is restored and row counts on every node match the original ones. + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.MetadataLayout +version: 1.0 + +The [clickhouse-backup] utility SHALL store an embedded `ON CLUSTER` backup in the remote storage with the following layout: + +* `//.backup` - native [ClickHouse] backup descriptor +* `//metadata.json` - [clickhouse-backup] backup metadata +* `//shards//replicas//metadata/...` and `//shards//replicas//data/...` - native [ClickHouse] schema and data of each shard and replica +* `//shards//replicas//metadata//
.json` - [clickhouse-backup] per-table metadata of each shard and replica + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.DeleteLocal +version: 1.0 + +The [clickhouse-backup] utility SHALL support `delete local ` for an embedded `ON CLUSTER` backup: the node +where the command is issued SHALL remove its own local copy of the backup and SHALL make every other node of the cluster +remove its local copy as well, so that no node of the cluster keeps `/var/lib/clickhouse/backup/` afterwards. + +## References + +* **clickhouse-backup**: https://github.com/Altinity/clickhouse-backup +* **ClickHouse:** https://clickhouse.tech +* **GitHub Repository:** https://github.com/Altinity/clickhouse-backup/blob/master/test/testflows/clickhouse_backup/requirements/requirements.md +* **Revision History:** https://github.com/Altinity/clickhouse-backup/commits/master/test/testflows/clickhouse_backup/requirements/requirements.md +* **Git:** https://git-scm.com/ + +[SRS]: #srs +[clickhouse-backup]: https://github.com/Altinity/clickhouse-backup +[ClickHouse]: https://clickhouse.tech +[GitHub Repository]: https://github.com/Altinity/clickhouse-backup/blob/master/test/testflows/clickhouse_backup/requirements/requirements.md +[Revision History]: https://github.com/Altinity/clickhouse-backup/commits/master/test/testflows/clickhouse_backup/requirements/requirements.md +[Git]: https://git-scm.com/ +[GitHub]: https://github.com/ diff --git a/test/testflows/clickhouse_backup/requirements/requirements.py b/test/testflows/clickhouse_backup/requirements/requirements.py index 54202a523..1005e0fcc 100644 --- a/test/testflows/clickhouse_backup/requirements/requirements.py +++ b/test/testflows/clickhouse_backup/requirements/requirements.py @@ -1,6 +1,6 @@ # These requirements were auto generated # from software requirements specification (SRS) -# document by TestFlows v1.7.211116.1144005. +# document by TestFlows v2.0.250110.1002922. # Do not edit by hand but re-generate instead # using 'tfs requirements generate' command. from testflows.core import Specification @@ -18,10 +18,11 @@ description=( 'The [clickhouse-backup] utility SHALL support backing up and restoring tables containing any existing data type.\n' '\n' - ), + ), link=None, level=3, - num='4.1.1') + num='4.1.1' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_RestorePartiallyDropped = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.RestorePartiallyDropped', @@ -33,10 +34,11 @@ description=( 'The [clickhouse-backup] utility SHALL support restoring table that has lost some piece of data.\n' '\n' - ), + ), link=None, level=3, - num='4.1.2') + num='4.1.2' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_BackupDuringMutation = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.BackupDuringMutation', @@ -48,10 +50,11 @@ description=( 'The [clickhouse-backup] utility SHALL be able to create a backup if there is an ongoing mutation. Restoring from such backups SHALL also be possible.\n' '\n' - ), + ), link=None, level=3, - num='4.1.3') + num='4.1.3' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_AllTables = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.AllTables', @@ -63,10 +66,11 @@ description=( 'The [clickhouse-backup] utility SHALL support easy backup and restore of all [ClickHouse] tables that SHALL result in table having identical state as at the time of backup.\n' '\n' - ), + ), link=None, level=3, - num='4.1.4') + num='4.1.4' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_SpecificTables = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables', @@ -79,10 +83,11 @@ 'The [clickhouse-backup] utility SHALL support easy backup and restore of specific [ClickHouse] tables that SHALL result in table having identical state as at the time of backup.\n' '\n' '\n' - ), + ), link=None, level=3, - num='4.1.5') + num='4.1.5' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_SpecificTables_Performance = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables.Performance', @@ -94,10 +99,11 @@ description=( 'The [clickhouse-backup] utility SHALL create backups for a particular table with high performance due to avoiding scanning whole `system.tables` when set `table` query string parameter or `--tables` CLI parameter.\n' '\n' - ), + ), link=None, level=3, - num='4.1.6') + num='4.1.6' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_SpecificTables_ManyColumns = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.SpecificTables.ManyColumns', @@ -110,10 +116,11 @@ 'The [clickhouse-backup] utility SHALL support creating backups and restoring from them for a table with high number of columns.\n' '\n' '\n' - ), + ), link=None, level=3, - num='4.1.7') + num='4.1.7' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_ReplicatedTable = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.ReplicatedTable', @@ -126,10 +133,11 @@ 'The [clickhouse-backup] utility SHALL support easy backup and restore of [ClickHouse] replicated tables from the backup\n' 'that SHALL result in table having identical state as at the time of backup.\n' '\n' - ), + ), link=None, level=3, - num='4.1.8') + num='4.1.8' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_MultipleBackups_EfficientStorageByUsingHardLinks = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.MultipleBackups.EfficientStorageByUsingHardLinks', @@ -142,10 +150,11 @@ 'The [clickhouse-backup] utility SHALL support efficient storage of multiple backups on the file system by\n' 'manipulating hardlinks used by [ClickHouse] parts.\n' '\n' - ), + ), link=None, level=3, - num='4.1.9') + num='4.1.9' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_CompressionLevel = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.CompressionLevel', @@ -157,10 +166,11 @@ description=( 'The [clickhouse-backup] utility SHALL support uploading to/downloading from remote storage with specified compression level.\n' '\n' - ), + ), link=None, level=3, - num='4.1.10') + num='4.1.10' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_ClickHouseVersions = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.ClickHouseVersions', @@ -172,10 +182,11 @@ description=( 'The [clickhouse-backup] utility SHALL support [ClickHouse] versions above `1.1.54390`.\n' '\n' - ), + ), link=None, level=3, - num='4.1.11') + num='4.1.11' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_TieredStorage = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.TieredStorage', @@ -187,10 +198,11 @@ description=( 'The [clickhouse-backup] utility SHALL support `tiered storage` that was added in ClickHouse `19.15.2.2`.\n' '\n' - ), + ), link=None, level=3, - num='4.1.12') + num='4.1.12' +) RQ_SRS_013_ClickHouse_BackupUtility_Generic_EncryptedStorage = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Generic.EncryptedStorage', @@ -202,10 +214,11 @@ description=( 'The [clickhouse-backup] utility SHALL support creating backups from [encrypted storages](https://clickhouse.tech/docs/en/operations/storing-data/#encrypted-virtual-file-system) and restoring them.\n' '\n' - ), + ), link=None, level=3, - num='4.1.13') + num='4.1.13' +) RQ_SRS_013_ClickHouse_BackupUtility_Views_View = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Views.View', @@ -217,10 +230,11 @@ description=( 'The [clickhouse-backup] utility SHALL support proper handling of `View` over `MergeTree` table engines.\n' '\n' - ), + ), link=None, level=3, - num='4.2.1') + num='4.2.1' +) RQ_SRS_013_ClickHouse_BackupUtility_Views_MaterializedView = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Views.MaterializedView', @@ -232,10 +246,11 @@ description=( 'The [clickhouse-backup] utility SHALL support proper handling of `MaterializedView` over `MergeTree` table engines.\n' '\n' - ), + ), link=None, level=3, - num='4.2.2') + num='4.2.2' +) RQ_SRS_013_ClickHouse_BackupUtility_Views_LiveView = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Views.LiveView', @@ -247,10 +262,11 @@ description=( 'The [clickhouse-backup] utility SHALL support proper handling of `LiveView` over `MergeTree` table engines.\n' '\n' - ), + ), link=None, level=3, - num='4.2.3') + num='4.2.3' +) RQ_SRS_013_ClickHouse_BackupUtility_Views_WindowView = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Views.WindowView', @@ -262,10 +278,11 @@ description=( 'The [clickhouse-backup] utility SHALL support proper handling of `WindowView` over `MergeTree` table engines.\n' '\n' - ), + ), link=None, level=3, - num='4.2.4') + num='4.2.4' +) RQ_SRS_013_ClickHouse_BackupUtility_Views_NestedViews = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Views.NestedViews', @@ -277,10 +294,11 @@ description=( 'The [clickhouse-backup] utility SHALL support proper handling of nested views, e.g. `View` over `View`.\n' '\n' - ), + ), link=None, level=3, - num='4.2.5') + num='4.2.5' +) RQ_SRS_013_ClickHouse_BackupUtility_TableEngines_MergeTree = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.MergeTree', @@ -292,10 +310,11 @@ description=( 'The [clickhouse-backup] utility SHALL support only `MergeTree` family table engines.\n' '\n' - ), + ), link=None, level=3, - num='4.3.1') + num='4.3.1' +) RQ_SRS_013_ClickHouse_BackupUtility_TableEngines_OtherEngines = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.OtherEngines', @@ -307,10 +326,11 @@ description=( 'The [clickhouse-backup] utility workflow SHALL NOT be broken by presence of tables with engines other that listed above.\n' '\n' - ), + ), link=None, level=3, - num='4.3.2') + num='4.3.2' +) RQ_SRS_013_ClickHouse_BackupUtility_TableEngines_OtherEngines_Kafka = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.TableEngines.OtherEngines.Kafka', @@ -322,10 +342,11 @@ description=( 'The [clickhouse-backup] utility SHALL only support creating table schema backup and restore for `Kafka` table engine.\n' '\n' - ), + ), link=None, level=3, - num='4.3.3') + num='4.3.3' +) RQ_SRS_013_ClickHouse_BackupUtility_DatabaseEngines_Atomic = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.Atomic', @@ -337,10 +358,11 @@ description=( 'The [clickhouse-backup] utility SHALL support `MergeTree` family table engines with `Atomic` database engine.\n' '\n' - ), + ), link=None, level=3, - num='4.4.1') + num='4.4.1' +) RQ_SRS_013_ClickHouse_BackupUtility_DatabaseEngines_MaterializedMySQL = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.MaterializedMySQL', @@ -352,10 +374,11 @@ description=( 'The [clickhouse-backup] utility SHALL support `MergeTree` family table engines with `MaterializedMySQL` database engine.\n' '\n' - ), + ), link=None, level=3, - num='4.4.2') + num='4.4.2' +) RQ_SRS_013_ClickHouse_BackupUtility_DatabaseEngines_MaterializedPostgreSQL = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.DatabaseEngines.MaterializedPostgreSQL', @@ -367,10 +390,11 @@ description=( 'The [clickhouse-backup] utility SHALL support `MergeTree` family table engines with `MaterializedPostgreSQL` database engine.\n' '\n' - ), + ), link=None, level=3, - num='4.4.3') + num='4.4.3' +) RQ_SRS_013_ClickHouse_BackupUtility_CloudStorage = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage', @@ -382,10 +406,11 @@ description=( 'The [clickhouse-backup] utility SHALL support creating and uploading backups on cloud storage.\n' '\n' - ), + ), link=None, level=3, - num='4.5.1') + num='4.5.1' +) RQ_SRS_013_ClickHouse_BackupUtility_CloudStorage_IncrementalBackups = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.IncrementalBackups', @@ -409,10 +434,11 @@ '* `ALTER TABLE
CLEAR COLUMN`\n' '* `ALTER TABLE
DROP COLUMN`\n' '\n' - ), + ), link=None, level=3, - num='4.5.2') + num='4.5.2' +) RQ_SRS_013_ClickHouse_BackupUtility_CloudStorage_CompressionFormats = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.CompressionFormats', @@ -424,10 +450,11 @@ description=( 'The [clickhouse-backup] utility SHALL support storing backup on a cloud storage in following compression formats: `tar`, `bzip2`, `gzip`, `sz`, `xz`, `br/brotli`, `zstd`. In case of other value, error shall be thrown and no backup uploaded.\n' '\n' - ), + ), link=None, level=3, - num='4.5.3') + num='4.5.3' +) RQ_SRS_013_ClickHouse_BackupUtility_CloudStorage_S3_Minio = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.S3.Minio', @@ -439,10 +466,11 @@ description=( 'The [clickhouse-backup] utility SHALL support storing backups on a Minio S3 storage. It also SHALL be possible to restore from such backups.\n' '\n' - ), + ), link=None, level=3, - num='4.5.4') + num='4.5.4' +) RQ_SRS_013_ClickHouse_BackupUtility_CloudStorage_S3_AWS = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.S3.AWS', @@ -454,10 +482,11 @@ description=( 'The [clickhouse-backup] utility SHALL support storing backups on [AWS](https://aws.amazon.com/products/storage/) cloud servers. It also SHALL be possible to restore from such backups.\n' '\n' - ), + ), link=None, level=3, - num='4.5.5') + num='4.5.5' +) RQ_SRS_013_ClickHouse_BackupUtility_CloudStorage_GCS = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.GCS', @@ -469,10 +498,11 @@ description=( 'The [clickhouse-backup] utility SHALL support storing backups on [GCS](https://cloud.google.com/storage) cloud servers. It also SHALL be possible to restore from such backups.\n' '\n' - ), + ), link=None, level=3, - num='4.5.6') + num='4.5.6' +) RQ_SRS_013_ClickHouse_BackupUtility_CloudStorage_Azure = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.Azure', @@ -484,10 +514,11 @@ description=( 'The [clickhouse-backup] utility SHALL support storing backups on [Azure](https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction) cloud servers. It also SHALL be possible to restore from such backups.\n' '\n' - ), + ), link=None, level=3, - num='4.5.7') + num='4.5.7' +) RQ_SRS_013_ClickHouse_BackupUtility_CloudStorage_TencentCOS = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.TencentCOS', @@ -499,10 +530,11 @@ description=( 'The [clickhouse-backup] utility SHALL support storing backups on [Tencent COS](https://intl.cloud.tencent.com/product/cos) cloud servers. It also SHALL be possible to restore from such backups.\n' '\n' - ), + ), link=None, level=3, - num='4.5.8') + num='4.5.8' +) RQ_SRS_013_ClickHouse_BackupUtility_CloudStorage_FTP = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.FTP', @@ -514,10 +546,11 @@ description=( 'The [clickhouse-backup] utility SHALL support storing backups on a remote FTP server. It also SHALL be possible to restore from such backup.\n' '\n' - ), + ), link=None, level=3, - num='4.5.9') + num='4.5.9' +) RQ_SRS_013_ClickHouse_BackupUtility_CloudStorage_SFTP = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.SFTP', @@ -529,10 +562,11 @@ description=( 'The [clickhouse-backup] utility SHALL support storing backups on a remote SFTP (also known as Secure FTP) server. It also SHALL be possible to restore from such backup.\n' '\n' - ), + ), link=None, level=3, - num='4.5.10') + num='4.5.10' +) RQ_SRS_013_ClickHouse_BackupUtility_CloudStorage_GreaterThan5TB = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CloudStorage.GreaterThan5TB', @@ -545,10 +579,11 @@ 'The [clickhouse-backup] utility SHALL support uploading backups greater than `5TB` using partitioning by 5TB chunks.\n' 'More info can be found [here](https://stackoverflow.com/questions/54012602/is-there-a-file-size-limit-for-amazon-s3#:~:text=While%20the%20maximum%20file%20size,any%20files%20larger%20than%20100MB)\n' '\n' - ), + ), link=None, level=3, - num='4.5.11') + num='4.5.11' +) RQ_SRS_013_ClickHouse_BackupUtility_Configs_Backup = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Configs.Backup', @@ -560,7 +595,7 @@ description=( 'The [clickhouse-backup] utility SHALL support backing up ClickHouse configs.\n' '\n' - 'By configs we mean the following files:\n' + 'By configs, we mean the following files:\n' ' * `/etc/clickhouse-server/config.xml`\n' ' * `/etc/clickhouse-server/config.d/*.xml`\n' ' * `/etc/clickhouse-server/users.xml`\n' @@ -568,10 +603,11 @@ '\n' 'Listed above are default config files. However, these values may be overriden on clickhouse-server launch.\n' '\n' - ), + ), link=None, level=3, - num='4.6.1') + num='4.6.1' +) RQ_SRS_013_ClickHouse_BackupUtility_Configs_Restore = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Configs.Restore', @@ -583,7 +619,7 @@ description=( 'The [clickhouse-backup] utility SHALL support restoring ClickHouse configs from backup. This should be done automatically in case configs backup is present. If configs are found in backup, ClickHouse server SHALL restart automatically.\n' '\n' - 'By configs we mean the following files:\n' + 'By configs, we mean the following files:\n' ' * `/etc/clickhouse-server/config.xml`\n' ' * `/etc/clickhouse-server/config.d/*.xml`\n' ' * `/etc/clickhouse-server/users.xml`\n' @@ -591,10 +627,11 @@ '\n' 'Listed above are default config files. However, these values may be overriden on clickhouse-server launch.\n' '\n' - ), + ), link=None, level=3, - num='4.6.2') + num='4.6.2' +) RQ_SRS_013_ClickHouse_BackupUtility_RBAC_Backup = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.RBAC.Backup', @@ -611,10 +648,11 @@ ' * `SETTINGS PROFILE`\n' ' * `QUOTA`\n' '\n' - ), + ), link=None, level=3, - num='4.7.1') + num='4.7.1' +) RQ_SRS_013_ClickHouse_BackupUtility_RBAC_Restore = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.RBAC.Restore', @@ -631,10 +669,11 @@ ' * `SETTINGS PROFILE`\n' ' * `QUOTA`\n' '\n' - ), + ), link=None, level=3, - num='4.7.2') + num='4.7.2' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_Usage = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.Usage', @@ -673,10 +712,11 @@ ' --version, -v print the version\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.1') + num='4.8.1' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_Tables = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.Tables', @@ -697,10 +737,11 @@ '\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.2') + num='4.8.2' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_Create = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.Create', @@ -728,10 +769,11 @@ ' --table value, --tables value, -t value \n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.3') + num='4.8.3' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_CreateRemote = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.CreateRemote', @@ -762,10 +804,11 @@ '\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.4') + num='4.8.4' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_MatchExpression = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.MatchExpression', @@ -797,10 +840,11 @@ "\tlo '-' hi matches character c for lo <= c <= hi\n" '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.5') + num='4.8.5' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_Upload = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.Upload', @@ -823,10 +867,11 @@ ' --diff-from value \n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.6') + num='4.8.6' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_List = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.List', @@ -850,10 +895,11 @@ ' --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG]\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.7') + num='4.8.7' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_Download = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.Download', @@ -875,10 +921,11 @@ ' --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG]\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.8') + num='4.8.8' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_Restore = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.Restore', @@ -907,10 +954,11 @@ ' --rm, --drop Drop table before restore\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.9') + num='4.8.9' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_RestoreRemote = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.RestoreRemote', @@ -941,10 +989,11 @@ ' --schema, -s Schemas only\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.10') + num='4.8.10' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_Delete = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.Delete', @@ -967,10 +1016,11 @@ ' --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG]\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.11') + num='4.8.11' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_DefaultConfig = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.DefaultConfig', @@ -990,10 +1040,11 @@ ' --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG]\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.12') + num='4.8.12' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_Server = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.Server', @@ -1013,10 +1064,11 @@ ' --config FILE, -c FILE Config FILE name. (default: "/etc/clickhouse-backup/config.yml") [$CLICKHOUSE_BACKUP_CONFIG]\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.8.13') + num='4.8.13' +) RQ_SRS_013_ClickHouse_BackupUtility_CLI_Help = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.CLI.Help', @@ -1028,10 +1080,11 @@ description=( 'The [clickhouse-backup] utility SHALL support `help` option to show the list of commands or help for one command.\n' '\n' - ), + ), link=None, level=3, - num='4.8.14') + num='4.8.14' +) RQ_SRS_013_ClickHouse_BackupUtility_Configuration = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.Configuration', @@ -1144,10 +1197,11 @@ ' compression_level: 1 # SFTP_COMPRESSION_LEVEL\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.9.1') + num='4.9.1' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server', @@ -1160,10 +1214,11 @@ 'The [clickhouse-backup] utility SHALL support `clickhouse-backup server`\n' 'command to start REST API server.\n' '\n' - ), + ), link=None, level=3, - num='4.10.1') + num='4.10.1' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_AsyncPostQuery = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.AsyncPostQuery', @@ -1175,10 +1230,11 @@ description=( 'The [clickhouse-backup] utility SHALL return `code 423` if a POST request is received while previous operation is not finished yet. The request SHALL NOT be executed in such case.\n' '\n' - ), + ), link=None, level=3, - num='4.10.2') + num='4.10.2' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_OutputFormat_Default = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.OutputFormat.Default', @@ -1190,12 +1246,13 @@ description=( 'The [clickhouse-backup] utility REST API server SHALL return results\n' "in a format compatible with [ClickHouse]'s `JSONEachRow` format\n" - 'for all the the endpoints.\n' + 'for all the endpoints.\n' '\n' - ), + ), link=None, level=3, - num='4.10.3') + num='4.10.3' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_GetTables = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetTables', @@ -1213,10 +1270,11 @@ '\n' 'to print the list of tables. For example, `curl -s localhost:7171/backup/tables | jq`.\n' '\n' - ), + ), link=None, level=3, - num='4.10.4') + num='4.10.4' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_PostCreate = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostCreate', @@ -1250,10 +1308,11 @@ '{"status":"acknowledged","operation":"create","backup_name":""}\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.10.5') + num='4.10.5' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_PostCreate_Remote = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostCreate.Remote', @@ -1274,10 +1333,11 @@ 'This operation SHALL be async, so the API will return once the operation has been started.\n' '\n' '\n' - ), + ), link=None, level=3, - num='4.10.6') + num='4.10.6' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_PostUpload = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostUpload', @@ -1301,10 +1361,11 @@ '\n' 'This operation SHALL be async, so the API will return once the operation has been started.\n' '\n' - ), + ), link=None, level=3, - num='4.10.7') + num='4.10.7' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_GetList = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList', @@ -1331,10 +1392,11 @@ '...\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.10.8') + num='4.10.8' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_GetList_Local = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList.Local', @@ -1361,10 +1423,11 @@ '...\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.10.9') + num='4.10.9' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_GetList_Remote = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetList.Remote', @@ -1389,10 +1452,11 @@ '...\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.10.10') + num='4.10.10' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_PostDownload = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDownload', @@ -1414,10 +1478,11 @@ '\n' 'This operation SHALL be async, so the API will return once the operation has been started.\n' '\n' - ), + ), link=None, level=3, - num='4.10.11') + num='4.10.11' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_PostRestore = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostRestore', @@ -1443,10 +1508,11 @@ '* Optional query argument `rbac` SHALL work the same as the `--rbac` CLI argument (restore RBAC only).\n' '* Optional query argument `configs` SHALL work the same as the `--configs` CLI argument (restore configs only).\n' '\n' - ), + ), link=None, level=3, - num='4.10.12') + num='4.10.12' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_PostRestore_Remote = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostRestore.Remote', @@ -1464,10 +1530,11 @@ '\n' 'to restore from an existing remote backup.\n' '\n' - ), + ), link=None, level=3, - num='4.10.13') + num='4.10.13' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_PostDelete_Remote = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDelete.Remote', @@ -1493,10 +1560,11 @@ '\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.10.14') + num='4.10.14' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_PostDelete_Local = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostDelete.Local', @@ -1522,10 +1590,11 @@ '{"status":"success","operation":"delete","backup_name":"","location":"local"}\n' '```\n' '\n' - ), + ), link=None, level=3, - num='4.10.15') + num='4.10.15' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_GetStatus = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetStatus', @@ -1549,10 +1618,11 @@ "{'command': '', 'status': 'in progress', 'start': '2021-10-26 12:11:34'}\n" '```\n' '\n' - ), + ), link=None, level=3, - num='4.10.16') + num='4.10.16' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_GetActions = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetActions', @@ -1570,10 +1640,11 @@ '\n' 'to display a list of current async operations.\n' '\n' - ), + ), link=None, level=3, - num='4.10.17') + num='4.10.17' +) RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_PostActions = Requirement( name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostActions', @@ -1592,17 +1663,136 @@ 'to create a new action with parameters being the same as for the CLI commands.\n' 'All CLI actions such as `create`, `upload` etc. SHALL be supported.\n' '\n' - ), + ), link=None, level=3, - num='4.10.18') + num='4.10.18' +) + +RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster = Requirement( + name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster', + version='1.0', + priority=None, + group=None, + type=None, + uid=None, + description=( + 'The [clickhouse-backup] utility SHALL support `clickhouse.use_embedded_backup_restore_cluster: ""`\n' + '(macros such as `{cluster}` SHALL be allowed). When it is set together with `clickhouse.use_embedded_backup_restore: true`\n' + 'and `clickhouse.embedded_backup_disk: ""`, the node where `create_remote ` or `restore_remote ` is issued\n' + '(the initiator) SHALL run `BACKUP ... ON CLUSTER TO S3(...)` (resp. `RESTORE ... ON CLUSTER FROM S3(...)`)\n' + 'so that the data of every shard and replica of the cluster is written to (resp. read from) the shared remote storage\n' + 'configured by `general.remote_storage` in a single operation.\n' + '\n' + ), + link=None, + level=3, + num='4.11.1' +) + +RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_Worker = Requirement( + name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Worker', + version='1.0', + priority=None, + group=None, + type=None, + uid=None, + description=( + 'The [clickhouse-backup] utility SHALL support the `--embedded-on-cluster-worker` flag for the `create`, `upload`, `download`,\n' + '`restore`, `create_remote` and `restore_remote` commands. For every non-local host of the cluster listed in `system.clusters`\n' + 'the initiator SHALL execute\n' + "`INSERT INTO FUNCTION remote(':', 'system', 'backup_actions', '', '') (command) VALUES ('create_remote --embedded-on-cluster-worker ')`\n" + '(resp. `restore_remote --embedded-on-cluster-worker `), SHALL wait for those worker operations to finish by polling\n' + 'the same remote `system.backup_actions` table, and SHALL report the operation as failed when any worker fails or cannot be reached.\n' + '\n' + ), + link=None, + level=3, + num='4.11.2' +) + +RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_Requirements = Requirement( + name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Requirements', + version='1.0', + priority=None, + group=None, + type=None, + uid=None, + description=( + 'Every [ClickHouse] node of the cluster SHALL run its own `clickhouse-backup server` with `api.create_integration_tables: true`\n' + '(which creates the `system.backup_actions`, `system.backup_list` and `system.backup_version` `ENGINE=URL` tables pointing at\n' + '`api.integration_tables_host:`), and all servers SHALL point at the same remote storage destination\n' + '(same bucket and same `path`).\n' + '\n' + ), + link=None, + level=3, + num='4.11.3' +) + +RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_RestoreOrder = Requirement( + name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.RestoreOrder', + version='1.0', + priority=None, + group=None, + type=None, + uid=None, + description=( + 'During `restore_remote` the initiator SHALL make the workers process the backup metadata (download and prepare the\n' + 'per-table metadata of their own shard and replica) before the `RESTORE ... ON CLUSTER` SQL command is executed,\n' + 'so that the data of every node of the cluster is restored and row counts on every node match the original ones.\n' + '\n' + ), + link=None, + level=3, + num='4.11.4' +) + +RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_MetadataLayout = Requirement( + name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.MetadataLayout', + version='1.0', + priority=None, + group=None, + type=None, + uid=None, + description=( + 'The [clickhouse-backup] utility SHALL store an embedded `ON CLUSTER` backup in the remote storage with the following layout:\n' + '\n' + '* `//.backup` - native [ClickHouse] backup descriptor\n' + '* `//metadata.json` - [clickhouse-backup] backup metadata\n' + '* `//shards//replicas//metadata/...` and `//shards//replicas//data/...` - native [ClickHouse] schema and data of each shard and replica\n' + '* `//shards//replicas//metadata//
.json` - [clickhouse-backup] per-table metadata of each shard and replica\n' + '\n' + ), + link=None, + level=3, + num='4.11.5' +) + +RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_DeleteLocal = Requirement( + name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.DeleteLocal', + version='1.0', + priority=None, + group=None, + type=None, + uid=None, + description=( + 'The [clickhouse-backup] utility SHALL support `delete local ` for an embedded `ON CLUSTER` backup: the node\n' + 'where the command is issued SHALL remove its own local copy of the backup and SHALL make every other node of the cluster\n' + 'remove its local copy as well, so that no node of the cluster keeps `/var/lib/clickhouse/backup/` afterwards.\n' + '\n' + ), + link=None, + level=3, + num='4.11.6' +) QA_SRS013_ClickHouse_Backup_Utility = Specification( - name='QA-SRS013 ClickHouse Backup Utility', + name='QA-SRS013 ClickHouse Backup Utility', description=None, author=None, - date=None, - status=None, + date=None, + status=None, approved_by=None, approved_date=None, approved_version=None, @@ -1702,6 +1892,13 @@ Heading(name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetStatus', level=3, num='4.10.16'), Heading(name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetActions', level=3, num='4.10.17'), Heading(name='RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostActions', level=3, num='4.10.18'), + Heading(name='Embedded Backup', level=2, num='4.11'), + Heading(name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster', level=3, num='4.11.1'), + Heading(name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Worker', level=3, num='4.11.2'), + Heading(name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Requirements', level=3, num='4.11.3'), + Heading(name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.RestoreOrder', level=3, num='4.11.4'), + Heading(name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.MetadataLayout', level=3, num='4.11.5'), + Heading(name='RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.DeleteLocal', level=3, num='4.11.6'), Heading(name='References', level=1, num='5'), ), requirements=( @@ -1777,8 +1974,14 @@ RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_GetStatus, RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_GetActions, RQ_SRS_013_ClickHouse_BackupUtility_REST_API_Server_PostActions, - ), - content=''' + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster, + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_Worker, + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_Requirements, + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_RestoreOrder, + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_MetadataLayout, + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_DeleteLocal, + ), + content=r''' # QA-SRS013 ClickHouse Backup Utility # Software Requirements Specification @@ -1873,6 +2076,13 @@ * 4.10.16 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetStatus](#rqsrs-013clickhousebackuputilityrestapiservergetstatus) * 4.10.17 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetActions](#rqsrs-013clickhousebackuputilityrestapiservergetactions) * 4.10.18 [RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.PostActions](#rqsrs-013clickhousebackuputilityrestapiserverpostactions) + * 4.11 [Embedded Backup](#embedded-backup) + * 4.11.1 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster](#rqsrs-013clickhousebackuputilityembeddedbackuponcluster) + * 4.11.2 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Worker](#rqsrs-013clickhousebackuputilityembeddedbackuponclusterworker) + * 4.11.3 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Requirements](#rqsrs-013clickhousebackuputilityembeddedbackuponclusterrequirements) + * 4.11.4 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.RestoreOrder](#rqsrs-013clickhousebackuputilityembeddedbackuponclusterrestoreorder) + * 4.11.5 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.MetadataLayout](#rqsrs-013clickhousebackuputilityembeddedbackuponclustermetadatalayout) + * 4.11.6 [RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.DeleteLocal](#rqsrs-013clickhousebackuputilityembeddedbackuponclusterdeletelocal) * 5 [References](#references) ## Revision History @@ -1883,9 +2093,10 @@ ## Introduction -[ClickHouse] does not support any convenient method to backup and restore databases or tables. +[ClickHouse] does not support any convenient method to make backup and restore databases or tables. +BACKUP/RESTORE command in non production ready status. This [SRS] covers the requirements for the [clickhouse-backup] utility that supports -easy [ClickHouse] backup and restore with cloud storages support. +easy [ClickHouse] backup and restore tool with cloud storages support. ## Terminology @@ -2117,7 +2328,7 @@ The [clickhouse-backup] utility SHALL support backing up ClickHouse configs. -By configs we mean the following files: +By configs, we mean the following files: * `/etc/clickhouse-server/config.xml` * `/etc/clickhouse-server/config.d/*.xml` * `/etc/clickhouse-server/users.xml` @@ -2130,7 +2341,7 @@ The [clickhouse-backup] utility SHALL support restoring ClickHouse configs from backup. This should be done automatically in case configs backup is present. If configs are found in backup, ClickHouse server SHALL restart automatically. -By configs we mean the following files: +By configs, we mean the following files: * `/etc/clickhouse-server/config.xml` * `/etc/clickhouse-server/config.d/*.xml` * `/etc/clickhouse-server/users.xml` @@ -2484,7 +2695,7 @@ gcs: credentials_file: "" # GCS_CREDENTIALS_FILE credentials_json: "" # GCS_CREDENTIALS_JSON - credentials_json_encoded: "" # GCS_CREDENTIALS_JSON_ENCODED' + credentials_json_encoded: "" # GCS_CREDENTIALS_JSON_ENCODED bucket: "" # GCS_BUCKET path: "" # GCS_PATH compression_level: 1 # GCS_COMPRESSION_LEVEL @@ -2545,7 +2756,7 @@ The [clickhouse-backup] utility REST API server SHALL return results in a format compatible with [ClickHouse]'s `JSONEachRow` format -for all the the endpoints. +for all the endpoints. #### RQ.SRS-013.ClickHouse.BackupUtility.REST.API.Server.GetTables version: 1.0 @@ -2797,6 +3008,63 @@ to create a new action with parameters being the same as for the CLI commands. All CLI actions such as `create`, `upload` etc. SHALL be supported. +### Embedded Backup + +The [clickhouse-backup] utility SHALL support the native [ClickHouse] `BACKUP` / `RESTORE` SQL commands +(`clickhouse.use_embedded_backup_restore: true`) and SHALL be able to orchestrate them across a whole cluster. + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster +version: 1.0 + +The [clickhouse-backup] utility SHALL support `clickhouse.use_embedded_backup_restore_cluster: ""` +(macros such as `{cluster}` SHALL be allowed). When it is set together with `clickhouse.use_embedded_backup_restore: true` +and `clickhouse.embedded_backup_disk: ""`, the node where `create_remote ` or `restore_remote ` is issued +(the initiator) SHALL run `BACKUP ... ON CLUSTER TO S3(...)` (resp. `RESTORE ... ON CLUSTER FROM S3(...)`) +so that the data of every shard and replica of the cluster is written to (resp. read from) the shared remote storage +configured by `general.remote_storage` in a single operation. + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Worker +version: 1.0 + +The [clickhouse-backup] utility SHALL support the `--embedded-on-cluster-worker` flag for the `create`, `upload`, `download`, +`restore`, `create_remote` and `restore_remote` commands. For every non-local host of the cluster listed in `system.clusters` +the initiator SHALL execute +`INSERT INTO FUNCTION remote(':', 'system', 'backup_actions', '', '') (command) VALUES ('create_remote --embedded-on-cluster-worker ')` +(resp. `restore_remote --embedded-on-cluster-worker `), SHALL wait for those worker operations to finish by polling +the same remote `system.backup_actions` table, and SHALL report the operation as failed when any worker fails or cannot be reached. + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.Requirements +version: 1.0 + +Every [ClickHouse] node of the cluster SHALL run its own `clickhouse-backup server` with `api.create_integration_tables: true` +(which creates the `system.backup_actions`, `system.backup_list` and `system.backup_version` `ENGINE=URL` tables pointing at +`api.integration_tables_host:`), and all servers SHALL point at the same remote storage destination +(same bucket and same `path`). + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.RestoreOrder +version: 1.0 + +During `restore_remote` the initiator SHALL make the workers process the backup metadata (download and prepare the +per-table metadata of their own shard and replica) before the `RESTORE ... ON CLUSTER` SQL command is executed, +so that the data of every node of the cluster is restored and row counts on every node match the original ones. + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.MetadataLayout +version: 1.0 + +The [clickhouse-backup] utility SHALL store an embedded `ON CLUSTER` backup in the remote storage with the following layout: + +* `//.backup` - native [ClickHouse] backup descriptor +* `//metadata.json` - [clickhouse-backup] backup metadata +* `//shards//replicas//metadata/...` and `//shards//replicas//data/...` - native [ClickHouse] schema and data of each shard and replica +* `//shards//replicas//metadata//
.json` - [clickhouse-backup] per-table metadata of each shard and replica + +#### RQ.SRS-013.ClickHouse.BackupUtility.EmbeddedBackup.OnCluster.DeleteLocal +version: 1.0 + +The [clickhouse-backup] utility SHALL support `delete local ` for an embedded `ON CLUSTER` backup: the node +where the command is issued SHALL remove its own local copy of the backup and SHALL make every other node of the cluster +remove its local copy as well, so that no node of the cluster keeps `/var/lib/clickhouse/backup/` afterwards. + ## References * **clickhouse-backup**: https://github.com/Altinity/clickhouse-backup @@ -2812,4 +3080,5 @@ [Revision History]: https://github.com/Altinity/clickhouse-backup/commits/master/test/testflows/clickhouse_backup/requirements/requirements.md [Git]: https://git-scm.com/ [GitHub]: https://github.com/ -''') +''' +) diff --git a/test/testflows/clickhouse_backup/tests/embedded_on_cluster.py b/test/testflows/clickhouse_backup/tests/embedded_on_cluster.py new file mode 100644 index 000000000..8b6f53fff --- /dev/null +++ b/test/testflows/clickhouse_backup/tests/embedded_on_cluster.py @@ -0,0 +1,296 @@ +import json +import os +import time + +from clickhouse_backup.requirements.requirements import * +from clickhouse_backup.tests.common import * +from clickhouse_backup.tests.steps import * + +CLUSTER = "sharded_cluster" +TABLE = "embedded_oc" +MINIO_URL = "http://minio:9000" +MINIO_BUCKET = "clickhouse" +MINIO_ACCESS_KEY = "access_key" +MINIO_SECRET_KEY = "it_is_my_super_secret_key" +WORKER_FLAG = "--embedded-on-cluster-worker" + + +@TestStep(When) +def post_action(self, url, command): + """POST a command to /backup/actions and expect it to be acknowledged.""" + r = api_request(endpoint=f"{url}/backup/actions", request_type="post", payload={"command": command}) + with Then("the command is acknowledged"): + assert r.status_code in (200, 201), error(r.text) + # async commands (create_remote, restore_remote) are acknowledged, `delete` is executed synchronously + assert r.json()["status"] in ("acknowledged", "success"), error(r.text) + + +@TestStep(Then) +def wait_action(self, url, command, timeout=600): + """Poll /backup/actions until the last row for ``command`` leaves ``in progress`` and return it.""" + deadline = time.time() + timeout + row = None + while time.time() < deadline: + rows = [json.loads(s) for s in api_request(endpoint=f"{url}/backup/actions").text.splitlines() if s] + matching = [r for r in rows if r["command"] == command] + if matching: + row = matching[-1] + if row["status"] != "in progress": + return row + time.sleep(2) + fail(f"`{command}` did not finish in {timeout}s, last row: {row}") + + +@TestStep(Given) +def wait_integration_tables(self, node, timeout=120): + """Wait until clickhouse-backup server created system.backup_actions on ``node``.""" + for _ in range(timeout): + if node.query("EXISTS TABLE system.backup_actions", no_checks=True).output.strip() == "1": + return + time.sleep(1) + fail(f"system.backup_actions not created on {node.name} in {timeout}s") + + +@TestStep(When) +def remote_objects(self, node, backup_name): + """List object keys of ``backup_name`` in the minio bucket under both s3.path and s3.object_disk_path. + + Returns ``(keys, path, object_disk_path)``: ClickHouse writes the native backup (``.backup``, + ``shards/N/replicas/M/{metadata,data}``) under ``object_disk_path``, clickhouse-backup uploads its own + ``metadata.json`` and per-node ``shards/N/replicas/M/metadata//
.json`` under ``path``. + """ + with open(self.context.backup_config_file) as f: + s3 = yaml.safe_load(f)["s3"] + prefixes = ",".join(sorted({s3["path"], s3["object_disk_path"]})) + r = node.query( + f"SELECT DISTINCT _path FROM s3('{MINIO_URL}/{MINIO_BUCKET}/{{{prefixes}}}/{backup_name}/**', " + # LineAsString instead of One: format One exists only since 23.8, the backup is tiny + f"'{MINIO_ACCESS_KEY}', '{MINIO_SECRET_KEY}', 'LineAsString') ORDER BY _path FORMAT TSVRaw" + ).output + return [line for line in r.splitlines() if line], s3["path"], s3["object_disk_path"] + + +def row_count(node, table=TABLE): + return int(node.query(f"SELECT count() FROM default.{table}").output.strip()) + + +@TestScenario +@Requirements( + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster("1.0"), + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_Worker("1.0"), + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_MetadataLayout("1.0"), +) +def create_remote_on_cluster(self): + """create_remote issued on one node backs up every shard of the cluster to the shared S3 bucket.""" + ch1, ch2 = self.context.nodes + url = self.context.url + backup_name = self.context.backup_name + + with Given(f"I create a MergeTree table ON CLUSTER {CLUSTER} and insert different rows on each shard"): + ch1.query(f"CREATE TABLE default.{TABLE} ON CLUSTER {CLUSTER} (id UInt64, v String) " + f"ENGINE=MergeTree ORDER BY id") + ch1.query(f"INSERT INTO default.{TABLE} SELECT number, toString(number) FROM numbers(10)") + ch2.query(f"INSERT INTO default.{TABLE} SELECT number, toString(number) FROM numbers(100, 25)") + assert {ch1.name: row_count(ch1), ch2.name: row_count(ch2)} == self.context.expected_rows, error() + + with When("I issue create_remote on the initiator"): + post_action(url=url, command=f"create_remote {backup_name}") + + with Then("the operation finishes successfully"): + row = wait_action(url=url, command=f"create_remote {backup_name}") + assert row["status"] == "success", error(row) + + with And("the backup is listed as remote via system.backup_list on the initiator node"): + r = ch1.query(f"SELECT location FROM system.backup_list WHERE name='{backup_name}' FORMAT TSVRaw").output + assert "remote" in r, error(r) + + with And("the worker on clickhouse2 reports a finished create_remote worker command"): + r = ch2.query(f"SELECT status, error FROM system.backup_actions " + f"WHERE command LIKE 'create_remote {WORKER_FLAG} %{backup_name}%' ORDER BY start DESC LIMIT 1 " + f"FORMAT TSVRaw").output + assert r.startswith("success"), error(r) + + with And("the backup metadata is tagged as embedded for the cluster"): + metadata = json.loads(self.context.backup.cmd(f"cat /var/lib/clickhouse/backup/{backup_name}/metadata.json").output) + assert f"embedded,cluster={CLUSTER}" in metadata.get("tags", ""), error(metadata.get("tags")) + + with And("the bucket contains the ON CLUSTER layout for both shards"): + keys, path, disk_path = remote_objects(node=ch1, backup_name=backup_name) + joined = "\n".join(keys) + for expected in ( + f"{disk_path}/{backup_name}/.backup", + f"{path}/{backup_name}/metadata.json", + f"{disk_path}/{backup_name}/shards/1/replicas/1/data/default/{TABLE}/", + f"{disk_path}/{backup_name}/shards/2/replicas/1/data/default/{TABLE}/", + # the CREATE query is identical on both shards, ClickHouse `deduplicate_files=1` stores one physical + # copy under the first host prefix and references it from `.backup`, so only shard 1 has the `.sql` + f"{disk_path}/{backup_name}/shards/1/replicas/1/metadata/default/{TABLE}.sql", + f"{path}/{backup_name}/shards/1/replicas/1/metadata/default/{TABLE}.json", + f"{path}/{backup_name}/shards/2/replicas/1/metadata/default/{TABLE}.json", + ): + with By(f"expect an object matching {expected}"): + assert expected in joined, error(joined) + + +@TestScenario +@Requirements( + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster("1.0"), + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_RestoreOrder("1.0"), +) +def restore_remote_on_cluster(self): + """restore_remote issued on one node restores every shard of the cluster.""" + ch1, ch2 = self.context.nodes + url = self.context.url + backup_name = self.context.backup_name + + with Given("I drop the table on every shard"): + ch1.query(f"DROP TABLE IF EXISTS default.{TABLE} ON CLUSTER {CLUSTER} SYNC") + for node in (ch1, ch2): + assert node.query(f"EXISTS TABLE default.{TABLE}").output.strip() == "0", error() + + with When("I issue restore_remote on the initiator"): + post_action(url=url, command=f"restore_remote {backup_name}") + + with Then("the operation finishes successfully"): + row = wait_action(url=url, command=f"restore_remote {backup_name}") + assert row["status"] == "success", error(row) + + with And("the worker on clickhouse2 reports a finished restore_remote worker command"): + r = ch2.query(f"SELECT status, error FROM system.backup_actions " + f"WHERE command LIKE 'restore_remote {WORKER_FLAG} %{backup_name}%' ORDER BY start DESC LIMIT 1 " + f"FORMAT TSVRaw").output + assert r.startswith("success"), error(r) + + with And("row counts on every shard match the originals"): + for node in (ch1, ch2): + with By(f"check {node.name}"): + assert row_count(node) == self.context.expected_rows[node.name], error() + + +@TestScenario +@Requirements( + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_DeleteLocal("1.0"), +) +def delete_local_on_cluster(self): + """delete local issued on one node removes the local backup on every node of the cluster.""" + url = self.context.url + backup_name = self.context.backup_name + backup_nodes = (self.context.backup, self.context.backup2) + backup_dir = f"/var/lib/clickhouse/backup/{backup_name}" + + with Given("the local backup directory exists on every node"): + for node in backup_nodes: + with By(f"check {node.name}"): + node.cmd(f"test -d {backup_dir}", exitcode=0) + + with When("I issue delete local on the initiator"): + post_action(url=url, command=f"delete local {backup_name}") + + with Then("the operation finishes successfully"): + row = wait_action(url=url, command=f"delete local {backup_name}") + assert row["status"] == "success", error(row) + + with And("the local backup directory is gone on every node"): + for node in backup_nodes: + with By(f"check {node.name}"): + node.cmd(f"test -d {backup_dir}", exitcode=1) + + +@TestScenario +@Requirements( + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_Worker("1.0"), + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_Requirements("1.0"), +) +def worker_unavailable(self): + """create_remote fails when the clickhouse-backup server of another node is not reachable.""" + url = self.context.url + backup2 = self.context.backup2 + backup_name = f"{self.context.backup_name}_noworker" + + try: + with Given("I stop the clickhouse-backup server of clickhouse2"): + backup2.stop_server() + + with When("I issue create_remote on the initiator"): + post_action(url=url, command=f"create_remote {backup_name}") + + with Then("the operation fails and the error points at the unreachable worker"): + row = wait_action(url=url, command=f"create_remote {backup_name}") + assert row["status"] == "error", error(row) + err = row.get("error", "").lower() + assert any(s in err for s in ("clickhouse2", "backup2", "backup_actions")), error(row) + + finally: + with Finally("I start the clickhouse-backup server of clickhouse2 again"): + backup2.start_server(timeout=120) + + with And("I remove leftovers of the failed backup"): + self.context.backup.cmd(f"clickhouse-backup delete local {backup_name}", exitcode=None) + self.context.backup.cmd(f"clickhouse-backup delete remote {backup_name}", exitcode=None) + backup2.cmd(f"clickhouse-backup delete local {backup_name}", exitcode=None) + + +@TestFeature +@Requirements( + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster("1.0"), + RQ_SRS_013_ClickHouse_BackupUtility_EmbeddedBackup_OnCluster_Requirements("1.0"), +) +def embedded_on_cluster(self): + """Embedded BACKUP/RESTORE ON CLUSTER orchestration between two clickhouse-backup servers (#928).""" + # BACKUP ... ON CLUSTER TO S3(...) needs the S3 backup engine and production ready BACKUP/RESTORE, + # same gate as TestEmbeddedS3 in test/integration, https://github.com/ClickHouse/ClickHouse/issues/39416 + if os.environ.get("CLICKHOUSE_VERSION", "26.8") < "23.3": + skip("embedded BACKUP/RESTORE ON CLUSTER needs ClickHouse 23.3+") + ch1, ch2 = self.context.nodes + backup = self.context.backup + backup2 = self.context.backup2 + self.context.url = f"http://localhost:{self.context.backup_api_port}" + self.context.backup_name = f"embedded_oc_{time.time_ns()}" + # scenario contexts are not shared between sibling scenarios, so the expected row counts live on the feature + self.context.expected_rows = {ch1.name: 10, ch2.name: 25} + + with open(self.context.backup_config_origin) as f: + s3_path = yaml.safe_load(f)["s3"]["path"] + + try: + with Given("I configure both clickhouse-backup servers for embedded ON CLUSTER backups to minio"): + config_modifier(fields={ + "general": {"remote_storage": "s3"}, + "s3": { + "access_key": MINIO_ACCESS_KEY, + "secret_key": MINIO_SECRET_KEY, + "endpoint": MINIO_URL, + "bucket": MINIO_BUCKET, + "disable_ssl": True, + "force_path_style": True, + "region": "us-west-2", + "object_disk_path": f"{s3_path}_embedded_disk", + }, + "clickhouse": { + # ValidateConfig requires clickhouse.timeout >= 240m with use_embedded_backup_restore, + # the base config has 5m and the server log.Fatal()s on it inside the actions handler + "timeout": "4h", + "use_embedded_backup_restore": True, + "use_embedded_backup_restore_cluster": CLUSTER, + "embedded_backup_disk": "", + }, + }) + + with And("both clickhouse-backup servers are running with integration tables"): + if not backup2.is_server_running(): + backup2.start_server(timeout=120) + for node in (ch1, ch2): + wait_integration_tables(node=node) + + for scenario in loads(current_module(), Scenario, Suite): + Scenario(run=scenario) + + finally: + with Finally("I remove the backup and the table"): + backup.cmd(f"clickhouse-backup delete remote {self.context.backup_name}", exitcode=None) + backup.cmd(f"clickhouse-backup delete local {self.context.backup_name}", exitcode=None) + backup2.cmd(f"clickhouse-backup delete local {self.context.backup_name}", exitcode=None) + ch1.query(f"DROP TABLE IF EXISTS default.{TABLE} ON CLUSTER {CLUSTER} SYNC", no_checks=True) + + with And("I restore the original config"): + config_modifier() diff --git a/test/testflows/helpers/cluster.py b/test/testflows/helpers/cluster.py index 1b905791c..a58f328cb 100644 --- a/test/testflows/helpers/cluster.py +++ b/test/testflows/helpers/cluster.py @@ -992,7 +992,7 @@ def node(self, node_name): """Get object with node bound methods :param node_name: name of service name """ - if node_name in ("clickhouse_backup", "clickhouse_backup_fips"): + if node_name in ("clickhouse_backup", "clickhouse_backup2", "clickhouse_backup_fips"): return BackupNode(self, node_name) if node_name.startswith("clickhouse"): return ClickHouseNode(self, node_name) @@ -1712,6 +1712,12 @@ def _do_up(self, clickhouse_image, clickhouse_version, zookeeper_image, zookeepe "GCS_CREDENTIALS_JSON_ENCODED": gcs_cred_json_encoded, "CLICKHOUSE_HOST": "clickhouse1", "CLICKHOUSE_BACKUP_CONFIG": "/etc/clickhouse-backup/config.yml", + # system.backup_actions / backup_list / backup_version ENGINE=URL tables on + # clickhouse1 point at this container; the embedded ON CLUSTER orchestration + # (#928) drives every node through them. Env overrides win over the shared + # config.yml so each backup container can advertise its own hostname. + "API_CREATE_INTEGRATION_TABLES": "true", + "API_INTEGRATION_TABLES_HOST": "backup", "TZ": "Europe/Moscow", "GOCOVERDIR": "/tmp/_coverage_/", }, @@ -1738,6 +1744,54 @@ def _do_up(self, clickhouse_image, clickhouse_version, zookeeper_image, zookeepe ) _wait_for_container_healthy(self._docker_client, self._container_ids["clickhouse_backup"], timeout=300) + # 5b. clickhouse_backup2 (shares volumes from clickhouse2) + # Second backup server for the embedded BACKUP/RESTORE ON CLUSTER orchestration + # (#928): the initiator drives clickhouse2 through its system.backup_actions + # URL table, which POSTs to this container. The container itself only sleeps; + # the server is started via BackupNode.start_server below so tests can stop + # and restart it (a server running as the container command would take the + # container down with it). + with And("starting clickhouse_backup2"): + self._start_container( + name="clickhouse_backup2", + image="ubuntu:latest", + hostname="backup2", + env={ + "DEBIAN_FRONTEND": "noninteractive", + "LOG_LEVEL": log_level, + "GCS_CREDENTIALS_JSON": gcs_cred_json, + "GCS_CREDENTIALS_JSON_ENCODED": gcs_cred_json_encoded, + "CLICKHOUSE_HOST": "clickhouse2", + "CLICKHOUSE_BACKUP_CONFIG": "/etc/clickhouse-backup/config.yml", + "API_CREATE_INTEGRATION_TABLES": "true", + "API_INTEGRATION_TABLES_HOST": "backup2", + "TZ": "Europe/Moscow", + "GOCOVERDIR": "/tmp/_coverage_/", + }, + volumes=backup_volumes, + volumes_from_name="clickhouse2", + ports=["7171"], + entrypoint=["/bin/bash"], + command=[ + "-c", + "set -x && " + "apt-get update && " + "apt-get install -y ca-certificates tzdata bash curl && " + "update-ca-certificates && " + "exec sleep infinity" + ], + cap_add=["SYS_NICE"], + healthcheck={ + "Test": ["CMD-SHELL", "command -v curl >/dev/null && test -x /bin/clickhouse-backup"], + "Interval": 5 * 1_000_000_000, + "Timeout": 5 * 1_000_000_000, + "Retries": 40, + "StartPeriod": 10 * 1_000_000_000, + }, + ) + _wait_for_container_healthy(self._docker_client, self._container_ids["clickhouse_backup2"], timeout=300) + self.node("clickhouse_backup2").start_server(timeout=120) + # 6. clickhouse_backup_fips (optional; same volumes view as clickhouse1) # This container hosts the FIPS-compatible binary built by ``make build-race-fips``. # It does not auto-start clickhouse-backup so tests can start/stop/kill it