Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion Manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,10 @@ NAME:
USAGE:
clickhouse-backup restore_remote [--schema] [--data] [-t, --tables=<db>.<table>] [-m, --restore-database-mapping=<originDB>:<targetDB>[,<...>]] [--tm, --restore-table-mapping=<originTable>:<targetTable>[,<...>]] [--partitions=<partitions_names>] [--rm, --drop] [-i, --ignore-dependencies] [--rbac] [--configs] [--named-collections] [--resumable] [--skip-empty-tables] <backup_name>

DESCRIPTION:
A remote backup with ClickHouse Cloud / native BACKUP layout (.backup without metadata.json, `cloud` in `list remote`) is restored via restore_cloud, only with remote_storage s3 or azblob and without mapping, --schema, --data, --rbac, --configs, --named-collections, --resume and --streaming options
https://github.com/Altinity/clickhouse-backup/issues/1574

OPTIONS:
--table string, --tables string, -t string Download and restore objects which matched with table name patterns, separated by comma, allow ? and * as wildcard
--restore-database-mapping string, -m string [ --restore-database-mapping string, -m string ] Define the rule to restore data. For the database not defined in this struct, the program will not deal with it.
Expand Down Expand Up @@ -324,13 +328,14 @@ DESCRIPTION:
Credentials and defaults are taken from the s3 config section (also works for GCS via s3->endpoint=https://storage.googleapis.com with HMAC keys), or from the azblob config section when --container / --azblob-restore-url is passed or general->remote_storage is azblob
When s3->assume_role_arn is set, the manifest is read and RESTORE ... FROM S3(..., extra_credentials(role_arn='...')) is executed with the assumed AWS IAM role, the static keys only sign the STS AssumeRole call (requires ClickHouse 25.8+);
without any static keys the STS AssumeRole call is signed by the ambient AWS identity instead: shared credentials file / IRSA / EC2-ECS instance profile for the manifest reads, and the ClickHouse server's own environment for the RESTORE statement
For an incremental backup the base backups chain is resolved from <base_backup> of each .backup inside the same bucket/container, RESTORE gets SETTINGS base_backup with the configured credentials
https://github.com/Altinity/clickhouse-backup/issues/1508

OPTIONS:
--bucket string S3 bucket with the ClickHouse Cloud backup, overrides s3->bucket from config
--region string AWS region of the bucket, overrides s3->region from config
--endpoint string Custom S3 endpoint (MinIO, etc.), overrides s3->endpoint from config
--base-prefix string S3 key prefix of the base backup, for incremental backups with use_base files
--base-prefix string Key prefix of the base backup for incremental backups, by default the chain of base backups is resolved from <base_backup> of the .backup manifest, use it when the backups were copied from another bucket/prefix
--s3-restore-url string URL passed to RESTORE ... FROM S3('...'), default https://s3.<region>.amazonaws.com/<bucket>/<prefix>
--container string AzureBlobStorage container with the ClickHouse Cloud backup, overrides azblob->container from config and switches the source to AzureBlobStorage
--azblob-restore-url string Blob endpoint passed to RESTORE ... FROM AzureBlobStorage(...), e.g. http://azurite:10000/devstoreaccount1, when it differs from azblob config section, switches the source to AzureBlobStorage
Expand Down
9 changes: 8 additions & 1 deletion ReadMe.md
Original file line number Diff line number Diff line change
Expand Up @@ -786,6 +786,8 @@ Note: this operation is asynchronous, so the API will return once the operation

Download and restore data from remote backup: `curl -s localhost:7171/backup/restore_remote/<BACKUP_NAME> -X POST | jq .`

A backup with ClickHouse Cloud / native `BACKUP ... TO S3/AzureBlobStorage` layout (`.backup` without `metadata.json`, shown as `cloud` in `/backup/list/remote`) is restored the same way as `POST /backup/restore_cloud`, see the `restore_remote` CLI command description.

- Optional boolean query argument `dry-run` or `dry_run` works the same as the `--dry-run` CLI argument (show the number of tables and the data size which would be downloaded and restored, without downloading and restoring). A dry-run request executes synchronously, is allowed even when another operation is in progress regardless of `api.allow_parallel`, and returns the report as JSON in the response body instead of the asynchronous acknowledgement.
- Optional string query argument `table` works the same as the `--table value` CLI argument.
- Optional string query argument `partitions` works the same as the `--partitions value` CLI argument.
Expand Down Expand Up @@ -1152,6 +1154,10 @@ NAME:
USAGE:
clickhouse-backup restore_remote [--schema] [--data] [-t, --tables=<db>.<table>] [-m, --restore-database-mapping=<originDB>:<targetDB>[,<...>]] [--tm, --restore-table-mapping=<originTable>:<targetTable>[,<...>]] [--partitions=<partitions_names>] [--rm, --drop] [-i, --ignore-dependencies] [--rbac] [--configs] [--named-collections] [--resumable] [--skip-empty-tables] <backup_name>

DESCRIPTION:
A remote backup with ClickHouse Cloud / native BACKUP layout (.backup without metadata.json, `cloud` in `list remote`) is restored via restore_cloud, only with remote_storage s3 or azblob and without mapping, --schema, --data, --rbac, --configs, --named-collections, --resume and --streaming options
https://github.com/Altinity/clickhouse-backup/issues/1574

OPTIONS:
--table string, --tables string, -t string Download and restore objects which matched with table name patterns, separated by comma, allow ? and * as wildcard
--restore-database-mapping string, -m string [ --restore-database-mapping string, -m string ] Define the rule to restore data. For the database not defined in this struct, the program will not deal with it.
Expand Down Expand Up @@ -1204,13 +1210,14 @@ DESCRIPTION:
Credentials and defaults are taken from the s3 config section (also works for GCS via s3->endpoint=https://storage.googleapis.com with HMAC keys), or from the azblob config section when --container / --azblob-restore-url is passed or general->remote_storage is azblob
When s3->assume_role_arn is set, the manifest is read and RESTORE ... FROM S3(..., extra_credentials(role_arn='...')) is executed with the assumed AWS IAM role, the static keys only sign the STS AssumeRole call (requires ClickHouse 25.8+);
without any static keys the STS AssumeRole call is signed by the ambient AWS identity instead: shared credentials file / IRSA / EC2-ECS instance profile for the manifest reads, and the ClickHouse server's own environment for the RESTORE statement
For an incremental backup the base backups chain is resolved from <base_backup> of each .backup inside the same bucket/container, RESTORE gets SETTINGS base_backup with the configured credentials
https://github.com/Altinity/clickhouse-backup/issues/1508

OPTIONS:
--bucket string S3 bucket with the ClickHouse Cloud backup, overrides s3->bucket from config
--region string AWS region of the bucket, overrides s3->region from config
--endpoint string Custom S3 endpoint (MinIO, etc.), overrides s3->endpoint from config
--base-prefix string S3 key prefix of the base backup, for incremental backups with use_base files
--base-prefix string Key prefix of the base backup for incremental backups, by default the chain of base backups is resolved from <base_backup> of the .backup manifest, use it when the backups were copied from another bucket/prefix
--s3-restore-url string URL passed to RESTORE ... FROM S3('...'), default https://s3.<region>.amazonaws.com/<bucket>/<prefix>
--container string AzureBlobStorage container with the ClickHouse Cloud backup, overrides azblob->container from config and switches the source to AzureBlobStorage
--azblob-restore-url string Blob endpoint passed to RESTORE ... FROM AzureBlobStorage(...), e.g. http://azurite:10000/devstoreaccount1, when it differs from azblob config section, switches the source to AzureBlobStorage
Expand Down
5 changes: 4 additions & 1 deletion cmd/clickhouse-backup/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -746,6 +746,8 @@ func newRootCommand() *cli.Command {
Name: "restore_remote",
Usage: "Download and restore",
UsageText: "clickhouse-backup restore_remote [--schema] [--data] [-t, --tables=<db>.<table>] [-m, --restore-database-mapping=<originDB>:<targetDB>[,<...>]] [--tm, --restore-table-mapping=<originTable>:<targetTable>[,<...>]] [--partitions=<partitions_names>] [--rm, --drop] [-i, --ignore-dependencies] [--rbac] [--configs] [--named-collections] [--resumable] [--skip-empty-tables] <backup_name>",
Description: "A remote backup with ClickHouse Cloud / native BACKUP layout (.backup without metadata.json, `cloud` in `list remote`) is restored via restore_cloud, only with remote_storage s3 or azblob and without mapping, --schema, --data, --rbac, --configs, --named-collections, --resume and --streaming options\n" +
" https://github.com/Altinity/clickhouse-backup/issues/1574",
Action: func(ctx context.Context, c *cli.Command) error {
b := backup.NewBackuper(config.GetConfigFromCli(c))
b.DryRun = c.Bool("dry-run")
Expand Down Expand Up @@ -909,6 +911,7 @@ func newRootCommand() *cli.Command {
" Credentials and defaults are taken from the s3 config section (also works for GCS via s3->endpoint=https://storage.googleapis.com with HMAC keys), or from the azblob config section when --container / --azblob-restore-url is passed or general->remote_storage is azblob\n" +
" When s3->assume_role_arn is set, the manifest is read and RESTORE ... FROM S3(..., extra_credentials(role_arn='...')) is executed with the assumed AWS IAM role, the static keys only sign the STS AssumeRole call (requires ClickHouse 25.8+);\n" +
" without any static keys the STS AssumeRole call is signed by the ambient AWS identity instead: shared credentials file / IRSA / EC2-ECS instance profile for the manifest reads, and the ClickHouse server's own environment for the RESTORE statement\n" +
" For an incremental backup the base backups chain is resolved from <base_backup> of each .backup inside the same bucket/container, RESTORE gets SETTINGS base_backup with the configured credentials\n" +
" https://github.com/Altinity/clickhouse-backup/issues/1508",
Action: func(ctx context.Context, c *cli.Command) error {
if c.Args().First() == "" {
Expand Down Expand Up @@ -956,7 +959,7 @@ func newRootCommand() *cli.Command {
&cli.StringFlag{
Name: "base-prefix",
Hidden: false,
Usage: "S3 key prefix of the base backup, for incremental backups with use_base files",
Usage: "Key prefix of the base backup for incremental backups, by default the chain of base backups is resolved from <base_backup> of the .backup manifest, use it when the backups were copied from another bucket/prefix",
},
&cli.StringFlag{
Name: "s3-restore-url",
Expand Down
3 changes: 3 additions & 0 deletions pkg/backup/download.go
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,9 @@ func (b *Backuper) downloadRemoteBackupInfo(ctx context.Context, backupName, tab
if !found {
return storage.Backup{}, nil, nil, errors.Errorf("'%s' is not found on remote storage", backupName)
}
if remoteBackup.DataFormat == storage.CloudBackupDataFormat {
return storage.Backup{}, nil, nil, errors.Errorf("'%s' has ClickHouse Cloud / native BACKUP layout (.backup without metadata.json), it can't be downloaded, use `restore_remote %s` or `restore_cloud`", backupName, backupName)
}
// Download file manifest for Walk-free restore (falls back gracefully if not present)
backupManifest := b.dst.DownloadManifest(ctx, backupName)

Expand Down
Loading