Skip to content

download of a regular backup fails with "Chown backup disk" when embedded_backup_disk is configured #1653

Closed

Description

@louisantoinem

Version: clickhouse-backup 2.8.2, ClickHouse 26.8.2.7

Config

clickhouse:
  use_embedded_backup_restore: true
  embedded_backup_disk: backups_s3   # disk of type s3 declared in the server config

Steps

  1. Create backups with use_embedded_backup_restore: false (regular backups, tag regular in list remote).
  2. Switch the config to use_embedded_backup_restore: true with an embedded_backup_disk.
  3. clickhouse-backup download <regular backup>

Expected
The download works: the mode is picked from the tags of the backup (b.isEmbedded = strings.Contains(remoteBackup.Tags, "embedded")), so a regular backup should be downloaded as before.

Actual
The data is downloaded, then the command fails:

FTL cmd/clickhouse-backup/main.go:1414 > , error=Chown backup disk: lstat /var/lib/clickhouse/disks/backups_s3/full_monitoring_monitoring_20261025: no such file or directory
stack:
github.com/Altinity/clickhouse-backup/v2/pkg/backup.(*Backuper).downloadEpilogue()
    github.com/Altinity/clickhouse-backup/v2/pkg/backup/download.go:430
github.com/Altinity/clickhouse-backup/v2/pkg/backup.(*Backuper).Download()

Cause
In downloadEpilogue (pkg/backup/download.go), the loop that chowns <backup disk path>/<backup name> for every disk with IsBackup runs whatever the kind of the backup.
For a regular backup nothing is written on the backup disk, so the directory does not exist and Chown fails.
The block just above, which downloads the .backup file, is guarded by b.isEmbedded && b.cfg.ClickHouse.EmbeddedBackupDisk != "": the chown loop probably needs the same guard.

Workaround
Download and restore a regular backup with CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE=false CLICKHOUSE_EMBEDDED_BACKUP_DISK="".

Activity

  1. added this to the 2.8.3 milestone on Oct 29, 2026
  2. Slach commented on Oct 29, 2026

    @Slach
    Collaborator

    could you share
    SELECT * FROM system.disk ?

  3. louisantoinem commented on Oct 29, 2026

    @louisantoinem
    Author

    Sure, here it is (ClickHouse 26.8.2.7, clickhouse-backup 2.8.2):

       ┌─name───────┬─path──────────────────────────────────┬─type──────────┬─object_storage_type─┬─metadata_type─┬─is_remote─┬─is_read_only─┐
    1. │ backups_s3 │ /var/lib/clickhouse/disks/backups_s3/ │ ObjectStorage │ S3                  │ Local         │         1 │            0 │
    2. │ default    │ /var/lib/clickhouse/                  │ Local         │ None                │ None          │         0 │            0 │
       └────────────┴───────────────────────────────────────┴───────────────┴─────────────────────┴───────────────┴───────────┴──────────────┘
    

    backups_s3 is declared in config.d as a disk of type s3, in the same bucket as the remote storage of clickhouse-backup.

    Thanks for #1654, it matches what we see: with the guard on b.isEmbedded, our workaround for regular backups (CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE=false CLICKHOUSE_EMBEDDED_BACKUP_DISK="") should no longer be needed.

  4. added a commit that references this issue on Oct 30, 2026
  5. added a commit that references this issue on Oct 30, 2026
    6183c58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions