Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 12 additions & 84 deletions addons/clickhouse/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ There are two key components in the ClickHouse cluster:

| Major Versions | Description |
| -------------- | ------------------------- |
| 22 | 22.3.18, 22.3.20, 22.8.21 |
| 22 | 22.3.18, 22.3.20, 22.6.1, 22.8.21, 22.9.4 |
| 24 | 24.7.2, 24.8.3, 24.12.2 |
| 25 | 25.4.4, 25.9.7 |

Expand Down Expand Up @@ -802,58 +802,9 @@ kubectl apply -f examples/clickhouse/verticalscale.yaml

### Switchover for Clickhouse Keeper

#### Switchover without preferred candidates
#### Switchover with specified candidate

Switchover a specified instance as the new primary or leader of the cluster

```yaml
# cat examples/clickhouse/keeper-switchover.yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
name: keeper-switchover
namespace: demo
spec:
# Specifies the name of the Cluster resource that this operation is targeting.
clusterName: clickhouse-cluster
type: Switchover
# Lists Switchover objects, each specifying a Component to perform the switchover operation.
switchover:
# Specifies the name of the Component.
- componentName: ch-keeper
# Specifies the instance whose role will be transferred.
# A typical usage is to transfer the leader role in a consensus system.
instanceName: "clickhouse-cluster-ch-keeper-0"

```

```yaml
# cat examples/clickhouse/keeper-switchover.yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
name: keeper-switchover
namespace: demo
spec:
# Specifies the name of the Cluster resource that this operation is targeting.
clusterName: clickhouse-cluster
type: Switchover
# Lists Switchover objects, each specifying a Component to perform the switchover operation.
switchover:
# Specifies the name of the Component.
- componentName: ch-keeper
# Specifies the instance whose role will be transferred.
# A typical usage is to transfer the leader role in a consensus system.
instanceName: "clickhouse-cluster-ch-keeper-0"
```

```bash
kubectl apply -f examples/clickhouse/keeper-switchover.yaml
```

#### Switchover-specified-instance

Switchover a specified instance as the new primary or leader of the cluster
Switchover the current leader to a selected candidate instance. ClickHouse Keeper switchover currently requires `candidateName`.

```yaml
# cat examples/clickhouse/keeper-switchover-specified-instance.yaml
Expand All @@ -873,40 +824,17 @@ spec:
# Specifies the instance whose role will be transferred.
# A typical usage is to transfer the leader role in a consensus system.
instanceName: "clickhouse-cluster-ch-keeper-0"
# Specifies the instance that will become the new leader, if not specify, the first non leader instance will become candidate.
# Need to ensure the candidate instance is catch up logs of the quorum, otherwise the switchover will transfer the leader to other instance.
# ClickHouse Keeper switchover currently requires candidateName.
# Need to ensure the candidate instance has caught up with quorum logs.
candidateName: "clickhouse-cluster-ch-keeper-1"

```

```yaml
# cat examples/clickhouse/keeper-switchover-specified-instance.yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
name: keeper-switchover
namespace: demo
spec:
# Specifies the name of the Cluster resource that this operation is targeting.
clusterName: clickhouse-cluster
type: Switchover
# Lists Switchover objects, each specifying a Component to perform the switchover operation.
switchover:
# Specifies the name of the Component.
- componentName: ch-keeper
# Specifies the instance whose role will be transferred.
# A typical usage is to transfer the leader role in a consensus system.
instanceName: "clickhouse-cluster-ch-keeper-0"
# Specifies the instance that will become the new leader, if not specify, the first non leader instance will become candidate.
# Need to ensure the candidate instance is catch up logs of the quorum, otherwise the switchover will transfer the leader to other instance.
candidateName: "clickhouse-cluster-ch-keeper-1"
```

```bash
kubectl apply -f examples/clickhouse/keeper-switchover-specified-instance.yaml
```

You may need to update the `opsrequest.spec.switchover.instanceName` field to your desired instance name.
You may need to update the `opsrequest.spec.switchover.instanceName` and `opsrequest.spec.switchover.candidateName` fields to your desired instance names.

### Expand volume

Expand Down Expand Up @@ -1101,7 +1029,7 @@ spec:
configMap:
name: custom-user-configuration-tpl # refers to the configmap with your customized configuration info.
variables:
max_threads: "16"
udf_max_threads: "16"
systemAccounts:
- name: admin
secretRef:
Expand Down Expand Up @@ -1479,9 +1407,9 @@ The incremental backup's `parentBackupName` must refer to an existing backup cre
#### Restore Settings

> [!NOTE]
> Restoring a TLS-enabled cluster directly from backup is NOT supported. You should restore the cluster with TLS disabled first, and then enable TLS manually after the restore process is complete.
> When the restore target has TLS enabled, the restore job mounts the ClickHouse TLS files and connects through the secure TCP port.

Restore process will restore schema and rbac first, then restore data. You can tune schema-ready waiting behavior for restore jobs via Helm values:
Restore process will restore schema and RBAC metadata first, then restore data. When the restore target has TLS enabled, the restore job uses the secure ClickHouse TCP port and Keeper TLS settings for RBAC metadata. You can tune schema-ready waiting behavior for restore jobs via Helm values:

```yaml
restore:
Expand Down Expand Up @@ -1531,7 +1459,7 @@ spec:
storage: 10Gi
shardings:
- name: clickhouse
shards: 3
shards: 2
template:
name: clickhouse
componentDef: clickhouse-1
Expand Down Expand Up @@ -1833,10 +1761,10 @@ spec:
services:
# Exposes ClickHouse service using NodePort
- name: clickhouse-nodeport
# Sharding associated with this service
componentSelector: clickhouse
# Type of the exposed service. Valid options are 'ClusterIP', 'NodePort', and 'LoadBalancer'.
serviceType: NodePort
# Sharding associated with this service
shardingSelector: clickhouse
# Role selector for the component service.
# In a ClickHouse cluster, we only expose nodes without specifying roles
# since all nodes can handle read/write requests
Expand Down
7 changes: 4 additions & 3 deletions addons/clickhouse/configs/00_default_overrides.xml.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -106,10 +106,9 @@
<server>
<certificateFile>{{$CERT_FILE}}</certificateFile>
<privateKeyFile>{{$KEY_FILE}}</privateKeyFile>
<!--
Use relaxed verification for ClickHouse to skip hostname check, while still supporting TLS encryption.
-->
<verificationMode>relaxed</verificationMode>
<!-- Disable peer hostname matching; certificate validation is controlled by verificationMode. -->
<extendedVerification>false</extendedVerification>
<caConfig>{{$CA_FILE}}</caConfig>
<cacheSessions>true</cacheSessions>
<disableProtocols>sslv2,sslv3</disableProtocols>
Expand All @@ -124,6 +123,8 @@
<disableProtocols>sslv2,sslv3</disableProtocols>
<preferServerCiphers>true</preferServerCiphers>
<verificationMode>strict</verificationMode>
<!-- Disable peer hostname matching; certificate validation remains strict. -->
<extendedVerification>false</extendedVerification>
<invalidCertificateHandler>
<name>RejectCertificateHandler</name>
</invalidCertificateHandler>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,9 @@
<server>
<certificateFile>{{$CERT_FILE}}</certificateFile>
<privateKeyFile>{{$KEY_FILE}}</privateKeyFile>
<!--
Use relaxed verification for Keeper to skip hostname check, while still supporting TLS encryption.
-->
<verificationMode>relaxed</verificationMode>
<!-- Disable peer hostname matching; certificate validation is controlled by verificationMode. -->
<extendedVerification>false</extendedVerification>
<caConfig>{{$CA_FILE}}</caConfig>
<cacheSessions>true</cacheSessions>
<disableProtocols>sslv2,sslv3</disableProtocols>
Expand All @@ -103,6 +102,8 @@
<disableProtocols>sslv2,sslv3</disableProtocols>
<preferServerCiphers>true</preferServerCiphers>
<verificationMode>strict</verificationMode>
<!-- Disable peer hostname matching; certificate validation remains strict. -->
<extendedVerification>false</extendedVerification>
<invalidCertificateHandler>
<name>RejectCertificateHandler</name>
</invalidCertificateHandler>
Expand Down
56 changes: 56 additions & 0 deletions addons/clickhouse/dataprotection/common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,9 @@ function init_datasafed() {
function clickhouse_query() {
local ch_port="${CLICKHOUSE_PORT:-9000}"
local ch_args=(--user "${CLICKHOUSE_USERNAME}" --password "${CLICKHOUSE_PASSWORD}" --host "${CLICKHOUSE_HOST}" --port "$ch_port" --connect_timeout=5)
if [[ "${CLICKHOUSE_SECURE:-false}" == "true" ]]; then
ch_args+=(--secure --accept-invalid-certificate)
fi
clickhouse-client "${ch_args[@]}" --query "$1"
}

Expand Down Expand Up @@ -468,6 +471,9 @@ function ch_query() {
local query="$1"
local ch_port="${CLICKHOUSE_PORT:-9000}"
local ch_args=(--user "${CLICKHOUSE_USERNAME}" --password "${CLICKHOUSE_PASSWORD}" --host "${CLICKHOUSE_HOST}" --port "$ch_port" --connect_timeout=5)
if [[ "${CLICKHOUSE_SECURE:-false}" == "true" ]]; then
ch_args+=(--secure --accept-invalid-certificate)
fi
clickhouse-client "${ch_args[@]}" --query "$query"
}

Expand Down Expand Up @@ -499,6 +505,37 @@ function fetch_backup() {
DP_log "Backup '$backup_name' is available locally."
}

function stage_required_backup_metadata() {
local target_base_path="$1"
shift
local backup_name
local metadata_file

# clickhouse-backup expects incremental parents under the same S3_PATH.
# KubeBlocks stores each Backup under its own path, so stage parent metadata only.
export PATH="$PATH:$DP_DATASAFED_BIN_PATH"
for backup_name in "$@"; do
[[ -z "$backup_name" ]] && continue
metadata_file=$(mktemp) || {
DP_error_log "Failed to create temporary metadata file"
return 1
}
export DATASAFED_BACKEND_BASE_PATH="$DP_BACKUP_ROOT_PATH/$backup_name/$DP_TARGET_RELATIVE_PATH"
if ! datasafed pull "$backup_name/metadata.json" "$metadata_file"; then
rm -f "$metadata_file"
DP_error_log "Failed to pull metadata for backup '$backup_name'"
return 1
fi
export DATASAFED_BACKEND_BASE_PATH="$target_base_path"
if ! datasafed push "$metadata_file" "$backup_name/metadata.json"; then
rm -f "$metadata_file"
DP_error_log "Failed to stage metadata for backup '$backup_name'"
return 1
fi
rm -f "$metadata_file"
done
}

function delete_backups_except() {
local latest_backup=$1
DP_log "delete backup except $latest_backup"
Expand All @@ -523,6 +560,25 @@ function save_backup_size() {
DP_save_backup_status_info "$backup_size"
}

function detect_restore_mode() {
local first_entry="${ALL_COMBINED_SHARDS_POD_FQDN_LIST%%,*}"
local first_component="${first_entry%%:*}"
if [[ -z "$first_component" ]]; then
DP_error_log "Invalid ALL_COMBINED_SHARDS_POD_FQDN_LIST" >&2
return 1
fi

if [[ -z "${CH_KEEPER_POD_FQDN_LIST:-}" ]]; then
echo "standalone"
return 0
fi

if [[ "$first_component" == "$first_entry" ]]; then
first_component="${CURRENT_SHARD_COMPONENT_SHORT_NAME}"
fi
echo "cluster:$first_component"
}

# Restore schema and wait for sync across shards
function restore_schema_and_sync() {
local backup_name="$1"
Expand Down
25 changes: 7 additions & 18 deletions addons/clickhouse/dataprotection/incremental-restore.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,36 +8,25 @@ trap handle_exit EXIT
generate_backup_config
set_clickhouse_backup_config_env

if [[ "${CLICKHOUSE_SECURE}" = "true" ]]; then
DP_error_log "ClickHouse restore does not support TLS"
exit 1
fi

# 1. Download full (base) backup
export S3_PATH="$DP_BACKUP_ROOT_PATH/$DP_BASE_BACKUP_NAME/$DP_TARGET_RELATIVE_PATH"
fetch_backup "$DP_BASE_BACKUP_NAME"
downloaded_backups=("$DP_BASE_BACKUP_NAME")

# 2. Download ancestor incremental backups
IFS=',' read -r -a ancestors <<<"${DP_ANCESTOR_INCREMENTAL_BACKUP_NAMES}"
for parent_name in "${ancestors[@]}"; do
[[ -z "$parent_name" ]] && continue
export S3_PATH="$DP_BACKUP_ROOT_PATH/$parent_name/$DP_TARGET_RELATIVE_PATH"
stage_required_backup_metadata "$S3_PATH" "${downloaded_backups[@]}" || exit 1
fetch_backup "$parent_name"
downloaded_backups+=("$parent_name")
done

# 3. Detect topology mode: standalone (no ':' in FQDN) or cluster
# 3. Detect topology mode: standalone or cluster
export S3_PATH="${DP_BACKUP_BASE_PATH}"
first_entry="${ALL_COMBINED_SHARDS_POD_FQDN_LIST%%,*}"
first_component="${first_entry%%:*}"
if [[ -z "$first_component" ]]; then
DP_error_log "Invalid ALL_COMBINED_SHARDS_POD_FQDN_LIST"
exit 1
fi
if [[ "$first_component" == "$first_entry" ]]; then
mode_info="standalone"
DP_log "Standalone mode detected"
else
mode_info="cluster:$first_component"
fi
stage_required_backup_metadata "$S3_PATH" "${downloaded_backups[@]}" || exit 1
mode_info=$(detect_restore_mode) || exit 1

# 4. Restore schema + data + marker
do_restore "${DP_BACKUP_NAME}" "$mode_info" || exit 1
Expand Down
20 changes: 2 additions & 18 deletions addons/clickhouse/dataprotection/restore.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,24 +8,8 @@ trap handle_exit EXIT
generate_backup_config
set_clickhouse_backup_config_env

if [[ "${CLICKHOUSE_SECURE}" = "true" ]]; then
DP_error_log "ClickHouse restore does not support TLS"
exit 1
fi

# 1. Detect topology mode: standalone (no ':' in FQDN) or cluster
first_entry="${ALL_COMBINED_SHARDS_POD_FQDN_LIST%%,*}"
first_component="${first_entry%%:*}"
if [[ -z "$first_component" ]]; then
DP_error_log "Invalid ALL_COMBINED_SHARDS_POD_FQDN_LIST"
exit 1
fi
if [[ "$first_component" == "$first_entry" ]]; then
mode_info="standalone"
DP_log "Standalone mode detected"
else
mode_info="cluster:$first_component"
fi
# 1. Detect topology mode: standalone or cluster
mode_info=$(detect_restore_mode) || exit 1

# 2. Restore schema + data + marker
do_restore "${DP_BACKUP_NAME}" "$mode_info" || exit 1
Expand Down
25 changes: 25 additions & 0 deletions addons/clickhouse/scripts/bootstrap-keeper.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
#!/bin/bash
set -euo pipefail

function chown_data_dir() {
data_dir=/bitnami/clickhouse/data
if [ ! -d "${data_dir}" ]; then
Expand All @@ -10,7 +12,30 @@ function chown_data_dir() {
chown -R root:root ${data_dir}
fi
}

function wait_peer_dns_ready() {
local peers="${CH_KEEPER_POD_FQDN_LIST:-}"
local attempts=30
local sleep_seconds=2

[[ -z "$peers" ]] && return 0

for peer in ${peers//,/ }; do
for attempt in $(seq 1 $attempts); do
if getent hosts "$peer" >/dev/null 2>&1; then
break
fi
if [[ $attempt -eq $attempts ]]; then
echo "$(date) ERROR: peer DNS $peer is still unresolved after $((attempts * sleep_seconds))s" >&2
return 1
fi
sleep "$sleep_seconds"
done
done
}

chown_data_dir
wait_peer_dns_ready
HOSTNAME="$(hostname -s)"
if grep -q "<id>0</id>" /opt/bitnami/clickhouse/etc/conf.d/ch-keeper_00_default_overrides.xml; then
# compatible old version
Expand Down
Loading
Loading