FIx hostname generation issue preventing distributed operations #1833
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sunsingerus
added
research required
Member
|
Thank you @dcaputo-harmoni for detailed analysis and fix. It seems to be correct. We are concerned, however, that we do not see problems you are referring to. So it might be something specific to your Kubernetes cluster network setup. The fix may also break monitoring metrics (see failed test), since labels will likely change. |
Author
|
@alex-zaitsev Thanks for the feedback! The reason I submitted this change is that the current hostname logic doesn’t line up with what Kubernetes documents as the standard for StatefulSet pods behind a headless service. The canonical pattern is I understand the concern about metrics labels breaking. To avoid disruption, we could introduce a |
Member
|
@dcaputo-harmoni , makes sense. We always use CoreDNS, but I think it resolves short names by default. Broken pod name label for monitoring is probably not a big deal. I am more concerned of an upgrade that will result in re-generation of remote_servers completely. I've just realized that the real problem is in replicated tables. The replica name is stored in (Zoo)Keeper. And when changes -- ClickHouse will loose it. So at the very least we need to keep replica macro unchanged. We need to do some more testing and decide if it safe to include in 0.25.5 or better to wait for a next major release. As a side note -- it was always confusing to me that ClickHouse reports the different host name by itself, compared to what is in remote_servers (it did not stop it from resolving is_local properly though). Thanks again. |
Author
|
@alex-zaitsev all sounds good to me, and yeah the hostname issue was a real head scratcher for me too. We are running these changes in our production environment now via a custom built image, and everything has been working well - but I'll continue to follow this PR, and just let me know if I can be of assistance in any way! |
|
I think i'm hitting the same issue. I've set up a brand new cluster and can't get ReplicatedMergeTree table working. My first clue was this log line Log: Debugging queries: So This is the remote servers config that clickhouse-operator generated. And finally manifests for chk and chi @alex-zaitsev you mention that the current host naming setup is working for you but how? Unless the problem isn't with hostnames but something else? |
|
I discovered that the operator has an option to enable fully qualified names, enabling that solved my problem. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
The ClickHouse operator generates incorrect hostnames in
remote_servers.xmlconfiguration, causing DNS resolution failures and breaking distributed operations in multi-node clustered deployments (sharded and/or replicated setups).Issue Details:
chi-db-clickhouse-db-{shard}-{replica}(e.g.,chi-db-clickhouse-db-0-0)chi-db-clickhouse-db-{shard}-{replica}-{ordinal}(e.g.,chi-db-clickhouse-db-0-0-0)This mismatch causes all nodes to show
is_local=0insystem.clusters, breaking distributed operations andON CLUSTERcommands.Root Cause
The operator was not following Kubernetes StatefulSet DNS naming conventions. StatefulSets use a specific DNS pattern:
The
createPodFQDNfunction was incorrectly usingcreatePodHostname()(service name) instead ofcreatePodName()(actual pod name with-0ordinal suffix). While the service name would work for network connectivity, ClickHouse'sis_localdetection requires the hostname inremote_servers.xmlto exactly match the pod's actual hostname for proper cluster node identification.Solution: Fixed Both Hostname Generation Functions
Modified both
createPodHostnameandcreatePodFQDNfunctions in CHI and CHK namers:1. Fixed
createPodHostname()Before (broken): Returned service name without ordinal
After (fixed): Returns actual pod name with ordinal
2. Fixed
createPodFQDN()Before (broken): Used service name in FQDN
After (fixed): Uses proper StatefulSet DNS pattern
This ensures both functions return pod names that match actual StatefulSet pod hostnames, enabling proper
is_localdetection and DNS resolution.Files Changed:
pkg/model/chi/namer/name.go- Implemented proper StatefulSet DNS pattern for CHIpkg/model/chk/namer/name.go- Implemented proper StatefulSet DNS pattern for CHKCompatibility with namespaceDomainPattern
This fix is fully compatible with the existing
namespaceDomainPatternfunctionality. When users specify a custom domain pattern like:The implementation properly handles both cases:
<pod-name>.<headless-service-name>.<namespace>.svc.cluster.local<pod-name>.<headless-service-name>.<custom-domain-pattern>The
%splaceholder in namespaceDomainPattern gets replaced with the namespace name, maintaining full backward compatibility while fixing the underlying DNS resolution issues.Impact
ON CLUSTERoperations: Distributed DDL now works in sharded configurationsis_local=0issue: All cluster nodes correctly identify themselves as localextraConfigremote_servers overridesnamespaceDomainPatternoverridesOperator Log Messages Explained
This fix resolves continuous operator log messages like:
These occur because the operator's
IsHostInCluster()function queries:When hostnames mismatch, this always returns 0 (no local node found), causing the operator to repeatedly log that hosts are "outside" the cluster even when they're functioning correctly.
Testing & Validation
Production Tested:
ON CLUSTERDDL operations on 4-node cluster (2 shards, 2 replicas each)namespaceDomainPatterncompatibilityTechnical Details: StatefulSet DNS Pattern Implementation
The fix implements the standard Kubernetes StatefulSet DNS pattern by ensuring FQDNs follow:
Key Components:
chi-clickhouse-clickhouse-0-0-0(includes-0ordinal)chi-clickhouse-clickhouse-0-0(StatefulSet service)<namespace>.svc.cluster.local(or custom vianamespaceDomainPattern)This ensures proper DNS resolution for StatefulSet pods while maintaining compatibility with all existing cluster configurations and custom domain patterns.
Verification
After this fix, users will no longer need manual
extraConfigoverrides. The operator automatically generates correct hostnames inremote_servers.xmlthat match actual StatefulSet pod names and DNS patterns.Example of corrected hostname generation:
chi-db-clickhouse-db-0-0.namespace.svc.cluster.localchi-db-clickhouse-db-0-0-0.chi-db-clickhouse-db-0-0.namespace.svc.cluster.localThis change ensures ClickHouse can properly identify local replicas and enables all distributed operations to work correctly out of the box with proper StatefulSet DNS resolution.