Skip to content

feat: change kafka to redpanda #1771

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: develop
Choose a base branch
from
Draft

feat: change kafka to redpanda #1771

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1afa22f
feat: change kafka to redpanda
TartanLeGrand May 14, 2025
3721a22
feat: update configuration to support Redpanda alongside Kafka
TartanLeGrand May 15, 2025
ef19a4d
feat: update Redpanda configuration and adjust Kafka fallback settings
TartanLeGrand May 26, 2025
7314b28
feat: update Redpanda configuration and listeners settings
TartanLeGrand May 27, 2025
073ffeb
feat: add Redpanda configuration for auto topic creation and disable …
TartanLeGrand May 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions charts/sentry/Chart.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ dependencies:
- name: kafka
repository: oci://registry-1.docker.io/bitnamicharts
version: 29.3.14
- name: redpanda
repository: https://charts.redpanda.com
version: 5.9.23
- name: clickhouse
repository: https://sentry-kubernetes.github.io/charts
version: 3.14.1
Expand All @@ -23,5 +26,5 @@ dependencies:
- name: nginx
repository: oci://registry-1.docker.io/bitnamicharts
version: 18.2.5
digest: sha256:2b19e9605468921ff96afb9f393ffe09d3121e5b91f32789e025851f0b66ff63
generated: "2025-01-17T18:17:43.022337376+06:00"
digest: sha256:b1ffac97e8807f10c6632d050164355cb18c132f484f1ec9ee190e718dbd84be
generated: "2025-05-27T23:04:20.600779955+02:00"
4 changes: 4 additions & 0 deletions charts/sentry/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ dependencies:
repository: oci://registry-1.docker.io/bitnamicharts
version: 29.3.14
condition: kafka.enabled
- name: redpanda
repository: https://charts.redpanda.com
version: 5.9.23
condition: redpanda.enabled
- name: clickhouse
repository: https://sentry-kubernetes.github.io/charts
version: 3.14.1
Expand Down
44 changes: 33 additions & 11 deletions charts/sentry/templates/_helper.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,11 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this
{{- end -}}

{{- define "sentry.kafka.fullname" -}}
{{- if .Values.kafka.enabled -}}
{{- printf "%s-%s" .Release.Name "kafka" | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name "redpanda" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}

{{- define "sentry.zookeeper.fullname" -}}
Expand Down Expand Up @@ -443,7 +447,7 @@ True
Set Kafka Confluent host
*/}}
{{- define "sentry.kafka.host" -}}
{{- if .Values.kafka.enabled -}}
{{- if or (.Values.kafka.enabled) (.Values.redpanda.enabled) -}}
{{- template "sentry.kafka.fullname" . -}}
{{- else if and (.Values.externalKafka) (not (.Values.externalKafka.cluster)) -}}
{{ required "A valid .Values.externalKafka.host is required" .Values.externalKafka.host }}
Expand All @@ -454,7 +458,9 @@ Set Kafka Confluent host
Set Kafka Confluent port
*/}}
{{- define "sentry.kafka.port" -}}
{{- if and (.Values.kafka.enabled) (.Values.kafka.service.ports.client) -}}
{{- if .Values.redpanda.enabled -}}
{{- .Values.redpanda.listeners.kafka.port }}
{{- else if and .Values.kafka.enabled .Values.kafka.service.ports.client -}}
{{- .Values.kafka.service.ports.client }}
{{- else if and (.Values.externalKafka) (not (.Values.externalKafka.cluster)) -}}
{{ required "A valid .Values.externalKafka.port is required" .Values.externalKafka.port }}
Expand All @@ -465,7 +471,9 @@ Set Kafka Confluent port
Set Kafka Confluent Controller port
*/}}
{{- define "sentry.kafka.controller_port" -}}
{{- if and (.Values.kafka.enabled) (.Values.kafka.service.ports.controller ) -}}
{{- if .Values.redpanda.enabled -}}
{{- default 9092 (dig "kafka" "port" nil .Values.redpanda) }}
{{- else if and (.Values.kafka.enabled) (.Values.kafka.service.ports.controller ) -}}
{{- .Values.kafka.service.ports.controller }}
{{- else if and (.Values.externalKafka) (not (.Values.externalKafka.cluster)) -}}
{{ required "A valid .Values.externalKafka.port is required" .Values.externalKafka.port }}
Expand All @@ -476,7 +484,7 @@ Set Kafka Confluent Controller port
Set Kafka bootstrap servers string
*/}}
{{- define "sentry.kafka.bootstrap_servers_string" -}}
{{- if or (.Values.kafka.enabled) (not (.Values.externalKafka.cluster)) -}}
{{- if or (.Values.kafka.enabled) (.Values.redpanda.enabled) (not (.Values.externalKafka.cluster)) -}}
{{ printf "%s:%s" (include "sentry.kafka.host" .) (include "sentry.kafka.port" .) }}
{{- else -}}
{{- range $index, $elem := .Values.externalKafka.cluster -}}
Expand All @@ -496,7 +504,9 @@ SASL auth setings for Kafka:
Set Kafka security protocol
*/}}
{{- define "sentry.kafka.security_protocol" -}}
{{- if .Values.kafka.enabled -}}
{{- if .Values.redpanda.enabled -}}
{{ default "plaintext" .Values.redpanda.listeners.kafka.authenticationMethod }}
{{- else if .Values.kafka.enabled -}}
{{ default "plaintext" .Values.kafka.listeners.client.protocol }}
{{- else -}}
{{ default "plaintext" .Values.externalKafka.security.protocol }}
Expand All @@ -509,7 +519,9 @@ Set Kafka sasl mechanism
{{- define "sentry.kafka.sasl_mechanism" -}}
{{- $CheckProtocol := include "sentry.kafka.security_protocol" . -}}
{{- if (regexMatch "^SASL_" $CheckProtocol) -}}
{{- if .Values.kafka.enabled -}}
{{- if .Values.redpanda.enabled -}}
{{ default "None" .Values.redpanda.auth.sasl.mechanism }}
{{- else if .Values.kafka.enabled -}}
{{ default "None" (split "," .Values.kafka.sasl.enabledMechanisms)._0 }}
{{- else -}}
{{ default "None" .Values.externalKafka.sasl.mechanism }}
Expand All @@ -525,7 +537,9 @@ Set Kafka sasl username
{{- define "sentry.kafka.sasl_username" -}}
{{- $CheckProtocol := include "sentry.kafka.security_protocol" . -}}
{{- if (regexMatch "^SASL_" $CheckProtocol) -}}
{{- if .Values.kafka.enabled -}}
{{- if .Values.redpanda.enabled -}}
{{ default "None" (first (default tuple .Values.redpanda.auth.sasl.users).name) }}
{{- else if .Values.kafka.enabled -}}
{{ default "None" (first (default tuple .Values.kafka.sasl.client.users)) }}
{{- else -}}
{{ default "None" .Values.externalKafka.sasl.username }}
Expand All @@ -541,7 +555,9 @@ Set Kafka sasl password
{{- define "sentry.kafka.sasl_password" -}}
{{- $CheckProtocol := include "sentry.kafka.security_protocol" . -}}
{{- if (regexMatch "^SASL_" $CheckProtocol) -}}
{{- if .Values.kafka.enabled -}}
{{- if .Values.redpanda.enabled -}}
{{ default "None" (first (default tuple .Values.redpanda.auth.sasl.users).password) }}
{{- else if .Values.kafka.enabled -}}
{{ default "None" (first (default tuple .Values.kafka.sasl.client.passwords)) }}
{{- else -}}
{{ default "None" .Values.externalKafka.sasl.password }}
Expand All @@ -555,7 +571,9 @@ Set Kafka sasl password
Set Senty compression.type for Kafka
*/}}
{{- define "sentry.kafka.compression_type" -}}
{{- if .Values.kafka.enabled -}}
{{- if .Values.redpanda.enabled -}}
{{ default "" .Values.sentry.kafka.compression.type }}
{{- else if .Values.kafka.enabled -}}
{{ default "" .Values.sentry.kafka.compression.type }}
{{- else -}}
{{ default "" .Values.externalKafka.compression.type }}
Expand All @@ -566,7 +584,9 @@ Set Senty compression.type for Kafka
Set Senty message.max.bytes for Kafka
*/}}
{{- define "sentry.kafka.message_max_bytes" -}}
{{- if .Values.kafka.enabled -}}
{{- if .Values.redpanda.enabled -}}
{{ default 50000000 .Values.sentry.kafka.message.max.bytes | int64 }}
{{- else if .Values.kafka.enabled -}}
{{ default 50000000 .Values.sentry.kafka.message.max.bytes | int64 }}
{{- else -}}
{{ default 50000000 .Values.externalKafka.message.max.bytes | int64 }}
Expand All @@ -577,7 +597,9 @@ Set Senty message.max.bytes for Kafka
Set Senty socket.timeout for Kafka
*/}}
{{- define "sentry.kafka.socket_timeout_ms" -}}
{{- if .Values.kafka.enabled -}}
{{- if .Values.redpanda.enabled -}}
{{ default 1000 .Values.sentry.kafka.socket.timeout.ms | int64 }}
{{- else if .Values.kafka.enabled -}}
{{ default 1000 .Values.sentry.kafka.socket.timeout.ms | int64 }}
{{- else -}}
{{ default 1000 .Values.externalKafka.socket.timeout.ms | int64 }}
Expand Down
4 changes: 2 additions & 2 deletions charts/sentry/templates/hooks/sentry-db-check.job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ spec:
done
echo "Zookeeper is up"
{{- end }}
{{- if .Values.kafka.kraft.enabled }}
{{- if or (.Values.kafka.kraft.enabled) (.Values.kafka.redpanda.enabled) }}
KAFKA_REPLICAS={{ .Values.kafka.controller.replicaCount | default 3 }}
echo "Kafka Kraft is enabled, checking if Kraft controllers are up"
KRAFT_STATUS=0
Expand All @@ -162,7 +162,7 @@ spec:
done
echo "Kraft controllers are up"
{{- end }}
{{- else if (not (.Values.externalKafka.cluster)) }}
{{- else if and (not (.Values.externalKafka.cluster)) (not (.Values.redpanda.enabled)) }}
KAFKA_HOST={{ .Values.externalKafka.host }}
if ! nc -z "$KAFKA_HOST" {{ $kafkaPort }}; then
KAFKA_STATUS=0
Expand Down
22 changes: 21 additions & 1 deletion charts/sentry/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2196,10 +2196,30 @@ zookeeper:
# persistence:
# size: 8Gi

redpanda:
enabled: true
fullnameOverride: "sentry-redpanda"
config:
cluster:
auto_create_topics_enabled: true
statefulset:
replicas: 1
listeners:
kafka:
port: 9092
tls:
enabled: false
authenticationMethod: "plaintext"
console:
enabled: false
auth:
sasl:
enabled: false
Comment on lines +2215 to +2217
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
auth:
sasl:
enabled: false
auth:
sasl:
enabled: false
external:
enabled: false
logging:
usageStats:
enabled: false

I suggest disabling external access as I don't think it is necessary and is best left disabled as a security measure.

Also, usage stats sent to Redpanda are enabled by default.


# Settings for Kafka.
# See https://github.com/bitnami/charts/tree/master/bitnami/kafka
kafka:
enabled: true
enabled: false
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi!
I think the default broker should be set as kafka, changing it would cause every user to lost all the data stored in kafka.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it will be that much of a big deal, one could lose some unprocessed events in the transition but isn't that's something to be excepted when doing an upgrade anyways?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I make a guide for upgrade.

provisioning:
## Increasing the replicationFactor enhances data reliability during Kafka pod failures by replicating data across multiple brokers.
# Note that existing topics will remain with replicationFactor: 1 when updated.
Expand Down
Loading