Sentry helm externalKafka SSL config

[kalan89]

Hi Sentry Team,

When trying to configure the sentry fresh installation with externalKafka SSL.
Getting below error and no options exist to enable SSL Cert/key on helm values.yaml.

Values.yaml:

# Disable the in-cluster Kafka
kafka:
  enabled: false

# Configure external Kafka
externalKafka:
  host: data-test1.example.com
  port: 32500
  enabled: true
  security:
    protocol: SSL

Error:

%6|1736793971.761|FAIL|rdkafka#producer-46| [thrd:data-test1.example.com:32500/bootstrap]: data-test1.example.com:32500/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 36ms in state APIVERSION_QUERY, 1 identical error(s) suppressed)

Please share if we can enable cert and key on helm values.yaml or other workaround.

[kalan89]

When using the backend node as externalKafaka host, getting below error.

Let me know to kafka SSL location ssl.ca.location in the chart values.yaml file.

cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2025-01-16 16:58:16,630 Attempting to connect to Kafka (attempt 24)...
%3|1737046696.813|FAIL|rdkafka#producer-25| [thrd:ssl://kafka.example.com:9093/bootstrap]: ssl://kafka.example.com:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1890:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 85ms in state SSL_HANDSHAKE)
%3|1737046696.998|FAIL|rdkafka#producer-25| [thrd:ssl://kafka.example.com:9093/bootstrap]: ssl://kafka.example.com:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1890:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 94ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)
2025-01-16 16:58:17,647 Connection to Kafka failed (attempt 24)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 72, in bootstrap

[patsevanton]

my config:

    externalKafka:
      cluster:
        - host: rc1a-xxx.mdb.cloud.net.
          port: 9092
        - host: rc1b-xxx.mdb.cloud.net.
          port: 9092
        - host: rc1d-xxx.mdb.cloud.net.
          port: 9092
      sasl:
        mechanism: SCRAM-SHA-512
        username: sentry
      security:
        protocol: SASL_PLAINTEXT

[kalan89]

my config:

    externalKafka:
      cluster:
        - host: rc1a-xxx.mdb.cloud.net.
          port: 9092
        - host: rc1b-xxx.mdb.cloud.net.
          port: 9092
        - host: rc1d-xxx.mdb.cloud.net.
          port: 9092
      sasl:
        mechanism: SCRAM-SHA-512
        username: sentry
      security:
        protocol: SASL_PLAINTEXT

Thanks for your response.
I am trying to use kafa SSL with cert and key.
It looks like you are using SASL.

[Cheunn-Panaa]

You can use a sketchy workaround, as there is no other way you can set this up in the externalKafka part of the values.yml file.

config:
  configYml: {}
  sentryConfPy: |
      KAFKA_CLUSTERS["default"] = {
      "common": {
          # See https://github.com/getsentry/arroyo/blob/main/arroyo/backends/kafka/configuration.py#L16-L38 for the supported options
          "bootstrap.servers": "$KAFKA_HOST:$KAFKA_PORT",
          "ssl.ca.location": "/usr/local/share/ca-certificates/cert.pem",
          # If you need cert + key for users
          # "ssl.certificate.location": os.environ.get("KAFKA_SSL_CERT_PATH", ""),
          # "ssl.key.location": os.environ.get("KAFKA_SSL_KEY_PATH", ""),
          "security.protocol": "SASL_SSL",
          "sasl.mechanism": "PLAIN",
          "sasl.username": "$KAFKA_USER_NAME",
          "sasl.password": "$KAFKA_USER_OPS",
        }
      }
  snubaSettingsPy: |
    BROKER_CONFIG: Mapping[str, Any] = {
      # See https://github.com/getsentry/arroyo/blob/main/arroyo/backends/kafka/configuration.py#L16-L38 for the supported options
          "bootstrap.servers": "$KAFKA_HOST:$KAFKA_PORT",
          "ssl.ca.location": "/usr/local/share/ca-certificates/cert.pem",
          # If you need cert + key for users
          # "ssl.certificate.location": os.environ.get("KAFKA_SSL_CERT_PATH", ""),
          # "ssl.key.location": os.environ.get("KAFKA_SSL_KEY_PATH", ""),
          "security.protocol": "SASL_SSL",
          "sasl.mechanism": "PLAIN",
          "sasl.username": "$KAFKA_USER_NAME",
          "sasl.password": "$KAFKA_USER_OPS",
    }

I had to dig up in the snuba and sentry configuration to find what i needed. (also in the helm helpers)

This configuration works for me but i only hit a single server, you might be able to adjust it with some more digging sentry_helper.

For the maintainers maybe adding something like (sadly i dont have the time to contribute)

kafka:
  ssl:
    ca:
      secret: my-ca-cert-secret
      path: '/usr/local/share/cert.pem'

      {{- $sentryKafkaSslCaPath := include "sentry.kafka.ssl.ca.path" .}}
      {{- if $sentryKafkaSslCaPath) }}
          "ssl.ca.location": {{ $sentryKafkaSslCaPath | quote}},
      {{- end }}

        {{- $sentryKafkaSslCaPath := include "sentry.kafka.ssl.ca.path" .}}
        {{- if $sentryKafkaSslCaPath) }}
        - name: my-ca-cert-secret
          mountPath: $sentryKafkaSslCaPath
        {{ end }}