Document how to update / rotate erlang cookie on a running rabbitmq cluster

[YuryHrytsuk]

Is your feature request related to a problem? Please describe.

I have a RabbitMQ Cluster that consists for 3 Nodes. I want to know what procedure should I follow to update erlang cookie (e.g. update erlang cookie file + restart node one by one)

Describe the solution you'd like

Clear documentation on how to update erlang cookie on a running cluster in PROD

Describe alternatives you've considered

Additional context

Related Discussion that did not get much attention

• How do I rotate the erlang cookie on a production cluster? #9784

---

I do not ask how to automate it with docker swarm. I only want to understand the procedure (even if done manually)

My docker compose file for Docker Swarm

services:
  rabbit01:
    image: itisfoundation/rabbitmq:4.1.2-management
    init: true
    hostname: "{{.Service.Name}}"
    environment:
      RABBITMQ_DEFAULT_USER: ${RABBIT_USER}
      RABBITMQ_DEFAULT_PASS: ${RABBIT_PASSWORD}
      RABBITMQ_NODENAME: "rabbit@{{.Service.Name}}"
    secrets:
      - source: rabbit_erlang_cookie
        target: /var/lib/rabbitmq/.erlang.cookie
        mode: 0600
        uid: "999"
        gid: "999"
    configs:
      - source: rabbitmq.conf
        target: /etc/rabbitmq/rabbitmq.conf
        mode: 0600
        uid: "999"
        gid: "999"
    volumes:
      - rabbit01_data:/var/lib/rabbitmq
    networks:
      - rabbit
    healthcheck:
      test: rabbitmq-diagnostics ping
      interval: 60s
      timeout: 10s
      retries: 2
      start_period: 30s
      start_interval: 10s

  # <redacted> same for rabbit02 and rabbit03 ...

volumes: 
  rabbit01_data:
    name: ${STACK_NAME}01_data
  rabbit02_data:
    name: ${STACK_NAME}02_data
  rabbit03_data:
    name: ${STACK_NAME}03_data

networks:
  rabbit:
    name: ${RABBIT_DOCKER_SWARM_NETWORK_NAME}
    driver: overlay
    attachable: true

configs:
  rabbitmq.conf:
    file: ./rabbitmq.conf
    name: ${STACK_NAME}_rabbitmq_conf_a7186458e1

secrets:
  rabbit_erlang_cookie:
    name: ${STACK_NAME}_erlang_cookie
    file: ./erlang.cookie.secret

[michaelklishin]

Since all nodes must agree on the shared secret or they won't be able to join the cluster, there is only one option for the Erlang cookie file swaps: a full stop upgrade.

Well, Blue-Green deployments are also always an option.

I don't feel confident documenting a specific procedure, other than a Blue-Green deployment, because there can be factors that are entirely deployment-specific. For example, the K8S Cluster Operator allows the cookie to be stored in Vault, so a rolling restart can be an option in practice, depending on how a pod startup failure is handled.

So at beast I can promise a very brief note in the upgrades guide.

[YuryHrytsuk]

@michaelklishin thanks for heads up 🙏

you can also say that this is not possible without downtime in documentation (with exceptions / concerns that you mention). This will really help and save time since I believe this question is very common (aka rotating "credentials")