4.1.3 - stream queues - api GET /api/queues and stream_status report a different queue leader

[jcosmao]

Describe the bug

rabbitmq-server: 4.1.3

Stream queue leader reported by api may be different than what "rabbitmq-streams stream_status" report as writer
Only few stream queues are reporting a mismatch
On 3.12, it's ok

Reproduction steps

curl  -X GET  http://0:15672/api/queues/%2f/scheduler_fanout  | jq 
{
  "arguments": {
    "x-queue-type": "stream"
  },
  "auto_delete": false,
  "durable": true,
  "exclusive": false,
  "leader": "rabbit@rabbit1",
  "members": [
    "rabbit@rabbit1",
    "rabbit@rabbit2",
    "rabbit@rabbit3"
  ],
  "name": "scheduler_fanout",
  "node": "rabbit@rabbit1",
  "online": [
    "rabbit@rabbit1",
    "rabbit@rabbit2",
    "rabbit@rabbit3"
  ],
  "state": "running",
  "type": "stream",
  "vhost": "/"
}

rabbitmq-streams stream_status scheduler_fanout --formatter json | jq
[
  {
    "role": "replica",
    "node": "rabbit@rabbit1",
    "epoch": 6,
    "offset": 74820,
    "committed_offset": 74820,
    "first_offset": 65991,
    "readers": 8,
    "segments": 3
  },
  {
    "role": "replica",
    "node": "rabbit@rabbit2",
    "epoch": 6,
    "offset": 74820,
    "committed_offset": 74820,
    "first_offset": 65991,
    "readers": 18,
    "segments": 3
  },
  {
    "role": "writer",
    "node": "rabbit@rabbit3",
    "epoch": 6,
    "offset": 74820,
    "committed_offset": 74820,
    "first_offset": 65991,
    "readers": 14,
    "segments": 3
  }
]

Expected behavior

stream queue leader reported by api == stream_status queue writer

Additional context

No response

[michaelklishin]

@jcosmao your claim is based on a few assumptions are are incorrect.

Firstly, you assume that a queue and a stream have exactly identical sets of members and their functions. For the "functions" part, that's not exactly the case, streams have a coordinator part that queues do not.

Secondly, you assume that the HTTP API uses the same data source and the CLI commands. That is not the case, CLI tools usually poll the member [Erlang] processes while the HTTP API almost always uses a separate data store with emitted metrics, a store that pre-dates streams by many years.

This is the case for rabbitmq-streams stream_status which uses rabbit_stream_queue:tracking_status/2.

I also don't see how this discrepancy can make any meaningful difference. Both endpoints return a full set of members. The leader can move to a different node. Most importantly, the data locality factor which applies to queues does not really apply to streams when they are used properly: RabbitMQ Stream Protocol clients can connect to multiple nodes at once and consume from secondary members (replicas).

Our small team has much bigger fish to fry. That said, RabbitMQ is open source software, so if you care enough, you are welcome to investigate how exactly streams report their leader metrics that the HTTP API then reports.