Static shovels do not work in v4 when the topology is imported from a definitions file at boot time

[remus95]

Describe the bug

Given two RabbitMQ servers, rabbitmq-src and rabbitmq-dest, I want to transfer messages from the quorum queue my-queue on rabbitmq-src to the quorum queue my-queue on rabbitmq-dest using the shovel plugin with statically configured shovels and amqp 0-9-1. The quorum queues and the rest of the topology are automatically imported from a definitions file at boot time. Only a single static shovel is defined in the advanced.config file.

When using RabbitMQ 4.0.1, the static shovel worker fails (see the error in the attached image). It connects to the destination RabbitMQ server, but then immediately stops and closes the connection due to this error.

This issue does not occur with the latest RabbitMQ v3 release (3.13.7). Therefore, I believe the breaking behavior was introduced in version 4.0.1.

Since the topology is automatically imported from a definitions file at startup, I also tried configuring the Shovel plugin to wait until the queue is available by adding the following line shovel.topology.predeclared = true to the rabbitmq.conf as documented. However, this causes the server to fail at boot time with the following error:

rabbitmq-src  | BOOT FAILED
rabbitmq-src  | ===========
rabbitmq-src  | Error during startup: {error,
rabbitmq-src  |                        {rabbitmq_shovel,
rabbitmq-src  |                         {bad_return,
rabbitmq-src  |                          {{rabbit_shovel,start,[normal,[]]},
rabbitmq-src  |                           {'EXIT',
rabbitmq-src  |                            {{case_clause,"true"},
rabbitmq-src  |                             [{rabbit_amqp091_shovel,decl_fun,1,
rabbitmq-src  |                               [{file,"rabbit_amqp091_shovel.erl"},{line,612}]},
rabbitmq-src  |                              {rabbit_amqp091_shovel,parse,2,
rabbitmq-src  |                               [{file,"rabbit_amqp091_shovel.erl"},{line,59}]},
rabbitmq-src  |                              {rabbit_shovel_config,parse_current,2,
rabbitmq-src  |                               [{file,"rabbit_shovel_config.erl"},{line,140}]},
rabbitmq-src  |                              {rabbit_shovel_config,parse,2,
rabbitmq-src  |                               [{file,"rabbit_shovel_config.erl"},{line,76}]},
rabbitmq-src  |                              {rabbit_shovel_sup,parse_configuration,3,
rabbitmq-src  |                               [{file,"rabbit_shovel_sup.erl"},{line,77}]},
rabbitmq-src  |                              {rabbit_shovel_sup,start_link,0,
rabbitmq-src  |                               [{file,"rabbit_shovel_sup.erl"},{line,16}]},
rabbitmq-src  |                              {application_master,start_it_old,4,
rabbitmq-src  |                               [{file,"application_master.erl"},
rabbitmq-src  |                                {line,293}]}]}}}}}}

Please note that these issues also occurs with the latest RabbitMQ releases, 4.1.4 and 4.2-rc.

Reproduction steps

1. Start the two containers
   docker-compose -f docker-compose.yml up -d
2. Inspect the shovels status when the containers are up and running (notice the error)
   docker-compose -f docker-compose.yml exec -it rabbitmq-src rabbitmqctl shovel_status
   Or access the management UI http://0.0.0.0:15671/#/shovels (user admin, pass admin)
3. Make clean
   docker-compose -f docker-compose.yml down && docker-compose -f docker-compose.yml rm -sf
4. Configure shovel.topology.predeclared = true in ./configs/rabbitmq.conf
5. Start again the two containers
   docker-compose -f docker-compose.yml up
6. Notice the error
   docker-compose -f docker-compose.yml logs -f rabbitmq-src
7. Make clean
   docker-compose -f docker-compose.yml down && docker-compose -f docker-compose.yml rm -sf

Expected behavior

The static shovel should be in a running state and not display any errors.

Additional context

Versions: Rabbitmq 4.0.1, Erlang 26 (or newer)

I have uploaded an intentionally simplified docker-compose file to demonstrate the issue. Thank you!

rabbitmq-issue.zip

[michaelklishin]

This is not a bug. Static shovels are not particularly smart, they try to run their resource declarations and assume that all those operations will succeed. We won't try to make them smarter.
There is nothing a shovel can do about a queue with a queue property mismatch (an existing queue has different properties than what a shovel is configured to use), for example.

Static shovels are a feature from 2009, that is, before dynamic shovels (or even the runtime parameters subsystem they depend on) existed. It has effectively no good use cases today, in particular in the described scenario where a definition file is imported.
Definitions will include dynamic shovel definitions, which operators do not need to use CLI tools or the HTTP API to set such shovels up after cluster provisioning.

As of RabbitMQ 4.0 #11793, shovels — and here I mean dynamic shovels because that's what 99% of shovel users use — allow you to opt out of topology declaration entirely, which avoids effectively all topology declaration issues.

Or, well, all but one: the relative order of shovel startup and definition import. This was addressed in 2020 in #2396, and further improved in the already mentioned change in #11793.

Both the dynamic shovel doc guide and static shovel one cover the predeclared topology feature.

To quote one of those guides:

Shovels can use a pre-declared topology instead of declaring its source and destination.

For example, this may be necessary when the topology is imported from a definitions file at boot time. 
Using a pre-declared topology avoids a chicken-and-egg problem between the import of definitions
and Shovel plugin startup: the plugin must be enabled for definitions
to pass validation (of the runtime parameters part).