[Question] Redis autopipelining

[loris]

ioredis, used by bull, added support for autopipelining.
It states This feature can dramatically improve throughput and avoids HOL blocking. In our benchmarks, the improvement was between 35% and 50%.

I was wondering if it would also improve bull performance, or on the contrary is not compatible / will impact negatively how bull works? Anyone had some experience with that feature?

[manast]

It needs to be tested, could be useful when using high concurrency settings.

[loris]

FYI, we tried it on our bull connections (Queue, Worker and QueueScheduler) and we immediately saw tons of "Job is not in the active list 50178660 delayed" errors. So we rollbacked and the errors stopped.
Does it sound familiar to you?

[manast]

no it doesn't. I do not know how the autopipelining is implemented, in theory as it states in the documentation "In auto pipelining mode, all commands issued during an event loop are enqueued in a pipeline automatically managed by ioredis. At the end of the iteration, the pipeline is executed and thus all commands are sent to the server at the same time.", it should work. However I am not so sure it would give a lot of gains when using BullMQ specially if you already are using high concurrency factors and several workers each with their dedicated connection to redis.

[manast]

a way to start testing this would be to run BullMQ tests enabling this feature is some of them. Beware that running the tests deletes data in your local redis instance.