Redis doesn't seem to interleave Lua XADD calls made in a loop with XREAD or XREADGROUP that is running in parallel

[kant777]

Hi All,

I have a dedupe script that looks like this and Runs in Process A

func GetDedupeScript() string {
		return `if redis.call("EXISTS", ARGV[1]) == 0 then
			redis.call("SETEX", ARGV[1], ARGV[2], "1");
		
			redis.call("XADD", ARGV[3], "MAXLEN", "~", ARGV[4], "*",
			"jsonRpcMethodName", ARGV[5],
                        "value", ARGV[6]
		)
		end`
}

I invoke this script in a loop like below.

for i := 0;  i<len(messages); i++ { // len(messages) ~ 500
    go func() {
         redisClient.Run("GetDedupeScript()", messages[i])
     }()
 }

I have another process called it Process B that runs XREAD with COUNT 200 AND BLOCK 5000 from the same topic Process A is sending messages to.

when I do redis-cli monitor I see the following pattern

EXITS
SETEX
XADD

EXITS
SETEX
XADD

EXITS
SETEX
XADD

...500 times
XREAD 

As you can see XREAD is getting called after the entire loop finishes running although both XADD and XREAD are running in a separate processes but on the same machine. This pattern resulted in an increased LATENCY like 100-200 ms. Any ideas why this is happening?

I was thinking what happens if I introduce sleep after every iteration so I did this

for i := 0;  i<len(messages); i++ { // len(messages) ~ 500
    go func() {
         redisClient.Run("GetDedupeScript()", messages[i])
     }()
    time.Sleep(2 * time.Millisecond)
 }

This resulted in the following pattern

EXITS
SETEX
XADD
XREAD 

EXITS
SETEX
XADD
XREAD 

EXITS
SETEX
XADD
XREAD 

...500 times

Now I see the calls are getting interleaved nicely and results in Latency of 1ms which is awesome. I am trying to see how I can achieve the same result without introducing time.Sleep? I tried pipe-lining but that did not work it resulted in the same pattern as the one prior to introduction of sleep (like XADD 500 times and then one XREAD).