How to handle rate limiting and synchronous pauses?

[postmodern]

I'm trying to interface to a 3rd-party API service (cert.sh) with a bunch of async tasks to perform queries concurrently. However, if too many HTTP requests are sent within a period of time, the API will then start returning timeout error responses, which requires that I wait for a few seconds before sending another request. Is there a way to create some kind of global shared timeout lock that I could enable when a timeout error response which would temporarily pause all async tasks trying to access the API endpoint?

[ioquatix]

Does the server respond with a rate limiting header?

[ioquatix]

There is a specification for rate limiting headers: https://www.ietf.org/archive/id/draft-polli-ratelimit-headers-02.html

If this is present, you could adopt this in your client - i.e. a rate limiting semaphore etc.

If not, there are two options:

• Arbitrary back-off - i.e. configure some fix back-off.
• Shared global state for rate limiting - e.g. using redis and a count key for doing your own rate limiting to avoid receiving 429.

[postmodern]

@ioquatix is there a clean way to somehow have an Async primitive which all of the async tasks would check, and when one of the async tasks receives a 429 response I could somehow tell the global lock to sleep for N seconds, which would then cause all other async tasks to pause until the sleep timer runs out and the lock opens. I could probably implement from scratch, but it would probably involve busy pooling or some kind of ugly Time.now check to determine if the rate limit timer has expired.

[ioquatix]

Within the same reactor (thread), something like this would work:

class RateLimiter < Protocol::HTTP::Middleware
  def initialize(delegate)
    super
    @rate_limiting = 0
  end
  
  def call(request)
    while true
      while @rate_limiting > 0
        sleep 1
      end

      response = super
      
      if response.status == 429
        @rate_limiting += 1
        sleep 1
        @rate_limiting -= 1
      else
        return response
      end
    end
  end
end

Wrap it around the Async::HTTP::Client instance.

IF you need global state, use async-redis to store the rate_limiting value.

[bruno-]

@postmodern I wrote async-limiter. It allows you to specify all kinds of limits for a certain time window. Limiters can be nested, so you can specify conditions like 60 requests per minute, 1000 per hour, 5000 per day. And you can dynamically change the limits on the fly.

I run it in production and it works great for me with async 1.28. Unfortunately I didn't test it with Async 2.

[postmodern]

@bruno- I will keep async-limiter in mind, however I want to apply rate limiting once the upstream HTTP API starts limiting me; so some kind of "rate limit encountered" condition. Luckily for me the upstream API I've been working with changed their API to pause for several seconds before returning a response, instead of immediately returning a HTTP 429 response. However, I expect I'll run into hard rate limiting again with other 3rd-party APIs.