Fix to acquire the permit in a blocking way

[kozistr]

What does this PR do?

Fixes #662
Fixes #470 (?)

For a batch input, it spawns multiple asynchronous tasks, and each task calls an async-awaitable permit acquisition, whereas a single input does not.

Changes

• Use try_acquire_permit() instead of acquire_permit() for the batch case so that it immediately errors out instead of awaiting
• Increase the Queue buffer size to twice to prevent the failure (maybe related to thread 'tokio-runtime-worker' panicked at core/src/queue.rs:72:14 #470)

Also, I came across while testing this, for many small requests, likewise [a, b, c, d, e], the Queue rarely fails to perform append() or next_batch() due to a lack of capacity of the queue buffer. So, I temporarily doubled the buffer size to prevent this, which is obviously not a good way to handle this. (Maybe it'd be better to raise a 429 error when .try_send() failed, too)

We can reproduce it by following these steps.

./target/release/text-embeddings-router --model-id ../Qwen3-Embedding-0.6B/ --pooling last-token --port 8080 --dtype float32 --auto-truncate --max-batch-tokens 512 --max-concurrent-requests 5

// problem
for _ in range(25):
    print(
        requests.post(
            'http://127.0.0.1:8080/embed',
            data=json.dumps({'inputs': ['a', 'b', 'c', 'd', 'e']}),
            headers={'Content-type': 'application/json'},
        ).status_code
    )

thread 'tokio-runtime-worker' (295607) panicked at core/src/queue.rs:87:14:
Queue background task dropped the receiver or the receiver is too behind. This is a bug.: "Full(..)"

Log

./target/release/text-embeddings-router --model-id ../Qwen3-Embedding-0.6B/ --pooling last-token --port 8080 --dtype float32 --auto-truncate --max-batch-tokens 512 --max-concurrent-requests 5
2025-09-20T14:56:50.194026Z  INFO text_embeddings_router: router/src/main.rs:203: Args { model_id: "../Qwe**-*********-0.6B/", revision: None, tokenization_workers: None, dtype: Some(Float32), pooling: Some(LastToken), max_concurrent_requests: 1, max_batch_tokens: 512, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: true, default_prompt_name: None, default_prompt: None, dense_path: None, hf_api_token: None, hf_token: None, hostname: "0.0.0.0", port: 8080, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: None, payload_limit: 2000000, api_key: None, json_output: false, disable_spans: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", prometheus_port: 9000, cors_allow_origin: None }
2025-09-20T14:56:50.561317Z  WARN text_embeddings_router: router/src/lib.rs:190: Could not find a Sentence Transformers config
2025-09-20T14:56:50.561368Z  INFO text_embeddings_router: router/src/lib.rs:194: Maximum number of tokens per request: 32768
2025-09-20T14:56:50.561552Z  INFO text_embeddings_core::tokenization: core/src/tokenization.rs:38: Starting 8 tokenization workers
2025-09-20T14:56:50.943401Z  INFO text_embeddings_router: router/src/lib.rs:242: Starting model backend
2025-09-20T14:56:50.948236Z  INFO text_embeddings_backend_candle: backends/candle/src/lib.rs:305: Starting Qwen3 model on Cpu
2025-09-20T14:56:53.843729Z  INFO text_embeddings_router: router/src/lib.rs:260: Warming up model
2025-09-20T14:56:58.844160Z  WARN text_embeddings_router: router/src/lib.rs:269: Backend does not support a batch size > 4
2025-09-20T14:56:58.844211Z  WARN text_embeddings_router: router/src/lib.rs:270: forcing `max_batch_requests=4`
2025-09-20T14:56:58.847053Z  INFO text_embeddings_router::http::server: router/src/http/server.rs:1862: Starting HTTP server: 0.0.0.0:8080
2025-09-20T14:56:58.847103Z  INFO text_embeddings_router::http::server: router/src/http/server.rs:1863: Ready
2025-09-20T14:57:00.601838Z ERROR embed:try_acquire_permit: text_embeddings_core::infer: core/src/infer.rs:102: no permits available

requests.post(
    'http://127.0.0.1:8080/embed',
    data=json.dumps({'inputs': ['a', 'b', 'c', 'd', 'e', 'f']}),
    headers={'Content-type': 'application/json'},
).content
b'{"error":"Model is overloaded","error_type":"Overloaded"}'

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the documentation guidelines.
• Did you write any new necessary tests? If applicable, did you include or update the insta snapshots?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@Narsil @alvarobartt