[FEATURE] Python: configurable per-batch concurrency for ConcurrentToolExecutor #4425

SDK Language

Python

Problem Statement

ConcurrentToolExecutor starts every tool use in a model-generated batch concurrently. The built-in alternative is SequentialToolExecutor, but there is no built-in way to retain parallel execution while bounding the number of active calls within a batch.

A model can request several independent retrieval or API calls in one response. Starting all of them together can create a burst of connections or downstream requests. Serial execution avoids that burst but also removes useful concurrency.

Proposed Solution

Add an optional, keyword-only max_concurrency: int | None = None to ConcurrentToolExecutor:

from strands import Agent
from strands.tools.executors import ConcurrentToolExecutor

agent = Agent(
    tools=[...],
    tool_executor=ConcurrentToolExecutor(max_concurrency=3),
)

• Omitted or None: retain the current execution behavior.
• Positive integer N: admit at most N foreground tool executions at a time within each batch. Admit waiting work as a running execution completes.
• Preserve streamed events and the current final tool-result ordering by original tool-use order.
• Keep the configuration on the executor; no new Agent parameter.

A batch-local asyncio.Semaphore would keep admission control separate from the existing event queue and result collection, without storing an event-loop-bound semaphore on a reusable executor.

This is not a global, per-tool, cross-Agent, or requests-per-second limit. It does not introduce a custom thread pool or limit the lifetime of background tasks. Cancellation must preserve existing contracts; it cannot forcibly stop a synchronous tool's underlying thread.

Use Case

An application with capacity for two concurrent downstream operations should be able to process a batch of six tool calls two at a time, without making the model generate smaller batches or replacing the built-in executor with a custom implementation.

Alternatives Solutions

• SequentialToolExecutor: appropriate for serial execution, but cannot express an intermediate concurrency level.
• A semaphore inside each application tool: useful for application-wide or service-specific limits, but requires coordination across tools to bound an entire batch.
• A custom executor: already possible, but requires applications to maintain execution-policy code for this small, reusable capability.
• Prompting the model to request fewer tools: not an enforced runtime concurrency limit.

Additional Context

Inspected main at 54ca0befa69a1e8e8d7f3083da62a8e9d341f050. Two offline characterization checks passed with single-process pytest (-n 0):

1. Six calls to an async @tool increment an active counter and wait on a shared asyncio.Event. Before releasing that event, all six have entered the tool body and none has completed. The observed peak is six. After release, all six results are collected in tool-use order.
2. ConcurrentToolExecutor(max_concurrency=2) raises TypeError because the constructor does not accept the option.

The first check uses an event barrier and a counter, not elapsed execution time; a timeout only guards against a hung test. Neither check calls a real model or service. These characterize current behavior; feature tests and implementation have not yet been completed.

Related work reviewed:

• [FEATURE] Allow applications to control thread pools for Agent tool calls. #476 and [BUG] Configurable Threads #769 concern thread-pool control and thread context. This proposal does not fully address them.
• fix(python): abort stalled concurrent MCP tool waits on cancel #4407 changes cancellation handling in the same executor. It does not add a concurrency limit; implementation will need to account for any overlapping edits.
• [PORT] Graph max_concurrency cap (TS→Py) #3156 / feat(graph): add Python concurrency limit #3448 concern Graph concurrency, not tool batches.
• fix: keep concurrent tool results in request order #2340 established tool-result ordering that this proposal must preserve.

Searches of open/closed Issues and PRs for max_concurrency, ConcurrentToolExecutor, and tool concurrency did not identify an equivalent proposal or implementation as of this filing. Maintainer confirmation of scope is welcome before implementation.

Proposed acceptance tests:

• Six tool calls with a limit of two reach a peak of exactly two and never exceed it.
• Completing one call admits waiting work while another call remains active.
• Default behavior, streamed events, and final tool-result ordering are preserved.
• Handled tool errors release slots; executor-aborting errors and cancellation clean up active and waiting tasks under the existing contracts.
• Consecutive batches have independent admission budgets.
• An offline mock-model Agent test covers tool uses, bounded execution, tool-result messages, and the final Agent result.