Stack overflow: Cosmos container resolution can overflow the default 1 MiB Windows main-thread stack in an unoptimized build

[dmdenmsft]

Bug Title

Cosmos container resolution can overflow the default 1 MiB Windows main-thread stack in an unoptimized build. The failure is triggered by large, nested async poll frames in azure_data_cosmos_driver and is amplified by Microsoft ms-prod Rust code generation.

Crate Name

azure_data_cosmos

Crate Version

0.36.0

Description

Bug Title

Cosmos container resolution can overflow the default 1 MiB Windows main-thread stack in an unoptimized build. The failure is triggered by large, nested async poll frames in azure_data_cosmos_driver and is amplified by Microsoft ms-prod Rust code generation.

Crate Name

azure_data_cosmos (the dominant frames are in its internal dependency azure_data_cosmos_driver)

Crate Version

• azure_data_cosmos = 0.36.0
• azure_data_cosmos_driver = 0.5.0
• Both are the latest crates.io versions as of 2026-07-16.
• Current repository main contains azure_data_cosmos_driver = 0.6.0; that unreleased version has not yet been tested with this repro.

Description

Summary

On Windows x64 MSVC, a normal unoptimized Cargo build can terminate with
STATUS_STACK_OVERFLOW while resolving a Cosmos container. A first-chance CDB
dump shows a finite, non-recursive async poll chain through the Cosmos operation,
dataflow, cache, transport, Reqwest, and Hyper layers.

The Azure Cosmos driver owns approximately 666 KiB, or 66%, of the measured
1,009,696-byte symbolic stack. Its largest frame is
execute_operation_pipeline::async_fn$0.

Machine-code prologue measurements for that same function are:

Compiler/configuration	Active frame allocation
Microsoft ms-prod Rust 1.95, opt-level=0	155,048 bytes (0x25DA8)
Upstream Rust 1.95, opt-level=0	82,280 bytes (0x14168)
Microsoft ms-prod Rust 1.95, driver only at opt-level=1	51,616 bytes (0xC9A0)

The ms-prod debug frame is 72,768 bytes, or 88.4%, larger than the upstream
Rust 1.95 frame. This means the SDK has substantial debug stack pressure, while
the observed 1 MiB crash also has a compiler-codegen component.

A direct one-function SDK sample succeeds at 1 MiB. The layered repro below,
which uses an ordinary main -> run -> AppState::from_config startup shape,
fails under ms-prod. Please preserve those async boundaries when reproducing.

Expected behavior

A routine container-resolution call should not exhaust the default Windows
main-thread stack in a normal debug workflow. The SDK should retain enough stack
headroom for ordinary application async wrappers, or document and test a minimum
stack requirement.

Actual behavior

PE stack reserve bytes: 1048576
Exit code: 0xC00000FD

thread 'main' has overflowed its stack

The exception occurs after account properties are retrieved and HTTP/2 is
negotiated, during container metadata resolution.

Steps to Reproduce

1. Create Cargo.toml

[package]
name = "cosmos-stack-repro"
version = "0.1.0"
edition = "2021"
publish = false

[dependencies]
azure_core = "1.0.0"
azure_data_cosmos = "0.36.0"
azure_identity = "1.0.0"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

2. Create src/main.rs

use std::{env, error::Error, sync::Arc};

use azure_core::credentials::TokenCredential;
use azure_data_cosmos::options::Region;
use azure_data_cosmos::{AccountEndpoint, AccountReference, CosmosClient, RoutingStrategy};
use azure_identity::DeveloperToolsCredential;

struct Config {
    endpoint: String,
    database: String,
    container: String,
}

struct AppState;

impl AppState {
    async fn from_config(config: Config) -> Result<Self, Box<dyn Error>> {
        let credential: Arc<dyn TokenCredential> = DeveloperToolsCredential::new(None)?;
        let endpoint: AccountEndpoint = config.endpoint.parse()?;
        let account = AccountReference::with_credential(endpoint, credential);
        let client = CosmosClient::builder()
            .build(account, RoutingStrategy::ProximityTo(Region::EAST_US_2))
            .await?;
        let database = client.database_client(&config.database);
        let _container = database.container_client(&config.container).await?;

        Ok(Self)
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    run().await
}

async fn run() -> Result<(), Box<dyn Error>> {
    let config = Config {
        endpoint: env::var("COSMOS_ENDPOINT")?,
        database: env::var("COSMOS_DATABASE")?,
        container: env::var("COSMOS_CONTAINER")?,
    };
    let _state = AppState::from_config(config).await?;

    println!("Container resolved");
    Ok(())
}

3. Authenticate and set environment variables

DeveloperToolsCredential must be able to authenticate to a Cosmos account.

$env:COSMOS_ENDPOINT = "https://<account>.documents.azure.com/"
$env:COSMOS_DATABASE = "<database>"
$env:COSMOS_CONTAINER = "<container>"

4. Run an unoptimized Windows build with the Microsoft toolchain

The executable has no custom linker stack setting, so the MSVC default reserve
is 1 MiB.

cargo run

Expected repro result:

thread 'main' has overflowed its stack
error: process didn't exit successfully (exit code: 0xc00000fd, STATUS_STACK_OVERFLOW)

Control command

With upstream Rust 1.95 on the same machine and same source/dependencies:

cargo +1.95.0-x86_64-pc-windows-msvc run

This control succeeds and prints Container resolved.

Environment

• Windows 11 Enterprise x64, build 26200
• Target: x86_64-pc-windows-msvc
• Microsoft compiler: rustc 1.95.0-ms-20260618.5, commit ed80dadd6a
• Upstream control compiler: rustc 1.95.0, commit 59807616e, LLVM 22.1.2
• reqwest = 0.13.4
• hyper-util = 0.1.20
• Reproduced with both native_tls and rustls

Dump Measurements

CDB stopped on first-chance 0xC00000FD in __chkstk:

StackBase:  000000ed67160000
StackLimit: 000000ed67061000
RSP:        000000ed67069360
RAX:        0000000000004d18

At failure:

• Main-thread stack consumption from StackBase to RSP: 1,010,848 bytes.
• Parsed symbolic stack: 100 frames spanning 1,009,696 bytes.
• No exact symbolic frame repeats; this is not recursive growth.
• Hyper's next call_async poll frame requested another 19,736 bytes (0x4D18).
• The guard-page probe could not extend the stack inside the 1 MiB reserve.

Measured frame ownership:

Component	Frames	Approximate bytes	Share
azure_data_cosmos_driver	28	666,144	66.0%
HTTP/futures stack	44	167,528	16.6%
Application startup wrappers	15	128,304	12.7%
azure_data_cosmos	2	46,944	4.6%
Rust/runtime/other	10	776	0.1%

Largest measured frames in the failing build:

Function	Approximate bytes
execute_operation_pipeline	155,072
execute_operation_direct	69,520
execute_operation inner async block	68,608
execute_request	46,544
async cache get_or_insert_with	36,112

The adjacent stack-pointer estimate for execute_operation_pipeline is 155,072
bytes. Its machine-code prologue independently confirms a 155,048-byte local
allocation:

mov eax, 25DA8h
call __chkstk
sub rsp, rax

Controlled Results

Compiler/build	TLS	Stack reserve	Result
ms-prod, debug opt-level=0	native TLS	1 MiB	0xC00000FD
ms-prod, debug opt-level=0	rustls	1 MiB	0xC00000FD
ms-prod, debug opt-level=0	native TLS	2 MiB	Succeeds
ms-prod, release optimized	native TLS	1 MiB	Succeeds
ms-prod, debug, driver package only at opt-level=1	native TLS	1 MiB	Succeeds
Upstream Rust 1.95, debug opt-level=0	rustls	1 MiB	Succeeds
ms-prod, committed mitigation	native TLS	8 MiB	Succeeds

Optimizing only azure_data_cosmos_driver is sufficient:

[profile.dev.package.azure_data_cosmos_driver]
opt-level = 1

Under this configuration the active operation-pipeline frame drops from 155,048
to 51,616 bytes, a 66.7% reduction, and the 1 MiB application succeeds while the
application itself remains unoptimized.

Source Characteristics

In driver 0.5.0, execute_operation_pipeline is approximately:

• 744 source lines
• 37 local let bindings
• 8 await sites
• one large retry/orchestration loop

Current repository main already contains a TODO in execute_operation saying
that boxing is a temporary fix to avoid a large future and that refactoring is
needed to shrink it.

Related issue #4748 is not a duplicate. It investigates binary size, but it also
independently identifies large monomorphized driver async functions as a major
code-generation cost. This report concerns runtime debug stack consumption.

Requested Investigation

1. Reproduce against current main / azure_data_cosmos_driver 0.6.0 with both
   upstream Rust and ms-prod Rust.
2. Determine why ms-prod allocates an 88.4% larger debug poll frame for
   execute_operation_pipeline than upstream Rust 1.95.
3. Reduce peak debug stack use by splitting the 744-line operation pipeline into
   smaller async stage functions or introducing deliberate heap-backed boundaries
   where they reduce poll-frame residency.
4. Review other large driver frames in execute_operation_direct,
   execute_operation, cache resolution, and transport execution.
5. Add a Windows debug stack regression test, for example running container
   resolution on a thread with a 1 MiB stack and an emulator/mock transport.
6. Consider a CI check that records the largest async poll frame allocation from
   MSVC unwind/prologue data.

Workarounds

• Reserve a larger Windows executable stack, such as 8 MiB.
• Optimize only azure_data_cosmos_driver in the dev profile.
• Use a release/optimized build.

Artifact Sharing

Safe to attach publicly:

• The standalone repro crate above.
• Sanitized CDB stack/prologue logs.
• The frame-size and controlled-test tables.

Do not attach the full 72 MiB process dump publicly without security review. A
full-memory dump may contain credentials, tokens, request data, or account
metadata. It can be shared privately with Microsoft maintainers through an
approved secure channel if required.

Checklist

• Follow the Azure SDK for Rust Code of Conduct.
• Searched existing issues. No Cosmos stack-overflow duplicate was found.
• Included a standalone layered reproduction.

Steps to Reproduce

Steps to Reproduce

1. Create Cargo.toml

[package]
name = "cosmos-stack-repro"
version = "0.1.0"
edition = "2021"
publish = false

[dependencies]
azure_core = "1.0.0"
azure_data_cosmos = "0.36.0"
azure_identity = "1.0.0"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

2. Create src/main.rs

use std::{env, error::Error, sync::Arc};

use azure_core::credentials::TokenCredential;
use azure_data_cosmos::options::Region;
use azure_data_cosmos::{AccountEndpoint, AccountReference, CosmosClient, RoutingStrategy};
use azure_identity::DeveloperToolsCredential;

struct Config {
    endpoint: String,
    database: String,
    container: String,
}

struct AppState;

impl AppState {
    async fn from_config(config: Config) -> Result<Self, Box<dyn Error>> {
        let credential: Arc<dyn TokenCredential> = DeveloperToolsCredential::new(None)?;
        let endpoint: AccountEndpoint = config.endpoint.parse()?;
        let account = AccountReference::with_credential(endpoint, credential);
        let client = CosmosClient::builder()
            .build(account, RoutingStrategy::ProximityTo(Region::EAST_US_2))
            .await?;
        let database = client.database_client(&config.database);
        let _container = database.container_client(&config.container).await?;

        Ok(Self)
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    run().await
}

async fn run() -> Result<(), Box<dyn Error>> {
    let config = Config {
        endpoint: env::var("COSMOS_ENDPOINT")?,
        database: env::var("COSMOS_DATABASE")?,
        container: env::var("COSMOS_CONTAINER")?,
    };
    let _state = AppState::from_config(config).await?;

    println!("Container resolved");
    Ok(())
}

3. Authenticate and set environment variables

DeveloperToolsCredential must be able to authenticate to a Cosmos account.

$env:COSMOS_ENDPOINT = "https://<account>.documents.azure.com/"
$env:COSMOS_DATABASE = "<database>"
$env:COSMOS_CONTAINER = "<container>"

4. Run an unoptimized Windows build with the Microsoft toolchain

The executable has no custom linker stack setting, so the MSVC default reserve
is 1 MiB.

cargo run

Expected repro result:

thread 'main' has overflowed its stack
error: process didn't exit successfully (exit code: 0xc00000fd, STATUS_STACK_OVERFLOW)

Checklist

• Follow our Code of Conduct
• Check that there isn't already an issue that request the same bug to avoid creating a duplicate.
• The provided reproduction is a minimal reproducible example of the bug.

[github-actions]

Thank you for your feedback. Tagging and routing to the team members best able to assist. cc @Pilchie

Generated by Agentic Triage for #4794 · 24.3 AIC · ⌖ 8.2 AIC · ⊞ 8.2K · ◷

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/issue-triage.md@8e6d7c86bba37371d2d0eee1a23563db3e561eb5

[github-actions]

🎯 Agentic Issue Triage

Summary: On Windows x64 MSVC with the Microsoft ms-prod Rust toolchain, calling database.container_client(...) during an unoptimized (opt-level=0) debug build causes a STATUS_STACK_OVERFLOW (0xC00000FD) because azure_data_cosmos_driver's execute_operation_pipeline allocates a 155 KiB stack frame — 88% larger than the upstream Rust 1.95 equivalent — exhausting the default 1 MiB Windows main-thread stack.

📋 Issue Details

• Crate / module: azure_data_cosmos / internal dependency azure_data_cosmos_driver
• Affected API: DatabaseClient::container_client, internally execute_operation_pipeline
• Scenario: Resolving a Cosmos container in a layered async application (main → run → AppState::from_config) under an unoptimized debug build with the Microsoft ms-prod Rust compiler
• Root ask: Reduce peak debug async poll-frame stack consumption in azure_data_cosmos_driver so that routine debug workflows succeed within the default 1 MiB Windows main-thread stack

🔎 Debugging / Reproduction Notes

The crash is reproducible with ms-prod Rust 1.95 at opt-level=0 and does not reproduce with upstream Rust 1.95 at the same opt level. A table of controlled results is included in the issue, showing that:

• Optimizing only azure_data_cosmos_driver (opt-level=1 in [profile.dev.package]) reduces the frame from 155 KiB to 52 KiB and eliminates the crash.
• Setting the PE stack reserve to 2 MiB also avoids the crash.
• The 744-line execute_operation_pipeline function has 37 local bindings and 8 await sites and is the dominant frame owner.
• The repository main branch already has a TODO noting the need to refactor this function.

Suggested investigation steps:

1. Reproduce against azure_data_cosmos_driver 0.6.0 on main with both ms-prod and upstream Rust 1.95 to confirm whether the unreleased version reduces frame size.
2. Refactor execute_operation_pipeline by splitting the 744-line function into smaller async stage functions or introducing Box::pin-backed async boundaries to shrink per-frame residency.
3. Review the other large frames identified (execute_operation_direct, execute_operation, cache get_or_insert_with, execute_request) for similar splitting opportunities.
4. Add a regression test that runs container resolution on a thread with a 1 MiB stack against a mock/emulator transport to catch future regressions on Windows.

🏷️ Label Confidence

• Category: Client — azure_data_cosmos does not start with azure_resourcemanager_; this is a client library issue.
• Service: Cosmos — the issue explicitly names azure_data_cosmos and azure_data_cosmos_driver; maps to sdk/cosmos/.
• Confidence: High — the crate, service, and category are unambiguous.

👥 Owner Routing

• Matched CODEOWNERS entry: # ServiceLabel: %Cosmos — matched because the predicted service label is Cosmos.
• AzureSDKOwners: none listed
• ServiceOwners: @Pilchie
• Routing action: Added Service Attention and needs-team-attention labels; issue left unassigned per routing rules for ServiceOwners-only entries.

Generated by Agentic Triage for #4794 · 24.3 AIC · ⌖ 8.2 AIC · ⊞ 8.2K · ◷

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/issue-triage.md@8e6d7c86bba37371d2d0eee1a23563db3e561eb5