Skip to content

[cosmos] Use fixed self-owned accounts for key-based live tests#49735

Draft
tvaron3 wants to merge 9 commits into
Azure:mainfrom
tvaron3:tvaron3/cosmos-livetest-sub-tenant
Draft

[cosmos] Use fixed self-owned accounts for key-based live tests#49735
tvaron3 wants to merge 9 commits into
Azure:mainfrom
tvaron3:tvaron3/cosmos-livetest-sub-tenant

Conversation

@tvaron3

@tvaron3 tvaron3 commented Jul 6, 2026

Copy link
Copy Markdown
Member

Summary

Point the key-based Cosmos live tests at fixed, self-owned Cosmos accounts (resource group sdk-ci) instead of provisioning a new account per pipeline run. This removes the central EngSys service-connection / on-the-fly-provisioning dependency for these tests, which is required so the tests can run against accounts in an ephemeral tenant that is recreated every ~90 days.

This PR wires up two stages as the first step: the main Cosmos_live_test stage and Cosmos_Live_Test_Http2.

What changed

  • New sdk/cosmos/pipeline/ mechanism — a single ADO variable-group secret (sub-config-cosmos-azure-cloud-test-resources) holds a versioned JSON mapping a logical account name → { endpoint, key, secondaryKey, ... }:
    • live-test-accounts.schema.json + live-test-accounts.sample.json
    • resolve-cosmos-test-account.sh (bash + jq): resolves one account by selector and exports ACCOUNT_HOST / ACCOUNT_KEY (+ SECONDARY_ACCOUNT_KEY). The key uses the azure-sdk double-set convention (_ACCOUNT_KEY secret to mask it in logs + a plain ACCOUNT_KEY so it propagates to the Maven test env). It does not emit ACCOUNT_CONSISTENCY / PREFERRED_LOCATIONS — those stay matrix-controlled.
    • resolve-cosmos-test-account.tests.sh (8 local tests, all pass)
    • resolve-test-account-steps.yml (reusable PreTestRunSteps template)
    • README.md
  • tests.ymlCosmos_live_test and Cosmos_Live_Test_Http2 set DisableAzureResourceCreation: true and run the resolve pre-step.
  • live-platform-matrix.json — each leg gets an AccountSelector so the multi-config main stage picks the right fixed account per leg.

Not included (follow-ups)

Left on the existing provisioning path because they need feature-enabled accounts or a separate pipeline:

  • Thin-client stages (×3) and GSI — accounts need thin-client / GSI-preview enablement.
  • Kafka — entangled with AAD auth; belongs in a separate AAD pipeline.
  • Spring Data Cosmos — uses its own test-resources (AZURE_SPRING_TENANT_ID).

The companion account-provisioning + Key Vault scripts live in the cosmos-sdk-copilot-toolkit repo (separate PR).

Verification

  • resolve-cosmos-test-account.sh: 8/8 local unit tests pass; YAML/JSON validated.
  • Fixed accounts + secret provisioned; each account carries a secondaryKey (required by AzureKeyCredentialTest in the -Pfast legs).
  • ⚠️ Needs a live pipeline run to fully validate before merge.

Draft

Draft while the two wired stages are validated end-to-end in a real pipeline run.

Point the main Cosmos live-test stage and the Http2 stage at fixed,
self-owned Cosmos accounts (RG sdk-ci) instead of provisioning an account
per pipeline run. This removes the central EngSys service-connection /
on-the-fly-provisioning dependency for the key-based tests.

- Add sdk/cosmos/pipeline/ with a single-secret JSON mechanism:
  - live-test-accounts.schema.json + sample
  - resolve-cosmos-test-account.sh (bash+jq) resolves one account from the
    single secret (sub-config-cosmos-azure-cloud-test-resources) by selector
    and exports ACCOUNT_HOST/ACCOUNT_KEY(+SECONDARY) using the azure-sdk
    double-set convention so the key is masked in logs yet reaches the test env
  - resolve-cosmos-test-account.tests.sh (local tests)
  - resolve-test-account-steps.yml (reusable PreTestRunSteps template)
- tests.yml: Cosmos_live_test + Cosmos_Live_Test_Http2 set
  DisableAzureResourceCreation:true and run the resolve pre-step.
- live-platform-matrix.json: add per-leg AccountSelector.

Thin-client, GSI, Kafka and Spring stages are intentionally left on the
existing provisioning path (they need feature-enabled accounts or a separate
AAD pipeline) and will be migrated in follow-ups.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added the Cosmos label Jul 6, 2026
@tvaron3

tvaron3 commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

@tvaron3

tvaron3 commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

…test-sub-tenant

# Conflicts:
#	sdk/cosmos/live-platform-matrix.json
@tvaron3

tvaron3 commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

tvaron3 and others added 2 commits July 8, 2026 11:06
…count-safe

Follow-up to the fixed-account live-test wiring: bring the account
provisioning tooling into the repo and fix tests that assumed each run
gets its own freshly-provisioned account.

- Add sdk/cosmos/pipeline/account-provisioning/ (moved from the
  cosmos-sdk-copilot-toolkit repo):
  - New-CosmosLiveTestAccounts.ps1 creates the fixed sdk-ci accounts and
    outputs the cosmos-live-test-accounts JSON; reconciles capabilities on
    existing accounts (EnableNoSQLVectorSearch) via ARM PATCH.
  - Set-CosmosLiveTestAccountsSecret.ps1 publishes that JSON to the secret.
  - cosmos-live-test-accounts.definition.json defines the accounts; the 5
    general-purpose accounts enable EnableNoSQLVectorSearch to match the
    ARM test-resources.json capability set.
- ReadFeedDatabasesTest: readAllDatabases is account-global; on shared
  fixed accounts other runs create/delete DBs concurrently. Assert only
  that the databases created by this test are present (containment) via a
  new FeedResponseListValidator.containsResourceIds, instead of asserting
  the exact account-wide set/count/page-count.

Verified locally: ReadFeedDatabasesTest failed 3/3 under concurrent DB
churn before and passes 3/3 after; NonStreamingOrderByQueryVectorSearchTest
passes 4/4 once EnableNoSQLVectorSearch is enabled on the account.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Build Analyze (cspell) stage flagged domain/code tokens in the new
sdk/cosmos/pipeline files (sdkci, pmerge, Pfast, issecret, esac, WHATIF).
Add a scoped cspell override for sdk/cosmos/pipeline/** so these expected
words don't fail spellcheck, without touching the global word list.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3

tvaron3 commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

…/pipeline

- Remove Set-CosmosLiveTestAccountsSecret.ps1 and its references; the Key
  Vault / ADO variable secret is now updated manually with the JSON that
  New-CosmosLiveTestAccounts.ps1 outputs. Docs updated accordingly (removed
  the KV-publish usage, the dated "Verified run" note, and the macOS auth
  note).
- Move the cspell allow-list for pipeline domain/code tokens (sdkci, pmerge,
  Pfast, issecret, esac, WHATIF) out of the root .vscode/cspell.json into a
  nested sdk/cosmos/pipeline/cspell.json, which cspell merges automatically.
  Keeps the change scoped to files under sdk/cosmos.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3

tvaron3 commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

readOffers is an account-global read. With live tests running against
shared fixed accounts, concurrent legs create/delete containers (and
their offers), so asserting the exact account-wide offer set/count/page
count is non-deterministic. Assert containment of only the offers for
containers created by this test instead.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3

tvaron3 commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@tvaron3

tvaron3 commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - spark

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@tvaron3

tvaron3 commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - kafka

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

… guards

- Align fixed-account regions with the ARM bicep (test-resources.json):
  regionDefaults -> Central US (single), Central US + East US 2 (multi).
  This fixes the fault-injection availability-strategy tests whose region
  assertions require the preferred region (East US 2) to exist on the account.
  Rewire New-CosmosLiveTestAccounts.ps1 to consume regionDefaults as the single
  source of truth. Remove a stale gsi-single-session preferredLocations landmine.

- Isolate the partition-split-forcing profiles (-Pcfp-split, -Psplit) onto
  dedicated accounts (single-session-split, single-strong-split,
  multimaster-multiregion-session-split) so their split churn no longer
  contends with the shared query/fast/direct legs on the same account.

- Harden ReadFeedOffersTest and ReadFeedDatabasesTest so the containment
  assertion cannot silently degrade to a no-op on an empty expected set.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3

tvaron3 commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

Live tests now share fixed accounts across concurrent CI legs, so
database/container CRUD can hit control-plane throttling (metadata 429 /
substatus 3200) and 408 timeouts that the default client retry (9/30s)
does not absorb. Add TestSuiteBase.executeControlPlaneWithRetry, which
retries only transient control-plane failures (408/429/500/503) and
rethrows everything else immediately so negative-path assertions (400/409)
are preserved. Wrap the createContainer calls in
MultiMasterConflictResolutionTest and the createDatabaseIfNotExists call
in CosmosTracerTest (with a tracer reset between attempts so a throttled
attempt's spans do not leak into the trace assertions).

The default SDK client throttling behavior is intentionally left unchanged
so tests that assert on retry counts (MaxRetryCountTests, the
fault-injection availability-strategy tests) are unaffected.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3

tvaron3 commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

These change-feed tests are known-flaky (see Azure#48064, Azure#45216); running on
shared fixed accounts surfaces transient 408/429/503 more often during
forced partition splits and change-feed setup.

- Broaden SplitTestsRetryAnalyzer to retry not only SplitTimeoutException
  but also a transient CosmosException (408/429/503) anywhere in the cause
  chain, so verifyLeasesOnRestart_AfterSplit tolerates the 503s a split
  legitimately produces.
- Add FlakyTestRetryAnalyzer to readFeedDocumentsStartFromCustomDateForMultiWrite_test
  (pkversion + epkversion) and ChangeFeedTest.changesFromPartitionKey_AfterInsertingNewDocuments,
  matching the precedent set for their sibling CFP tests in Azure#48064.

No production SDK code or default client behavior is changed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3

tvaron3 commented Jul 10, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant