[cosmos] Use fixed self-owned accounts for key-based live tests#49735
Draft
tvaron3 wants to merge 9 commits into
Draft
[cosmos] Use fixed self-owned accounts for key-based live tests#49735tvaron3 wants to merge 9 commits into
tvaron3 wants to merge 9 commits into
Conversation
Point the main Cosmos live-test stage and the Http2 stage at fixed,
self-owned Cosmos accounts (RG sdk-ci) instead of provisioning an account
per pipeline run. This removes the central EngSys service-connection /
on-the-fly-provisioning dependency for the key-based tests.
- Add sdk/cosmos/pipeline/ with a single-secret JSON mechanism:
- live-test-accounts.schema.json + sample
- resolve-cosmos-test-account.sh (bash+jq) resolves one account from the
single secret (sub-config-cosmos-azure-cloud-test-resources) by selector
and exports ACCOUNT_HOST/ACCOUNT_KEY(+SECONDARY) using the azure-sdk
double-set convention so the key is masked in logs yet reaches the test env
- resolve-cosmos-test-account.tests.sh (local tests)
- resolve-test-account-steps.yml (reusable PreTestRunSteps template)
- tests.yml: Cosmos_live_test + Cosmos_Live_Test_Http2 set
DisableAzureResourceCreation:true and run the resolve pre-step.
- live-platform-matrix.json: add per-leg AccountSelector.
Thin-client, GSI, Kafka and Spring stages are intentionally left on the
existing provisioning path (they need feature-enabled accounts or a separate
AAD pipeline) and will be migrated in follow-ups.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/azp run java - cosmos - tests |
|
Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command. |
Member
Author
|
/azp run java - cosmos - tests |
|
Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command. |
…test-sub-tenant # Conflicts: # sdk/cosmos/live-platform-matrix.json
Member
Author
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…count-safe
Follow-up to the fixed-account live-test wiring: bring the account
provisioning tooling into the repo and fix tests that assumed each run
gets its own freshly-provisioned account.
- Add sdk/cosmos/pipeline/account-provisioning/ (moved from the
cosmos-sdk-copilot-toolkit repo):
- New-CosmosLiveTestAccounts.ps1 creates the fixed sdk-ci accounts and
outputs the cosmos-live-test-accounts JSON; reconciles capabilities on
existing accounts (EnableNoSQLVectorSearch) via ARM PATCH.
- Set-CosmosLiveTestAccountsSecret.ps1 publishes that JSON to the secret.
- cosmos-live-test-accounts.definition.json defines the accounts; the 5
general-purpose accounts enable EnableNoSQLVectorSearch to match the
ARM test-resources.json capability set.
- ReadFeedDatabasesTest: readAllDatabases is account-global; on shared
fixed accounts other runs create/delete DBs concurrently. Assert only
that the databases created by this test are present (containment) via a
new FeedResponseListValidator.containsResourceIds, instead of asserting
the exact account-wide set/count/page-count.
Verified locally: ReadFeedDatabasesTest failed 3/3 under concurrent DB
churn before and passes 3/3 after; NonStreamingOrderByQueryVectorSearchTest
passes 4/4 once EnableNoSQLVectorSearch is enabled on the account.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Build Analyze (cspell) stage flagged domain/code tokens in the new sdk/cosmos/pipeline files (sdkci, pmerge, Pfast, issecret, esac, WHATIF). Add a scoped cspell override for sdk/cosmos/pipeline/** so these expected words don't fail spellcheck, without touching the global word list. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…/pipeline - Remove Set-CosmosLiveTestAccountsSecret.ps1 and its references; the Key Vault / ADO variable secret is now updated manually with the JSON that New-CosmosLiveTestAccounts.ps1 outputs. Docs updated accordingly (removed the KV-publish usage, the dated "Verified run" note, and the macOS auth note). - Move the cspell allow-list for pipeline domain/code tokens (sdkci, pmerge, Pfast, issecret, esac, WHATIF) out of the root .vscode/cspell.json into a nested sdk/cosmos/pipeline/cspell.json, which cspell merges automatically. Keeps the change scoped to files under sdk/cosmos. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
readOffers is an account-global read. With live tests running against shared fixed accounts, concurrent legs create/delete containers (and their offers), so asserting the exact account-wide offer set/count/page count is non-deterministic. Assert containment of only the offers for containers created by this test instead. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Member
Author
|
/azp run java - cosmos - spark |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Member
Author
|
/azp run java - cosmos - kafka |
|
Azure Pipelines successfully started running 1 pipeline(s). |
… guards - Align fixed-account regions with the ARM bicep (test-resources.json): regionDefaults -> Central US (single), Central US + East US 2 (multi). This fixes the fault-injection availability-strategy tests whose region assertions require the preferred region (East US 2) to exist on the account. Rewire New-CosmosLiveTestAccounts.ps1 to consume regionDefaults as the single source of truth. Remove a stale gsi-single-session preferredLocations landmine. - Isolate the partition-split-forcing profiles (-Pcfp-split, -Psplit) onto dedicated accounts (single-session-split, single-strong-split, multimaster-multiregion-session-split) so their split churn no longer contends with the shared query/fast/direct legs on the same account. - Harden ReadFeedOffersTest and ReadFeedDatabasesTest so the containment assertion cannot silently degrade to a no-op on an empty expected set. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Live tests now share fixed accounts across concurrent CI legs, so database/container CRUD can hit control-plane throttling (metadata 429 / substatus 3200) and 408 timeouts that the default client retry (9/30s) does not absorb. Add TestSuiteBase.executeControlPlaneWithRetry, which retries only transient control-plane failures (408/429/500/503) and rethrows everything else immediately so negative-path assertions (400/409) are preserved. Wrap the createContainer calls in MultiMasterConflictResolutionTest and the createDatabaseIfNotExists call in CosmosTracerTest (with a tracer reset between attempts so a throttled attempt's spans do not leak into the trace assertions). The default SDK client throttling behavior is intentionally left unchanged so tests that assert on retry counts (MaxRetryCountTests, the fault-injection availability-strategy tests) are unaffected. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
These change-feed tests are known-flaky (see Azure#48064, Azure#45216); running on shared fixed accounts surfaces transient 408/429/503 more often during forced partition splits and change-feed setup. - Broaden SplitTestsRetryAnalyzer to retry not only SplitTimeoutException but also a transient CosmosException (408/429/503) anywhere in the cause chain, so verifyLeasesOnRestart_AfterSplit tolerates the 503s a split legitimately produces. - Add FlakyTestRetryAnalyzer to readFeedDocumentsStartFromCustomDateForMultiWrite_test (pkversion + epkversion) and ChangeFeedTest.changesFromPartitionKey_AfterInsertingNewDocuments, matching the precedent set for their sibling CFP tests in Azure#48064. No production SDK code or default client behavior is changed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Point the key-based Cosmos live tests at fixed, self-owned Cosmos accounts (resource group
sdk-ci) instead of provisioning a new account per pipeline run. This removes the central EngSys service-connection / on-the-fly-provisioning dependency for these tests, which is required so the tests can run against accounts in an ephemeral tenant that is recreated every ~90 days.This PR wires up two stages as the first step: the main
Cosmos_live_teststage andCosmos_Live_Test_Http2.What changed
sdk/cosmos/pipeline/mechanism — a single ADO variable-group secret (sub-config-cosmos-azure-cloud-test-resources) holds a versioned JSON mapping a logical account name →{ endpoint, key, secondaryKey, ... }:live-test-accounts.schema.json+live-test-accounts.sample.jsonresolve-cosmos-test-account.sh(bash + jq): resolves one account by selector and exportsACCOUNT_HOST/ACCOUNT_KEY(+SECONDARY_ACCOUNT_KEY). The key uses the azure-sdk double-set convention (_ACCOUNT_KEYsecret to mask it in logs + a plainACCOUNT_KEYso it propagates to the Maven test env). It does not emitACCOUNT_CONSISTENCY/PREFERRED_LOCATIONS— those stay matrix-controlled.resolve-cosmos-test-account.tests.sh(8 local tests, all pass)resolve-test-account-steps.yml(reusablePreTestRunStepstemplate)README.mdtests.yml—Cosmos_live_testandCosmos_Live_Test_Http2setDisableAzureResourceCreation: trueand run the resolve pre-step.live-platform-matrix.json— each leg gets anAccountSelectorso the multi-config main stage picks the right fixed account per leg.Not included (follow-ups)
Left on the existing provisioning path because they need feature-enabled accounts or a separate pipeline:
AZURE_SPRING_TENANT_ID).The companion account-provisioning + Key Vault scripts live in the
cosmos-sdk-copilot-toolkitrepo (separate PR).Verification
resolve-cosmos-test-account.sh: 8/8 local unit tests pass; YAML/JSON validated.secondaryKey(required byAzureKeyCredentialTestin the-Pfastlegs).Draft
Draft while the two wired stages are validated end-to-end in a real pipeline run.