[WIP] Thin-client endpoint probe drill-test: probe + HPK + QueryPlan + HTTP/2 ping + perf/Kusto tooling#49725
Draft
jeet1995 wants to merge 125 commits into
Draft
[WIP] Thin-client endpoint probe drill-test: probe + HPK + QueryPlan + HTTP/2 ping + perf/Kusto tooling#49725jeet1995 wants to merge 125 commits into
jeet1995 wants to merge 125 commits into
Conversation
… QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x002B) and x-ms-cosmos-query-version (0x002C) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. Previously these headers were only set as HTTP headers by QueryPlanRetriever and were lost when QueryPlan was routed through the proxy path, since ThinClientStoreModel serializes requests as RNTBD (not HTTP headers). IDs match server-side proxy definitions per ADO PR 1982503. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add testThinClientChangeFeedFullRange covering FeedRange.forFullRange() across multiple partition keys, and testThinClientChangeFeedPartitionKey covering FeedRange.forLogicalPartition with exact doc count + PK validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Documents all 59 thin client E2E tests across query (50), point operations (3), change feed (3), and stored procedures (3) with SQL, query features covered, and known account-side blockers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x00F0) and x-ms-cosmos-query-version (0x00F1) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. IDs are provisional (0x00F0, 0x00F1) — must be coordinated with server-side proxy team. See ADO PR 1982503 for the proxy-side design. Note: The design doc listed 0x002B/0x002C but those are already assigned to PartitionKey/PartitionKeyRangeId in the Java SDK. Using 0x00F0/0x00F1 to avoid ID collision until final server-side IDs are assigned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…BD instructions - Fix testGetCurrentDateTime: assert ISO 8601 format instead of exact match (gateway and proxy return slightly different timestamps) - Add DefaultAzureCredential support via COSMOS.USE_AAD_AUTH system property for accounts with disableLocalAuth=true - Add RNTBD class reference as .github/instructions/rntbd.instructions.md - Add pom.xml system properties for THINCLIENT_ENABLED, HTTP2_ENABLED, USE_AAD_AUTH - Add beforeSuiteReuse mode for degraded accounts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Switch baseline from Gateway V1 to Direct TCP to avoid JVM config interference (THINCLIENT_ENABLED/HTTP2_ENABLED affect Gateway V1) - Assert :10250 endpoint only on Gateway V2 results (not baseline) - Rename helpers: assertDirectAndThinClientMatch (was gateway) - Document seedTestData schema in Javadoc - Remove 'Expected to fail' comments (account has vector search enabled) - Clean up class/method Javadoc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…d align with ThinClientQueryE2ETest
…beClient cleanup - Configs: COSMOS.THINCLIENT_ENABLED is tri-state (null=unset default-on, TRUE=opt-in, FALSE=opt-out); consumers interpret null. Remove probe kill switch (COSMOS.THINCLIENT_PROBE_ENABLED) and probe-path config (path is fixed /connectivity-probe). - ThinClientConnectivityConfig (new): single authority for enablement + the shouldUseThinClientStoreModel routing gate, taking the nullable probe decision so null never collapses into a boolean clause at the call site. - GlobalEndpointManager.getProxyProbeDecision(): tri-state Boolean (null = no decision: probe not wired). - EndpointProbeClient: hard-code PROBE_PATH; remove cycleId correlation and unused EndpointProbeResult.reason; crispen runProbeCycle (single currentGate no-op Mono, removeIf delta). - Tests: repoint routing-gate tests to the 5-arg tri-state signature (+null/opt-in cases); remove THINCLIENT_PROBE_ENABLED usage from E2E tests; drop removed-config tests.
…-unhealthy latch and per-cycle copies - Replace volatile knownEndpoints Set + probeSucceeded CHM with one ConcurrentHashMap<URI,Boolean> (probeEndpointToProbeSuccessState): keys = current topology, value = proven/not-yet-proven. Gate is non-empty AND all TRUE. - Remove the force-unhealthy latch: an empty/null topology now empties the map via retainAll so the gate falls to false naturally (route to Gateway V1). - runProbeCycle reconciles and probes by iterating the provided Collection<URI> directly (reference only, no LinkedHashSet copies) and drops the delta/empty short-circuit fast paths; the Flux filters not-yet-proven endpoints inline. - applyCycleResult uses replace() so a region pruned by a concurrent reconcile is never resurrected. Unit suite green: 2579 tests, 0 failures.
… query Adds 7 thin-client E2E tests covering cross-partition aggregate merge (COUNT/SUM/AVG/MIN/MAX) and GROUP BY, plus two helpers that assert Direct vs thin-client value parity and verify the query genuinely fanned out (distinctPartitionKeyRangesContacted > 1), exercising the cross-partition MERGE path the single-partition aggregate tests miss. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… tests For a partial (prefix) hierarchical partition key, the thin-client / Gateway V2 path previously sent only StartEpkHash/EndEpkHash routing headers and no doc-level EPK filter, so the proxy resolved the request to the owning physical partition and returned every co-located document (e.g. 18 instead of 6). ThinClientStoreModel.wrapInHttpRequest now sets READ_FEED_KEY_TYPE= EffectivePartitionKeyRange, START_EPK and END_EPK on the request headers before RntbdRequest.from() so RntbdRequestHeaders.addStartAndEndKeys serializes the prefix EPK sub-range as the backend doc-level filter (hex string as UTF-8 bytes), mirroring the Direct/RNTBD FeedRangeEpkImpl path. StartEpkHash/EndEpkHash routing headers are retained. Adds ThinClientQueryE2ETest coverage: QueryPlan parity across sources and a hierarchical prefix half-open range test asserting thin-client matches Direct TCP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
For a partial (prefix) hierarchical (MULTI_HASH) partition key, the thin-client (Gateway V2) store model previously sent only StartEpkHash/EndEpkHash routing headers and no doc-level EPK filter. The proxy therefore resolved the request to the owning physical partition and returned every co-located document (an over-span) instead of only those matching the prefix. ThinClientStoreModel.wrapInHttpRequest now detects a prefix MULTI_HASH key and sets READ_FEED_KEY_TYPE=EffectivePartitionKeyRange, START_EPK and END_EPK on the request headers before RntbdRequest.from(), so RntbdRequestHeaders serializes the prefix EPK sub-range [hash(prefix), hash(prefix) + "FF") as the backend doc-level filter (hex string as UTF-8 bytes), mirroring the Direct/RNTBD FeedRangeEpkImpl path. StartEpkHash/EndEpkHash routing headers are retained. QueryPlan requests and full (non-prefix) keys are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Ports ThinClientQueryE2ETest and its ThinClientTestBase helper. The suite self-baselines every query against a Direct (RNTBD) client and the thin-client (:10250 HTTP/2) path, asserting identical results. This directly guards the MULTI_HASH prefix partition-key EPK over-span fix: prefix (single-component) hierarchical-partition-key queries must return only the narrow matching set, matching Direct, rather than over-spanning to all co-located documents. Both files are runtime self-enabling (enableThinClientForTest() + builder-driven HTTP/2 via setEnabled(true)); no module property or TestSuiteBase changes needed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The earlier fix (499f928) only scoped the prefix MULTI_HASH read when the request carried a non-null partition key. The parallel prefix-query path intentionally nulls the partial hierarchical partition key and instead carries the narrow prefix effective-partition-key range on the request's feedRange, so that guard never fired there and the thin-client read still over-spanned to the whole physical partition. Mark such requests (prefixPartitionKeyQuery) in ParallelDocumentQueryExecutionContextBase and, in ThinClientStoreModel, reuse the already-computed FeedRangeEpkImpl range as the doc-level EPK filter (START_EPK/END_EPK + StartEpkHash/EndEpkHash) instead of recomputing getEPKRangeForPrefixPartitionKey from a partition key we no longer have. This mirrors the Direct/GatewayV1 decision points and honors DRY/SRP. Also relax assertThinClientEndpointUsed so an all-QueryPlan diagnostics context passes: in thin-client mode QueryPlan calls resolve via the classic gateway and are the only requests permitted on a non-thin-client endpoint; any non-QueryPlan (data) request on a non-thin-client endpoint still fails the assertion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ag on clone, add readAllItems/readMany prefix HPK tests ThinClientStoreModel.wrapInHttpRequest now sets all five EPK-specific RNTBD headers (ReadFeedKeyType, StartEpk/EndEpk, StartEpkHash/EndEpkHash) in a single block directly on the RNTBD request, replacing the split request-header-map + post-construction approach. Wire encoding is unchanged (hex-string UTF-8 bytes for StartEpk/EndEpk, decoded hash bytes for the hash headers). RxDocumentServiceRequest.clone() now copies prefixPartitionKeyQuery so hedged/availability-strategy/retry clones keep the prefix signal and do not over-span. ThinClientQueryE2ETest adds prefix-HPK regression coverage for readAllItems (ReadFeed) and readManyByPartitionKeys, asserting Direct-vs-thin-client parity and per-tenant scoping (no over-span).
…onContextBase Assign isPrefixPartitionKeyQuery directly from isPartialPartitionKeyQuery(...) and guard only the PARTITION_KEY-setting branch with !isPrefixPartitionKeyQuery, removing the if/else. Behavior is unchanged: a partial (prefix) key still leaves partitionKeyInternal null so the prefix FeedRangeEpkImpl survives createDocumentServiceRequestWithFeedRange.
…cutionContextBase Keep main's single 'if (!isPartialPartitionKeyQuery)' structure; only capture the predicate in isPrefixPartitionKeyQuery and pass it to request.setPrefixPartitionKeyQuery. Removes the if/else.
In ThinClientStoreModel.wrapInHttpRequest, check the cached prefix feed-range path (isPrefixPartitionKeyQuery + FeedRangeEpkImpl) first and fall back to recomputing from the partition key only for the PK-bearing prefix case. Behavior is unchanged (the two cases are mutually exclusive).
… assertion TestSuiteBase.assertThinClientEndpointUsed now validates every request instead of early-returning on the first thin-client match, so a mixed scenario (some data requests via the classic gateway) fails; QueryPlan requests remain exempt and null endpoints are handled. ThinClientTestBase.assertThinClientEndpointUsed delegates to the shared TestSuiteBase implementation, removing duplicated logic that could NPE on a null endpoint and reject valid QueryPlan-via-gateway scenarios.
Replace the explicit MULTI_HASH prefix-detection and pre-serialization range computation with a header-presence branch: when the request already carries START_EPK/END_EPK HTTP headers (populated upstream by FeedRangeEpkImpl for prefix hierarchical partition keys and explicit EPK-range reads), set only the additive StartEpkHash/EndEpkHash RNTBD tokens. The ReadFeedKeyType/StartEpk/ EndEpk string tokens are copied verbatim from those HTTP headers by RntbdRequestHeaders#addStartAndEndKeys during RntbdRequest.from(), so they no longer need to be set here. Removes now-unused imports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Following the simplification of the prefix EPK fix to reuse the existing StartEpk/EndEpk HTTP headers (a56db9a), the RxDocumentServiceRequest prefixPartitionKeyQuery flag is no longer read anywhere. Remove the field, its accessors, the clone copy, and the caller assignment in ParallelDocumentQueryExecutionContextBase. Test changes: - Retrofit CosmosMultiHashTest into the thinclient TestNG group so the MULTI_HASH prefix over-span coverage also runs against GatewayV2 (proxy :10250) over HTTP/2, not just the emulator gateway. - ThinClientQueryE2ETest: correct the prefix-scope Javadoc to describe the actual header-copy behavior and add a deterministic prefix-partition-key readAllItems regression. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The prefix-query flag on RxDocumentServiceRequest was removed in 03a7c67 after the fix was simplified to reuse the StartEpk/EndEpk HTTP headers. The only functional edit in this file (the setter call) went away with it, leaving behind unnecessary refactor noise. Restore the file to its upstream/main state so the hotfix touches no query execution code. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ort' into thin-client-probe-flow # Conflicts: # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosNotFoundTests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ConfigsTests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/EndpointProbeClientTests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ReadManyByPartitionKeyQueryPlanRoutingTest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ThinClientProbeWiringTests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ThinClientRoutingGateTests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/query/QueryPlanRetrieverSupportedFeaturesTest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/ThinClientQueryE2ETest.java # sdk/cosmos/azure-cosmos/CHANGELOG.md # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/Configs.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/EndpointProbeClient.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/GlobalEndpointManager.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxDocumentClientImpl.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/query/PartitionedQueryExecutionInfo.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/query/QueryPlanRetriever.java
…stream-activity' into thin-client-probe-flow # Conflicts: # sdk/cosmos/azure-cosmos/CHANGELOG.md # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/http/ReactorNettyClient.java
…epk-overspan' into thin-client-probe-flow # Conflicts: # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/OperationPoliciesTest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/PerPartitionCircuitBreakerE2ETests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ThinClientE2ETest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/TestSuiteBase.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/ThinClientQueryE2ETest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/ThinClientTestBase.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/ThinClientStoreModel.java
The merge of AzCosmos_GatewayV2_QueryPlanSupport (Azure#47759) dropped probe's thin-client-probe supporting code while keeping the method bodies that reference it, breaking compilation: - GlobalEndpointManager: restore imports (http.HttpClient, java.util.Set) and the thinClientProbeClient AtomicReference field. - LocationCache: restore getThinClientRegionalEndpoints() and the collectThinClientEndpointsAllOrNothing() helper. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…retry The @BeforeSuite container-readiness probe issues a QueryPlan against a freshly created container. With thin-client default-on, a fresh container in a multi-region account can transiently return 404/subStatus 1003 (OWNER_RESOURCE_NOT_EXISTS) during regional metadata propagation. The classifier did not allowlist 1003, so the probe treated it as non-retryable and cascade-skipped the entire thinclient suite. Add OWNER_RESOURCE_NOT_EXISTS to the NOTFOUND retryable allowlist in isRetryableCollectionReadinessFailure so the probe retries through propagation and the suite runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…Flavor cold-start knobs Instrument azure-cosmos-benchmark for the cold-start Create-latency matrix driving the thin-client probe (PR Azure#49437): - pom: parameterize azure-cosmos + azure-cosmos-encryption versions (-Dazure-cosmos.version=...) so GA 4.79/4.80/4.81 and the probe build can be benchmarked from one harness. - Common tags (RunId, SdkVersion, ConnectionMode, EndpointFlavor, Phase) applied to every metric so a shared Grafana dashboard can slice runs. - endpointFlavor knob (ComputeGateway|ThinClient) maps to COSMOS.THINCLIENT_ENABLED at client build to drill both the Gateway V1 and Gateway V2 (thin-client) HTTP/2 paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add two opt-in Micrometer meters (gated on COSMOS.HTTP2_STREAM_METRICS_ENABLED): - cosmos_client_http2_connectionsOpened (Counter) - cosmos_client_http2_requestsPerConnection (DistributionSummary) Recorded in ReactorNettyClient via doOnConnected / closeFuture; requests-per- connection uses a deferred-drain queue drained on the CpuMemoryMonitor 5s tick so counts survive per-cycle registry teardown. Used to diagnose the cold-start HTTP/2 connection burst on the thin-client / Gateway-V2 Create path. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add an opt-in KUSTO reporting destination that streams all benchmark metrics with full dimensions directly into an Azure Data Explorer table via the streaming-ingest REST endpoint (bearer auth, no extra Maven dependency). Run-level common tags (RunId/SdkVersion/ConnectionMode/ EndpointFlavor/Phase) map straight to first-class BenchmarkMetrics columns for downstream Grafana dashboards. - KustoMetricsReporter: streaming-ingest reporter over a SimpleMeterRegistry - KustoReporterConfig: JSON config (cluster/db/table/auth) - ReportingDestination.KUSTO + BenchmarkConfig/Orchestrator wiring - Guard null tenantId in applicationName derivation for multi-cycle runs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Introduces best-effort eager warm-up of the shared thin-client HTTP/2 connection pool at client-build time so the first data-plane request avoids the cold TLS+HTTP/2 handshake burst. Adds COSMOS.HTTP2_EAGER_CONNECTION_WARMUP_ENABLED (off by default) and a bounded timeout config, an EndpointProbeClient.warmUpConnections method that never fails client init, and wires it into RxDocumentClientImpl/GlobalEndpointManager. Includes unit tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This is the consolidated branch that folds the thin-client endpoint-probe work together with its dependencies plus the perf benchmark tooling used to drill-test cold start. It corresponds to the endpoint-probe line of work (PR #49437) with the following merged in:
Benchmark / perf tooling (on top of the SDK fixes)
test(cosmos-benchmark): parameterize SDK version + add RunId/EndpointFlavor cold-start knobsperf(cosmos): add opt-in HTTP/2 connection instrumentation metricstest(cosmos-benchmark): add Azure Data Explorer (Kusto) metrics sink— streams benchmark metrics (withRunId/SdkVersion/ConnectionMode/EndpointFlavor/Phasedimensions) into ADX for the perf Grafana dashboards.Notes
main(three-dot).