feat(shard-distributor): Add canary pinger for periodic shard ownership verification #7487
+1,732
−32
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously GetShardOwner only returned the executor ID string. Now it returns an ExecutorOwnership struct containing both the executor ID and the full metadata map. This allows callers to access additional executor information like the gRPC address needed for peer routing. Changes: - Add ExecutorOwnership struct with ExecutorID and Metadata fields - Update GetShardOwner signature to return *ExecutorOwnership - Update spectatorImpl to build and return ownership info - Update tests to verify metadata is included in responses Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Use BlockUntil() to properly synchronize with the fake clock before advancing time. This prevents the test from hanging while waiting for the goroutine to process the time advancement. Signed-off-by: Jakob Haahr Taankvist <[email protected]>
jakobht
requested review from
3vilhamster,
Shaddoll,
davidporter-id-au,
demirkayaender,
dkrotx,
neil-xie,
sankari165,
shijiesheng and
taylanisikdemir
as code owners
jakobht
changed the title
Encapsulate the spectators map in a struct to hide implementation details and make it easier to extend in the future without breaking the API. Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Implement a YARPC peer chooser that routes requests to the correct
executor based on shard ownership. This is the shard distributor
equivalent of Cadence's RingpopPeerChooser.
Flow:
1. Client calls RPC with yarpc.WithShardKey("shard-key")
2. Chooser queries Spectator for shard owner
3. Extracts grpc_address from owner metadata
4. Creates/reuses peer for that address
5. Returns peer to YARPC for connection
The peer chooser maintains a cache of peers and handles concurrent
access safely. It uses the x-shard-distributor-namespace header to
determine which namespace's spectator to query.
Dependencies:
- Requires spectator GetShardOwner to return metadata (see previous commit)
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
The tests have dependency issues with mock generation that need to be resolved separately. The peer chooser implementation is complete and functional. Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Tests cover: - Success path with peer creation - Peer reuse on subsequent calls - Error cases (missing shard key, namespace header, spectator not found) - Lifecycle methods (Start, Stop, IsRunning) - SetSpectators method Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Define ShardDistributorExecutorCanaryAPI service that allows one executor to ping another executor that owns a specific shard. This is used for canary testing to verify that: 1. Executors can route to each other based on shard ownership 2. Shard ownership information is accurate 3. The shard distributor is functioning correctly The Ping RPC takes a shard key and namespace, and returns whether the receiving executor owns that shard along with the executor ID. Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Implement the client-side pinger that periodically pings random shard owners to verify: 1. Executors can route to each other based on shard ownership 2. Shard ownership information is accurate 3. The shard distributor is functioning correctly The pinger: - Selects random shards at regular intervals (1s with 10% jitter) - Sends ping requests to the executor owning each shard - Validates that the receiving executor actually owns the shard - Logs warnings when ownership is incorrect Dependencies: - Requires ShardDistributorExecutorCanaryAPI proto and client - Will use SpectatorPeerChooser for routing (wired in later commit) Signed-off-by: Jakob Haahr Taankvist <[email protected]>
jakobht
force-pushed
the
add-canary-pinger-new
branch
from
34d56e5 to
ddf0d74
Compare
jakobht
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on #7475 and #7478 being merged
What changed?
Added canary pinger component that periodically sends ping requests to shard owners to verify executor-to-executor communication and shard ownership.
Why?
The canary pinger provides active monitoring of the shard distributor's routing and ownership mechanisms by:
This is part of the canary ping/pong implementation that validates end-to-end executor-to-executor gRPC communication.
How did you test it?
Unit tests
Potential risks
Release notes
Documentation Changes