sql: introduce canary stats API #156307
Draft
sql: introduce canary stats API #156307
+397
−156
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
ZhouXing19
force-pushed
the
canary-settings
branch
3 times, most recently
from
cb7438c to
ab4f8de
Compare
ZhouXing19
force-pushed
the
canary-settings
branch
from
ab4f8de to
af0a3f6
Compare
This commit introduces a new table-level storage parameter `canary_window` that can be specified in CREATE TABLE or ALTER TABLE statements. The canary window specifies how long newly collected statistics will remain in "canary" state before being promoted to stable, enabling gradual rollout of new statistics to mitigate performance regressions. When set to a non-zero duration, this parameter enables canary statistics rollout for the table. During the canary window, the cluster setting sql.stats.canary_fraction (introduced in the next commit) determines what percentage of queries use the new canary statistics versus the previous stable statistics, providing a buffer period for observation and intervention. The window is capped by 48 hours to avoid an outrageously long canary window. Note that this commit adds only syntax support and storage for the parameter. The actual canary statistics selection logic will be implemented in subsequent commits. Release note (sql change): A new table storage parameter `canary_window` has been introduced to enable gradual rollout of newly collected table statistics. It takes a duration string as the value, with maximum allowed duration 48 hours. When set with a non-negative duration, the new statistics remain in a "canary" state for the specified duration before being promoted to stable. This allows for controlled exposure and intervention opportunities before statistics are fully deployed across all queries.
See release note for details. Note that this cluster setting doesn't apply to internal queries. Release note (sql change): introduce the cluster setting `sql.stats.canary_fraction` which takes a float number within range [0, 1]. Its value determines what fraction of queries will use "canary statistics" (newly collected stats within their canary window) versus "stable statistics" (previously proven stats). For example, a value of 0.2 means 20% of queries will test canary stats while 80% use stable stats. The selection is atomic per query: if a query is chosen for canary evaluation, it will use canary statistics for ALL tables it references (where available). A query never uses a mix of canary and stable statistics.
ZhouXing19
force-pushed
the
canary-settings
branch
from
af0a3f6 to
39e50de
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Informs: #150015
This PR introduce 2 key configurations for the Canary Statistics Rollout feature. Note that this PR just to introduce the configuration settings. The core implementation for canary stats rollout will be in #156385.
Storage parameter
canary_window(duration)This duration value determines specifies how long the newly collected statistics will be eligible for selection along
with the most recent full statistics for the optimizer. It is needed for the canary statistics rollout feature. Only tables with a non-zero canary window will have canary statistics rollout enabled.
This setting is capped at a duration of 48 hours or less to prevent excessively long canary windows.
Release note (sql change): A new table storage parameter
canary_windowhas been introduced to enable gradual rollout of newly collected table statistics. It takes a duration string as the value, capped with max duration 48 hours. When set with a non-negative duration, the new statistics remain in a "canary" state for the specified duration before being promoted to stable. This allows for controlled exposure and intervention opportunities before statistics are fully deployed across all queries.Cluster setting
sql.stats.canary_fraction(float in [0 - 1])This
canaryFractioncontrols the probabilistic sampling rate for queries participating in the canary statistics rollout feature.It determines what fraction of queries will use "canary statistics" (newly collected stats within their canary window) versus "stable statistics" (previously proven stats).
For example, a value of 0.2 means 20% of queries will test canary stats while 80% use stable stats.
The selection is atomic per query: if a query is chosen for canary evaluation, it will use canary statistics for ALL tables it references (where available). A query never uses a mix of canary and stable statistics.
Release note (sql change): introduce the cluster setting
sql.stats.canary_fractionwhich takes a float number within range [0, 1]. Its value determines what fraction of queries will use "canary statistics" (newly collected stats within their canary window) versus "stable statistics" (previously proven stats). For example, a value of 0.2 means 20% of queries will test canary stats while 80% use stable stats. The selection is atomic per query: if a query is chosen for canary evaluation, it will use canary statistics for ALL tables it references (where available). A query never uses a mix of canary and stablestatistics.