sql: introduce canary full stats rollout and stats_as_of session var
#156385
+1,412
−164
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
ZhouXing19
force-pushed
the
canary-main
branch
from
22531c4 to
6c87a57
Compare
ZhouXing19
force-pushed
the
canary-main
branch
from
6c87a57 to
0d7a246
Compare
This commit introduces a new table-level storage parameter `canary_window` that can be specified in CREATE TABLE or ALTER TABLE statements. The canary window specifies how long newly collected statistics will remain in "canary" state before being promoted to stable, enabling gradual rollout of new statistics to mitigate performance regressions. When set to a non-zero duration, this parameter enables canary statistics rollout for the table. During the canary window, the cluster setting sql.stats.canary_fraction (introduced in the next commit) determines what percentage of queries use the new canary statistics versus the previous stable statistics, providing a buffer period for observation and intervention. The window is capped by 48 hours to avoid an outrageously long canary window. Note that this commit adds only syntax support and storage for the parameter. The actual canary statistics selection logic will be implemented in subsequent commits. Release note (sql change): A new table storage parameter `canary_window` has been introduced to enable gradual rollout of newly collected table statistics. It takes a duration string as the value, with maximum allowed duration 48 hours. When set with a non-negative duration, the new statistics remain in a "canary" state for the specified duration before being promoted to stable. This allows for controlled exposure and intervention opportunities before statistics are fully deployed across all queries.
See release note for details. Note that this cluster setting doesn't apply to internal queries. Release note (sql change): introduce the cluster setting `sql.stats.canary_fraction` which takes a float number within range [0, 1]. Its value determines what fraction of queries will use "canary statistics" (newly collected stats within their canary window) versus "stable statistics" (previously proven stats). For example, a value of 0.2 means 20% of queries will test canary stats while 80% use stable stats. The selection is atomic per query: if a query is chosen for canary evaluation, it will use canary statistics for ALL tables it references (where available). A query never uses a mix of canary and stable statistics.
ZhouXing19
force-pushed
the
canary-main
branch
from
0d7a246 to
9468dc3
Compare
Potential Bug(s) DetectedThe three-stage Claude Code analysis has identified potential bug(s) in this PR that may warrant investigation. Next Steps: Note: When viewing the workflow output, scroll to the bottom to find the Final Analysis Summary. After you review the findings, please tag the issue as follows:
|
github-actions
bot
added
the
o-AI-Review-Potential-Issue-Detected
…indow This commit implements the core logic for canary statistics rollout, allowing gradual deployment of newly collected full statistics. Previously, all queries would immediately use the most recent full statistics, which could cause performance regressions if the new full statistics were inaccurate. The implementation adds a `CanaryWindowSize` field in table descriptors and catalog interfaces to define the canary period, along with logic in the statistics builder to skip "canary" statistics (the latest stats within the canary window) when not using the canary path. The cluster setting `sql.stats.canary_fraction` controls what percentage of queries use canary statistics. Release note (sql change): implement canary full statistics rollout core logic, which is configurable via the table-level storage paramter (`canary_window`) and the cluster setting `sql.stats.canary_fraction`.
… selection This commit adds a new session variable `stats_as_of` that allows controlling statistics selection based on a specific timestamp rather than the current time. Previously, statistics selection was always relative to the current wall clock time, making it difficult to get consistent query plans for historical analysis or testing. This feature is only for debugging and troubleshooting, and should not be used in production. The implementation is also integrated into the existing canary statistics logic to respect the as-of timestamp when determining canary window boundaries. Release note (sql change): adds a new session variable `stats_as_of` that allows controlling statistics selection based on a specific timestamp rather than the current time.
This commit adds tests for usage of canary stats rollout in makeTableStatistics(), which is the main entry point where statistics are selected for query optimization. This commit focuses on unit testing the makeTableStatistics() function and does not include end-to-end logic tests, which would require additional changes to Builder.maybeAnnotateWithEstimates() to support EXPLAIN ANALYZE output showing which statistics were used during planning. To enable testing, this commit adds: - Handler in opttester for setting the canary window storage parameter - Testing knob for controlling the canary fraction setting - Three new test files covering basic canary stats, histogram canary stats, and multi-column canary stats scenarios Release note: None
ZhouXing19
force-pushed
the
canary-main
branch
from
9468dc3 to
86c78c8
Compare
|
Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Informs: #150015
Rebased from #156307
Release note: TBD