-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[PLAT-16134][PLAT-16253] YCQL microseconds precision alert + remote b…
…ootstrap alert enhancement
Summary:
With default gflags YCQL operations with microsecond precision may cause issues with user workload, as inserts will support microseconds, which selects won't.
Special gflag wa introduced recently to address this issue, but we need to make sure no microsecond precision operations are executed before using this gFlag.
Adding an alert to make sure user is aware of any microsecond precision operations on the cluster.
Also, current increased remote bootstraps alert is using absolute number of bootstraps for threshold, and default value is 0.
It does not work typically as number of bootstraps highly depend on total number of tablets, etc, and few bootstraps running is not a big deal.
Made a change to use remote bootstrap count over 10 minutes percentage compared to total number of tablets and uses 10% as default threshold..
Test Plan:
Upgrade YBA.
Run the following YCQL operations:
CREATE TABLE test (
id int PRIMARY KEY,
created_date timestamp
) WITH default_time_to_live = 0
AND transactions = {'enabled': 'true'};
INSERT INTO test(id, created_date) VALUES (1, currenttimestamp()) ;
INSERT INTO test(id, created_date) VALUES (2, '2024-09-25 15:52:40.768819+0000') ;
Make sure YCQL operations with microsecond precision alert is raised and has a link to https://docs.yugabyte.com/preview/releases/techadvisories/ta-23476/.
Wait for around 10 minutes. Make sure alert is resolved.
Create Increase in remote bootstraps alert policy.
Make sure alert expression is valid.
Reviewers: skurapati, #yba-api-review!
Reviewed By: skurapati
Subscribers: sanketh, yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D40716- Loading branch information
1 parent
2c5c8bd
commit e5a8422
Showing
7 changed files
with
87 additions
and
20 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 2 additions & 0 deletions
2
managed/src/main/resources/db/migration/default_/common/V385__Add_Minute_Threshold_Unit.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| alter table alert_configuration drop constraint ck_ac_threshold_unit; | ||
| alter table alert_configuration add constraint ck_ac_threshold_unit check (threshold_unit in ('STATUS','COUNT','PERCENT','MILLISECOND','SECOND','MINUTE','DAY','MEGABYTE')); |
29 changes: 29 additions & 0 deletions
29
...migration/default_/postgres/V386__Create_YCQL_Microsecond_And_Update_Bootstraps_Alert.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| update alert_configuration | ||
| set | ||
| threshold_unit = 'PERCENT', | ||
| thresholds = '{"WARNING":{"condition":"GREATER_THAN","threshold":10.0}}' | ||
| where template = 'INCREASED_REMOTE_BOOTSTRAPS'; | ||
|
|
||
| update alert_definition set config_written = false where configuration_uuid IN | ||
| (select uuid from alert_configuration where template = 'INCREASED_REMOTE_BOOTSTRAPS'); | ||
|
|
||
| -- YCQL microseconds precision inserts detected alert | ||
| insert into alert_configuration | ||
| (uuid, customer_uuid, name, description, create_time, target_type, target, thresholds, threshold_unit, template, duration_sec, active, default_destination) | ||
| select | ||
| gen_random_uuid(), | ||
| uuid, | ||
| 'YCQL inserts with microseconds precision', | ||
| 'YCQL inserts with microseconds precision detected, which is not fully supported.', | ||
| current_timestamp, | ||
| 'UNIVERSE', | ||
| '{"all":true}', | ||
| '{"WARNING":{"condition":"GREATER_THAN","threshold":60.0}}', | ||
| 'MINUTE', | ||
| 'YCQL_MICROSECOND_TIMESTAMPS_DETECTED', | ||
| 0, | ||
| true, | ||
| true | ||
| from customer; | ||
|
|
||
| select create_universe_alert_definitions('YCQL inserts with microseconds precision'); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.