[Feature](Schema Change)light schema change support add key column #42911
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
clang-tidy review says "All clean, LGTM! 👍" |
mongo360
force-pushed
the
light_schema_change_support_add_key_column
branch
2 times, most recently
from
0fe8150 to
9a99b34
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
mongo360
force-pushed
the
light_schema_change_support_add_key_column
branch
from
9a99b34 to
72f1191
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
be/src/vec/olap/block_reader.cpp
Outdated
| @@ -15,8 +15,6 @@ | |||
| // specific language governing permissions and limitations | |||
| // under the License. | |||
|
|
|||
| #include "vec/olap/block_reader.h" | |||
|
|
|||
| #include <gen_cpp/olap_file.pb.h> | |||
There was a problem hiding this comment.
warning: 'gen_cpp/olap_file.pb.h' file not found [clang-diagnostic-error]
#include <gen_cpp/olap_file.pb.h>
^
mongo360
force-pushed
the
light_schema_change_support_add_key_column
branch
from
72f1191 to
98197ee
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 41363 ms |
TPC-DS: Total hot run time: 194832 ms |
ClickBench: Total hot run time: 32.08 s |
mongo360
force-pushed
the
light_schema_change_support_add_key_column
branch
from
ec7d59a to
79d0dc6
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
mongo360
force-pushed
the
light_schema_change_support_add_key_column
branch
from
79d0dc6 to
4c5aefd
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
mongo360
force-pushed
the
light_schema_change_support_add_key_column
branch
from
4c5aefd to
fbdeccd
Compare
|
run buildall |
TPC-H: Total hot run time: 32617 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 196644 ms |
ClickBench: Total hot run time: 31.8 s |
mongo360
force-pushed
the
light_schema_change_support_add_key_column
branch
from
fbdeccd to
324eed6
Compare
|
run buildall |
mongo360
force-pushed
the
light_schema_change_support_add_key_column
branch
2 times, most recently
from
0ab8b9a to
a7be770
Compare
|
run buildall |
TPC-H: Total hot run time: 32512 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 197498 ms |
ClickBench: Total hot run time: 30.93 s |
yiguolei
reviewed
Dec 31, 2024
| ALTER TABLE add_keys_light_schema_change ADD COLUMN new_key_column INT default "2" AFTER user_id PROPERTIES ("timeout"="604800") | ||
| """ | ||
|
|
||
| def jobStateResult = waitJobFinish(""" SHOW ALTER TABLE COLUMN WHERE IndexName='add_keys_light_schema_change' ORDER BY createtime DESC LIMIT 1 """, 9) |
There was a problem hiding this comment.
Do not need wait, because light weight schema change is sync process. Not need background process.
There was a problem hiding this comment.
jobStateResult is used to check schema change is use light schema change or not.
yiguolei
reviewed
Dec 31, 2024
| `sex` TINYINT COMMENT "用户性别", | ||
|
|
||
| `cost` BIGINT DEFAULT "0" COMMENT "用户总消费") | ||
| UNIQUE KEY(`user_id`, `date`, `city`, `age`, `sex`) DISTRIBUTED BY HASH(`user_id`) |
There was a problem hiding this comment.
also need test unique key merge on read
There was a problem hiding this comment.
👌🏻，i will add case 4.1 for this scene
yiguolei
reviewed
Dec 31, 2024
| BUCKETS 4 | ||
| PROPERTIES ( | ||
| "replication_num" = "1", | ||
| "light_schema_change" = "true", |
There was a problem hiding this comment.
all test cases should disable compaction
yiguolei
reviewed
Dec 31, 2024
| INSERT INTO add_keys_light_schema_change VALUES | ||
| (1, '2017-10-01', 'Beijing', 10, 1, 20) | ||
| """ | ||
| qt_sc_12 """ select * from add_keys_light_schema_change order by user_id """ |
There was a problem hiding this comment.
The test case should follow this process:
- insert values
- add key column with default value
- insert values
- check select value
yiguolei
reviewed
Dec 31, 2024
| qt_sc_12 """ select * from add_keys_light_schema_change order by user_id """ | ||
|
|
||
| sql """ | ||
| ALTER TABLE add_keys_light_schema_change ADD COLUMN new_mv_key1 INT KEY default "2" |
There was a problem hiding this comment.
should also test delete condition:
- insert values
- add key column with default value
- delete where new keycolumn = xxx
- insert values
- check select value
yiguolei
reviewed
Dec 31, 2024
| qt_sc_12 """ select * from add_keys_light_schema_change order by user_id """ | ||
|
|
||
| sql """ | ||
| ALTER TABLE add_keys_light_schema_change ADD COLUMN new_mv_key1 INT KEY default "2" |
There was a problem hiding this comment.
If the org table is key(col1,col2)
you add col3, the new key is (col1,col2,col3) or (col3,col1,col2) or (col1,col3,col2) ?
I think you should test all these cases.
yiguolei
reviewed
Dec 31, 2024
| sql """ DROP TABLE IF EXISTS add_keys_light_schema_change """ | ||
| sql """ | ||
| CREATE TABLE IF NOT EXISTS add_keys_light_schema_change ( | ||
| `user_id` LARGEINT NOT NULL COMMENT "用户id", |
There was a problem hiding this comment.
should test all typed key columns such as string,date,int.
For example, the original key column could be one of string,date,int
and you add a new key column, it could also be one of string,date,int
yiguolei
reviewed
Dec 31, 2024
| jobStateResult = waitJobFinish(""" SHOW ALTER TABLE COLUMN WHERE IndexName='add_keys_light_schema_change' ORDER BY createtime DESC LIMIT 1 """, 9) | ||
| assertNotEquals(jobStateResult[0][8], "-1") | ||
|
|
||
| // case 2.1: light schema change aggreage : with multi version rowset and compaction |
There was a problem hiding this comment.
Should check these query values:
- select * from new table
- select * from newtable where key=xxx
yiguolei
reviewed
Dec 31, 2024
| """ | ||
| table_tablets = sql """ SHOW TABLETS FROM add_keys_light_schema_change ORDER BY RowCount DESC LIMIT 1 """ | ||
|
|
||
| qt_41_unique_multi_rowset """ SELECT * FROM add_keys_light_schema_change; """ |
There was a problem hiding this comment.
For all test cases, you should check the compaction logic:
- first disable compaction and run all cases.
- call compaction API to do compaction for the table, and then run all test cases.
mongo360
force-pushed
the
light_schema_change_support_add_key_column
branch
from
a7be770 to
23f7672
Compare
|
run buildall |
TPC-H: Total hot run time: 32354 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 196160 ms |
ClickBench: Total hot run time: 30.82 s |
mongo360
force-pushed
the
light_schema_change_support_add_key_column
branch
from
23f7672 to
2c7af74
Compare
|
run buildall |
TPC-H: Total hot run time: 32826 ms |
TPC-DS: Total hot run time: 197143 ms |
ClickBench: Total hot run time: 31.48 s |
|
TeamCity be ut coverage result: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Background
Currently, the LightSchemaChange feature in Doris only supports adding value columns, not supports the regular key column adding. If we add a regular key column to a simple table that not in the short key columns, it will do with HardLinkedSchemaChange; But for the cooldown tablets that save on remote system like S3, it will degrade to DirectSchemaChange, The schema change time is very slow. This modification is intended to replace HardLinkedSchemaChange with LightSchemaChange for normal key column adding.
Modification
Modify LightSchemaChange check logic in FE, adding key column adding scene.