Skip to content

Commit a562874

Browse files
BlargianBlargian
authored
Merge pull request #3653 from ClickHouse/mysql-gtid-steps
MySQL ClickPipe: GTID steps and parameter group screenshots
2 parents f1f117e + 80bbaaa commit a562874

File tree

10 files changed

+120
-6
lines changed

10 files changed

+120
-6
lines changed

‎docs/integrations/data-ingestion/clickpipes/mysql/source/aurora.md

+58-2
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,12 @@ import rds_backups from '@site/static/images/integrations/data-ingestion/clickpi
99
import parameter_group_in_blade from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/parameter_group_in_blade.png';
1010
import security_group_in_rds_mysql from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/rds/security-group-in-rds-mysql.png';
1111
import edit_inbound_rules from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/edit_inbound_rules.png';
12+
import aurora_config from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/rds_config.png';
13+
import binlog_format from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/binlog_format.png';
14+
import binlog_row_image from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/binlog_row_image.png';
15+
import binlog_row_metadata from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/binlog_row_metadata.png';
16+
import edit_button from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/edit_button.png';
17+
import enable_gtid from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/enable_gtid.png';
1218
import Image from '@theme/IdealImage';
1319

1420
# Aurora MySQL source setup guide
@@ -33,15 +39,65 @@ mysql=> call mysql.rds_set_configuration('binlog retention hours', 24);
3339
```
3440
If this configuration isn't set, Amazon RDS purges the binary logs as soon as possible, leading to gaps in the binary logs.
3541

36-
If not already configured, make sure to set these in the parameter group:
37-
1. `binlog_format` to `ROW`
42+
## Configure binlog settings in the parameter group {#binlog-parameter-group-aurora}
43+
44+
The parameter group can be found when you click on your MySQL instance in the RDS Console, and then heading over to the `Configurations` tab.
45+
46+
<Image img={aurora_config} alt="Where to find parameter group in RDS" size="lg" border/>
47+
48+
Upon clicking on the parameter group link, you will be taken to the page for it. You will see an Edit button in the top-right.
49+
50+
<Image img={edit_button} alt="Edit parameter group" size="lg" border/>
51+
52+
The following settings need to be set as follows:
53+
54+
1. `binlog_format` to `ROW`.
55+
56+
<Image img={binlog_format} alt="Binlog format to ROW" size="lg" border/>
57+
3858
2. `binlog_row_metadata` to `FULL`
59+
60+
<Image img={binlog_row_metadata} alt="Binlog row metadata" size="lg" border/>
61+
3962
3. `binlog_row_image` to `FULL`
63+
64+
<Image img={binlog_row_image} alt="Binlog row image" size="lg" border/>
65+
66+
Then click on `Save Changes` in the top-right. You may need to reboot your instance for the changes to take effect - a way of knowing this is if you see `Pending reboot` next to the parameter group link in the Configurations tab of the RDS instance.
4067
<br/>
4168
:::tip
4269
If you have a MySQL cluster, the above parameters would be found in a [DB Cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) parameter group and not the DB instance group.
4370
:::
4471

72+
## Enabling GTID Mode {#gtid-mode-aurora}
73+
Global Transaction Identifiers (GTIDs) are unique IDs assigned to each committed transaction in MySQL. They simplify binlog replication and make troubleshooting more straightforward.
74+
75+
If your MySQL instance is MySQL 5.7, 8.0 or 8.4, we recommend enabling GTID mode so that the MySQL ClickPipe can use GTID replication.
76+
77+
To enable GTID mode for your MySQL instance, follow the steps as follows:
78+
1. In the RDS Console, click on your MySQL instance.
79+
2. Click on the `Configurations` tab.
80+
3. Click on the parameter group link.
81+
4. Click on the `Edit` button in the top-right corner.
82+
5. Set the following parameters:
83+
- `gtid_mode` to `OFF_PERMISSIVE`
84+
- `enforce-gtid-consistency` to `ON`
85+
6. Click on `Save Changes` in the top-right corner.
86+
7. Reboot your instance for the changes to take effect.
87+
8. Edit the parameter group again and set `gtid_mode` to `ON_PERMISSIVE`. Ensure `enforce-gtid-consistency` is `ON`.
88+
9. Click on `Save Changes` in the top-right corner.
89+
10. Reboot your instance for the changes to take effect.
90+
11. Edit the parameter group again and set `gtid_mode` to `ON`. Ensure `enforce-gtid-consistency` is `ON`.
91+
12. Click on `Save Changes` in the top-right corner.
92+
13. Reboot your instance for the changes to take effect.
93+
94+
<Image img={enable_gtid} alt="GTID enabled" size="lg" border/>
95+
96+
<br/>
97+
:::tip
98+
The MySQL ClickPipe also supports replication without GTID mode. However, enabling GTID mode is recommended for better performance and easier troubleshooting.
99+
:::
100+
45101
## Configure a database user {#configure-database-user-aurora}
46102

47103
Connect to your Aurora MySQL instance as an admin user and execute the following commands:

‎docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md

+61-4
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,12 @@ import rds_backups from '@site/static/images/integrations/data-ingestion/clickpi
99
import parameter_group_in_blade from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/parameter_group_in_blade.png';
1010
import security_group_in_rds_mysql from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/rds/security-group-in-rds-mysql.png';
1111
import edit_inbound_rules from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/edit_inbound_rules.png';
12+
import rds_config from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/rds_config.png';
13+
import binlog_format from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/binlog_format.png';
14+
import binlog_row_image from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/binlog_row_image.png';
15+
import binlog_row_metadata from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/binlog_row_metadata.png';
16+
import edit_button from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/edit_button.png';
17+
import enable_gtid from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/enable_gtid.png';
1218
import Image from '@theme/IdealImage';
1319

1420
# RDS MySQL source setup guide
@@ -33,16 +39,67 @@ To specify the number of hours to retain binary logs on a DB instance, use the m
3339
mysql=> call mysql.rds_set_configuration('binlog retention hours', 24);
3440
```
3541

36-
If not already configured, make sure to set these in the parameter group:
42+
## Configure binlog settings in the parameter group {#binlog-parameter-group-rds}
43+
44+
The parameter group can be found when you click on your MySQL instance in the RDS Console, and then heading over to the `Configurations` tab.
45+
46+
<Image img={rds_config} alt="Where to find parameter group in RDS" size="lg" border/>
47+
48+
Upon clicking on the parameter group link, you will be taken to the page for it. You will see an Edit button in the top-right.
49+
50+
<Image img={edit_button} alt="Edit parameter group" size="lg" border/>
51+
52+
The following settings need to be set as follows:
53+
3754
1. `binlog_format` to `ROW`.
38-
2. `binlog_expire_logs_seconds` to a value >= `86400` (1 day).
39-
3. `binlog_row_metadata` to `FULL`
40-
4. `binlog_row_image` to `FULL`
55+
56+
<Image img={binlog_format} alt="Binlog format to ROW" size="lg" border/>
57+
58+
2. `binlog_row_metadata` to `FULL`
59+
60+
<Image img={binlog_row_metadata} alt="Binlog row metadata" size="lg" border/>
61+
62+
3. `binlog_row_image` to `FULL`
63+
64+
<Image img={binlog_row_image} alt="Binlog row image" size="lg" border/>
65+
66+
Then click on `Save Changes` in the top-right. You may need to reboot your instance for the changes to take effect - a way of knowing this is if you see `Pending reboot` next to the parameter group link in the Configurations tab of the RDS instance.
67+
4168
<br/>
4269
:::tip
4370
If you have a MySQL cluster, the above parameters would be found in a [DB Cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) parameter group and not the DB instance group.
4471
:::
4572

73+
## Enabling GTID Mode {#gtid-mode-rds}
74+
Global Transaction Identifiers (GTIDs) are unique IDs assigned to each committed transaction in MySQL. They simplify binlog replication and make troubleshooting more straightforward.
75+
76+
If your MySQL instance is MySQL 5.7, 8.0 or 8.4, we recommend enabling GTID mode so that the MySQL ClickPipe can use GTID replication.
77+
78+
To enable GTID mode for your MySQL instance, follow the steps as follows:
79+
1. In the RDS Console, click on your MySQL instance.
80+
2. Click on the `Configurations` tab.
81+
3. Click on the parameter group link.
82+
4. Click on the `Edit` button in the top-right corner.
83+
5. Set the following parameters:
84+
- `gtid_mode` to `OFF_PERMISSIVE`
85+
- `enforce-gtid-consistency` to `ON`
86+
6. Click on `Save Changes` in the top-right corner.
87+
7. Reboot your instance for the changes to take effect.
88+
8. Edit the parameter group again and set `gtid_mode` to `ON_PERMISSIVE`. Ensure `enforce-gtid-consistency` is `ON`.
89+
9. Click on `Save Changes` in the top-right corner.
90+
10. Reboot your instance for the changes to take effect.
91+
11. Edit the parameter group again and set `gtid_mode` to `ON`. Ensure `enforce-gtid-consistency` is `ON`.
92+
12. Click on `Save Changes` in the top-right corner.
93+
13. Reboot your instance for the changes to take effect.
94+
95+
<Image img={enable_gtid} alt="GTID enabled" size="lg" border/>
96+
97+
<br/>
98+
:::tip
99+
The MySQL ClickPipe also supports replication without GTID mode. However, enabling GTID mode is recommended for better performance and easier troubleshooting.
100+
:::
101+
102+
46103
## Configure a database user {#configure-database-user-rds}
47104

48105
Connect to your RDS MySQL instance as an admin user and execute the following commands:

‎scripts/aspell-ignore/en/aspell-dict.txt

+1
Original file line numberDiff line numberDiff line change
@@ -380,6 +380,7 @@ Fuzzers
380380
GDPR
381381
GHCN
382382
GTID
383+
GTIDs
383384
GTest
384385
GUID
385386
Gb

‎static/images/integrations/data-ingestion/clickpipes/mysql/enable_gtid.png

96.5 KB
Loading

‎static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/aurora_config.png

177 KB
Loading

‎static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/binlog_format.png

25.2 KB
Loading

‎static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/binlog_row_image.png

24.8 KB
Loading

‎static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/binlog_row_metadata.png

24.7 KB
Loading

‎static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/edit_button.png

43.6 KB
Loading

‎static/images/integrations/data-ingestion/clickpipes/mysql/parameter_group/rds_config.png

98 KB
Loading

0 commit comments

Comments
 (0)