Skip to content

Apply variable for livesync + cleanup #4059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Apr 29, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions _partials/_livesync-configure-source-database-awsrds.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Updating parameters on a PostgreSQL instance will cause an outage. Choose a time
Changing parameters will cause an outage. Wait for the database instance to reboot before continuing.
1. Verify that the settings are live in your database.

1. **Create a user for livesync and assign permissions**
1. **Create a user for $LIVESYNC and assign permissions**

1. Create `<livesync username>`:

Expand Down Expand Up @@ -63,7 +63,7 @@ Updating parameters on a PostgreSQL instance will cause an outage. Choose a time
EOF
```

If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing.:
If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing:
```sql
psql $SOURCE <<EOF
GRANT USAGE ON SCHEMA <schema> TO <livesync username>;
Expand Down
4 changes: 2 additions & 2 deletions _partials/_livesync-configure-source-database.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import EnableReplication from "versionContent/_partials/_migrate_live_setup_enab

This will require a restart of the PostgreSQL source database.

1. **Create a user for livesync and assign permissions**
1. **Create a user for $LIVESYNC and assign permissions**

1. Create `<livesync username>`:

Expand Down Expand Up @@ -47,7 +47,7 @@ import EnableReplication from "versionContent/_partials/_migrate_live_setup_enab
EOF
```

If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing.:
If the tables you are syncing are not in the `public` schema, grant the user permissions for each schema you are syncing:
```sql
psql $SOURCE <<EOF
GRANT USAGE ON SCHEMA <schema> TO <livesync username>;
Expand Down
35 changes: 18 additions & 17 deletions _partials/_livesync-console.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,16 @@ import TuneSourceDatabaseAWSRDS from "versionContent/_partials/_livesync-configu

- Ensure that the source $PG instance and the target $SERVICE_LONG have the same extensions installed.

LiveSync does not create extensions on the target. If the table uses column types from an extension,
$LIVESYNC_CAP does not create extensions on the target. If the table uses column types from an extension,
first create the extension on the target $SERVICE_LONG before syncing the table.

## Limitations

* Indexes(including Primary Key and Unique constraints) are not migrated by $SERVICE_LONG.
* Indexes (including Primary Key and Unique constraints) are not migrated by $SERVICE_LONG.

We recommend that you create only necessary indexes on the target $SERVICE_LONG depending on your query patterns.
We recommend that you create only the necessary indexes on the target $SERVICE_LONG depending on your query patterns.

* Tables with user defined types are not migrated by $SERVICE_LONG.
* Tables with user-defined types are not migrated by $SERVICE_LONG.

You need to create the user defined types on the target $SERVICE_LONG before syncing the table.

Expand Down Expand Up @@ -74,41 +74,42 @@ To sync data from your PostgreSQL database to your $SERVICE_LONG using $CONSOLE:
1. **Connect to your $SERVICE_LONG**

In [$CONSOLE][portal-ops-mode], select the service to sync live data to.
1. **Start livesync**
1. Click `Actions` > `livesync for PostgreSQL`.
1. **Start $LIVESYNC**
1. Click `Actions` > `Livesync for PostgreSQL`.

1. **Connect the source database and target $SERVICE_SHORT**

![Livesync wizard](https://assets.timescale.com/docs/images/livesync-wizard.png)

In `livesync for PostgreSQL`:
In `Livesync for Postgres`:
1. Set the `Livesync Name`.
1. Set the` PostgreSQL Connection String` to point to the source database you want to sync to Timescale.
1. Set the `PostgreSQL Connection String` to point to the source database you want to sync to Timescale.

This is the connection string for [`<livesync username>`][livesync-tune-source-db].
1. Press `Continue`.
1. Click `Continue`.
$CONSOLE connects to the source database and retrieves the schema information.

1. **Optimize the data to synchronize in hypertables**

![livesync start](https://assets.timescale.com/docs/images/livesync-start.png)
1. Select the table to sync, and press `+`.
$CONSOLE checks the table schema and, if possible suggests the column to use as the time dimension in a hypertable.
1. Select the table to sync and click `+`.

$CONSOLE checks the table schema and, if possible, suggests the column to use as the time dimension in a hypertable.
1. Repeat this step for each table you want to sync.
1. Press `Start Livesync`.
1. Click `Start Livesync`.

$CONSOLE starts livesync between the source database and the target $SERVICE_SHORT and displays the progress.
$CONSOLE starts $LIVESYNC between the source database and the target $SERVICE_SHORT and displays the progress.

1. **Monitor syncronization**
1. To view the progress of the livesync, click the name of the livesync process:
1. To view the progress of the $LIVESYNC, click the name of the $LIVESYNC process:
![livesync view status](https://assets.timescale.com/docs/images/livesync-view-status.png)
1. To pause and restart livesync, click the buttons on the right of the livesync process and select an action:
1. To pause and restart $LIVESYNC, click the buttons on the right of the $LIVESYNC process and select an action:
![livesync start stop](https://assets.timescale.com/docs/images/livesync-start-stop.png)

</Procedure>

And that is it, you are using Livesync to synchronize all the data, or specific tables, from a PostgreSQL database
instance to your $SERVICE_LONG in real-time.
And that is it, you are using $LIVESYNC to synchronize all the data, or specific tables, from a PostgreSQL database
instance to your $SERVICE_LONG in real time.

[install-psql]: /integrations/:currentVersion:/psql/
[portal-ops-mode]: https://console.cloud.timescale.com/dashboard/services
Expand Down
2 changes: 1 addition & 1 deletion _partials/_livesync-limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
the same changes to the source PostgreSQL instance.
* Ensure that the source $PG instance and the target $SERVICE_LONG have the same extensions installed.

LiveSync does not create extensions on the target. If the table uses column types from an extension,
$LIVESYNC_CAP does not create extensions on the target. If the table uses column types from an extension,
first create the extension on the target $SERVICE_LONG before syncing the table.
* There is WAL volume growth on the source PostgreSQL instance during large table copy.
* This works for PostgreSQL databases only as source. TimescaleDB is not yet supported.
24 changes: 12 additions & 12 deletions _partials/_livesync-terminal.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ import TuneSourceDatabaseAWSRDS from "versionContent/_partials/_migrate_live_tun

- Ensure that the source $PG instance and the target $SERVICE_LONG have the same extensions installed.

LiveSync does not create extensions on the target. If the table uses column types from an extension,
$LIVESYNC_CAP does not create extensions on the target. If the table uses column types from an extension,
first create the extension on the target $SERVICE_LONG before syncing the table.

- [Install Docker][install-docker] on your sync machine.

You need a minimum of a 4 CPU/16GB EC2 instance to run Livesync.
You need a minimum of a 4 CPU/16GB EC2 instance to run $LIVESYNC.

- Install the [PostgreSQL client tools][install-psql] on your sync machine.

Expand All @@ -26,7 +26,7 @@ import TuneSourceDatabaseAWSRDS from "versionContent/_partials/_migrate_live_tun

<LivesyncLimitations />

- The Schema is not migrated by Livesync, you use pg_dump/restore to migrate schema
- The schema is not migrated by $LIVESYNC, you use `pg_dump`/`pg_restore` to migrate it.

## Set your connection strings

Expand Down Expand Up @@ -61,7 +61,7 @@ The `<user>` in the `SOURCE` connection must have the replication role granted i

## Migrate the table schema to the $SERVICE_LONG

Use pg_dump to:
Use `pg_dump` to:

<Procedure>

Expand Down Expand Up @@ -129,22 +129,22 @@ events data, and tables that are already partitioned using PostgreSQL declarativ

## Synchronize data to your $SERVICE_LONG

You use the Livesync docker image to synchronize changes in real-time from a PostgreSQL database
You use the $LIVESYNC docker image to synchronize changes in real-time from a PostgreSQL database
instance to a $SERVICE_LONG:

<Procedure>

1. **Start Livesync**
1. **Start $LIVESYNC**

As you run Livesync continuously, best practice is to run it as a background process.
As you run $LIVESYNC continuously, best practice is to run it as a background process.

```shell
docker run -d --rm --name livesync timescale/live-sync:v0.1.11 run --publication analytics --subscription livesync --source $SOURCE --target $TARGET
```

1. **Trace progress**

Once Livesync is running as a docker daemon, you can also capture the logs:
Once $LIVESYNC is running as a docker daemon, you can also capture the logs:
```shell
docker logs -f livesync
```
Expand All @@ -168,7 +168,7 @@ instance to a $SERVICE_LONG:

- r: table is ready, synching live changes

1. **Stop Livesync**
1. **Stop $LIVESYNC**

```shell
docker stop live-sync
Expand All @@ -191,9 +191,9 @@ instance to a $SERVICE_LONG:

## Specify the tables to synchronize

After the Livesync docker is up and running, you [`CREATE PUBLICATION`][create-publication] on the SOURCE database to
After the $LIVESYNC docker is up and running, you [`CREATE PUBLICATION`][create-publication] on the SOURCE database to
specify the list of tables which you intend to synchronize. Once you create a PUBLICATION, it is
automatically picked by Livesync, which starts synching the tables expressed as part of it.
automatically picked by $LIVESYNC, which starts syncing the tables expressed as part of it.

For example:

Expand Down Expand Up @@ -223,7 +223,7 @@ For example:
ALTER PUBLICATION analytics SET(publish_via_partition_root=true);
```

1. **Stop synching a table in the `PUBLICATION` with a call to `DROP TABLE`**
1. **Stop syncing a table in the `PUBLICATION` with a call to `DROP TABLE`**

```sql
ALTER PUBLICATION analytics DROP TABLE tags;
Expand Down
2 changes: 1 addition & 1 deletion _partials/_migrate_import_prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Before you migrate your data:
Each $SERVICE_LONG has a single database that supports the
[most popular extensions][all-available-extensions]. $SERVICE_LONGs do not support tablespaces,
and there is no superuser associated with a $SERVICE_SHORT.
Best practice is to create a $SERVICE_LONGs with at least 8 CPUs for a smoother experience. A higher-spec instance
Best practice is to create a $SERVICE_LONG with at least 8 CPUs for a smoother experience. A higher-spec instance
can significantly reduce the overall migration window.

- To ensure that maintenance does not run during the process, [adjust the maintenance window][adjust-maintenance-window].
Expand Down
6 changes: 3 additions & 3 deletions _partials/_migrate_live_setup_enable_replication.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Replica identity assists data replication by identifying the rows being modified. Your options are that
each table and hypertable in the source database should either have:
- **A primary key**: Data replication defaults to the primary key of the table being replicated.
- **A primary key**: data replication defaults to the primary key of the table being replicated.
Nothing to do.
- **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns
marked as `NOT NULL`. If a UNIQUE index does not exists, create one to assist the migration. You can delete if after
marked as `NOT NULL`. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after
migration.

For each table, set `REPLICA IDENTITY` to the viable unique index:
Expand All @@ -19,4 +19,4 @@
```
For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows. This results
in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE` operations on the table,
best practice is to not use `FULL`
best practice is to not use `FULL`.
8 changes: 4 additions & 4 deletions _partials/_migrate_live_tune_source_database_awsrds.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Updating parameters on a PostgreSQL instance will cause an outage. Choose a time
1. In [https://console.aws.amazon.com/rds/home#databases:][databases],
select the RDS instance to migrate.

1. Click `Configuration`, scroll down and note the `DB instance parameter group`, then click `Parameter Groups`
1. Click `Configuration`, scroll down and note the `DB instance parameter group`, then click `Parameter groups`

<img class="main-content__illustration"
src="https://assets.timescale.com/docs/images/migrate/awsrds-parameter-groups.png"
Expand All @@ -23,10 +23,10 @@ Updating parameters on a PostgreSQL instance will cause an outage. Choose a time
- `old_snapshot_threshold` set to `-1`: prevent PostgreSQL from treating the data in a snapshot as outdated.
- `rds.logical_replication` set to `1`: record the information needed for logical decoding.

1. In RDS, navigate back to your [databases][databases], select the RDS instance to migrate and click `Modify`.
1. In RDS, navigate back to your [databases][databases], select the RDS instance to migrate, and click `Modify`.

1. Scroll down to `Database options` select your new parameter group and click `Continue`.
1. Either `Apply immediately` or choose a maintence window, then click `Modify DB instance`.
1. Scroll down to `Database options`, select your new parameter group, and click `Continue`.
1. Either `Apply immediately` or choose a maintenance window, then click `Modify DB instance`.

Changing parameters will cause an outage. Wait for the database instance to reboot before continuing.
1. Verify that the settings are live in your database.
Expand Down
2 changes: 1 addition & 1 deletion _partials/_migrate_prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Before you move your data:
Each Timescale Cloud service has a single database that supports the
[most popular extensions][all-available-extensions]. $SERVICE_LONGs do not support tablespaces,
and there is no superuser associated with a $SERVICE_SHORT.
Best practice is to create a $SERVICE_LONGs with at least 8 CPUs for a smoother experience. A higher-spec instance
Best practice is to create a $SERVICE_LONG with at least 8 CPUs for a smoother experience. A higher-spec instance
can significantly reduce the overall migration window.

- To ensure that maintenance does not run while migration is in progress, best practice is to [adjust the maintenance window][adjust-maintenance-window].
Expand Down
6 changes: 3 additions & 3 deletions about/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ All the latest features and updates to Timescale products.

[Livesync for S3](https://docs.timescale.com/migrate/latest/livesync-for-s3/) is our second livesync offering in
Timescale Console, following livesync for PostgreSQL. This feature helps users sync data in their S3 buckets to a
Timescale Cloud service, and simplifies data importing. Livesync handles both existing and new data in real-time,
Timescale Cloud service, and simplifies data importing. Livesync handles both existing and new data in real time,
automatically syncing everything into a Timescale Cloud service. Users can integrate Timescale Cloud alongside S3, where
S3 stores data in raw form as the source for multiple destinations.

Expand Down Expand Up @@ -133,7 +133,7 @@ To see the job information page, in [$CONSOLE][console], select the $SERVICE_SHO
## 🤩 In-Console Livesync for PostgreSQL
<Label type="date">March 21, 2025</Label>

You can now set up an active data ingestion pipeline with Livesync for PostgreSQL in Timescale Console. This tool enables you to replicate your source database tables into Timescale's hypertables indefinitely. Yes, you heard that right—keep Livesync running for as long as you need, ensuring that your existing source PostgreSQL tables stay in sync with Timescale Cloud. Read more about setting up and using [Livesync for PostgreSQL](https://docs.timescale.com/migrate/latest/livesync-for-postgresql/).
You can now set up an active data ingestion pipeline with livesync for PostgreSQL in Timescale Console. This tool enables you to replicate your source database tables into Timescale's hypertables indefinitely. Yes, you heard that right—keep livesync running for as long as you need, ensuring that your existing source PostgreSQL tables stay in sync with Timescale Cloud. Read more about setting up and using [Livesync for PostgreSQL](https://docs.timescale.com/migrate/latest/livesync-for-postgresql/).

![Livesync in Timescale Console](https://assets.timescale.com/docs/images/timescale-cloud-livesync-tile.png)

Expand Down Expand Up @@ -350,7 +350,7 @@ We have built a new solution that helps you continuously replicate all or some o

[Livesync](https://docs.timescale.com/migrate/latest/livesync-for-postgresql/) allows you to keep a current Postgres instance such as RDS as your primary database, and easily offload your real-time analytical queries to Timescale Cloud to boost their performance. If you have any questions or feedback, talk to us in [#livesync in Timescale Community](https://app.slack.com/client/T4GT3N2JK/C086NU9EZ88).

This is just the beginning—you'll see more from Livesync in 2025!
This is just the beginning—you'll see more from livesync in 2025!

## In-Console import from S3, I/O Boost, and Jobs Explorer
<Label type="date">December 13, 2024</Label>
Expand Down
10 changes: 5 additions & 5 deletions migrate/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,14 +36,14 @@ see [Ingest data from other sources][data-ingest].

## Livesync your data

You use $LIVESYNC to synchronize all or some of your data to your $SERVICE_LONG in real-time. You run $LIVESYNC
You use $LIVESYNC to synchronize all or some of your data to your $SERVICE_LONG in real time. You run $LIVESYNC
continuously, using your data as a primary database and your $SERVICE_LONG as a logical replica. This enables you
to leverage $CLOUD_LONG’s real-time analytics capabilities on your replica data.

| $LIVESYNC options | Downtime requirements |
|----------------------------------------|-----------------------|
| [$LIVESYNC for $PG][livesync-postgres] | None |
| [$LIVESYNC for S3][livesync-s3] | None |
| $LIVESYNC_CAP options | Downtime requirements |
|--------------------------------------------|-----------------------|
| [$LIVESYNC_CAP for $PG][livesync-postgres] | None |
| [$LIVESYNC_CAP for S3][livesync-s3] | None |



Expand Down
Loading