Skip to content

feature: Add schema support to postgres connector #1070 #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

feature: Add schema support to postgres connector #1070 #71

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5c653eb
schema parameter in postgres.md
siditaduli Oct 5, 2023
6f55a1c
Update postgres.md
siditaduli Oct 5, 2023
b25f19e
Update data-sources.md
siditaduli Oct 5, 2023
8c671c8
Update data-sources.md
siditaduli Oct 5, 2023
c870eba
Update postgres.md
siditaduli Oct 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/configuration/data-sources.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ connections:
host: localhost
port: 5433
database: film
schema: movies

- name: kafka_store
config: !Kafka
Expand All @@ -25,7 +26,7 @@ Each connection typically requires the following details:
- Port number.
- Database name.
- User credentials: username and password.
- Specific details depending on the service type (e.g., schema registry for Kafka).
- Specific details depending on the service type, e.g. schema registry for Kafka, schema for PostgreSQL.

For a comprehensive list of supported connectors and their configurations, refer to [Dozer's supported connectors documentation](/category/data-sources).

Expand Down Expand Up @@ -54,4 +55,4 @@ sources:
columns: # Optional
- txn_id
- description
```
```
15 changes: 9 additions & 6 deletions docs/sources/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
description: Connects to PostgreSQL, leveraging replication slots for continuosly receiving CDC updates.
description: Connects to PostgreSQL, leveraging replication slots for continuously receiving CDC updates.
---

# PostgreSQL

Dozer PostgreSQL connector serves as an essential link for real-time data replication from a PostgreSQL database to Dozer. Upon its initial start, the connector begins by taking a snapshot of the existing data in specified tables. This process provides a foundation from which further data changes can be efficiently managed. After snapshotting, the connector taps into PostgreSQL's logical replication by connecting to a designated replication slot, monitoring for data changes in real-time. Any changes occurring in the PostgreSQL database are detected and instantly sent to Dozer, ensuring a continuous stream of updated data. The specific tables to observe are customizable within the connector configuration, ensuring a targeted and resource-optimized data-handling process.
Dozer PostgreSQL connector serves as an essential link for real-time data replication from a PostgreSQL database to Dozer. Upon its initial start, the connector begins by taking a snapshot of the existing data in specified tables. This process provides a foundation from which further data changes can be efficiently managed. After snapshotting, the connector taps into PostgreSQL's logical replication by connecting to a designated replication slot, monitoring for data changes in real time. Any changes occurring in the PostgreSQL database are detected and instantly sent to Dozer, ensuring a continuous stream of updated data. The specific tables to observe are customizable within the connector configuration, ensuring a targeted and resource-optimized data-handling process.

### Configuration
The following configuration block can be used in `dozer-config.yaml` to define a new PostgreSQL connection:
Expand All @@ -18,16 +18,18 @@ connections:
host: localhost
port: 5433
database: film
```
# in case schema is not the default public
schema: movies

```
Alternatively, it's possible to specify a connection string:
```yaml
connections:
- name: pagila_conn
config: !Postgres
# connection_url can be used instead of above config
# connection_url can be used instead of the above config
connection_url: postgresql://postgres:postgres@localhost:5433/film
# ssl_mode with disable, prefer and require (optional)
# ssl_mode with disable, prefer, and require (optional)
ssl_mode: require
```

Expand All @@ -36,10 +38,11 @@ connections:
| **Parameter Name** | **Type** | **Description** |
|--------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `user` | String | The username required for authenticating the user's access to the PostgreSQL instance. |
| `password` | String | The password corresponding to the above username, required for secure authentication to the PostgreSQL instance. |
| `password` | String | The password corresponding to the above username, is required for secure authentication to the PostgreSQL instance. |
| `host` | String or IP address | The host address of the PostgreSQL instance. It could be an IP address or a valid hostname. |
| `port` | Integer | The specific port on which the PostgreSQL service is running. |
| `database` | String | The specific database within the PostgreSQL instance to which the connector needs to establish a connection. |
| `schema` | String | The specific schema within the PostgreSQL instance. The default value is `public`. In case the schema in the PostgreSQL instance is different from `public`, specify it in this parameter.
| `SSLMode` | String | The choice of SSL TCP/IP connection negotiation priority with the server can be configured with three available options: "require," "prefer," or "disable." These options allow you to define how the SSL connection will be established between the client and the server. |
| `connection_url` | String | Postgres connection string takes precedence over all other configurations in Dozer when connecting to the database. running. |

Expand Down