0.2.0 (2025-04-04)

TL;DR

• Completely new UI.
• Add support for FileSystem connections.
• Add support for simple tranformations within transfer - filter files, filter rows, change columns.
• Add support for incremental read strategy.
• Add support for running transfers by schedule.
• Add support for changing SparkSession resource limits.

Huge thanks to @dmitry-pedchenko, @maxim-lixakov, @IlyasDevelopment, @Zabilsya.

Breaking Changes

• Implement a single error handling format to improve consistency (#95)
• Change response format for GET /v1/groups - add current user role for each group (#97)
• Change response format for GET /v1/groups/:id - add current user role for group (#109)
• Now migrations are executed in a dedicated one-off container, instead of being run as a part of backend container. (#163)
• Delete transfers, queues and groups records instead of marking them as deleted (#168)
• Move the type field from nested connection_data field to the root level of the Connection response. (#169)
• Decouple auth_data.type from the connection type (e.g. postgres) and link it to the authentication type (e.g. basic). (#169)
• Add deletion of connections records instead of marking them as deleted (#170)
• Use PUT instead of PATCH for Connection and Transfer models (#215)
• Return new connection and transfer object in POST /v1/transfer/copy and POST /v1/connection/copy endpoints.
• Change response status from 200 to 204 for all DELETE endpoints.

Features

• Add hive to known types (#67)
• Allow search for groups, users, connections, transfers, queues (#92, #94, #99, #100, #101, #103)
• Add filters for connections, transfers and runs (#94, #102, #106)
• Implement a scheduler to run celery tasks on a schedule. This can be done by setting Transfer.is_scheduled=True and Transfer.schedule="..." (cron-like expression). The Run model now has a type field with options MANUAL and SCHEDULED. (#114)
• Add GET v1/monitoring/metrics endpoint to provide basic HTTP server metrics in Prometheus format (#121)
• Implemented KeycloakAuthProvider for Single Sign-On (SSO) authentication.(#123)
• Implemented DummyAuthProvider for development and testing environments. (#123)
• Add API schemas for new DB sources - Clickhouse, MSSQL, MySQL (#124, #125, #126, #160)
• Add logic for handling FTP, FTPS, SFTP, Samba, WebDAV transfers (#189, #191, #192, #194)
• Add API schemas for file sources - SFTP, FTP, FTPS, WebDAV, Samba (#187)
• Add API schemas for file formats - Excel, XML, ORC, Parquet (#140, #142, #143, #144)
• Add compression options to file formats CSV, JSON, JSONLine, Excel, ORC, Parquet, XML (#159, #161)
• Add transformations for Transfers with dataframe row filtering (#184)
• Add transformations for Transfers with dataframe column filtering (#186)
• Add transformations for Transfers with file filtering (#198)
• Add increment_by field to strategy_params (#202)
• Implement increment strategy for transfers with file sources (#209)
• Implement increment strategy for transfers with database sources (#211)
• Add resources field to Transfer. (#214)
• Add file_name_template field to target_params (#196, #201)

Improvements

• Updated User model to include email, first_name, middle_name, and last_name fields, all optional. (#123)
• Read env variable SYNCMASTER__ENTRYPOINT__SUPERUSERS to promote users to SUPERUSER role during server startup. (#137)
• Enabled dynamic selection of authentication provider via environment variable SYNCMASTER__AUTH__PROVIDER. (#123)
• Enable parallel reading from JDBC sources. (#219)
• Reset HWM when changing strategy from incremental to full. (#217)
• Grant read-only permissions for the previous group owner when ownership is transferred (#135)

Bug Fixes

• Use Hadoop AWS magic commiter only if transfer target is S3. (#46)
• Check that service_name and sid are mutually exclusive when editing Oracle connection. (#52)
• Queue name is unique within a group, new field slug is globally-unique. (#54, #119)
• Prohibit updating connection type it if there is a transfer associated with this connection. (#55)
• Fix error when is_scheduled field value was ignored. (#57)
• Group without any users assigned was missing in groups list. (#62)
• Dump connection credentials while starting a transfer. (#63)