Dinky v1.0.0-rc4

Feature:

• Realize the synchronization of the entire database and orderly data under multiple degrees of parallelism
• Implement HDFS HA in the resource center
• Implement permission control for global configuration in the configuration center
• Implement configurable alarm anti retransmission function
• Implement DB SQL that can be scheduled by DolphinSchedule
• New resource center synchronization directory based on configured resource storage type (currently implemented as oss)

Fix:

• Fix the issue of incorrect heartbeat detection in cluster instances
• Fix delimiter issues
• Fix the issue of Jar task submission not being able to use set syntax
• Fix the issue of NPE when obtaining user related information through LDAP
• Fix the issue of being unable to carry the higher-level ID when assigning menu permissions
• Fix the issue of version history not being updated properly when switching keys in data development
• Fix some default value issues in PG SQL files
• Fix Dinky's inability to start due to resource configuration errors
• Fix the issue of default route redirection in permission control
• Fix the issue of data development ->task list section not being able to fold
• Fix the issue of duplicate sending of alarm information under multithreading
• Fix the issue of tag height in data development ->open job
• Fix authentication related issues when integrating Gitlab
• Fix the issue where jobmanager logs in the operation and maintenance center's job details cannot be displayed properly
• Fix issues with CataLog NPE
• Fix the issue of Yarn's port being 0
• Fix front-end form status issues with data sources
• Fix the issue with Kubeconfig acquisition
• Fix the issue of pre job task status errors
• Fix syntax issues with add customjar
• Fix some web NPE exceptions
• Fix a bug in enabling SSL when the alarm instance is of email type
• Fix the issue of inability to monitor jar tasks

Optimization & Improve:

• Optimize the UDF download logic of K8S
• Optimize CDC3.0 related logic
• Optimize the synchronization of the entire database by sub database and sub table
• Optimize and integrate LDAP logic
• Optimize the logic of redirecting the data source list to the details page in the registry ->Data source list
• Optimization of homework configuration logic (homework configuration cannot be edited in the published status)
• Optimize the cluster instance rendering logic for job configuration in data development
• Optimize the startup script to enable configuration of environment variables for startup

Document：

• Optimize documents for database and table partitioning
• Optimize regular deployment documents
• Add relevant documents on alarm anti resend
• Optimize OpenAPI documentation
• Add HDFS HA configuration document

Contributors

@aiwenmo
@gaoyan1998
@izouxv
@JiaLiangC
@kylinmac
@leechor
@yangzehan
@yqwoe
@zackyoungh

Dinky v1.0.0-rc3

New Feature

• The default Flink startup version is modified to 1.16
• Implement CodeShow component line break button
• Implement Flink instance stopping function
• Realize deletion of defined task monitoring layout

Optimization

• The method of obtaining the version is optimized to be returned by the backend interface
• Optimize the CANCEL job logic, and can forcefully stop the lost connection job
• Optimize the detection reference logic when part of the registration center is deleted
• You can specify a job template when creating an optimization job
• Optimize Task deletion logic
• Optimize some front-end internationalization
• Optimize Dinky process PID detection logic
• Optimize automatic switching between console and result tag during execution preview

Fix

• Fixed alarm instance form rendering issue
• Fixed the problem that FlinkSQLEnv could not be checked
• Fixed the problem that set statement could not take effect
• Fixed the problem of invalid yarn cluster configuration, customized Flink and hadoop configuration
• Fix some problems in Prejob mode
• Fixed the problem that the checkpoint information of the operation and maintenance center cannot be obtained
• Fixed the problem that the status cannot be detected after the Yarn Application job is completed
• Fixed the problem that the console log failed to print when yarn job submission failed.
• Fix the problem of getting savepoint list 404
• Fixed the issue where Flink instances started from the cluster configuration could not be selected in the job configuration.
• Fixed RECONNECT status job status recognition error
• Fixed the problem that the end time of the operation and maintenance center list is 1970-01-01
• Fixed the problem of submitting FlinkJar tasks to PreJob mode
• Fixed the repeated introduction of dependencies in the alarm module, causing conflicts
• Fix the problem of Dinky startup detection pid
• Fix the problem of conflict caused by inconsistent version of built-in Paimon and user integration (implemented using shader)
• Fix the syntax regular issue of execute jar
• Fixed the problem that the CheckPoint parameter does not take effect in the Application mode of the FlinkJar task
• Fixed the issue where the name and remark information were updated incorrectly when modifying Task operations
• Fixed the problem that password is required when registering data source

Document

• Add some data development related documents
• Optimize some documents of the registration center
• Remove some deprecated/wrong documentation
• Adjust some document structures
• Add quick start document
• Add deployment documents

@aiwenmo
@drgnchan
@gaoyan1998
@gitfortian
@gitjxm
@leechor
@leeoo
@Logout-y
@MaoMiMao
@Pandas886
@yangzehan
@YardStrong
@zackyoungh
@Zzm0809

Dinky v1.0.0-rc2

Fix:
[Fix-2739] Fix bug that complete the missing path in auto.sh's CLASS_PATH
[Fix-2740] Fixed issue of re-rendering task list after publishing or offline
[Fix] Fix flink 1.18 set operator not work and configure null error
[Fix] Fix the bug of save_point_strategy in submission history
[Fix] Fix the bug of print flink table
[Fix] Fix the bug of create view to ddl catalog
[Fix] Fix flink application not throw exception
[Fix] Fix the alert option is incorrect
[Fix] Fix the bug of job life cycle
[Fix-2754] Fix the YAML of K8s form in the cluster is not displayed
[Fix-2756] Fix the devops job list duration formate error
[Fix-2777] Fix flink dag tooltip
[Fix-2782]Fix checkpoint path not found
[Fix] Fix the locations bug in pushing task to DolphinScheduler
[Fix-2806] The job parameters are not effective when the set parameters key and value contain single quotes
[Fix-2811] Upgrade jmx_prometheus_javaagent to 0.20.0 to fix some CVE
[Fix-2814] Fix checkpoint overview error
[Fix] Fix Flink catalog does not take effect with add_jar
[Fix] Fix some devops bug
[Fix-2832] Fix h2 driver no default packaging problem
[Fix] Fix sql bug
[Fix] Fixed jobInstance was always in the running state
[Fix-2843] Fix Yarn Application mode submission task failed and lack of log printing
[Fix] Fix the bug of udf in h2
[Fix-2823] Fix jobconfig cannot render yarn prejob cluster
[Fix] Fix URL misspelling causing the request to fail
[Fix-2855] Fix savepoint table params bug
[Fix-2776] Fix multi user login with the same token value insert error

Optimization & Improve:
[Improve] Improve extract yaml from execute pipeline command
[Optimization] Add key width for job configure item
[Optimization] Add dinky port configure in PrintNetSink
[Improve] Improve query catalog tree
[Optimization-2773] Optimize the data source directory tree has two scroll bars
[Optimization-2822] Optimize metrics page tips
[Optimization] Optimize Flink on yarn app submit
[Optimization] Optimize explainer class use user builder for result
[Optimization] Optimize document management
[Optimization] Implement operator with SPI
[Improve] Improve document form layout
[Optimization-2757] Optimize Flink instance render type
[Optimization-2755] Optimize datasource detail search box
[Optimization] Add resource implement for DinkyClassLoader

Document：
[Document] Improve the cluster instance list document for the registration center
[Document] Improve the alert document for the registration center
[Document] Improve the git project document for the registration center
[Document] Improve the k8s document for the quick start
[Document] Modify domain name
[Document] Improve documents in registration center and authentication center
[Document] Improve documents in developer guide
[Document] Add parameter description in CDCSOURCE and example for debezium.*
[Document-2830] Update download
[Document] Modify document struct

Contributors:
@aiwenmo
@gaoyan1998
@gitfortian
@leeoo
@leechor
@stdnt-xiao
@yangzehan
@zackyoungh
@Zzm0809

Dinky v1.0.0-rc1

Introduction

Dinky is a data development platform based on Apache Flink, which enables agile data development and deployment.

Upgrade instructions

Dinky 1.0 is a refactored version that restructures existing functions, adds several enterprise-level functions, and fixes some limitations of 0.7. Currently, it is not possible to directly upgrade from 0.7 to 1.0. An upgrade plan will be provided in the future.

Function

Its main functions are as follows:

• FlinkSQL data development: automatic prompt completion, syntax highlighting, statement beautification, syntax verification, execution plan, MetaStore, lineage analysis, version comparison, etc.
• Support FlinkSQL multi-version development and multiple execution modes: Local, Standalone, Yarn/Kubernetes Session, Yarn Per-Job, Yarn/Kubernetes Application
• Support Apache Flink ecosystem: Connector, FlinkCDC, Paimon, etc.
• Support FlinkSQL syntax enhancement: whole database synchronization, execution environment, global variables, statement merging, table value aggregation function, loading dependencies, row-level permissions, Jar submission, etc.
• Support FlinkCDC real-time warehousing of the entire database into the lake: multi-database output, automatic table creation, model evolution, sub-database and sub-table
• Supports SQL job development and metadata browsing: ClickHouse, Doris, Hive, Mysql, Oracle, Phoenix, PostgreSql, Presto, SqlServer, StarRocks, etc.
• Support Flink real-time online debugging preview TableData, ChangeLog, Operator, Catalog
• Support Flink job custom monitoring statistical analysis and custom alarm rules.
• Support real-time task operation and maintenance: online and offline, job information (supports obtaining checkpoint), job log, version information, job snapshot, monitoring, SQL lineage, alarm record, etc.
• Support real-time job alarms and alarm groups: DingTalk, WeChat business account, Feishu, email, SMS, etc.
• Supports automatically hosted SavePoint/CheckPoint recovery and triggering mechanisms: latest, earliest, specified, etc.
• Supports multiple resource management: cluster instances, cluster configurations, data sources, alarms, documents, global variables, Git projects, UDFs, system configurations, etc.
• Support enterprise-level management: tenants, users, roles, menus, tokens, data permissions

New Feature

• Added new homepage signboard
• Data development supports code tips
• Supports real-time printing of Flink table data
• The console supports real-time printing task submission log
• Support Flink CDC 3.0 entire database synchronization
• Support customized alarm rules and customized alarm information templates
• Comprehensive revision of the operation and maintenance center
• k8s and k8s operator support
• Support proxy Flink webui access
• Support Flink task monitoring
• Support Dinky jvm monitoring
• New resource center function and expanded rs protocol
• New Git UDF/JAR project hosting and overall construction process
• Supports full-mode application mode custom jar submission
• openapi supports custom parameter submission
• Permission system upgrade, supporting tenants, roles, tokens, and menu permissions
• LDAP authentication support
• New widget function on data development page
• Support pushing dependent tasks to DolphinScheduler

Dinky v0.7.5

Feature:
[Feature-2370][connect] Add jdbc scan filter in flink-connector-jdbc-1.16
[Feature-2381][udf] AGGTABLE supports udtaf in Flink 1.16

Fix:
[Fix][udf] Fix UDF's submission to submission with Checkpoint failed
[Fix-2392][metadata] Fix listColumnsSortByPK infinite recursion

Optimization & Improve:
[Optimization-2361][web] Optimize the display of job which unable to obtain the latest status
[Optimization][core] Optimize flink task query use enum value

Contributors:
@aiwenmo
@Wintle
@zackyoungh

Dinky v0.7.4

Feature:
[Feature-1930][client] Analyze custom functions from Flink SQL
[Feature-2126][connector] Add SQLServer Simple Connector in Flink 1.16
[Feature][cdc] Doris auto create table in CDCSOURCE task

Fix:
[Fix][metadata] Fix the issue of no data type when MySQL automatically generates DDL statements
[Fix][cdc] Fix issues with CDCSOURCE
[Fix-1958][cdc] Fix data misalignment occurs after synchronizing data hen PK do not declare at the table head
[Fix-2087][UDF] Fix udf cannot run on flink1.16 and flink 1.17
[Fix][metadata] Fix clickhouse Nullable columns
[Fix-2147][metadata] Fix missing precision and scale when generating MySql DDL
[Fix][metadata] Fix mssql tinyint Type conversion error
[Fix-2176][admin] Fix unknown jobid bug
[Fix-2108][admin] Fix updating the savepoint policy of the job during checkpoint recovery

Optimization & Improve:
[Optimization][cdc] Optimize CDCSOURCE in Flink 1.16
[Optimization-2089][flink] Upgrade Flink version

Contributors:
@aiwenmo
@HamaWhiteGG
@leeoo
@LiuHao0606
@seawenc
@tan-zhuo
@xiaofan2022
@zhengwenhao2017

Dinky v0.7.3

Feature:
[Feature-1709][core] Support use FlinkSQL CTE to preview data
[Feature-1774][flink] Support Apache Flink 1.17

Fix:
[Fix-797][web] Fix no fragment parameters were passed when getting lineage
[Fix-1731][gateway] Fix deploy flink k8s session error
[Fix-1787][metadata] Fix error in automatic table creation using CDCSOURCE Mysql field unsigned type problem
[Fix-1805][admin] Fix NPE occurred when publishing the job
[Fix][cdcsource] Fix data sending and receiving issues with synchronizing multiple Sinks in CDCSOURCE
[Fix][SqlSegment] Fix the issue of SQL configuration parsing failure in specific scenarios
[Fix-1784][admin] Fix task is deleted but task instances and history are not deleted consecutively
[Fix-1849][client] Fix CDCSOURCE kafkaSink transactionalIdPrefix NullPointerException
[Fix-1847][metadata] Fix postgreSql get DDL script error
[Fix][metadata] Fix generating MySQL DDL statements without quotes for normal default values

Optimization & Improve:
[Optimization-1715][admin] Optimize job monitor reconnect
[Optimization-1776][core,metadata] Fix typo
[Optimization-1657][client] Optimize MysqlJsonDebeziumDeserializationSchema use shaded lib in flink 1.13
[Optimization][client] Support scan.incremental.snapshot.chunk.size
[Optimization][client] Analyze lineage from the original relNode, not the optimized relNode, and add unit tests
[Optimization-1851][flink] Upgrade all versions of Flink to the latest version

Contributors:
@aiwenmo
@AntChen27
@cygnusdark
@HamaWhiteGG
@LiuHao0606

Dinky v0.7.2

Feature:
[Feature-1593] Add CDCSOURCE support sink catalog
[Feature-1608] Support get task lineage by taskid with openapi
[Feature-1641] Add Flink row level permissions
[Feature-1647] Yarn session support applicationId and resource manager config
[Feature-1547] Flink Jar support alert

Fix:
[Fix-1615] Fix submit task error but return succeed when using dolphinscheduler
[Fix-1624] Fix lineage api in data explore module returns empty result
[Fix-1652] Fix the task using the FlinkSQLEnv cannot obtain lineage
[Fix-1662] Fix OpenApi notLoginException
[Fix] Fix wrong Python type
[Fix-1672] Fix failed to get metadata due to jdbc session timeout in CDCSOURCE
[Fix-1638] Fix dlink web could not correct display Boolean type data
[Fix-1686] Fix ZipFile got wrong in K8s native app mode

Optimization & Improve:
[Optimization-1631] Optimize auto.sh and remove useless files
[Optimization-1666] Add job reconnecting status and optimize job monitor
[Optimization-1682] Optimize setting named tinyInt1isBit for mysql metadata

Contributors:
@aiwenmo
@liaotian1005

Dinky v0.7.1

Fix:
[Fix-1494] Fix tinyint(1) convet error
[Fix-1575] Fix unknown column 'tenant_id' in 'field list'
[Fix] Fix i18n
[Fix] Fix UDF function case insensitive
[Fix] Fix the bug that the schema name contains middleLine
[Fix-1556] Fix job submission succeeded but failed
[Fix-1503] Fix use yarn application mode to submit job reported null pointer exception
[Fix-1563] Fix NPE caused by querying data
[Fix] Fix add k8s ClusterConfiguration npe
[Fix] Fix parse type error when use bigint types

Optimization & Improve:
[Optimization-1548] Optimize datasource registration should can save config before validate
[Optimization] Optimize clear console and add clear console at right click
[Optimization] Update org.apache.httpcomponents:httpclient 4.4.1 to 4.5.13

Contributors:
@aiwenmo
@leechor
@zackyoungh
@zhoumengyks
@zhu-mingye

Dinky v0.7.0

Feature:
Supports Apache Flink 1.16.0
Add build java udf jar
Support Flink session mode to automatically load udf jars
Support Flink per-job mode to automatically load udf jars
Support Flink application mode to automatically load udf jars
Support python udf online development
Support scala udf online development
Support custom k8sapp submit at studio
Add file upload in flinkjar task
Add Column Lineage By Logic Plan in Flink all version
Flink JDBC support data filter
Flink JDBC scan partition support datetime
Add multi tenant management
Add select tenant at login
Add dolphinScheduler auto create task
Add system log console
Add execution progress console
Add Flink UDF template
Add Presto data source
Add the function of deleting job tree directory level
Add new sink connector 'datastream-doris-ext' for CDCSOURCE to support metadata writing
DataStudio metadata add refesh button
Add database copying
Add frontend internationalization
Add backend internationalization
Add K8S auto deploy application
Add local environment with web UI
Add FlinkSQL built-in date global variable
Add Kerberos Verification on session model
Add CDCSOURCE supports Doris schema evolution
Add user can change password
Add kill cluster
Add deploy session cluster by cluster configuration
Add jmx monitoring
Add clear console

Fix:
Resolve the exception when starting from the specified savepoint
Fix StarRocks databases display
Fix query exceptions caused by the system's failure to automatically clean up the selected table information when switching data sources
Fix Exception in obtaining SQLDDL of view
Fix invalid oracle validationQuery
Fix failed to get schema of PG database metadata
Fix kafka sink properties not enable
Fix jobConfig useAutoCancel parameter pass wrong value
Fix job monitoring bugs caused by multi tenancy
Fix the deletion of a cluster instance causes existing tasks to fail to stop
Fix application mode missing database variable
Fix continuous click task item will open multiply tabs problem
Fix cdcsource kafka product transactionalIdPrefix disable
Fix cluster display nothing when alias is not set
Fix task version query error
Fix The status of per-job and application always is unknown when the job completes
Fix guide page link error
Fix open API does not have tenant
Fix switch editor tab doesn't work

Optimization & Improve:
Add parameter configuration of MySQL CDC
Optimizate datastream kafka-json and datastream starrocks
Support metadata cache to memory or redis
Add uuid after doris label prefix
Optimize tenant selection
Change resource center to authentication center
Add spotless plugin
Optimize SQL files and differentiate versions by directory
Improve the automatic creation of MySQL tables
Improve postgres metadata information
Improve the generation of postgres table building statements
Optimizate Flink Oracle Connector
Optimizate maven assembly and profile
Compatible java 11
Remove duplicated init of datasource
Upgrade mysql-connector-java to 8.0.28
Upgrade Flink 1.14.5 to 1.14.6
Upgrade Guava and Lombok Version
Upgrade jackson and sa-token version
Optimize lineage to support watermark and udf and LOCALTIMESTAMP and cep and ROW_NUMBER
The authentication center is not visible under non-admin users

Document：
Optimizate readme.md
Update website
Optimize deploy doc
Add flink metrics monitor and optimize deploy document

Contributors:
@admxj
@aiwenmo
@billy-xing
@boolean-dev
@chengchuen
@czy868
@dzygcc
@Forus0322
@gujincheng
@HamaWhiteGG
@hxp0618
@ikiler
@leechor
@lewnn
@jinyanhui2008
@nylqd
@ren-jq101
@RookieGao
@siriume
@tgluon
@wellCh4n
@wfmh
@wmtbnbo
@wuzhenhua01
@zackyoungh
@zhongjingq
@zhu-mingye
@ziqiang-wang
@zq0757
@Zzih
@zzzzzzzs