Skip to content

refactor: Extract deployment file model classes to sqrl-deployment-model - #2238

Merged
ferenc-csaky merged 2 commits into
mainfrom
refactor/sqrl-deployment-model
Jul 29, 2026
Merged

refactor: Extract deployment file model classes to sqrl-deployment-model#2238
ferenc-csaky merged 2 commits into
mainfrom
refactor/sqrl-deployment-model

Conversation

@ferenc-csaky

@ferenc-csaky ferenc-csaky commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Extract deployment-file models into a dependency-light sqrl-deployment-model module. Deployment JSON is now produced from explicit file models rather than planner physical plans, keeping compiler state and persisted contracts separate.

Motivation

Deployment-file consumers should not require Calcite, Flink, or planner implementation dependencies. Persisting physical plans also exposes compiler-only state and can accidentally serialize unsupported runtime types.

Implementation

  • Add the sqrl-deployment-model Maven module, limited to JDK types and Jackson annotations.
  • Add CombinedPlanModel, FlinkPlanModel, JdbcPlanModel, JdbcStatementModel, KafkaPlanModel, KafkaNewTopicModel, and MutationDatabaseModel.
  • Add EnginePhysicalPlan.toFileModel() and update Packager to serialize that representation.
  • Add explicit mappings for Flink, JDBC, Kafka, and combined database plans.
  • Preserve the combined-plan plans wrapper while mapping every nested plan to its file model. This prevents Jackson from traversing Calcite RelDataType instances during deployment serialization.
  • Keep planner lifecycle behavior in sqrl-planner; KafkaNewTopic remains the EngineCreateTable wrapper around the deployment DTO.
  • Keep mutation-database creation and compatibility checks in planner helpers while persisting MutationDatabaseModel.
  • Update config loaders and CLI execution to consume deployment models when reading postgres.json, kafka.json, and mutation-database output.

Compatibility

  • Preserve existing deployment JSON structure, including the combined plans wrapper.
  • Preserve JDBC CREATE TABLE metadata: fields, primary key, partition key/type/count, and TTL.
  • Preserve Kafka topic convenience constructors and defaults.
  • Mark derived KafkaPlanModel.isEmpty() as ignored so kafka.json only contains topics and testRunnerTopics.
  • Retain Jackson deserialization support for persisted TestPlan.jdbcViews as GenericJdbcStatement.

Testing

  • Expand PlanModelSerializationTest to cover JDBC generic, CREATE TABLE, and extension statements; Flink fields; Kafka topic mapping/defaults; Kafka JSON properties; and combined-plan serialization containing a Calcite BIGINT type.
  • Add a CLI regression test for deserializing persisted TestPlan.jdbcViews.

@ferenc-csaky
ferenc-csaky force-pushed the refactor/sqrl-deployment-model branch 3 times, most recently from 5022f9a to d888abc Compare July 29, 2026 13:25
@ferenc-csaky
ferenc-csaky requested review from mateczagany, mbroecheler and velo and removed request for velo July 29, 2026 13:31
…odel`

Co-authored-by: Marvin Froeder <marvin@datasqrl.com>
@ferenc-csaky
ferenc-csaky force-pushed the refactor/sqrl-deployment-model branch from d888abc to 70671a9 Compare July 29, 2026 13:40
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.07692% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.44%. Comparing base (ea69f32) to head (5e7483c).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
.../com/datasqrl/engine/log/kafka/KafkaLogEngine.java 0.00% 9 Missing ⚠️
...li/src/main/java/com/datasqrl/cli/DatasqrlRun.java 0.00% 3 Missing ⚠️
.../java/com/datasqrl/compile/CompilationProcess.java 0.00% 2 Missing ⚠️
...engine/database/relational/AbstractJDBCEngine.java 0.00% 2 Missing ⚠️
...l/engine/database/relational/JdbcPhysicalPlan.java 93.54% 1 Missing and 1 partial ⚠️
...n/java/com/datasqrl/planner/FlinkPhysicalPlan.java 33.33% 2 Missing ⚠️
...om/datasqrl/planner/dag/plan/MutationDatabase.java 0.00% 2 Missing ⚠️
.../src/main/java/com/datasqrl/compile/DagWriter.java 0.00% 1 Missing ⚠️
...main/java/com/datasqrl/util/ConfigLoaderUtils.java 85.71% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2238      +/-   ##
============================================
+ Coverage     17.13%   17.44%   +0.30%     
- Complexity     1054     1072      +18     
============================================
  Files           613      612       -1     
  Lines         17802    17807       +5     
  Branches       2166     2167       +1     
============================================
+ Hits           3051     3107      +56     
+ Misses        14435    14382      -53     
- Partials        316      318       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

velo

This comment was marked as outdated.

@ferenc-csaky
ferenc-csaky merged commit 8b8622f into main Jul 29, 2026
20 of 22 checks passed
@ferenc-csaky
ferenc-csaky deleted the refactor/sqrl-deployment-model branch July 29, 2026 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants