v0.5.12

Enhancements

• #700 #709 #712 Skip dag instance creation if no new message is available in kafka

Internal tasks

• #703 Recognize dead scheduler instances without relying on synchronized clocks
• #706 Scalafmt update - implicit formatting

Added a property for testing
executors.enable.hyperdrive.executor (optional, default true). If set to false, HyperdriveExecutor will not be used and instead, the SparkExecutor will be used for all workflows.

On top of enabling the HyperdriveExecutor globally, jobs also need to opt-in by specifying the following app argument:
useHyperdriveExecutor=true

Both items are only intended during a testing period, after which the logic for both things should be removed.

New configuration for Tomcat
https://github.com/AbsaOSS/hyperdrive-trigger#symbolic-links-on-user-defined-files

Full Changelog: v0.5.11...v0.5.12

v0.5.11

Enhancements

• #689 Keep order of key value input fields ("Additional Spark Config" on UI)
• #686 Application running in YARN will have the same name as in UI
• #698 Possibility to set YARN tags via UI using "spark.yarn.tags"

Internal tasks

• #672 Added formatting checker
• #672 Added license checker

Full Changelog: v0.5.10...v0.5.11

v0.5.10

Bugfixes

• #677: Curly brace is lost from workflow's job app properties

Internal tasks

• #54: Catch and log Slick exceptions
• #666: Remove Hadoop binaries from the packaged application
• #665: Bump spring-boot-starter-web from 2.6.1 to 2.6.6
• #669: Bump async from 2.6.3 to 2.6.4 in /ui

Full Changelog: v0.5.9...v0.5.10

v0.5.9

Enhancements

• #651 : Show notification usage

Internal tasks

• #659: Reduce logging

Full Changelog: v0.5.8...v0.5.9

v0.5.8

Enhancements

• #287: Search workflows in side menu

Bugfixes

• #639: Add retry when sending notifications

Internal tasks

• #599: Replaced SparkLauncher with InProcessLauncher
• #647: Bump follow-redirects from 1.13.3 to 1.14.8 in /ui
• #648: Bump minimist from 1.2.5 to 1.2.6 in /ui

Application properties changes
New

• notification.delay. Optional, default value 0ms
• notification.max.retries. Optional, default value 5

Removed

• sparkYarnSink.hadoopConfDir
• sparkYarnSink.sparkHome

New configuration for Tomcat

• The Hadoop configuration directory needs to be added as the environment variable HADOOP_CONF_DIR and it has to be added to the web application's classpath.

• The environment variable can be added in <tomcat-root>/bin/setenv.sh, e.g. HADOOP_CONF_DIR=/opt/hadoop.

• To add the Hadoop configuration directory to the application classpath,
  in the file <tomcat-base>/conf/catalina.properties, append to the key shared.loader the hadoop conf dir, e.g. shared.loader="/opt/hadoop".

• Update env variable and application classpath as follows: <tomcat-base>/bin/setenv.sh. Add export SPARK_CONF_DIR=/opt/spark/conf
  /conf/catalina.properties
  Set shared.loader=/opt/spark/conf,/opt/hadoop_{environment} (It's important that /opt/spark/conf is before /opt/hadoop)

Full Changelog: v0.5.7...v0.5.8

v0.5.7

Enhancements

• #628 Text-area for string sequence inputs

Bugfixes

• #626 Screen stuck after re-login after session timeout

Internal tasks

• #627 Dedicated ThreadPool for SparkLauncher
• #630 Remove aws profile and aws region from config
• #336: Workflow optimistic locking
• #70: Update app properties in readme

Application properties changes
New

• spark.submit.thread.pool.size. Optional, default value 10

Removed

• spark.emr.awsProfile
• spark.emr.regionInternal

Full Changelog: v0.5.6...v0.5.7

v0.5.6

Enhancements

• #611 Expand side menu workflows/project when workflow is open

Bugfixes

• #622 Improve runs view performance
• #602 Workflows are shown to be unselected when moving across pages

Internal tasks

• #610 clarity upgrade
• #614 Bump follow-redirects from 1.13.3 to 1.14.8 in /ui
• #589 Bump engine.io from 4.1.1 to 4.1.2 in /ui
• #612 Bump karma from 6.3.2 to 6.3.14 in /ui
• #623 Bump h2 from 2.0.206 to 2.1.210
• #613 Bump ajv from 6.10.2 to 6.12.6 in /ui
• #615 Bump postgresql from 42.2.6 to 42.3.3
• #625 Bump karma from 6.3.14 to 6.3.16 in /ui

Full Changelog: v0.5.5...v0.5.6

v0.5.5

Bugfixes

• #570 For EMR jobs, always try to get status from steps api if yarn api is unavailable
• #603 Fix - sort jobs in workflow

v0.5.4

Enhancements

• #564 - Search workflows rest endpoint
• #563 #564 - Workflows main page refactoring (Use backend search endpoint)
• #583 - Revert Job template/Notification/Workflow feature

Bugfixes

• #562 - Get workflows to run refactoring

Internal tasks

• #548 - Upgrade jszip from 3.6.0 to 3.7.1 (FE)
• #549 - Upgrade path-parse from 1.0.6 to 1.0.7 (FE)
• #550 - Upgrade tar from 6.1.0 to 6.1.11 (FE)
• #551 - Upgrade url-parse from 1.5.1 to 1.5.3 (FE)
• #569 - Upgrade h2 from 1.4.200 to 2.0.202 (FE)
• #581 - Upgrade h2 from 2.0.202 to 2.0.206 (FE)
• #571 - Upgrade @npmcli/git from 2.0.6 to 2.1.0 (FE)
• #572 - Upgrade dns-packet from 1.3.1 to 1.3.4 (FE)
• #573 - Upgrade browserslist from 4.16.3 to 4.19.1 (FE)
• #574 - Upgrade ws from 6.2.1 to 6.2.2 (FE)
• #575 - Upgrade lodash from 4.17.15 to 4.17.20 (FE)
• #576 - Upgrade minimist from 1.2.0 to 1.2.5 (FE)
• #567 - Upgrade log4j-api from 2.15.0 to 2.16.0 (BE)
• #577 - Upgrade log4j-api from 2.16.0 to 2.17.0 (BE)
• #580 - Upgrade log4j-api from 2.17.0 to 2.17.1 (BE)
• #539 - Upgrade spring from 2.5.2 to 2.6.1 and spring.security.ldap from 5.5.1 to 5.6.1 (BE)
• #582 - Upgrade scala from 2.11to 2.12 (BE)
• #592 - Git build: add --no-transfer-progress to mvn build

v0.5.3.1

Bugfixes

• Override log4j version to 2.15.0