[RelEng] Split stream, branch and schedule configuration for I/Y builds

Split the configuration of built branches per stream and specify it
individually per build type (I- and Y-build). Additionally specify
individually per build type and stream the cron trigger schedule of the
corresponding build.
This allows to configure all managed builds individually.

When preparing a new release, add new and move old stream as before. But
preserve the old build's schedule and don't overwrite especially the
Y-build schedule with the new streams schedule even if a Java release is
imminent (this wrongly happened before).
For Y builds, also add the new stream immediately, but disable it, when
a java release is imminent and the the new BETA-Java branch is not
created. The old stream's Y-build can then continue until the Java
release to feed the P-build and the new build can be activated once
ready.

Author: HannesWell

JenkinsJobs/AutomatedTests/I_unit_tests.groovy

@@ -8,7 +8,7 @@ def TEST_CONFIGURATIONS = [
 	[os: 'win32' , ws:'win32', arch: 'x86_64' , javaVersion: 21, agentLabel: 'qa6xd-win11'        , javaHome: "'C:\\\\Program Files\\\\Eclipse Adoptium\\\\jdk-21.0.5.11-hotspot'" ],
 ]
 
-for (STREAM in config.Branches.keySet()){
+for (STREAM in config.I.streams.keySet()){
 	def MAJOR = STREAM.split('\\.')[0]
 	def MINOR = STREAM.split('\\.')[1]
 	for (TEST_CONFIG in TEST_CONFIGURATIONS){

JenkinsJobs/Builds/FOLDER.groovy

@@ -4,23 +4,20 @@ folder('Builds') {
   description('Eclipse periodic build jobs.')
 }
 
-for (entry in config.Branches.entrySet()){
-	def STREAM = entry.key
-	def BRANCH = entry.value
+config.I.streams.each{ STREAM, configuration ->
 
 	pipelineJob('Builds/I-build-' + STREAM){
 		description('Daily Eclipse Integration builds.')
 		properties {
 			pipelineTriggers {
 				triggers {
 					cron {
-						spec('''TZ=America/Toronto
+						spec(configuration.schedule ? """TZ=America/Toronto
 # Format: Minute Hour Day Month Day-of-week (1-7)
 # - - - Integration Eclipse SDK builds - - - 
 # Schedule: 6 PM every day until end of RC2
-0 18 * 8-10 *
-0 18 1-26 11 *
-''')
+${configuration.schedule}
+""" : '')
 					}
 				}
 			}
@@ -29,7 +26,7 @@ for (entry in config.Branches.entrySet()){
 			cpsScm {
 				lightweight(true)
 				scm {
-					github('eclipse-platform/eclipse.platform.releng.aggregator', BRANCH)
+					github('eclipse-platform/eclipse.platform.releng.aggregator', configuration.branch)
 				}
 				scriptPath('JenkinsJobs/Builds/build.jenkinsfile')
 			}

JenkinsJobs/JobDSL.json

@@ -1,5 +1,19 @@
 {
-    "Branches": {
-        "4.38": "master"
+    "I": {
+        "streams": {
+            "4.38": {
+                "branch": "master",
+                "schedule": "0 18 * 8-10 *\n0 18 1-26 11 *"
+            }
+        }
+    },
+    "Y": {
+        "streams": {
+            "4.38": {
+                "branch": "master",
+                "disabled": "true",
+                "schedule": "0 10 * 8-10 2,4,6\n0 10 1-26 11 2,4,6"
+            }
+        }
     }
-}
+}

JenkinsJobs/Releng/FOLDER.groovy

@@ -88,7 +88,7 @@ Useful for debugging and to very that the pipeline behaves as intended.
 		stringParam('RC1_DATE', null, 'Release-Candidate 1 end date in the format yyyy-mm-dd, for example: 2025-08-22')
 		stringParam('RC2_DATE', null, 'Release-Candidate 2 end date in the format yyyy-mm-dd, for example: 2025-08-29')
 		stringParam('GA_DATE', null, 'Final general availability release date in the format yyyy-mm-dd, for example: 2025-09-10')
-		stringParam('NEXT_JAVA_RELEASE_DATE', null, 'Release date of the next nww Java version, if it is released shortly after the Eclipse release (i.e. shortly after the <em>GA_DATE</em> specified above, usually for odd release versions), else left blank (usually for even releases). Value is in the format yyyy-mm-dd, for example: 2025-09-16')
+		stringParam('NEXT_JAVA_RELEASE_DATE', null, 'Release date of the next Java version, if it is released shortly after the Eclipse release (i.e. shortly after the <em>GA_DATE</em> specified above, <b>usual for odd release versions</b>), else left blank (<b>usual for even releases</b>). Value is in the format yyyy-mm-dd, for example: 2025-09-16')
 	}
 	definition {
 		cpsScm {

JenkinsJobs/Releng/prepareNextDevCycle.jenkinsfile

@@ -29,6 +29,10 @@ pipeline {
 					nextVersionMatcher = null // release matcher as it's not serializable
 
 					env.PREVIOUS_RELEASE_CANDIDATE_ID = readParameter('PREVIOUS_RELEASE_CANDIDATE_ID')
+					def buildPropertiesTxt = sh(script: "curl --fail https://download.eclipse.org/eclipse/downloads/drops4/${PREVIOUS_RELEASE_CANDIDATE_ID}/buildproperties.txt", returnStdout: true)
+					def buildProperties = readProperties(text: buildPropertiesTxt)
+					assignEnvVariable('IS_JAVA_RELEASE_IMMINENT', !buildProperties.NEXT_JAVA_RELEASE_DATE.replace('"','').isEmpty()) // Remove surrounding quotes
+					
 					def previousIdMatcher = null
 					if ((previousIdMatcher = env.PREVIOUS_RELEASE_CANDIDATE_ID =~ /(?<type>[SR])-(?<major>\d+)\.(?<minor>\d+)(\.(?<service>\d+))?(?<checkpoint>(M|RC)\d+[a-z]?)?-(?<date>\d{8})(?<time>\d{4})/).matches()) {
 						def checkpoint = previousIdMatcher.group('checkpoint')
@@ -39,8 +43,6 @@ pipeline {
 						assignEnvVariable('PREVIOUS_RELEASE_CANDIDATE_GIT_TAG', "${previousIdMatcher.group('type')}${PREVIOUS_RELEASE_VERSION_MAJOR}_${PREVIOUS_RELEASE_VERSION_MINOR}${(checkpoint || previousServiceVersion != '0') ? ('_' + previousServiceVersion) : ''}${checkpoint ? ('_' + checkpoint) : ''}")
 
 					} else if ((previousIdMatcher = env.PREVIOUS_RELEASE_CANDIDATE_ID =~ /I(?<date>\d{8})-(?<time>\d{4})/).matches()) {
-						def buildPropertiesTxt = sh(script: "curl --fail https://download.eclipse.org/eclipse/downloads/drops4/${PREVIOUS_RELEASE_CANDIDATE_ID}/buildproperties.txt", returnStdout: true)
-						def buildProperties = readProperties(text: buildPropertiesTxt)
 						assignEnvVariable('PREVIOUS_RELEASE_VERSION_MAJOR', buildProperties.STREAMMajor.replace('"','')) // Remove surrounding quotes
 						assignEnvVariable('PREVIOUS_RELEASE_VERSION_MINOR', buildProperties.STREAMMinor.replace('"','')) // Remove surrounding quotes
 						assignEnvVariable('PREVIOUS_RELEASE_CANDIDATE_TAG', "${PREVIOUS_RELEASE_CANDIDATE_ID}")
@@ -156,6 +158,7 @@ pipeline {
 					"STREAMMajor=\".*\"" : "STREAMMajor=\"${NEXT_RELEASE_VERSION_MAJOR}\"",
 					"STREAMMinor=\".*\"" : "STREAMMinor=\"${NEXT_RELEASE_VERSION_MINOR}\"",
 					"ECLIPSE_RUN_REPO=\".*\"" : "ECLIPSE_RUN_REPO=\"https://download.eclipse.org/eclipse/updates/${NEXT_RELEASE_VERSION}-I-builds/\"",
+					"NEXT_JAVA_RELEASE_DATE=\".*\"" : "NEXT_JAVA_RELEASE_DATE=\"${NEXT_JAVA_RELEASE_DATE}\"",
 				])
 				replaceInFile('eclipse.platform.releng.tychoeclipsebuilder/eclipse.platform.repository/platform.p2.inf', [
 					"${PREVIOUS_RELEASE_VERSION} Release" : "${NEXT_RELEASE_VERSION} Release",
@@ -170,17 +173,21 @@ pipeline {
 					"for ${PREVIOUS_RELEASE_VERSION}.0 builds" : "for ${NEXT_RELEASE_VERSION}.0 builds",
 				])
 				script {
-					utilities.modifyJSON('JenkinsJobs/JobDSL.json') { jobs ->
-						jobs.Branches["${PREVIOUS_RELEASE_VERSION}"] = env.MAINTENANCE_BRANCH
-						jobs.Branches["${NEXT_RELEASE_VERSION}"] = 'master'
+					utilities.modifyJSON('JenkinsJobs/JobDSL.json') { builds ->
+						// Create I-build for new stream and move previous I-build to maintenance branch to allow late re-spins
+						builds.I.streams["${NEXT_RELEASE_VERSION}"] = [ branch: 'master', schedule: env.I_BUILD_SCHEDULE ]
+						builds.I.streams["${PREVIOUS_RELEASE_VERSION}"].branch = env.MAINTENANCE_BRANCH
+						builds.I.streams["${PREVIOUS_RELEASE_VERSION}"].schedule = '' // schedule should already be inactive, but clear it to be sure
+						
+						//Create Y-build for new stream
+						builds.Y.streams["${NEXT_RELEASE_VERSION}"] = [ branch: 'master', disabled: env.IS_JAVA_RELEASE_IMMINENT, schedule: env.Y_BUILD_SCHEDULE ]
+						// If a new Java version is released shortly after the previous release, that Java release will happen in the next days from the point when this is executed.
+						// In that case Y-builds for the old stream continue on maintenance branch (with pre-defined schedule) to provide P-builds upon Java release.
+						// Then the Y-build for the new stream is initially disabled and enabled manually later, when the beta branch for the subsequent Java release is set up.
+						// If no Java release is scheduled soon, the old Y-build stream will just be dormat (like the old I-build), but don't reset its schedule to not overwrite it in the other case.
+						builds.Y.streams["${PREVIOUS_RELEASE_VERSION}"].branch = env.MAINTENANCE_BRANCH
 					}
 				}
-				replaceAllInFile('JenkinsJobs/Builds/FOLDER.groovy', [
-					"(?<prefix># Schedule:.*\\R)(?s).*(?<suffix>\\R'''\\))" : "\${prefix}${I_BUILD_SCHEDULE}\${suffix}",
-				])
-				replaceAllInFile('JenkinsJobs/YBuilds/FOLDER.groovy', [
-					"(?<prefix># Schedule:.*\\R)(?s).*(?<suffix>\\R'''\\))" : "\${prefix}${Y_BUILD_SCHEDULE}\${suffix}",
-				])
 
 				gitCommitAllExcludingSubmodules("Update versions to ${NEXT_RELEASE_VERSION} in build scripts")
 			}
@@ -248,8 +255,12 @@ pipeline {
 				sh 'git checkout -b prepareMaintenance ${MAINTENANCE_BRANCH}'
 
 				script {
-					utilities.modifyJSON('JenkinsJobs/JobDSL.json') { jobs ->
-						jobs.Branches["${PREVIOUS_RELEASE_VERSION}"] = env.MAINTENANCE_BRANCH
+					utilities.modifyJSON('JenkinsJobs/JobDSL.json') { builds ->
+						builds.values().each{ buildType ->
+							def build = buildType.streams["${PREVIOUS_RELEASE_VERSION}"]
+							build.branch = env.MAINTENANCE_BRANCH
+							build.schedule = ''
+						}
 					}
 				}
 				replaceInFile('JenkinsJobs/Builds/build.jenkinsfile', [
@@ -258,10 +269,6 @@ pipeline {
 				replaceInFile('JenkinsJobs/Builds/DockerImagesBuild.jenkinsfile', [
 					'-b master' : "-b ${MAINTENANCE_BRANCH}",
 				])
-				replaceAllInFile('JenkinsJobs/Builds/FOLDER.groovy', [
-					"spec\\('''(?s).+?'''\\)" : "spec('')",
-					"'master'" : "'${MAINTENANCE_BRANCH}'",
-				])
 				replaceInFile('cje-production/buildproperties.txt', [
 					'BRANCH="master"' : "BRANCH=\"${MAINTENANCE_BRANCH}\"",
 				])
@@ -468,8 +475,8 @@ def readParameter(String name) {
 }
 
 @NonCPS
-def assignEnvVariable(String name, String value) {
-	env."${name}" = value
+def assignEnvVariable(String name, Object value) {
+	env[name] = value?.toString()
 	println("${name}=${value}")
 }
 

JenkinsJobs/Releng/promoteBuild.jenkinsfile

@@ -218,8 +218,9 @@ pipeline {
 							'previousReleaseVersion=.*' : "previousReleaseVersion=${BUILD_MAJOR}.${BUILD_MINOR}",
 							'previousReleaseVersionRepo=.*' : "previousReleaseVersionRepo=${BUILD_MAJOR}.${BUILD_MINOR}",
 						])
-						utilities.modifyJSON('JenkinsJobs/JobDSL.json') { jobs -> // Remove old I-build job for this release
-							jobs.Branches.remove("${BUILD_MAJOR}.${BUILD_MINOR}")
+						utilities.modifyJSON('JenkinsJobs/JobDSL.json') { builds ->
+							// Remove old I/Y-build jobs for this release
+							builds.values().each{ build -> build.streams.remove("${BUILD_MAJOR}.${BUILD_MINOR}") }
 						}
 
 						utilities.gitCommitAllExcludingSubmodules("Update previous release version to ${BUILD_MAJOR}.${BUILD_MINOR} GA across build scripts")
@@ -239,10 +240,6 @@ pipeline {
 						utilities.replaceAllInFile('cje-production/buildproperties.txt', [
 							'ECLIPSE_RUN_REPO="https://download.eclipse.org/eclipse/updates/.*"' : "ECLIPSE_RUN_REPO=\"${RELEASE_P2_REPOSITORY}\"",
 						])
-						// Disable Y-build schedule only after the previous releases job is not updated (on the master) anymore, because Y-build jobs run beyond the Eclipse release if a new java release is imminent
-						utilities.replaceAllInFile('JenkinsJobs/YBuilds/FOLDER.groovy', [
-							"spec\\('''(?s).+?'''\\)" : "spec('')",
-						])
 						utilities.gitCommitAllExcludingSubmodules("Update ${MAINTENANCE_BRANCH} branch with release version for ${BUILD_MAJOR}_${BUILD_MINOR}+ changes")
 					}
 					// Switch back to master for subsequent parts of this pipeline

JenkinsJobs/YBuilds/FOLDER.groovy

@@ -5,23 +5,21 @@ folder('YPBuilds') {
   description('Builds and tests for the beta java builds.')
 }
 
-for (entry in config.Branches.entrySet()){
-	def STREAM = entry.key
-	def BRANCH = entry.value
+config.Y.streams.each{ STREAM, configuration ->
 
 	pipelineJob('YPBuilds/Y-build-' + STREAM){
 		description('Daily Maintenance Builds.')
+		disabled(configuration.disabled?.toBoolean() ?: false)
 		properties {
 			pipelineTriggers {
 				triggers {
 					cron {
-						spec('''TZ=America/Toronto
+						spec(configuration.schedule ? """TZ=America/Toronto
 # Format: Minute Hour Day Month Day-of-week (1-7)
 # - - - Beta Java Eclipse SDK builds - - - 
 # Schedule: 10 AM every second day (and every day in Java RC phase)
-0 10 * 8-10 2,4,6
-0 10 1-26 11 2,4,6
-''')
+${configuration.schedule}
+""" : '')
 					}
 				}
 			}
@@ -30,7 +28,7 @@ for (entry in config.Branches.entrySet()){
 			cpsScm {
 				lightweight(true)
 				scm {
-					github('eclipse-platform/eclipse.platform.releng.aggregator', BRANCH)
+					github('eclipse-platform/eclipse.platform.releng.aggregator', configuration.branch)
 				}
 				scriptPath('JenkinsJobs/Builds/build.jenkinsfile')
 			}

JenkinsJobs/YBuilds/Y_unit_tests.groovy

@@ -8,7 +8,7 @@ def TEST_CONFIGURATIONS = [
 	[os: 'win32' , ws:'win32', arch: 'x86_64' , javaVersion: 21, agentLabel: 'qa6xd-win11'        , javaHome: "'C:\\\\Program Files\\\\Eclipse Adoptium\\\\jdk-21.0.5.11-hotspot'" ],
 ]
 
-for (STREAM in config.Branches.keySet()){
+for (STREAM in config.Y.streams.keySet()){
 	def MAJOR = STREAM.split('\\.')[0]
 	def MINOR = STREAM.split('\\.')[1]
 	for (TEST_CONFIG in TEST_CONFIGURATIONS){

RELENG.md

@@ -6,14 +6,10 @@
 
 The (Create Jobs)[https://ci.eclipse.org/releng/job/Create%20Jobs/] job is used to populate the jenkins subfolders with the jobs defined in (JenkinsJobs)[JenkinsJobs] groovy files. There are 2 Process Job DSLs steps, the first looks for FOLDER.groovy files and creates the folders, the second creates the jobs themselves.
 
-Since not every folder needs to be updated every release cycle (for example (JenkinsJobs/YBuilds)[JenkinsJobs/YBuilds]) the currently active folders need to be explicitly listed in the Process Job DSLs step of the build. Likewise, unless you want the YBuilds to be recreated when they are no longer needed, the YBuilds folder will need to be removed from Create Jobs after the release. 
-
 Create Jobs *must be run manually*. Unfortunately JobDSL needs to be run by a specific user, so the build cannot be automatically started by a timer or when it detects jenkins changes without installing an additional plugin like (Authorize Project)[https://plugins.jenkins.io/authorize-project/], which supposedly still works but is abandoned and I (Sam) have not had time to investigate further or find alternatives. This means that while any committer can make changes to the Jenkins Jobs in git, someone with Jenkins rights will have to start the build to implement those changes.
 
-Exceptions: 
-  - (StartSmokeTests)[https://ci.eclipse.org/releng/job/Start-smoke-tests/] predates the rest of the groovy migrations and changing the script to fit JobDSL would have just complicated it with little gain so it was left as is. The source file ((StartSmokeTests.groovy)[JenkinsJobs/SmokeTests/StartSmokeTests.groovy]) is kept with the rest of the smoke test groovy files, but if JobDSL tries to build it it fails so instead of following the normal `JenkinsJobs/FOLDER/*.groovy` format, smoke tests are listed in Create Jobs as `JenkinsJobs/SmokeTests/smoke_*.groovy` specifically.
-
-Currently jobs also need to be deleted manually.
+Obsolete jobs have to be deleted manually.
+They are not deleted automatically as some may be are still in use for a short time (e.g. Y-builds if a java-release is imminent) or to serve as reference in case the new jobs have problems.
 
 **The JenkinsJobs Folder**
 
@@ -52,12 +48,12 @@ The builds themselves and their unit tests are in the (Y Builds)[JenkinsJobs/YBu
 
 ## Setting Up New Builds
 
-When the JDT team is ready they will raise an issue to create new Y and P builds and supply the name of the new branch, usually BETA_JAVA##.
+When the JDT team is ready they will raise an issue to create new Y builds and supply the name of the new branch, usually `BETA_JAVA##`.
 
 **Things to Do:**
-  * Update the Y-build (Y_build.groovy)[JenkinsJobs/YBuilds/Y_build.groovy].
+  * Update the Y-build configuration in the (build.jenkinsfile)[JenkinsJobs/Builds/build.jenkinsfile]
     - Update `branchLabel` and `typeName` to the name of the new java version
+  * Remove the disablement of the current stream in the Y-build configuration in the (JobDSL.json)[JenkinsJobs/JobDSL.json] (should be the only Y-build stream).
   * Update and rename the java repository files in (cje-production/streams)[cje-production/streams]
-    - Repos without a BETA_JAVA## branch should be set to master
-  * Add unit tests for the new java version in (JenkinsJobs/YBuilds)[JenkinsJobs/YBuilds]
-  * Add Y builds to (Create Jobs)[https://ci.eclipse.org/releng/job/Create%20Jobs/] in Jenkins if they've been removed
+    - Repos without a `BETA_JAVA##` branch should be set to master
+  * Add unit tests for the new java version in (JenkinsJobs/YBuilds)[JenkinsJobs/YBuilds] and (build.jenkinsfile)[JenkinsJobs/Builds/build.jenkinsfile]

cje-production/buildproperties.txt

@@ -33,6 +33,7 @@ TMP_DIR="tmp"
 DOWNLOAD_HOST="download.eclipse.org"
 BUILD_TO_COMPARE_SITE="ftp.osuosl.org/pub/eclipse/eclipse/updates"
 LOCAL_REPO="localMavenRepo"
+NEXT_JAVA_RELEASE_DATE=""
 
 # Base builder parameters
 BASEBUILDER_TAG="4.37"