Skip to content

[Build] Unify clean-ups jobs as Jenkins pipelines and inline scripts #3322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 26, 2025
Merged

[Build] Unify clean-ups jobs as Jenkins pipelines and inline scripts #3322

merged 1 commit into from
Sep 26, 2025

Conversation

HannesWell
Copy link
Member

  • Migrate all clean-up jobs to pure Jenkinsfile/pipeline based jobs.
  • Unify the daily clean-up into one job that handles Eclispe and Equinox builds in parallel.
  • Move remaining two jobs to the 'Releng' folder as it already contains the manually maintained variant of the clean-up job for release artifacts.
  • Automatically run clean-up of release artifacts when promoting a release.

@HannesWell
Copy link
Member Author

This is a draft as it requires:

And a few more things have to be cleaned-up.

@HannesWell HannesWell mentioned this pull request Sep 15, 2025
Open
@HannesWell HannesWell force-pushed the unify-cleanup-jobs branch 2 times, most recently from b81ccea to 18c7393 Compare September 25, 2025 18:05
@HannesWell HannesWell mentioned this pull request Sep 25, 2025
Merged
@HannesWell
Copy link
Member Author

This is now almost ready and I was able to remove the need for all parameters of the cleanup release artifacts job and derive them automatically but implementing the rules that where followed manually into the pipeline, for example that only three releases are kept. Removing the handling of P-builds also allowed more simplifications.

It also considers the NEXT_JAVA_RELEASE_DATE build property introduced with

... to determine if the Y-build p2-repositories should be cleaned up.

As suggested in #3328 (comment), this also changes the Daily Cleanup of old Builds to keep always the Drops! at the download page of the latest six Y-builds. This corresponds to two weeks of regular Y-builds. The P2-repository is not affected by this!
@stephan-herrmann, @jarthana or @mpalat is that fine for you?
Due to the special schedule of Y-builds when a Java release is due, it's really difficult to find another good, automated way to clean-up the Y-build drops otherwise and that was the best solution, with the existing facilities, jobs and properties, that I could find. I did that based on the assumption that you are usually only interested in the latest Y-build results (probably test results) anyway and inspect them regularly and soon after a Y-build has completed.
Please let me know if that's a no go for you.

As a countermeasure, in case a drop should be kept for longer (regardless of Y-build, I-build or even older releases), this now also adds new way to mark a build as kept indefinitely based on

If marked as kept indefinitely, all clean-up jobs (the daily one and the one for releases), don't delete these builds, even if they were selected for deletion.
So if you encounter a special Y-build drop that should be kept for longer, then this could be used.

HannesWell
HannesWell commented Sep 25, 2025
View reviewed changes
JenkinsJobs/Cleanup/cleanupReleaseArtifacts.jenkinsfile Outdated
Comment on lines 74 to 78
if (release != latestMinorRelease || (getBuildProperty("${EP_ECLIPSE_DROPS}", release, 'NEXT_JAVA_RELEASE_DATE')?.isEmpty() ?: false)) {
// Delete Y-build p2-repositories for the latest release only if no Java release is scheduled shortly after that release
utilities.removeDropsOnRemote("${EP_ECLIPSE_UPDATES}", ["${versions.version}-Y-builds"])
}
//TODO: maybe keep always the last 10 children, if this is the latest release? Regardless of a new Java-release is close or not
Copy link
Member Author

@HannesWell HannesWell Sep 25, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This considers the new NEXT_JAVA_RELEASE_DATE that was introduced with

... to determine if the Y-build p2-repositories should be cleaned up or if a java release is imminent and the p2-repo should be kept until the next release, just as you said in

@mpalat, @jarthana or @stephan-herrmann, but I think the clean-up of be more robust (i.e. robust against that this variable was accidentally not set when the release cycle was prepared and it wasn't noticed and added afterwards), if we for example would just keep always a sufficient number Y-build p2-repositories created for the latest release, regardless of a Java release is imminent or not. For the final P-builds you usually just use the latest one, don't you?
So my suggest would be to always keep the last 10 Y-build p2-repos of the latest release.

Would that be fine for you as well?

@HannesWell HannesWell force-pushed the unify-cleanup-jobs branch 3 times, most recently from 11f6399 to 2eec042 Compare September 26, 2025 23:10
@HannesWell
Copy link
Member Author

As suggested in #3328 (comment), this also changes the Daily Cleanup of old Builds to keep always the Drops! at the download page of the latest six Y-builds. This corresponds to two weeks of regular Y-builds. The P2-repository is not affected by this!

Unfortunately the I didn't get any response to this and the other questions as the addressees are on vacation or unavailable.
But as I want to proceed as this contains many other changes I have further relaxed the policy for now to keep the drops of nine Y-builds and to always keep all Y-build p2-repos of the latest release, regardless of if a java release is due. As Y-builds are currently not active, no new builds are added and thus nothing will be deleted.
I'll create a follow-up PR soon with the goal to tighten these bounds again, which will hopefully receive more feedback. Ideally this is done before Y-builds are reactivated for Java-26, then nothing should be deleted without consent.

@HannesWell
[Build] Unify clean-up jobs as Jenkins pipelines and inline scripts
b832bf0 
Migrate all clean-up jobs to pure Jenkinsfile based pipeline jobs.

In this process the following enhancements are applied additionally:
- Unify the daily clean-up builds into one job that handles Eclipse and
Equinox builds in parallel.
- Implement the logic for all release/stable and I-build drop removals
upon releases into the job for the cleanup of release artifacts, so that
it doesn't need any input parameters.
- Continuously clean-up Y-build drops, with the policy to keep the last
nine builds (corresponds to three consecutive weeks on the regular build
schedule).
- Introduce the ability to manually mark a as kept-indfinitivly in order
to prevent the clean-up jobs from deleting it.
- Move the update of the eclipse/updates/index.html to redirect to the
update-site of the latest release into the (release) publication job.
@HannesWell HannesWell marked this pull request as ready for review September 26, 2025 23:23
@HannesWell HannesWell merged commit c172c81 into eclipse-platform:master Sep 26, 2025
5 of 6 checks passed
@HannesWell HannesWell deleted the unify-cleanup-jobs branch September 26, 2025 23:25
@stephan-herrmann
Copy link
Contributor

Unfortunately the I didn't get any response to this and the other questions as the addressees are on vacation or unavailable.

Sorry for not responding. I personally don't see much need for retaining lots of "old" Y-builds. My only use for previous Y-builds is for comparing test results. And every now and then the latest successful Y-build will be used to produce a P-build. That's it.

How Y-builds are consumed by early adopters I simply don't know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@HannesWell @stephan-herrmann