feat(23570): Add controller for workspace backup #1530
Conversation
Allda
requested review from
akurinnoy,
dkwon17,
ibuziuk and
rohanKanojia
as code owners
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Allda The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Allda
force-pushed
the
23570
branch
2 times, most recently
from
42dd45c to
dffd7e6
Compare
|
@Allda : Really appreciate you taking the time to contribute this in such a short time. 🎉 Could you please also fill out the “Is it tested? How?” section in the PR template? It’ll help reviewers and future contributors verify the change more easily. Thanks again for your effort! 🙌 |
|
I tested this PR and it seems to work.
config:
workspace:
backupCronJob:
enable: true
schedule: "*/3 * * * *"
|
Allda
force-pushed
the
23570
branch
3 times, most recently
from
0bc74b1 to
8427ba5
Compare
|
/retest |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1530 +/- ##
==========================================
+ Coverage 34.09% 35.30% +1.21%
==========================================
Files 160 161 +1
Lines 13348 13802 +454
==========================================
+ Hits 4551 4873 +322
- Misses 8487 8599 +112
- Partials 310 330 +20 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Could you please share more details about the created Job? Are there any events or logs? I tried it today with a similar config (only difference is registry location) and the push Job passed. |
|
Here is the job's yaml: job-devworkspace-backup-29mb2.yaml There weren't any pod logs (since the pod never started) but only a lot of the Screen.Recording.2025-11-18.at.3.59.33.PM.movOutput of |
A registry configuration is now stored under a separated nested struct. Signed-off-by: Ales Raszka <[email protected]>
A SA is created for every backup workspace to avoid ownership conflict. Signed-off-by: Ales Raszka <[email protected]>
A switching a based image to podman allowed us to run a backup job as a regular user 1000 without any privileged escalation. Signed-off-by: Ales Raszka <[email protected]>
Signed-off-by: Ales Raszka <[email protected]>
Signed-off-by: Ales Raszka <[email protected]>
|
@Allda : Edit: I was able to resolve this issue by granting additional permissions to ServiceAccount , however I'm not sure whether this is required step or some issue: After updating ServiceAccount permissions, I'm able to see backup job getting executed and creating images on the configured registry: 
Question: Will the backup image will be platform dependent? I see only I'm also facing the same issue as David. I was trying to test your changes on CRC. Environment:OS: Linux CRC Version: Steps to Reproduce:
config:
workspace:
backupCronJob:
enable: true
registry:
authSecret: dockerhub-push-secret
path: docker.io/rohankanojia
schedule: '* * * * *'
# run From DWO root dir
oc create -f samples/code-latest.yaml
oc patch devworkspace code-latest \
--type=merge \
-p '{"spec": {"started": false}}'
Upon checking details I see this error: Perhaps pod creation is getting forbidden by OpenShift SecurityContextConstraints due to this: SecurityContext: &corev1.SecurityContext{
RunAsUser: ptr.To[int64](1000),
}, |
|
@rohanKanojia @dkwon17 I managed to find an alternative that doesn't violate any OCP-specific security constraints. The I tested it locally on Kind and remotely on OCP 4.20 cluster. |
|
@Allda : Thank you! I can confirm that this approach works without any explicit configuration. I can see the backup image being pushed to the configured registry. However, I see image has a different format I tested on CRC with OpenShift 4.20.1 |
Yes, oras artifact is not "real" image so it has a different config. The images that are produced by the oras should have following manifest: |
| } | ||
|
|
||
| func (r *BackupCronJobReconciler) copySecret(workspace *dw.DevWorkspace, ctx context.Context, sourceSecret *corev1.Secret, log logr.Logger) (namespaceSecret *corev1.Secret, err error) { | ||
| existingNamespaceSecret := &corev1.Secret{} |
There was a problem hiding this comment.
Can we use https://github.com/Allda/devworkspace-operator/blob/48b0eebf7a3b41bb7325c62010775da517675a3a/pkg/provision/sync/sync.go#L47 instead to create/update the secret?
There was a problem hiding this comment.
I tried that already but didn't manage to make it work. The secret was ussually created by this function but in many cases raised an error that caused it to fail.
There was a problem hiding this comment.
I think it was NotInSyncError, but only happening in the tests. Is this expected error that I should ignore/handle?
There was a problem hiding this comment.
It is hard to say, we have to know the reason.
Signed-off-by: Ales Raszka <[email protected]>
Signed-off-by: Ales Raszka <[email protected]>
project-backup/Containerfile
Outdated
| # | ||
|
|
||
| FROM quay.io/konflux-ci/oras:3d83c68 AS oras | ||
| FROM registry.access.redhat.com/ubi10/ubi@sha256:8405dd7146117f019670429f93ce044f0839f47ff81ec45bb53cf528f1f6ce11 |
There was a problem hiding this comment.
Not sure if it has been discussed, but what about using upstream version of the image?
https://quay.io/repository/devfile/base-developer-image?tab=tags
cc @dkwon17
There was a problem hiding this comment.
Good catch, for the upstream codebase, I'd prefer we use: quay.io/devfile/base-developer-image:ubi10-latest
| return err | ||
| } | ||
|
|
||
| if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, sa, func() error { return nil }); err != nil { |
There was a problem hiding this comment.
| } | ||
|
|
||
| func (r *BackupCronJobReconciler) copySecret(workspace *dw.DevWorkspace, ctx context.Context, sourceSecret *corev1.Secret, log logr.Logger) (namespaceSecret *corev1.Secret, err error) { | ||
| existingNamespaceSecret := &corev1.Secret{} |
| return pvcName, constants.DefaultProjectsSourcesRoot, nil | ||
|
|
||
| } else if _, ok := storageProvisioner.(*storage.CommonStorageProvisioner); ok { | ||
| pvcName := "claim-devworkspace" |
There was a problem hiding this comment.
I think we can reuse, probably move to constants.go
https://github.com/devfile/devworkspace-operator/blob/main/pkg/config/configmap/property.go#L25
There was a problem hiding this comment.
I moved it to constants and updated the controller to remove the hardcoded name.
| err := r.Get(ctx, req.NamespacedName, dwOperatorConfig) | ||
| if err != nil { | ||
| log.Error(err, "Failed to get DevWorkspaceOperatorConfig") | ||
| r.stopCron(log) |
There was a problem hiding this comment.
I don't think it is a reason to stop a cron.
There was a problem hiding this comment.
I followed the same logic you already have in the cleanup controller. Should I change it? https://github.com/devfile/devworkspace-operator/blob/main/controllers/cleanupcronjob/cleanupcronjob_controller.go#L161
There was a problem hiding this comment.
+1 to keep it as is, I believe this case is needed for when the global DWOC has been deleted, which in that case it would make sense to keep stopCron
|
@tolusha for some reason GitHub is not letting me comment on your comment: #1530 (comment) IMHO I would prefer if we didn't add an annotation to the DevWorkspaces to avoid potentially sending a high number of requests to the apiserver |
- Moving extraArgs to Oras config section - Unify default values - Change UBI base image - Use constant for the PVC name Signed-off-by: Ales Raszka <[email protected]>
|
|
||
| type RegistryConfig struct { | ||
| // A registry where backup images are stored. Images are stored | ||
| // in {registry}/backup-${DEVWORKSPACE_NAMESPACE}-${DEVWORKSPACE_NAME} |
There was a problem hiding this comment.
| // in {registry}/backup-${DEVWORKSPACE_NAMESPACE}-${DEVWORKSPACE_NAME} | |
| // in {path}/backup-${DEVWORKSPACE_NAMESPACE}-${DEVWORKSPACE_NAME} |
6 participants
A new backup controller orchestrates a backup process for workspace PVC. A new configuration option is added to DevWorkspaceOperatorConfig that enables running regular cronjob that is responsible for backup mechanism. The job executes following steps:
The last step is currently not fully implemented as it requires running a buildah inside the container and it will be delivered as a separate feature.
Issue: eclipse-che/che#23570
What does this PR do?
What issues does this PR fix or reference?
Is it tested? How?
The feature has been tested locally and using integration tests. Following configuration should be added to the config to enable this feature:
After a config is added, stop any workspace and wait till a backup job is created.
The job creates a backup and push image to registry
PR Checklist
/test v8-devworkspace-operator-e2e, v8-che-happy-pathto trigger)v8-devworkspace-operator-e2e: DevWorkspace e2e testv8-che-happy-path: Happy path for verification integration with Che