Add ability to configure custom init containers

[akurinnoy]

What does this PR do?

This PR adds the ability for cluster administrators to configure custom init containers that run in all workspace pods via the DWOC.
So, now administrators can:

• Inject arbitrary init containers into all workspaces via config.workspace.initContainers.
• Override the built-in init-persistent-home logic by providing a custom container with the same name.

What issues does this PR fix or reference?

https://issues.redhat.com/browse/CRW-9373
https://issues.redhat.com/browse/CRW-9367

Is it tested? How?

Deploy the DevWorkspace Operator with these changes

Test 1: Custom init-persistent-home script execution

1. Apply a DWOC with a custom init container:

   kubectl apply -f - <<EOF
   apiVersion: controller.devfile.io/v1alpha1
   kind: DevWorkspaceOperatorConfig
   metadata:
     name: devworkspace-operator-config
     namespace: $OPERATOR_NAMESPACE
   config:
     workspace:
       persistUserHome:
         enabled: true
       initContainers:
         - name: init-persistent-home
           args:
             - |
               echo "Custom home init executed" > /home/user/.custom_init_marker
               mkdir -p /home/user/custom-config
               echo "enterprise-config" > /home/user/custom-config/settings.txt
   EOF

2. Create a test workspace and wait for it to start

   kubectl apply -f - <<EOF
   apiVersion: workspace.devfile.io/v1alpha2
   kind: DevWorkspace
   metadata:
     name: test1-custom-home
     namespace: $WORKSPACE_NAMESPACE
   spec:
     started: true
     template:
       attributes:
         controller.devfile.io/storage-type: per-user
       components:
         - name: tooling
           container:
             image: quay.io/devfile/universal-developer-image:latest
             memoryLimit: 2Gi
             memoryRequest: 256Mi
   EOF

   Wait for workspace to run:

   kubectl wait --for=condition=Ready devworkspace/test1-custom-home -n $WORKSPACE_NAMESPACE --timeout=5m

3. Verify the init container was injected:

   POD_NAME=$(kubectl get pods -n $WORKSPACE_NAMESPACE -l controller.devfile.io/devworkspace_name=test1-custom-home -o jsonpath='{.items[0].metadata.name}')

   • Verify marker file

     kubectl exec -n $WORKSPACE_NAMESPACE $POD_NAME -c tooling -- cat /home/user/.custom_init_marker

     Expected output

     Custom home init executed
     

   • Verify config file

     kubectl exec -n $WORKSPACE_NAMESPACE $POD_NAME -c tooling -- cat /home/user/custom-config/settings.txt

     Expected output

     enterprise-config

4. Cleanup

   kubectl delete devworkspace test1-custom-home -n $WORKSPACE_NAMESPACE

Test 2: Default init-persistent-home behavior

1. Reset DWOC to default (no custom init)

   kubectl apply -f - <<EOF
   apiVersion: controller.devfile.io/v1alpha1
   kind: DevWorkspaceOperatorConfig
   metadata:
     name: devworkspace-operator-config
     namespace: $OPERATOR_NAMESPACE
   config:
     workspace:
       persistUserHome:
         enabled: true
   EOF

2. Create workspace with persistent home enabled

   kubectl apply -f - <<EOF
   apiVersion: workspace.devfile.io/v1alpha2
   kind: DevWorkspace
   metadata:
     name: test2-backward-compat
     namespace: $WORKSPACE_NAMESPACE
   spec:
     started: true
     template:
       attributes:
         controller.devfile.io/storage-type: per-user
       components:
         - name: tooling
           container:
             image: quay.io/devfile/universal-developer-image:latest
             memoryLimit: 2Gi
             memoryRequest: 256Mi
   EOF

   Wait for workspace to run:

   kubectl wait --for=condition=Ready devworkspace/test2-backward-compat -n $WORKSPACE_NAMESPACE --timeout=5m

3. Verify default stow logic still works

   kubectl logs -n $WORKSPACE_NAMESPACE -l controller.devfile.io/devworkspace_name=test2-backward-compat -c init-persistent-home

   Expected output

   Checking for stow command
   Running stow command
   

4. Cleanup

   kubectl delete devworkspace test2-backward-compat -n $WORKSPACE_NAMESPACE

Test 3: Validation of init-persistent-home configuration

1. Apply DWOC with invalid command

   kubectl apply -f - <<EOF
   apiVersion: controller.devfile.io/v1alpha1
   kind: DevWorkspaceOperatorConfig
   metadata:
     name: devworkspace-operator-config
     namespace: $OPERATOR_NAMESPACE
   config:
     workspace:
       persistUserHome:
         enabled: true
       initContainers:
         - name: init-persistent-home
           image: busybox:latest
           command: ["/bin/bash"]  # Invalid - must be ["/bin/sh", "-c"]
           args:
             - echo "test"
   EOF

2. Create workspace

   kubectl apply -f - <<EOF
   apiVersion: workspace.devfile.io/v1alpha2
   kind: DevWorkspace
   metadata:
     name: test3-invalid-command
     namespace: $WORKSPACE_NAMESPACE
   spec:
     started: true
     template:
       attributes:
         controller.devfile.io/storage-type: per-user
       components:
         - name: tooling
           container:
             image: quay.io/devfile/universal-developer-image:latest
             memoryLimit: 2Gi
             memoryRequest: 256Mi
   EOF

   Wait a few seconds for reconciliation

3. Verify workspace fails with validation error

   kubectl get devworkspace test3-invalid-command -n $WORKSPACE_NAMESPACE -o jsonpath='{.status.phase}'

   Expected output

   Failed

   kubectl get devworkspace test3-invalid-command -n $WORKSPACE_NAMESPACE -o jsonpath='{.status.message}'

   Expected output

   Invalid init-persistent-home container: command must be exactly [/bin/sh, -c]

4. Cleanup

   kubectl delete devworkspace test3-invalid-command -n $WORKSPACE_NAMESPACE

Test 4: Multiple Custom Init Containers

1. Apply DWOC with multiple init containers

   kubectl apply -f - <<EOF
   apiVersion: controller.devfile.io/v1alpha1
   kind: DevWorkspaceOperatorConfig
   metadata:
     name: devworkspace-operator-config
     namespace: $OPERATOR_NAMESPACE
   config:
     workspace:
       persistUserHome:
         enabled: true
       initContainers:
         - name: init-persistent-home
           args:
             - |
               echo "Step 1: Home init" > /home/user/init-sequence.log
               date >> /home/user/init-sequence.log
               echo "Home directory initialized" >> /home/user/init-sequence.log
         - name: install-tools
           image: quay.io/devfile/universal-developer-image:latest
           command: ["/bin/sh", "-c"]
           args:
             - |
               echo "Step 2: Installing tools" >> /home/user/init-sequence.log
               date >> /home/user/init-sequence.log
               echo "Tools setup completed" > /home/user/tools-installed.txt
               echo "wget simulation completed" >> /home/user/init-sequence.log
           volumeMounts:
             - name: persistent-home
               mountPath: /home/user/
         - name: config-setup
           image: quay.io/devfile/universal-developer-image:latest
           command: ["/bin/sh", "-c"]
           args:
             - |
               echo "Step 3: Final config" >> /home/user/init-sequence.log
               date >> /home/user/init-sequence.log
               echo "Configuration setup completed" > /home/user/config-complete.txt
           volumeMounts:
             - name: persistent-home
               mountPath: /home/user/
   EOF

2. Create workspace

   kubectl apply -f - <<EOF
   apiVersion: workspace.devfile.io/v1alpha2
   kind: DevWorkspace
   metadata:
     name: test4-multiple-init
     namespace: $WORKSPACE_NAMESPACE
   spec:
     started: true
     template:
       attributes:
         controller.devfile.io/storage-type: per-user
       components:
         - name: tooling
           container:
             image: quay.io/devfile/universal-developer-image:latest
             memoryLimit: 2Gi
             memoryRequest: 256Mi
   EOF

   Wait for workspace to run:

   kubectl wait --for=condition=Ready devworkspace/test4-multiple-init -n $WORKSPACE_NAMESPACE --timeout=5m

3. Verify all init containers were injected and ran in order

   POD_NAME=$(kubectl get pods -n $WORKSPACE_NAMESPACE -l controller.devfile.io/devworkspace_name=test4-multiple-init -o jsonpath='{.items[0].metadata.name}')

   • Verify all 3 init containers are present

     kubectl get pod -n $WORKSPACE_NAMESPACE $POD_NAME -o jsonpath='{.spec.initContainers[*].name}'

     Expected output

     init-persistent-home install-tools config-setup

   • Verify the sequence log shows all 3 steps

     kubectl exec -n $WORKSPACE_NAMESPACE $POD_NAME -c tooling -- cat /home/user/init-sequence.log

     Expected output

     Step 1: Home init
     [timestamp]
     Home directory initialized
     Step 2: Installing tools
     [timestamp]
     wget simulation completed
     Step 3: Final config
     [timestamp]
     

   • Verify marker files from each init container

     kubectl exec -n $WORKSPACE_NAMESPACE $POD_NAME -c tooling -- cat /home/user/tools-installed.txt

     Expected output

     Tools setup completed

     kubectl exec -n $WORKSPACE_NAMESPACE $POD_NAME -c tooling -- cat /home/user/config-complete.txt

     Expected output

     Configuration setup completed

4. Cleanup

   kubectl delete devworkspace test4-multiple-init -n $WORKSPACE_NAMESPACE

PR Checklist

• E2E tests pass (when PR is ready, comment /test v8-devworkspace-operator-e2e, v8-che-happy-path to trigger)
  • v8-devworkspace-operator-e2e: DevWorkspace e2e test
  • v8-che-happy-path: Happy path for verification integration with Che

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[rohanKanojia]

I tested with the steps provided in the PR description and can confirm it works as expected ✔️

[akurinnoy]

/retest

[tolusha]

I think we shouldn't take this clause into account
cc @dkwon17

[akurinnoy]

To me, it appears to align with the documentation:

Note: If persistUserHome.enabled is false, any init-persistent-home container is ignored.

[dkwon17]

To keep it simple, I agree with @tolusha ,

Note: If persistUserHome.enabled is false, any init-persistent-home container is ignored.

Unless I'm mistaken, the above documentation is introduced in this PR, so removing the clause is okay on my end

[akurinnoy]

@dkwon17 you're right about the documentation. It comes from this:

devworkspace-operator/apis/controller/v1alpha1/devworkspaceoperatorconfig_types.go

Lines 230 to 243 in 1110865

	type PersistentHomeConfig struct {
	// Determines whether the `/home/user/` directory in workspaces should persist between
	// workspace shutdown and startup.
	// Must be used with the 'per-user'/'common' or 'per-workspace' storage class in order to take effect.
	// Disabled by default.
	Enabled *bool `json:"enabled,omitempty"`
	// Determines whether the init container that initializes the persistent home directory should be disabled.
	// When the `/home/user` directory is persisted, the init container is used to initialize the directory before
	// the workspace starts. If set to true, the init container will not be created.
	// This field is not used if the `workspace.persistUserHome.enabled` field is set to false.
	// Enabled by default.
	DisableInitContainer *bool `json:"disableInitContainer,omitempty"`
	}

I understand that PersistentHomeConfig.Enabled and PersistentHomeConfig.DisableInitContainer serve different purposes and work together.

Without this check, custom init-persistent-home container will be processed even when Enabled: false, which is inconsistent.

[dkwon17]

@akurinnoy thank you for the PR,

I want to suggest a different approach for this feature:

After all of the initconatiners have been accumulated:

• initcontainers create from devfile
• container for persistent home (if enabled)
• container for project-clone

We can do a strategic merge around here for the init containers, where the patch is the content of config.workspace.initContainers.

We already do something similar here with container contributions, the code uses strategic merge from the devfile api to merge containers

WDYT cc @rohanKanojia @tolusha ?

[rohanKanojia]

@akurinnoy : Hello, I noticed E2E failure seems related to changes in this PR:

�[38;5;9m• [FAILED] [104.258 seconds]�[0m
�[0m[Custom Init Container Tests] �[38;5;243mCustom init-persistent-home container �[38;5;9m�[1m[It] Create workspace and verify custom init-persistent-home executed�[0m
�[38;5;243m/go/src/github.com/devfile/devworkspace-operator/test/e2e/pkg/tests/custom_init_container_tests.go:67�[0m

  �[38;5;9m[FAILED] Expected
      <string>: command terminated with exit code 1
      FAILED
      
  to contain substring
      <string>: SUCCESS�[0m
  �[38;5;9mIn �[1m[It]�[0m�[38;5;9m at: �[1m/go/src/github.com/devfile/devworkspace-operator/test/e2e/pkg/tests/custom_init_container_tests.go:94�[0m �[38;5;243m@ 11/19/25 14:53:40.407�[0m
�[38;5;243m------------------------------�[0m
�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m2025/11/19 14:53:50 Cleaning up test resources are disabled


�[38;5;9m�[1mSummarizing 1 Failure:�[0m
  �[38;5;9m[FAIL]�[0m �[0m[Custom Init Container Tests] �[38;5;243mCustom init-persistent-home container �[38;5;9m�[1m[It] Create workspace and verify custom init-persistent-home executed�[0m
  �[38;5;243m/go/src/github.com/devfile/devworkspace-operator/test/e2e/pkg/tests/custom_init_container_tests.go:94�[0m

�[38;5;9m�[1mRan 14 of 18 Specs in 157.983 seconds�[0m
�[38;5;9m�[1mFAIL!�[0m -- �[38;5;10m�[1m13 Passed�[0m | �[38;5;9m�[1m1 Failed�[0m | �[38;5;11m�[1m0 Pending�[0m | �[38;5;14m�[1m4 Skipped�[0m

[akurinnoy]

/retest

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: akurinnoy, tolusha
Once this PR has been reviewed and has the lgtm label, please assign dkwon17 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tolusha]

Good job!