test: enforce restricted PSS for CI user namespace by danish9039 · Pull Request #3487 · kubeflow/community-distribution

danish9039 · 2026-05-26T21:57:32Z

Summary of Changes

This PR makes CI validate Kubeflow user workloads under Kubernetes Pod Security Standards restricted.

Relabel the CI Profile namespace kubeflow-user-example-com to pod-security.kubernetes.io/enforce=restricted after the Profile Controller creates it.
Keep the production/default Profile Controller behavior unchanged.
Update workflow path filters so shared PSS test-script and local overlay changes rerun affected component workflows.
Make CI workload examples compatible with restricted PSS across Katib, KServe, Notebooks, Workspaces, Trainer, Training Operator, Ray, and Istio validation.
Move Katib, Trainer, and Training Operator restricted-PSS test mutations into local overlays instead of live kubectl patch or generated temp-overlay logic.
Move PR-added script debug dumps into GitHub Actions failure-log artifacts while keeping test waits fail-fast.

Notes

This is intentionally CI/test focused. It does not change the default customer Profile namespace policy.

The new application changes are local Kubeflow manifests overlays under applications/*/overlays. They do not edit synchronized applications/*/upstream folders or common/* manifests.

Follow-up notes are tracked separately for keeping the copied Katib CI config in sync during future Katib syncs and for removing the temporary Training Operator overlay after the final 26.03.1 path is available.

Validation

Latest head 7adb2d55 is green on all technical checks.

DCO passes.
Formatting checks pass for YAML, Bash, and Python.
Full Kubeflow integration passes.
Component workflows pass, including Katib, Trainer, Training Operator, notebooks, pipelines, workspaces, KServe, Ray, Istio validation, dashboard, and Dex/OAuth2.
Local validation included bash -n, shellcheck, YAML parsing, kustomize build for the new overlays, rendered field assertions, git diff --check, and a guard confirming no applications/*/upstream/** changes.

tide is still waiting for review labels.

Contributor Checklist

I have tested these changes with kustomize. See Installation Prerequisites.
All commits are signed-off to satisfy the DCO check.
I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue.

Overwrites the default baseline namespace label created by the profile controller exclusively during CI tests. This guarantees that test workloads simulate strict pod security standard (PSS) restricted enforcements without modifying production default baselines. Signed-off-by: abdullahpathan22 <abdullahpathan22@users.noreply.github.com>

- Broaden katib triggers: tests/katib_install.sh -> tests/katib* - Broaden pipeline triggers: individual files -> tests/pipeline* - Add tests/pipeline* trigger to pipeline_run_from_notebook workflow - Replace dead experimental/security/PSS/* path (directory no longer exists) with actual test files: tests/kubeflow_profile_install.sh and tests/PSS_enable.sh across all affected workflows Note: Dashboard/profiles directory paths are intentionally kept as-is since applications/profiles/ does not exist yet on master. Those paths will be updated once the directory restructure lands. Signed-off-by: abdullahpathan22 <abdullahpathan22@users.noreply.github.com>

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

github-actions · 2026-05-26T21:57:41Z

Welcome to the Kubeflow Manifests Repository

Thanks for opening your first PR. Your contribution means a lot to the Kubeflow community.

Before making more PRs:
Please ensure your PR follows our Contributing Guide.
Please also be aware that many components are synchronizes from upstream via the scripts in /scripts.
So in some cases you have to fix the problem in the upstream repositories first, but you can use a PR against kubeflow/manifests to test the platform integration.

Community Resources:

CNCF Slack channels: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
Community meetings: https://www.kubeflow.org/docs/about/community/

Thanks again for helping to improve Kubeflow.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR updates CI to validate/enforce Kubernetes Pod Security Standards (PSS) settings for Kubeflow profile namespaces, and wires the related test scripts into multiple GitHub Actions path filters so relevant workflows run when these scripts change.

Changes:

Update kubeflow_profile_install.sh to label the Kubeflow user namespace with pod-security.kubernetes.io/enforce=restricted and enforce-version=latest, and assert the labels are applied.
Add tests/kubeflow_profile_install.sh and tests/PSS_enable.sh to workflow on.pull_request.paths filters across several test workflows.
Broaden some workflow path filters (e.g., tests/pipeline*, tests/katib*) to cover more related changes.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/kubeflow_profile_install.sh	Labels the profile namespace for PSS restricted + adds assertions for applied labels.
.github/workflows/workspaces_pipeline_run_test.yaml	Triggers workflow on profile/PSS test script changes.
.github/workflows/volumes_web_application_test.yaml	Triggers workflow on profile install script changes.
.github/workflows/training_operator_test.yaml	Triggers workflow on profile/PSS test script changes.
.github/workflows/trainer_test.yaml	Triggers workflow on profile/PSS test script changes.
.github/workflows/pipeline_test.yaml	Broadens pipeline path trigger + triggers on profile/PSS test script changes.
.github/workflows/pipeline_run_from_notebook.yaml	Adds pipeline wildcard + profile/PSS script triggers.
.github/workflows/kserve_test.yaml	Triggers workflow on profile/PSS test script changes.
.github/workflows/kserve_models_web_application_test.yaml	Triggers workflow on profile install script changes.
.github/workflows/katib_test.yaml	Broadens katib path trigger + triggers on profile/PSS test script changes.
.github/workflows/istio_validation.yaml	Triggers workflow on profile/PSS test script changes.
.github/workflows/dex_oauth2-proxy_test.yaml	Triggers workflow on profile/PSS test script changes.

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

juliusvonkohout · 2026-06-01T14:27:06Z

 kubectl get pods -n kubeflow-system -l app.kubernetes.io/name=trainer
 kubectl get clustertrainingruntimes torch-distributed

+kubectl patch clustertrainingruntime torch-distributed --type=json -p='[


this must also be a proper overlay and maybe upstreamed.

There might already bea recent Pr for this.

Moved the torch-distributed runtime changes into applications/trainer/overlays/runtimes-restricted; upstream follow-up is noted separately.

Checked kubeflow/trainer#3066 ; it helps with override support, but this PR still needs the local runtime default overlay.

Split to kubeflow/trainer#3702; #3487 keeps the CI-only runtime overlay until that lands.

juliusvonkohout · 2026-06-01T14:29:00Z

please a proper overlay. this has to survive the next release and then we can already remove it. Please raise a follow up PR to remove it such that i can merge it afte rthe 26.03.1 release.

Moved this into applications/training-operator/overlays/kubeflow-restricted-pss and noted the follow-up removal after 26.03.1.

Kept as a proper overlay and logged the removal follow-up after 26.03.1.

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

tests/workspaces_pipeline_run_test.sh:1

This JSONPatch adds nested fields under /spec/podTemplate/securityContext/... and /spec/podTemplate/containerSecurityContext/.... JSONPatch add will fail if the parent object (e.g., /spec/podTemplate/securityContext) does not already exist. To make this robust across different base WorkspaceKind manifests, patch the parent objects in one operation (e.g., add/merge the full securityContext / containerSecurityContext objects) or use a strategic/merge patch (--type=merge) targeting the parent keys.
tests/kserve_test.sh:1
The polling logic treats any response other than 403 as success (twice). This can still false-positive on 404/503 while routes/backends are not ready—exactly the failure mode mentioned in the comment above the loop. Prefer checking for the expected success code (typically 200) or at least explicitly excluding 404/503 (and possibly 000) from counting toward STABLE_POLL_COUNT, to prevent passing the policy wait while the service is still unreachable.

#!/bin/bash

juliusvonkohout · 2026-06-05T11:32:09Z

Can you upstream this or make sure to only patch the securitycontext, but not the image for example ?

Split to kubeflow/katib#2683; the intended delta is only securityContext/injection config, not image changes.

juliusvonkohout · 2026-06-05T11:33:37Z

+        serviceAccountName: default-editor
+        containers:
+        - name: ray-worker
+          image: rayproject/ray:2.44.1-py311-cpu


may you update ray and kuberay to the latest available versions please

so not just this file, also the kuberay operator

Moved the Ray/KubeRay version update to #3528 and removed it from this PR.

juliusvonkohout · 2026-06-05T11:36:41Z


-cd applications/katib/upstream && kustomize build installs/katib-with-kubeflow | kubectl apply -f - && cd ../../../
-
+kustomize build applications/katib/overlays/security | kubectl apply -f -


is it possible to have 2 steps for this? since these scripts are also used for the single step install and would reach customers.

Maybe you have to use kustomize components instead of overlays to have a cleaner separation. If componnts do not work you can also stay with the overlay

you caould have an if around it as in our kind installation that you install the extra security stuff on top only if you are in GHA. That goes for all installation scripts.

katib_install.sh now keeps the default install outside GHA and applies the security overlay only in GHA.

juliusvonkohout · 2026-06-05T11:40:21Z

 export KSERVE_M2M_TOKEN="$(kubectl -n ${NAMESPACE} create token default-editor)"
 export KSERVE_TEST_NAMESPACE=${NAMESPACE}

+kubectl patch clusterstoragecontainer default --type=merge --patch '{


can you add this directly to the kserve installation as patch ? Maybe even upstream it ?

Split the durable KServe patch to #3527; #3487 keeps a GHA-only live patch until that lands.

juliusvonkohout · 2026-06-05T11:42:17Z

+kubectl label namespace "$KF_PROFILE" \
+  pod-security.kubernetes.io/enforce=restricted \
+  pod-security.kubernetes.io/enforce-version=latest \
+  --overwrite


only when you are in the GHA environments as in our kind script.

restricted relabeling now runs only under GITHUB_ACTIONS; regular installs stay at baseline.

juliusvonkohout · 2026-06-05T11:42:58Z

-kubectl -n kubeflow rollout status deployment/tensorboard-controller-deployment --timeout=60s
-kubectl -n kubeflow rollout status deployment/tensorboards-web-app-deployment --timeout=60s
-kubectl -n kubeflow rollout status deployment/volumes-web-app-deployment --timeout=60s
+kubectl -n kubeflow rollout status deployment/jupyter-web-app-deployment --timeout=180s


why do you increase the timeouts so much ?

Raised to 180s because CI hit real rollout delays; failures still surface with workflow artifacts.

juliusvonkohout · 2026-06-05T11:43:35Z

 kubectl wait --for=condition=Available deployment/jobset-controller-manager -n kubeflow-system --timeout=120s

-kustomize build upstream/overlays/runtimes | kubectl apply --server-side --force-conflicts -f -
+kustomize build overlays/runtimes-restricted | kubectl apply --server-side --force-conflicts -f -


the same comments as above. keep it like before for regular users, but if you are in gha add the extra security

trainer_install.sh now uses default runtimes outside GHA and restricted runtimes only in GHA.

juliusvonkohout · 2026-06-05T11:44:25Z


-cd applications/training-operator/upstream
-kustomize build overlays/kubeflow | kubectl apply --server-side --force-conflicts -f -
+kustomize build applications/training-operator/overlays/kubeflow-restricted-pss | kubectl apply --server-side --force-conflicts -f -


same here, only when you are in the GHA environment

training_operator_install.sh now uses the restricted overlay only in GHA.

juliusvonkohout · 2026-06-05T11:44:48Z

 kubectl apply -f /tmp/pytorch-job.yaml

-kubectl wait --for=jsonpath='{.status.conditions[0].type}=Created' pytorchjob.kubeflow.org/pytorch-simple -n $KF_PROFILE --timeout=60s
+kubectl wait --for=condition=Created pytorchjob.kubeflow.org/pytorch-simple -n "$KF_PROFILE" --timeout=180s


why do you increase the timeouts ?

Raised because CI can be slow before pods are Ready; the waits still fail on readiness/success conditions.

juliusvonkohout · 2026-06-05T11:45:34Z

if really needed this has to be upstreamed. CC @christian-heusel

Split this to kubeflow/notebooks#1229; #3487 keeps the CI-only live patch until that lands.

juliusvonkohout · 2026-06-05T11:46:24Z


+kubectl get clustertrainingruntime torch-distributed -o yaml


what is the purpose of this line ?

It prints the effective torch-distributed runtime so Trainer failures include the config under test.

juliusvonkohout · 2026-06-05T11:47:09Z

why do you change the timeouts ?

Raised because Katib trial creation/completion exceeded the old window in CI; waits still fail on conditions.

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

google-oss-prow · 2026-07-04T20:04:34Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kimwnasptd for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

danish9039 · 2026-07-04T20:43:57Z

Tracking the follow-up PRs split out from this restricted-PSS CI work:

Katib: config: add restricted PSS contexts for suggestions katib#2683 — adds restricted-compatible suggestion runtime contexts and metrics collector injection config.
KServe: kserve: set storage initializer security context #3527 — adds the durable storage initializer securityContext patch in manifests.
Trainer: runtimes: add restricted PSS security contexts trainer#3702 — updates Trainer v2 Torch runtime manifests and Helm templates for restricted PSS.
Workspaces/Notebooks: fix: set restricted PSS sample context notebooks#1229 — updates the JupyterLab WorkspaceKind sample on notebooks-v2.
Ray/KubeRay: ray: update KubeRay and Ray versions #3528 — aligns the experimental Ray/KubeRay versions separately from this CI gate.

Keeping Training Operator removal deferred until after 26.03.1.

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

abdullahpathan22 and others added 3 commits May 27, 2026 03:26

ci: complete restricted PSS trigger coverage

f3b8537

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

Copilot AI review requested due to automatic review settings May 26, 2026 21:57

google-oss-prow Bot added the do-not-merge/work-in-progress label May 26, 2026

google-oss-prow Bot requested review from juliusvonkohout and tarekabouzeid May 26, 2026 21:57

google-oss-prow Bot added the size/M label May 26, 2026

Copilot AI reviewed May 26, 2026

View reviewed changes

Comment thread tests/kubeflow_profile_install.sh Outdated

Comment thread tests/kubeflow_profile_install.sh Outdated

test: make CI workloads restricted-compatible

3eec781

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

google-oss-prow Bot added size/L and removed size/M labels May 27, 2026

danish9039 added 9 commits May 27, 2026 11:53

test: patch generated workloads for restricted PSS

4ab9185

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: fix restricted PSS CI follow-ups

56e3320

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: fix restricted generated workload checks

5ea5779

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: configure generated workload security contexts

6d5b8cf

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: wait for patched katib controller rollout

c78ed42

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: render katib controller security flag

f447b3d

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: enable katib security context config

aed22f0

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: patch katib suggestion security context

6a66e91

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: avoid kserve auth policy race

2b55af8

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

google-oss-prow Bot added size/XL and removed size/L labels May 27, 2026

danish9039 added 2 commits May 27, 2026 13:57

test: wait longer for notebooks rollout

40b41f0

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: patch workspacekind for restricted PSS

50bc03b

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

juliusvonkohout reviewed May 27, 2026

View reviewed changes

Comment thread tests/dashboard_install.sh Outdated

danish9039 added 3 commits May 28, 2026 00:10

test: add raw kserve security context

6a64a7f

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: add ray autoscaler security context

3a7d2ad

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: fail fast in dashboard debug

cb5cde5

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

juliusvonkohout reviewed Jun 1, 2026

View reviewed changes

Comment thread tests/notebooks_install.sh Outdated

juliusvonkohout reviewed Jun 1, 2026

View reviewed changes

juliusvonkohout mentioned this pull request Jun 4, 2026

test: enforce PSS restricted for CI user namespace #3444

Open

test: move restricted pss fixes into overlays

7adb2d5

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

danish9039 requested a review from Copilot June 5, 2026 08:07

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread .github/workflows/workspaces_pipeline_run_test.yaml

juliusvonkohout reviewed Jun 5, 2026

View reviewed changes

test: scope restricted pss changes to ci

074cd31

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: cover restricted pss pipeline gaps

caa14a1

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

google-oss-prow Bot added size/XXL and removed size/XL labels Jul 5, 2026


		cd applications/katib/upstream && kustomize build installs/katib-with-kubeflow \| kubectl apply -f - && cd ../../../

		kustomize build applications/katib/overlays/security \| kubectl apply -f -


		kubectl get clustertrainingruntime torch-distributed -o yaml

Uh oh!

Conversation

danish9039 commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of Changes

Notes

Related

Validation

Contributor Checklist

Uh oh!

github-actions Bot commented May 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danish9039 Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danish9039 commented May 26, 2026 •

edited

Loading

danish9039 Jun 5, 2026 •

edited

Loading

juliusvonkohout Jun 5, 2026 •

edited

Loading

juliusvonkohout Jun 5, 2026 •

edited

Loading