[KEP-2349]: Add MultiKueue support for external custom job

[khrm]

This KEP documents how we can add MultiKueue support for external custom job by using configmap and generic multikueue adaptor.

What type of PR is this?

/kind documentation

What this PR does / why we need it:

This introduces a new design for adding MultiKueue support for custom job.

Which issue(s) this PR fixes:

KEP for #2349

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	135555c
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/68b5de1008625a0008059dba

[k8s-ci-robot]

Welcome @khrm!

It looks like this is your first PR to kubernetes-sigs/kueue 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kueue has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @khrm. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[khrm]

/assign @tenzen-y @mimowo

[mimowo]

/ok-to-test
I will return to reviewing this after 0.13 is out.

[kannon92]

To me, this feels like something that should live with the external controller. This sounds like you are asking Kueue to create a mutating webhook for your controller.

Why can't you add this logic into tekton-kueue?

[khrm]

Let's say we want to remove something from manager cluster before copying to worker cluster, how will external controller handle this? The external controller itself shouldn't require the secret or mechanism to copy from manager to worker. Same during the sync from worker to manager.

[ravisantoshgudimetla]

Apologies for joining the party late. I feel the same - It looks like this is the responsibility of external controller. I am wondering what is wrong with external controller handling it? Is the cognitive load on plugin developer a concern. I felt like the user should be able to submit to the custom type -> have the logic to translate it a workload object -> Submit the workload to the management cluster -> Get the name of the cluster suggested by Multikueue -> Create and Sync the workload there. There are some workflow changes that can be tweaked but I'm wondering what is wrong with putting this burden on plugin developer and making it Multi-Kueue just a workload placement engine - similar to core k8s scheduler.

[khrm]

Actually, it's still the responsibility of external controller to generate workload.

The only thing we have exposed or made configurable is just ability of MultiKueue adaptor to sync resource and status between clusters.

The issue isn't the code. The issue is that we will need to run a sync controller of workload in the same namespace and give it access to the same secret which MultiKueue has access to. And there's no benefit to doing all this. It's just a sync behaviour which is common for all CRs.

Of course, if someone want to create an external controller, there's nothing stopping from doing that.

It made sense to have an external controller for local workload creation due to the complexity of lifecycle involved, but I don't see that in case of MultiKueue sync.

[ravisantoshgudimetla]

Actually, it's still the responsibility of external controller to generate workload.

You mean the kueue workload CR? I thought you are a registered type, you cannot?

The issue is that we will need to run a sync controller of workload in the same namespace and give it access to the same secret which MultiKueue has access to. And there's no benefit to doing all this. It's just a sync behaviour which is common for all CRs.

What I am suggesting is much simpler, you can create your own controller in another namespace and have RBAC specific to CRD to do syncing for workload clusters, no need to rely on secret present in multi-kueue namespace. This way, it truly makes sure that Multi-kueue is just deciding on the cluster where the workload should run and rest of the work is done by controller giving us clear separation of clusters.(This also means we might have to change Multi-Adapter interface without SyncJob method).

[tenzen-y]

@ravisantoshgudimetla As I described in https://kubernetes.slack.com/archives/C032ZE66A2X/p1755181736825669, this KEP aims to delegate only Workload resource creation capability. In other words, the kueue-controller-manager is still responsible to synchronize Workload between manager and worker clusters.

But, as I mentioned in #5981 (comment), I would like to make an effort to generalize the Workload synchronization, but it should be handled as a separate KEP (enhancement). So, if you are interested in that, I'm happy to review your proposal.

[kannon92]

I really don't like the idea of putting this information into the kueue configuration. In other KEPs we explored the idea of singleton CRs. We could explore a CRD for this.

I'm still not sure on this design.

[khrm]

I was thinking about a CRD too, but configmap could also suffice.

[mimowo]

I prefer the CRD paradigm too, but until we have it, I'm ok in the configMap.

[kannon92]

Maybe @mszadkow or @mwielgus could be a good reviewer here?

[khrm]

Instead of just a Jsonpatch, we should use CEL format like we have for mutating-admission-policy

I will modify KEP with that approach.

[kannon92]

Instead of just a Jsonpatch, we should use CEL format like we have for mutating-admission-policy

I will modify KEP with that approach.

I’d be careful with MAP. Are you interested in supporting this feature on older K8s clusters? I think MAP will be beta in 1.34 which is not out yet.

[gbenhaim]

I would prefer an alternative approach for supporting external frameworks which is similar to how the jobframework works. In short, I think that external frameworks should run their own multikueue admission check and Kueue should provide them the framework to write it. I've explained this approach in #6117

[khrm]

@kannon92 I don't mean supporting MAP. I meant changing our design to how MAP works. So for this KEP, changing it to use CEL instead of only Jsonpatch.

[mimowo]

I would suggest dropping this for now, we can introduce when proven this is needed.

[mimowo]

Let me summarize my view, and reasoning why I like this approach.

First, over time three major alternatives are proposed:

1. Support creating separate MultiKueue-like ACs by users and managing them KEP
2. Declarative API indicating how to create and sync Jobs (this KEP)
3. De-couple AC management and Job management and delegate Job creation and syncing to external controller

I think (1.) is clearly inferior to (2.) and (3.) as it requires opening API to external MultiKueue controllers which is very committal, and the most complex. Writing ACs is complex, and MK AC is particularly complex, so I really want users to avoid implementing them.

The downsides of (3.):
a) still API is needed to skip handling of Jobs by the built-in MultiKueue admission check of unknown Job CRDs. This would need to be a list of GVK which are externally handled.
b) it requires huge effort on the developers of the external controllers to read the secret and maintain connections with all the worker clusters. I think it would remain a legitimate feature request to support (2.) for the ease of implementing MK support even if we succeeded with (3.).
c) connecting with external clusters may likely result in errors which need to be reported by API, and the best place would be workload.admissionChecks[*] which is reserved to ACs, not external Job controllers.

I think the most likely outcome of (1.) and (3.) due to complexity is users forking or copy/pasting the Kueue's implementation and hacking on top of it. Design which essentially requires users to copy-paste Kueue code is far from ideal.

A valid concern raised by @tenzen-y about this proposal is that it requires commitment to the API to be maintained long-term. However:

• in the minimal form the API is really small (just the list of GVKs for which the mechanism is enabled). I think this is also unavoidable if we go with (3.).
• it is not blocking the external solution if we need to ever. For example, as the KEP proposes we could have ControllerType field indicating "External" if we ever need to have it.
• long term does not need to be forever - there is deprecation guideline in k8s which allows to deprecate and remove beta features

Yes, in the extended form the API is a more complex list of patches resembling MAP, but we don't yet have an indication it needs to be extended (AFAIK). Yes, Pods would not be supported by managedBy, but we can support them by a dedicated adapter. I expect most external jobs to be simple managedBy + status sync.

[tenzen-y]

Suggested change

	ExternalFrameworks []ExternalFramework `json:"externalFrameworks,omitempty"`
	ExternalFrameworks []string `json:"externalFrameworks,omitempty"`

As I mentioned in #5981 (comment), we can extract gvk from Kind.version.group.com, which is a standardized form.

[tenzen-y]

Based on the discussion in https://kubernetes.slack.com/archives/C032ZE66A2X/p1755181736825669, we decided to introduce ExternalFramework type, which has only a name field. In the future, we might introduce controllerType. But, it is not introduced in the alpha phase.

[mimowo]

As we discussed on slack, we are conservative about the API, but some degree of extensibility might be needed in the future. Some ideas:

• allows to specifiy controllerType: Generic / External to discriminate between the mechanisms
• position of the managedBy (but we should be careful about it and not plan for Alpha)
• syncPatches to customize the synchronization (but we should be careful about it and not plan for Alpha)

So, we propose to expose just

type MultiKueueExternalFramework struct {
	Name string `json:"name"`
}

[tenzen-y]

I think (1.) is clearly inferior to (2.) and (3.) as it requires opening API to external MultiKueue controllers which is very committal, and the most complex. Writing ACs is complex, and MK AC is particularly complex, so I really want users to avoid implementing them.

I also want external-controller developers to rely on the kueue-controller-manager AC approval mechanism. So, I am thinking that we should establish an interface to make those mechanisms easy for the external. That's my proposal and don't indicate that external developers implement (or copy/paste) AC approval mechanism. But, if we can introduce only externalFrameworks to realize the possibilities for the external Jobs. That's also acceptable. If we want more flexibility (Patch / ControllerName) as proposed in this KEP, we should consider establishing interface as opposed to expose some of APIs.

As a summary, in the short-term solution, we can introduce ONLY .multiKueue.externalFrameworks to make the duration of feature shipment short.
In the long term, or for more flexibility, we should establish the AC approval mechanism and expose it as an interface to external controllers.

[mimowo]

I also want external-controller developers to rely on the kueue-controller-manager AC approval mechanism.

Me too, this KEP is achieving this actually.

So, I am thinking that we should establish an interface to make those mechanisms easy for the external. That's my proposal and don't indicate that external developers implement (or copy/paste) AC approval mechanism.

I'm not clear how to achieve it other than in the KEP. I expect this would require exposing huge amount of code as a function at compile time. I think it will be really hard to ensure the interface does not break users in the future.

If we want more flexibility (Patch / ControllerName) as proposed in this KEP, we should consider establishing interface as opposed to expose some of APIs.

The KEP didn't mention ControllerName but ControllerType. The idea of controllerType is to discriminate between "External" and "Generic" in the future if we want to support both (2.) and (3.) long term. To me, I hope (2.) would be enough long term actually.

EDIT: I don't see use cases for the patches yet, but they may arise. Also, we may need to be able to specify the location of the managedBy field. In Kubeflow 1.x it is under spec.runPolicy.managedBy. Since Kubeflow 1.x has a built-in integration we don't need the API field for now, but it shows that some flexibility might be needed.

[mimowo]

This is not going to be implemented as Alpha for the KEP, so we can drop and move this part to the Alternatives section as a potential extension.

[khrm]

Sure. I would move this to alternative.

[tenzen-y]

I'm not clear how to achieve it other than in the KEP. I expect this would require exposing huge amount of code as a function at compile time. I think it will be really hard to ensure the interface does not break users in the future.

We can break the interface anytime, even with a significant break. That's why I want to select the interface rather than exposed API.

The KEP didn't mention ControllerName but ControllerType. The idea of controllerType is to discriminate between "External" and "Generic" in the future if we want to support both (2.) and (3.) long term. To me, I hope (2.) would be enough long term actually.

EDIT: I don't see use cases for the patches yet, but they may arise. Also, we may need to be able to specify the location of the managedBy field. In Kubeflow 1.x it is under spec.runPolicy.managedBy. Since Kubeflow 1.x has a built-in integration we don't need the API field for now, but it shows that some flexibility might be needed.

Yes, controllerType is what I wanted to mention. Rather than exposing the placement for managedBy, I would like to select (3). But, I agree that (3) needs significant effort. So, I would like to scrape the use cases by basic line proposed in this KEP. Which step should be customizable, which step should not be customizable.

[khrm]

@tenzen-y @mimowo I updated the KEP based on the slack discussion.

[tenzen-y]

Thank you for update.
Otherwise LGTM.

[tenzen-y]

@khrm LGTM, Thank you!
/approve
/lgtm

/hold for @mimowo

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: d5788a62018bed4e6778dddf116d7d296bad7f05

[mimowo]

/lgtm
/approve
/unhold
Thank you! Looking forward to the implementation

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: khrm, mimowo, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mimowo,tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment