[proposal]Improve YARN with Koordinator so that BE pods and YARN tasks can share batch resources

[hawkphantomnet]

What is your proposal:
The current Yarn with Koordinator solution synchronizes all batch resources to the YARN RM. We hope to improve this solution so that BE pods and YARN tasks can share batch resources. Therefore, it is necessary to enhance the mechanism for synchronizing and managing batch resources between Koord and YARN. It is necessary to introduce a new configuration thirdPartyResourceConfig to calculate the amount of batch resources that can be used by YARN and implementing real-time control over YARN tasks' cgroup based on this configuration.

Example:

slo-colocation-config：|
{  
   ...
   "thirdPartyResourceConfig": [
     {
       "thirdPartyName": "hadoop-yarn",
       "batchResourceRatio": {
         "batchCpu": 80,
         "batchMemory": 80
       },
       "cgroupPath": "/hadoop-yarn"
     }
   }
   ...
}

Why is this needed:
Described above

Is there a suggested solution, if so, please add it:
Here is an initial draft for the detailed design

[hormes]

Do you want to control the upper limit of batch resources that Yarn can use?
If I understand correctly, I would like to know what scenarios require setting an upper limit

[hawkphantomnet]

Yes, in our scenario, we want to deploy three kinds of workloads on k8s nodes:

1. online serving pods: use prod resources.
2. training pods: run as BE pods and use batch resources.
3. yarn tasks: managed as containers(not pod) by YARN to use the remaining batch resources after the training containers have been allocated their resources. Setting upper limit of batch resources that Yarn can use is to keep some buffer in case of immediate training pods.

[hormes]

Yes, in our scenario, we want to deploy three kinds of workloads on k8s nodes:

1. online serving pods: use prod resources.
2. training pods: run as BE pods and use batch resources.
3. yarn tasks: managed as containers(not pod) by YARN to use the remaining batch resources after the training containers have been allocated their resources. Setting upper limit of batch resources that Yarn can use is to keep some buffer in case of immediate training pods.

Can yarn nodemanager run on k8s as a Pod?

[hawkphantomnet]

NodeManager is running as Pod, yarn tasks(like spark driver/executors) are running as containers managed by NodeManager.
The proposal is based on this solution: https://koordinator.sh/docs/designs/koordinator-yarn

[stale]

This issue has been automatically marked as stale because it has not had recent activity.
This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the issue is closed
  You can:
• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close
  Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity.
This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the issue is closed
  You can:
• Reopen this PR with /reopen
  Thank you for your contributions.