Custom Scheduler for Amazon EKS with Binpacking

This is the custom scheduler with MostAllocated score strategy, that is known as the binpacking.

When running jobs on Kubernetes, the default kube-scheduler will balance the pods across the nodes which result in fragmentations causing cost increase. Job workloads like Spark, that is not long running, and many customers have different sizes of jobs. Without bin packing for kube-scheduler, node usage can not be increased and node can be scale-in in time, which cause cost increase.

In this repo, we provide an example of how to run custom scheduler with Amazon EKS 1.24+.

Installation

This project support installation either by Manifests or helm.

Prequisites

• Amazon EKS 1.24+
• kubectl

Manifests

Clone the repo

git clone https://github.com/aws-samples/custom-scheduler-eks
cd custom-scheduler-eks

Amazon EKS 1.24

kubectl apply -f deploy/manifests/custom-scheduler/amazon-eks-1.24-custom-scheduler.yaml

Amazon EKS 1.29

kubectl apply -f deploy/manifests/custom-scheduler/amazon-eks-1.29-custom-scheduler.yaml

Other Amazon EKS versions

• replace the related image URL(https://gallery.ecr.aws/eks-distro/kubernetes/kube-scheduler)

Helm

Add the Helm repository and install the chart:

helm repo add custom-eks-scheduler https://aws-samples.github.io/custom-scheduler-eks
helm repo update
helm install custom-eks-scheduler custom-eks-scheduler/custom-scheduler-eks \
  --set eksVersion="1.29" \
  --set schedulerName="second-k8s-scheduler" \
  -n kube-system

Available Values

Please refer to the values for more info.

Install eks-node-viewer

eks-node-viewer is a tool for visualizing dynamic node usage within a cluster, we will use this for tracking the binpacking.

HomeBrew

brew tap aws/tap
brew install eks-node-viewer

Manual

go install github.com/awslabs/eks-node-viewer/cmd/eks-node-viewer@latest

eks-node-viewer output example

eks-node-viewer：https://github.com/awslabs/eks-node-viewer

Testing

Launch eks-node-viewer

eks-node-viewer --extra-labels topology.kubernetes.io/zone

EKS node usage before the deployments

Apply a new deployment

kubectl apply -f deploy/deployments/nginx-deployment.yaml

EKS node usage after the deployments

Tips for selecting scheduler image

The difference between Amazon EKS 1.24 and Amazon EKS 1.29 are image version and apiversion. We do not recommend build the kube-scheduler by yourself, we can leverage the eks-distro kube-scheduler image.

Amazon EKS 1.29 Image: public.ecr.aws/eks-distro/kubernetes/kube-scheduler:v1.29.6-eks-1-29-18 Amazon EKS 1.24 Image: public.ecr.aws/eks-distro/kubernetes/kube-scheduler:v1.24.16-eks-1-24-25

The KubeSchedulerConfiguration API version is stable(v1) in Kubernetes 1.25, for those cluster prior to 1.25, should use kubescheduler.config.k8s.io/v1beta2.

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.