[EKS]: IAM Roles for Service Accounts enhancements for usage across multiple clusters

[mikestef9]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
IAM Roles for Service Accounts (IRSA) enables you to associate an IAM role with a Kubernetes service account, and follow the principle of least privilege by giving pods only the AWS API permissions they need, without sharing permissions to all pods running on the same node. This feature works well with a smaller number of clusters, but becomes more difficult to manage as the number of EKS clusters grows, notably:

• Needing to create an IAM OIDC provider for each cluster
• Updating the trust policy of every IAM role needed by that cluster with cluster’s OIDC provider URL that maps to a particular namespace and service account name.
  • Given the 2048 character limit of IAM trust policies, only around 5 clusters can be trusted per role.
  • After 5 clusters, you need to duplicate IAM roles to be used by additional clusters.
  • Coordination between devs and admins - Development teams that own IAM roles often need to re-run Terraform apply or CloudFormation scripts to update trust policies for new clusters.

Given these pain points, EKS is considering a change to the way IRSA works, moving credential vending to the EKS control plane (similar to how ECS and Lambda works). With this change, a trust policy would only need to be updated once to trust a service principal like eks-pods.amazonaws.com, then you would call an EKS API to provide the IAM role to service account mapping, ex.

aws eks associate-role \
    --cluster-name $CLUSTER_NAME \
    --role-arn $ROLE_ARN \
    --namespace $KUBERNETES_NAMESPACE \
    --service-account $SERVICE_ACCOUNT 

We are looking for your feedback on this proposal, and to hear any additional pain points encountered with IRSA today that would not be solved by such a solution.

Are you currently working around this issue?
Creating and managing duplicate roles to be used across multiple clusters

[gregoryfranklin]

This would be very useful as it is definitely a problem we run into.

Would this work cross-account? We often associate service accounts with roles in accounts other than the one in which EKS is running.

[ajohnstone]

@mikestef9 thanks for raising this.

When running large clusters/platforms on top of AWS and requiring IAM there are a few more constraints than just the trust relationship payload size.

1. Each AWS account has a hard limit of 5000 IAM roles per account. With the above it would be ideal to not be bound by a single accounts hard limit and to allow cross account assume roles. Running a multi-tenant workload can quite easily consume this limit with a few thousand deployments.

2. Do you have more details around the implementation of this? E.g. Would this require aws-sdk changes to the credential provider chain and therefore a bump for all tenants using AWS SDKs?

3. How would you limit the trust relationship to a single or group of clusters?

4. Would there be any caveats/gotchas that would need to consider as part of this proposal?

[mikestef9]

@gregoryfranklin yes we plan to make cross account roles work with this proposal.

@ajohnstone we haven't settled on implementation yet. One idea is to use a CSI driver to retrieve a pod’s OIDC token from the Kubelet and exchange the token for IAM Role credentials from EKS. Another is a new credential provider in all AWS SDKs, but that would be hampered by requiring all clients to update.

You manage trust through the EKS API now. The IAM trust policy only is updated once to trust the EKS service principal.

[phillebaba]

This might solve a challenge I am currently having with IRSA. There are situations where I have to replace an existing cluster with a new one and then shift traffic to the new cluster, either due to disaster recovery or a change which cant be applied without downtime. Currently this would require all developers to update their trust policy with the new OIDC provider, so that the applications can authenticate from the new cluster.

Would this new solution require all roles to be associated with the new cluster or could that be solved somehow when replacing a cluster?

[mikestef9]

You would still need to call the EKS API to add the mapping, but this can be automated using IaC tools as part of the new cluster creation process. There would be no need to update a role trust policy when creating a new cluster, and you no longer run into trust policy character limits when you need a role associated with many clusters.

[phillebaba]

It sounds like we would still have the same issues as we do today. In my scenario individual development teams setup their own resources and IAM roles with Terraform, this is a separate state from the one which creates the cluster. The manifests are deployed with FluxCD into the cluster. This enables us to replace a cluster over night without any development team being the wiser as the new cluster will pull down the same manifests as the old one does. The issue here today is that each development team would have to update the OIDC provider in their trust policy.

Correct me if I am wrong but it sounds like there is still a need in the new proposed solution for each team to associate their IAM roles with service accounts in the new cluster. Basically keeping the same issue that we have today but solved with a different IaC resource.

[gregoryfranklin]

Correct me if I am wrong but it sounds like there is still a need in the new proposed solution for each team to associate their IAM roles with service accounts in the new cluster. Basically keeping the same issue that we have today but solved with a different IaC resource.

If teams are creating the IAM roles with terraform, can they not lookup the cluster id using a data source?

We store the current list of OIDC providers in an SSM parameter when we create a cluster. Terraform is therefore able to lookup the appropriate IDs in order to generate the policy.
The same appears to be true with the new proposal, but instead of needing to template the trust policy there would just be a new resource type to create the mapping. The cluster name parameter would change each time the cluster is re-built but its a lot easier to just lookup the value for that single parameter than template the trust policy.

[mikestef9]

With this proposal, yes there would be an EKS API call to make for each cluster instead of a trust policy update. Additionally, there would no longer be a need to annotate a service account with the IAM role Arn.

So is what you are looking for more along the lines of some way to trust an IAM role for all clusters in an account, without having to specify each cluster?

[phillebaba]

@mikestef9 yeah that is basically it. I do not want to have to inform development teams to rerun their Terraform if I replace the cluster. It would be nice to either configure trust to an account of some sort of regex pattern for the cluster name.

[sidewinder12s]

If a CSI Driver is used, I hope it's not a daemonset. We're quickly running into issues with the number of daemonsets we need to run on nodes that consume resources we can then not use to run workloads (currently at about 7 per node). At large cluster sizes, this is not insignificant where you might end up with 10k workload pods and 28k daemonset pods. If using the vpc-cni-plugin and IP allocations out of a VPC, it's even worse since now you're consuming potentially scarce IPs.

We've also run into the trust policy size limits already.

Talking from personal experience of dealing with both the AssumeRoleWithWebIdentity & AWS SSO default credential provider chain updates to multiple language SDKs: It is a nice long term change but actually doing it was an absolute nightmare. None of the AWS SDK teams coordinate their work when service teams add new methods leaving all customers in the lurch to deal with it. When you are dealing with 100s of devs and multiple languages this was a giant time sink that AWS was not helpful with.

[xavipanda]

This is a must.
In the way it is right now, 1 policy ~400bytes, assume role max policy size 4096bytes. 9clusters are enough to reach this limit.

• IAM Variable interpolations for EKS/OIDC --> inexistent
• Length of the ARN -> Gigantic
• Length of OIDC URL -> Gigantic

IMHO, i can't label the today integration.. where EKS elements are treat as a foreign element of AWS as "production-grade".
Even if we produce 1 role per cluster, we will hit max roles... or max policies, or max attachments..

Does anyone knows a workaround, besides producing resource roles (nodes, cluster, etc) where everyone can access everyone else AWS resources.. ??

Do we have an ETA for this feature ? we kinda struggle..

[project0]

Just faced the same issue.
We do run multiple clusters, each in their own AWS account. Currently we use KIAM to assume roles for our production services. The KIAM servers of all clusters/account currently runs on a special node which attaches a instance role of one central account, so the product teams specify only this account to allow assuming the role and we can scale out new clusters without any IAM change.

I just stopped a initiative to migrate all our services to the new IRSA as the effort to migrate all assume role policies is too high (additional to the AWS SDK library updates...). We use cloudformation across the place, so its a nightmare to get all our product teams on board to update their roles.

Considering possible Disaster Recovery cases when i need to re-create everything or even deploy into another region would not be possible as we cannot assume the roles anymore 😞, there is just no good way to keep the roles synced with possible OIDC provider changes.