📖 Proposal external CA and cert sign #12739
Conversation
k8s-ci-robot
added
area/documentation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Hi @moh2a. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
size/L
| User -> CAPI : Provider admin certificate for kubeconfig | ||
| User -> CAPI : Provider etcd-client certificate for etcd healthcheck and others actions. |
There was a problem hiding this comment.
| User -> CAPI : Provider admin certificate for kubeconfig | |
| User -> CAPI : Provider etcd-client certificate for etcd healthcheck and others actions. | |
| User -> CAPI : Provides admin certificate for kubeconfig | |
| User -> CAPI : Provides etcd-client certificate for etcd healthcheck and others actions. |
| ## Upgrade Strategy | ||
|
|
||
| No special upgrade plan is required for this feature. Enabling external CA and external certificate signing does not break existing Cluster API behavior. Clusters that use the default internal CA workflow will remain fully functional. | ||
|
|
There was a problem hiding this comment.
@moh2a will it be possible to update an existing cluster (which was created the old way) to a cluster with external CA? I guess this will not be possible. Maybe add a about that note here.
There was a problem hiding this comment.
I didn't think about switching from internal CA to external CA. If you want to make this change, you'll need to ask yourself whether you want to keep the same CAs or use new ones. If you want to use new ones (I think this is the case for you, as your current CAs may already have been exposed), you will need to perform a rotation, which is not trivial: https://kubernetes.io/docs/tasks/tls/manual-rotation-of-ca-certificates/
| ## Proposal | ||
|
|
||
| To support external certificate signing and fully external CAs in CAPI, this proposal introduces a spec-level flag in the Cluster resource, e.g., externalCA, externalCAKey, or externalCertSigning. This flag will indicate that the cluster should operate in "external CA mode." | ||
|
|
There was a problem hiding this comment.
Currently, I see these unencrypted keys are on control-planes:
- /etc/kubernetes/pki/ca.key
- /etc/kubernetes/pki/etcd/ca.key
- /etc/kubernetes/pki/front-proxy-ca.key
- /etc/kubernetes/pki/sa.key
It would be great if the proposal would list these files and provides a hint how each file will be handled
There was a problem hiding this comment.
It is already listed in kubernetes documentation here: https://kubernetes.io/docs/setup/best-practices/certificates/. I can refer to the link directly.
There was a problem hiding this comment.
@moh2a thank you for the link. Please image that: An control-plane admin was fired. He is angry. He took all key-files from a node. With your proposal /etc/kubernetes/pki/ca.key is not in his hands (nice). But what about the other key-files in the above list?
There was a problem hiding this comment.
In my proposal, I take into account the CA keys (etcd, ca, and proxy). All other certificates derive from these CAs. Personally, I create certificates (clients/servers) with a fairly short validity period. An angry administrator with access to the control planes could only retrieve the client/server certificate keys, but would not be able to issue certificates indefinitely since the CA keys are not present on the control planes.
Regarding the service account key pair, it's a little more complicated since the kube-controller-manager needs it on demand. I suggest you read this proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/740-service-account-external-signing/README.md.
There was a problem hiding this comment.
i think this is something that has been desired,, which is clear from user stories, and it can ease CA rotation. but the implementation details need a lot more work and discussion here. i think also required changes in the default controllers should be pointed out.
as it stands, this seems to me more as an idea doc and less like a full CAEP.
|
|
||
| ### External CA mode | ||
|
|
||
| When the flag is enabled, CA keys are no longer required in the CAPI management cluster and must not be stored or validated in secrets. |
There was a problem hiding this comment.
you should note that this should be an opt-in feature. as much as it improves the security stance, external ca is not something the majority of users care about.
|
|
||
| ## Proposal | ||
|
|
||
| To support external certificate signing and fully external CAs in CAPI, this proposal introduces a spec-level flag in the Cluster resource, e.g., externalCA, externalCAKey, or externalCertSigning. This flag will indicate that the cluster should operate in "external CA mode." |
There was a problem hiding this comment.
i think externalCA aligns best with the entire picture but needs a loaded API field comment to cover what it actually entails.
ideally this field should be wrapped in a feature gate tracking it's graduate from alpha->beta->GA.
| Certificates such as the etcd client certificate and the admin certificate for the kubeconfig are normally generated by CAPI. In external CA mode, these certificates are still needed, but their provisioning and lifecycle management become the responsibility of the user. Users can leverage their own certificate management system, such as cert-manager or any other certificate management solution, to sign and rotate these certificates. | ||
|
|
||
| The etcd client certificate should be provided by the user in a secret named <cluster-name>-apiserver-etcd-client | ||
| The kubeconfig admin certificate should be provided by the user in a secret named <cluster-name>-admin |
There was a problem hiding this comment.
would controller logs be the location where errors would surface for users that did not prepare the cert secrets correctly?
| ### External CA mode | ||
|
|
||
| When the flag is enabled, CA keys are no longer required in the CAPI management cluster and must not be stored or validated in secrets. | ||
| CAPI will not generate or manage CA private keys. The CA certificates needs to be provided by the user in Kubernetes secrets with the current format: |
There was a problem hiding this comment.
this would require secret format spec, examples and encoding rules.
|
|
||
| ### Certificates necessary to Cluster API | ||
|
|
||
| Certificates such as the etcd client certificate and the admin certificate for the kubeconfig are normally generated by CAPI. In external CA mode, these certificates are still needed, but their provisioning and lifecycle management become the responsibility of the user. Users can leverage their own certificate management system, such as cert-manager or any other certificate management solution, to sign and rotate these certificates. |
There was a problem hiding this comment.
what about the client certificate keys?
| ### Kubeadm Bootstrap: ClusterConfiguration and InitConfiguration | ||
|
|
||
| When using Kubeadm bootstrap, the command kubeadm certs generate-csr requires a configuration file containing ClusterConfiguration and/or InitConfiguration. Currently these files are only provided to the initialization control plane. | ||
| These configurations must be provided to joining control plane nodes because they contain the necessary information to generate Certificate Signing Requests (CSRs) for their components. The joining control plane nodes will generate CSRs locally and submit them to the external CA or user-provided signing system. |
There was a problem hiding this comment.
how would the init/clusterconfiguration be transferred to joining control plane nodes?
what mechanism would take the CSRs from the hosts and send them to the external signer?
while ideally kubeadm CSRs should be the source of truth for any certs that kubeadm needs, calling kubeadm commands on the node hosts and then sending CSRs over the network sounds complicated.
also who is going to call the kubeadm commands on the node hosts?
i'd assume cloud init, but is that going to be somehow part of the default capi behavior if the externalCA mode is enabled?
|
|
||
| #### Flexible Bootstrap Provider Support | ||
|
|
||
| As a Cluster API user I want that cluster-api provide all necessary configuration to bootstrap providers to be compatible with external signing, so that I can maintain secure certificate flows regardless of the underlying bootstrap mechanism. |
There was a problem hiding this comment.
| As a Cluster API user I want that cluster-api provide all necessary configuration to bootstrap providers to be compatible with external signing, so that I can maintain secure certificate flows regardless of the underlying bootstrap mechanism. | |
| As a Cluster API user I want that it provides all necessary configuration to bootstrap providers to be compatible with external signing, so that I can maintain secure certificate flows regardless of the underlying bootstrap mechanism. |
There was a problem hiding this comment.
how can that be done in a bootstrap provider agnostic way? the proposal earlier speaks about kubeadm KCP and CABPK mechanisms like generate-csr but i didn't see any notes about generalization and abstraction.
|
|
||
| ## Upgrade Strategy | ||
|
|
||
| No special upgrade plan is required for this feature. Enabling external CA and external certificate signing does not break existing Cluster API behavior. Clusters that use the default internal CA workflow will remain fully functional. |
There was a problem hiding this comment.
here you'd have to cover how a cluster can enabled the feature on upgrade and roll it back on failure.
5 participants
What this PR does / why we need it:
This PR add a proposal fo fixe this issue
/area documentation