Trust Manager is reordering certificates injected by CA bundle

[sagarmujumale]

Trust Manager is reordering certificates injected by CA bundle in config maps. Pods are unable to communicate with service running in different namespace and complaining "unable to find valid certification path to requested target".

As we are using intermediate certs, all certs must be in order(intermediate and the root CA), however trust manager is changing the sequence and adding root CA above the intermediate which is not the correct sequence. Which is breaking the chain and service communication failing as there is no chain.

[erikgb]

@sagarmujumale, why do you need intermediate CAs in your trust bundle? It is usually much better/easier to add trust exclusively to self-signed/root CAs. We have an open issue with requiring only self-signed/root CAs in trust bundles by default: #4.

[sagarmujumale]

@erikgb We are using intermediate CA for highly important services. We dont want to use root CA in all of the environment to sign the certs. All these services are communicating with each other across all the environment. hence we need to use complete chain in bundle. The fetaure requiring only self-signed/root CAs in trust bundles by default: #4. which is good for some environment who are not using any signed certs for their services.

It will be better if we use "allowIntermediateCertificates" to keep existing feature along with default self signed certificate.

[inteon]

@sagarmujumale all certificates in a trust bundle are trusted normally. There should be no need to form a chain in your trust bundle. Normally, clients check each certificate starting from the leaf and stop when they find a match (in your case the first match will be the intermediate).

[SgtCoDFish]

Hey @sagarmujumale from looking at this I'm not sure I'm understanding why the ordering would matter here. I think I need more context to determine if there's anything we need to change here.

Do you have some software which requires a specific ordering of certs in a trust bundle? I've never encountered that before and I'd find the behaviour very surprising to say the least!

[sagarmujumale]

@SgtCoDFish We are distributing Service Root CA and Service Intermediate CA in bundle. All services are using service intermediate CA signed certificate. All K8's clusters has Intermediate service CA signed by root CA. Services are using this certificate to communicate with each other. Reordering certificate chain is causing cert trust issues. It need to have in certain order to form a chain.

[erikgb]

@sagarmujumale I think your services should serve certificate chains instead of adding intermediate CAs to the trust bundle. But even if you insist on trusting the intermediate CAs explicitly by adding them to the trust bundle, the order shouldn't matter. I would take a closer look at the software checking your chains.

[cert-manager-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale

[cert-manager-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
/lifecycle rotten
/remove-lifecycle stale

[cert-manager-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
/close

[cert-manager-prow]

@cert-manager-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[RafalMaleska]

using trust-manager to distribute *.pem CAs bundles.

after mounting the secret holding a mix of publicCAs and companies own CAs (root and intermidiate CAs) into jfrog artifactory. communication with an LDAP server broke down. the root cause is the wrong order of the CAs in the bundle. jfrog artifactory is based on java.

it was also mentioned here:
#380

using trust-manager v0.15.0.

not sure if this is the same issue as above but feels like it is.
tested it with a bundle that is referencing each CA separatly (by the key in a configmap) and with a bundle that is referencing each root/intermidiate CA pair by one key. the result is a random order which dosen't work with some java processes (jfrog artifactory ldap integration)

could we reopen this issue?

[erikgb]

You are not supposed to put intermediate CAs into the trust bundle. You typically trust self-signed (root) CAs. See #4. If you think you need intermediate certificates in the trust bundle, the clients/servers are misconfigured.