x509: certificate signed by unknown authority #178

apopaa · 2021-07-20T21:22:45Z

apopaa
Jul 20, 2021

Hi,

I'm getting the following error when running a signed image built from the connaisseur/setup/Dockerfile (verified by "docker trust inspect"):
kubectl run -n apns signed2 --image=myregistry.azurecr.io/apopa/testimage2:0.0.1 Error from server (InternalError): Internal error occurred: failed calling webhook "connaisseur-svc.connaisseur.svc": Post https://connaisseur-svc.connaisseur.svc:443/mutate?timeout=30s: x509: certificate signed by unknown authority

Is the controller trying to make an HTTPS call to the service via the Internet? Because my Kubernetes machines are proxied by a very restrictive corporate proxy, that does not allow arbitrary connections to the Internet.

Answered by apopaa

Jul 22, 2021

OK, so I've made some progress in that, as I had suspected, the "x509: certificate signed by unknown authority" error was related to my proxy, which is configured to inspect content and so decrypts any TLS traffic and then re-encrypts it with its own certificate, which of course is then seen as "unknown" by the connaisseur webhook. The solution was pretty simple: add ".svc" to the no_proxy environment variable in /etc/kubernetes/manifests/kube-apiserver.yaml, to make kube-apiserver bypass the proxy.

Now, in doing that, I think I uncovered a bug in the handling of the policy configuration from values.yml: because I changed the manifest of kube-apiserver, the kubelet attempted to reload it,…

View full answer

xopham · 2021-07-21T00:42:22Z

xopham
Jul 21, 2021
Maintainer

Hi @apopaa, Connaisseur will in general connect via HTTPS. It is hard to judge without knowing the details of your infrastructure, but I might give you a few hints:
First off, you seem to be using ACR from Azure. In general, Connaisseur has been tested with ACR and should work. However, since ACR does not provide a fully standardized notary server, you have to tell Connaisseur that this ACR via the isACR flag in helm/values.yaml. See here for the specialized guide: https://github.com/sse-secure-systems/connaisseur/tree/master/setup/acr#2-configure-specifics-to-acr
On the other hand, your error message and notes about the proxy seem to indicate that a self-signed certificate may be in use. In this case, you may have to configure the notary.selfsignedCert in helm/values.yaml as is explained for a local harbor install here: https://github.com/sse-secure-systems/connaisseur/tree/master/setup/acr#2-configure-specifics-to-acr

Let me know if that helps 🙏

16 replies

apopaa Jul 22, 2021
Author

Damn, "targetNamespaces" was such a convenient quick way of verifying ALL my images, since they all get deployed to the same namespace. I have to use the "policy" feature and provide a pattern to match my images.

Still being unable to install, though. Getting this error now:
bash-5.1# make install
...
helm install connaisseur helm --atomic --set alerting.cluster="kubernetes-epiphany"
Error: an error occurred while uninstalling the release. original install error: timed out waiting for the condition: timed out waiting for the condition
make: *** [Makefile:28: install] Error 1

What condition is it waiting for? Any error log somewhere that could tell me what's the "condition"?

apopaa Jul 22, 2021
Author

OK, the "condition" it is waiting for is the readiness check of the connaisseur pods: it is failing repeatedly with error 500; the liveness check otoh succeeds.

This is the end of the log of one of the pods; no errors:
[2021-07-22 05:01:34,296] INFO: 10.244.3.1 - - [22/Jul/2021 05:01:34] "GET /health HTTP/1.1" 200 -
[2021-07-22 05:01:39,175] INFO: 10.244.3.1 - - [22/Jul/2021 05:01:39] "GET /ready HTTP/1.1" 500 -
[2021-07-22 05:01:44,810] INFO: 10.244.3.1 - - [22/Jul/2021 05:01:44] "GET /health HTTP/1.1" 200 -
[2021-07-22 05:01:44,813] INFO: 10.244.3.1 - - [22/Jul/2021 05:01:44] "GET /ready HTTP/1.1" 500 -
[2021-07-22 05:01:48,001] INFO: 10.244.3.1 - - [22/Jul/2021 05:01:48] "GET /ready HTTP/1.1" 500 -
[2021-07-22 05:01:49,571] INFO: 10.244.3.1 - - [22/Jul/2021 05:01:49] "GET /health HTTP/1.1" 200 -

xopham Jul 22, 2021
Maintainer

Don't get me wrong, namespaced validation will continue to work, just better. It is not removed, but the way it is implemented rn on each install the namespaces to limit validation have to be created on each install and if they already exist...install will fail. In the 2.0 release, you will be able to activate it and then use labels to mark namespaces for validation.

To your errors: It is a bit hard to judge from the healthiness and readiness alone. However, since you use ACR have you set notary.isAcr: true in helm/values.yaml? ACR does not provide a proper notary health endpoint and therefore Connaisseur has to ignore notary health in that case which is achieved by the isAcr key. See here: https://github.com/sse-secure-systems/connaisseur/blob/master/helm/values.yaml#L54

apopaa Jul 22, 2021
Author

Yes, isAcr is true:

# if you use Azure Container Registry (ACR) for your notary
# changes some behaviour, such as health probes and how to retrieve auth tokens
# for compatibility with ACR set to true
isAcr: true

Besides, in my log the health probe returns 200; it is the ready probe that returns 500.

Update: It appears like the problems with the ready probe were of a transitory nature due (probably) to my network infrastructure being slow yesterday and timing out certain deployments, resulting in the Connaisseur controller not properly setting up. Today the network is fast again and Connaisseur deployed without problems.

apopaa Jul 22, 2021
Author

OK, so now that the deployment problems have been figured out, we're back to the opening issue: "x509: certificate signed by unknown authority"

My attempt to deploy an unsigned (testimage2) and a signed (testimage3) image:
[user@master ~]$ kubectl run -n apns unsigned2 --image=myregistry.azurecr.io/apopa/testimage2:0.0.1
Error from server (InternalError): Internal error occurred: failed calling webhook "connaisseur-svc.connaisseur.svc": Post https://connaisseur-svc.connaisseur.svc:443/mutate?timeout=30s: x509: certificate signed by unknown authority
[user@master ~]$ kubectl run -n apns signed3 --image=myregistry.azurecr.io/apopa/testimage3:1.0.1
Error from server (InternalError): Internal error occurred: failed calling webhook "connaisseur-svc.connaisseur.svc": Post https://connaisseur-svc.connaisseur.svc:443/mutate?timeout=30s: x509: certificate signed by unknown authority

apopaa · 2021-07-22T21:04:41Z

apopaa
Jul 22, 2021
Author

OK, so I've made some progress in that, as I had suspected, the "x509: certificate signed by unknown authority" error was related to my proxy, which is configured to inspect content and so decrypts any TLS traffic and then re-encrypts it with its own certificate, which of course is then seen as "unknown" by the connaisseur webhook. The solution was pretty simple: add ".svc" to the no_proxy environment variable in /etc/kubernetes/manifests/kube-apiserver.yaml, to make kube-apiserver bypass the proxy.

Now, in doing that, I think I uncovered a bug in the handling of the policy configuration from values.yml: because I changed the manifest of kube-apiserver, the kubelet attempted to reload it, but failed with the following error in systemd logs (journalctl):

Jul 22 16:17:56 masternode kubelet[38796]: E0722 16:17:56.313082 38796 kubelet.go:1664] Failed creating a mirror pod for "kube-apiserver-masternode_kube-system(6da95f69ab0a006e91f81c7c5d98f14f)": admission webhook "connaisseur-svc.connaisseur.svc" denied the request: "masternode:5000/k8s.gcr.io/kube-apiserver:v1.18.6" is not a valid image format.

That's because my K8s is set up to load system pods from a local repo installed on masternode machine, and no pattern in the policy in values.yml was matching "masternode:5000/k8s.gcr.io/kube-apiserver:v1.18.6", and so connaisseur rejected loading the image. I had expected it to accept any image it cannot match, by default; turns out it is rejects any image it cannot match, by default. Is this a bug, or a feature?
The workaround is simple: add a default policy for ":" with "verify: false", in addition to the specific policy that validates my images:
policy:

pattern: "myregistry.azurecr.io/apopa/:"
verify: true
pattern: ":"
verify: false

7 replies

xopham Jul 27, 2021
Maintainer

I think you might be correct and we have a bug there in the image format validation that still persists into version 2.0. Connaisseur seems to reject the two '.' in the "repository" part. Escalating to issue now ⬆️

xopham Jul 27, 2021
Maintainer

Issue #194 . We'll be looking into this.

phbelitz Jul 28, 2021
Maintainer

Actually, I think you have a bug in your 1.5 code because even with this setting in values.yml:
policy: - pattern: "*" verify: false
it still denies the request:
denied the request: "master-node:5000/k8s.gcr.io/kube-apiserver:v1.18.6" is not a valid image format.

I think the double ":" throws your code off.

I see 2.0 is out today, so this bug might be unimportant now.

Hey there! I assume you signed your image using:

DOCKER_CONTENT_TRUST=1 docker image push master-node:5000/k8s.gcr.io/kube-apiserver:v1.18.6

Under which of these two URL's is your trust data available?

https://<your-notary-url>/v2/master-node:5000/k8s.gcr.io/kube-apiserver/_trust/tuf/root.json
# OR
https://<your-notary-url>/v2/k8s.gcr.io/kube-apiserver/_trust/tuf/root.json

apopaa Jul 28, 2021
Author

That image is not signed. And I want to keep it that way, at least for now. All I want atm is to verify the images containing my application, ie "myregistry/apopa/testimage2" and the like.

phbelitz Jul 30, 2021
Maintainer

Well well well! We fixed the issue with our new v2.1.0 release. Now your master-node:5000/k8s.gcr.io/kube-apiserver:v1.18.6 image should be correctly accepted as a valid image.

CHEERS! 🍷

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x509: certificate signed by unknown authority #178

{{title}}

Replies: 2 comments 23 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

x509: certificate signed by unknown authority #178

apopaa Jul 20, 2021

Replies: 2 comments · 23 replies

xopham Jul 21, 2021 Maintainer

apopaa Jul 22, 2021 Author

apopaa Jul 22, 2021 Author

xopham Jul 22, 2021 Maintainer

apopaa Jul 22, 2021 Author

apopaa Jul 22, 2021 Author

apopaa Jul 22, 2021 Author

xopham Jul 27, 2021 Maintainer

xopham Jul 27, 2021 Maintainer

phbelitz Jul 28, 2021 Maintainer

apopaa Jul 28, 2021 Author

phbelitz Jul 30, 2021 Maintainer

apopaa
Jul 20, 2021

Replies: 2 comments 23 replies

xopham
Jul 21, 2021
Maintainer

apopaa Jul 22, 2021
Author

apopaa Jul 22, 2021
Author

xopham Jul 22, 2021
Maintainer

apopaa Jul 22, 2021
Author

apopaa Jul 22, 2021
Author

apopaa
Jul 22, 2021
Author

xopham Jul 27, 2021
Maintainer

xopham Jul 27, 2021
Maintainer

phbelitz Jul 28, 2021
Maintainer

apopaa Jul 28, 2021
Author

phbelitz Jul 30, 2021
Maintainer