Error: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "xyz" and path "/" is already defined in ingress xxx

[dev-samples]

I recently upgraded to k8s 1.22.4 and ingress-nginx version 4.0.16 and now its no longer possible to create two different ingress objects with the same host/path pointing to two different services (which need I need for a temporary workaround).

This worked fine in k8s 1.15 with e.g. something like this:

- apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  spec:
    rules:
    - host: my-host.internal
      http:
        paths:
        - backend:
            serviceName: my-app-a
            servicePort: 8080
          path: /
    tls:
    - hosts:
      - my-host.internal
  status:
    loadBalancer:
      ingress:
      - {}

- apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  spec:
    rules:
    - host: my-host.internal
      http:
        paths:
        - backend:
            serviceName: my-app-b
            servicePort: 8080
          path: /
    tls:
    - hosts:
      - my-host.internal
  status:
    loadBalancer:
      ingress:
      - {}

But in 1.22.4 with:

- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: my-app-a
  spec:
    rules:
    - host: my-host.internal
      http:
        paths:
        - backend:
            service: 
              name: my-app-a
              port:
                number: 8080
          path: /
    tls:
    - hosts:
      - my-host.internal
  status:
    loadBalancer:
      ingress:
      - {}

- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: my-app-b
  spec:
    rules:
    - host: my-host.internal
      http:
        paths:
        - backend:
            service: 
              name: my-app-b
              port:
                number: 8080
          path: /
    tls:
    - hosts:
      - my-host.internal
  status:
    loadBalancer:
      ingress:
      - {}

I get the error:
Error: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "xyz" and path "/" is already defined in ingress xxx

Is there some document/changelog describing this specific change or some option I can temporary set to allow for multiple ingresses to be created with same host/paths?

[k8s-ci-robot]

@dev-samples: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

Its not in the ingress api spec to implement controllers that can service 2 ingress objects with identical host & path. Imagine the practical consequences of that.

However another person seemed to have faced this same requirement https://kubernetes.slack.com/archives/CANQGM8BA/p1616518953121800

/remove-kind bug

[dev-samples]

Yes as I mentioned I need this for a temporary workaround and not a permanent configuration. Also it seems strange that it works fine for 1.15 so I was also wondering if I was missing some configuration option that allows for this.

Update. Ok the slack thread you link to goes into more depth with this, thanks! So seems the nginx admission controller has become more strict since 1.15. Will try and wrap my head around this details in that (long) slack thread :-)

[brianmay]

We have a genuine use case for this to work. We have two ingress-nginx instances, one for traffic from external sources, and one for traffic from internal sources.

We want to use an internal ingress to route traffic the the internal ingress-nginx, and the external ingress to route traffic to the external-nginx. There is no conflict or ambiguity. The IP routing makes sure that the traffic goes to the correct instance.

But as of the latest ingress-nginx it seems that this working setup is no longer working. As the two ingress objects conflict with each other despite having two different classes.

I consider this a regression and as a result a bug.

[brianmay]

Also while good somebody posted a link to a workaround, would be good if somebody could paste the gist here, as not everybody is subscribed to that slack channel...

My guess is that the workaround is to disable the webhook, i.e. set controller.admissionWebhooks.enabled=false

[jsalatiel]

Probably duplicate of #7546.
It has been merged and It will probably be fixed in the next release.

[unoexperto]

Whoever comes here from google search disabling admission webhook in controlled didn't help me, but kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission has.

[dud225]

This behaviour conflicts with the documentation:

If the same path for the same host is defined in more than one Ingress, the oldest rule wins.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pawellrus]

I was wrong.

[pawellrus]

/reopen

[pawellrus]

/remove-lifecycle rotten

[k8s-ci-robot]

@pawellrus: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[brianmay]

/reopen

[k8s-ci-robot]

@brianmay: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mesquitamv]

I'm getting the same issue. I have same scenario that @brianmay described (one ingress class for internal and another for external traffic) and I got the same error:

Error from server (BadRequest): error when creating "spec/ingress/nginx-ingress.yml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "foo" and path "bar" is already defined in ingress namespace/ingress-name

Anyway to workaround this problem?

I'm using ingress-nginx version 1.3.1

[mesquitamv]

/reopen

[k8s-ci-robot]

@mesquitamv: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[brianmay]

@mesquitamv I have a vague recollection I disabled the admission web hook. It isn't needed, it just does some sanity checks. Which it appears are a bit broken.

Looking at the helm chart, I think you can disable the hooks by setting admissionWebhooks.enabled to false.

We are now using skipper, that does not have this particular problem.

Unfortunately, since this bug was auto closed, I don't think we are getting the attention of the developers. Might need to open a new bug report, since we can't reopen this one. But I would suggest looking for an alternative ingress implementation, maybe one that is better supported.

[longwuyuan]

• I don't see the data that supports comments from @brianmay or @mesquitamv .
• One data that is publicly available is https://kubernetes.github.io/ingress-nginx/user-guide/k8s-122-migration/#how-can-i-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster that relates to running multiple instances of the controller in one cluster.
• Disabling or deleting hooks is a piece of data that leads to a state that invalid ingress configurations are flagged

[tao12345666333]

/reopen

Recently, I have seen some other feedback as well. #10318
Let me reopen this issue first, and provide some detailed explanations.

[k8s-ci-robot]

@tao12345666333: Reopened this issue.

In response to this:

/reopen

Recently, I have seen some other feedback as well. #10318
Let me reopen this issue first, and provide some detailed explanations.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tao12345666333]

About the admission webhooks response.

As you can see, sometimes you may see the following response.

(BadRequest) : admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "xyz" and path "/" is already defined in ingress xxx

We have implemented some logical checks in the admission webhook to assist us with data validation.
You can think of them as some enhanced features, because currently we are only using Validating Admission.

You will receive this response here for the following reasons:

• Consistency in user experience. In Kubernetes, we use declarative configuration, and the controller always reconciles the actual state to the desired state. Assuming we allow users to configure the same host and path in different Ingress resources, when a user wants to delete an Ingress, they will find that this does not really take effect, because there are other Ingress resources containing the same host and path. This will cause confusion.

How to solve it

• If you know what you are really doing, you can indeed disable admission. This does not affect the functionality of the Ingress-NGINX controller, you can continue to create and modify Ingress resources.

Not finished yet, I will come back when I have time.

[tobinus]

I can add my thoughts here as well, so that the details aren't spread across multiple issues.

Use case: "Renaming" an ingress object

My specific use case is attempting to "rename" an ingress, because we only needed one at first, but now I'd like to have two ingresses. So I'd like to add a new ingress and rename the old one, so that the names make sense.

This has been reported to Helm in the issue helm/helm#12157, but I don't think the fault here is with Helm. Since there is no such thing as renaming an object in Kubernetes, you need to:

1. Create the new ingress object with the new name, e.g. example-ingress-renamed
2. Delete the old one, e.g. example-ingress

The admission webhook does not allow the intermediate state between step 1 and step 2, where there would be two ingresses targeting the same hostname and path. Helm does the exact same thing that I would have done, if I had been tasked with doing this manually, as far as I can tell.

I included steps for how to reproduce this in #10318.

Why not delete first?

Going the other way around would not run into the same problem:

1. Delete the old ingress object, e.g. example-ingress
2. Create the new one with the new name, e.g. example-ingress-renamed

The reason this is not an acceptable workaround for everyone is that it would make the application unavailable between step 1 and step 2, where there would be no ingress targeting example.com.

A quick search on the Internet turns up a lot of answers telling me to delete the admission webhook policy, as you mention @tao12345666333. That would be unfortunate for a shared cluster like the one I'm using at work (and I don't think the cluster admins would allow it).

Thoughts about possible solutions

If Kubernetes had supported renaming objects or multi resource transactions (kubernetes/client-go#630) then this would not have been a problem. But Kubernetes does not support that, so I think ingress-nginx should provide some way to allow the transient state between creating a new ingress object and deleting the old one.

Some ideas:

• Automatically allow duplicate ingresses which share everything except name (including namespace and backends)
  • Downside: Users would need to be careful about never making other changes simultaneously with the "renaming"
  • Downside: This would not work across namespaces and different instances of the application
• Flag ingress conflicts by setting some status and reporting an event on the ingress object, but allow it to be created? I don't know if that's possible
• Handle duplicate ingresses in a deterministic manner, like always using whatever ingress comes first in the alphabet
• Accept a new annotation which the user can use to acknowledge that the ingress is sharing host and path with a different ingress, and that the admission webhook should let it pass. Instead of simply disabling the check, the admission webhook could require that the user puts the namespace and name of the offending ingress in the annotation, and deny the ingress if there were any other ingresses for the same host and path that were not mentioned. This way, you still require that the user knows about and acknowledges the specific ingresses that share host and path with theirs

I'm looking forward to seeing your full answer, tao 🙂

[alice-sawatzky]

@tobinus note that @dud225 pointed out earlier in the discussion, the currently documented behavior is "If the same path for the same host is defined in more than one Ingress, the oldest rule wins", so your suggestion of "handle duplicate ingresses in a deterministic manner" is actually already implemented. The situation is already handled gracefully, it's just prevented from occurring by the admission webhook.

In addition to it being the one already implemented I think, your suggestion of "handle duplicate ingresses in a deterministic manner" is the best option, since it allows for broader use cases than just "rename existing ingresses", but zero-downtime replacement of ingresses in general. This is useful for helm upgrades, where a template could be modified to change both the naming scheme and the configuration of an ingress at the same time (or different times, which then get compressed into one time by a user jumping multiple versions in a single upgrade).

I understand the reasoning of wanting to avoid potential confusion caused by multiple Ingress objects matching a single route, but to me this is a case of trying to mitigate a surface-level confusing behavior leading to more deeply confusing behavior down the road.

[hobti01]

We are also seeing this surprising behaviour from the admission webhook. It's surprising to us because without the ability to use the existing behaviour, cosmetically re-naming Ingress resources triggers downtime.

Are there no work-arounds other than disabling all webhook validation?

[longwuyuan]

no, there is no well-known and supported method for that.

[icy]

Can the admission webhook select what it should validate? I have two nginx ingress controller instances, and it seems there is only 1 admission webhook for both instances to validate all my ingress resources.

This is confusing and challenging. We really want to install some standby load balancer/ingress with the same name on the cluster.

[jsalatiel]

Afaik this has been fixed for several months at least.

[icy]

Afaik this has been fixed for several months at least.

hi @jsalatiel , how is the fix and in which version? I've tried the latest helm-chart and it seems the problem still persists. The controller version is 1.9.4

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.4
  Build:         846d251814a09d8a5d8d28e2e604bfc7749bcb49
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

[icy]

Afaik this has been fixed for several months at least.

My bad. I can confirm that is fixed. You need to change the controller.ingressClassResource.controllerValue to avoid collision with the existing controller.

# the first controller
  --set "controller.ingressClassResource.controllerValue=k8s.io/ingress-nginx" \
  --set "controller.ingressClass=nginx" \
  --set "controller.ingressClassResource.name=nginx" \

# the second controller
  --set "controller.ingressClassResource.controllerValue=k8s.io/ingress-nginx2" \
  --set "controller.ingressClass=nginx2" \
  --set "controller.ingressClassResource.name=nginx2" \

[jsalatiel]

Exactly.

[jsalatiel]

This ticket should be closed by the author

[longwuyuan]

/close
closing as per comments prior to this message. Can re-open if required but I know for sure that this problem is not a bug but almost certainly all symptoms are pointing at misconfiguration

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

/close
closing as per comments prior to this message. Can re-open if required but I know for sure that this problem is not a bug but almost certainly all symptoms are pointing at misconfiguration

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hobti01]

Confused as to why this is closed without any change to documentation or implementation.

• Docs state that duplicate host/path is permitted and oldest wins
• Implementation prevents duplicate host/path
• Users want the behaviour to permit duplicate host/path since this is not a validation failure, but want validation on Ingress resources (i.e. we don't want to disable the validating webhook since there are many other validations that are completely valid and aligned with ingress controller functionality)

We'd prefer the validating webhook did not mark duplicate host/path as invalid (since it is valid). Probably an argument to the webhook could make that user-configurable.

[jsalatiel]

This issue was about not being able to have the same path/host in different ingress controllers deploys/instances. It has been a regression created after the ingressClassName adoption.

[hobti01]

OK, I see that in the comments, but overlooked that in the Ingress resources as they have no ingressClassName.

So is it not considered a bug that the same host/path is not permitted in the same ingress controller despite the docs?
https://kubernetes.github.io/ingress-nginx/how-it-works/#building-the-nginx-model

If the same path for the same host is defined in more than one Ingress, the oldest rule wins.

[jsalatiel]

OK, I see that in the comments, but overlooked that in the Ingress resources as they have no ingressClassName.

So is it not considered a bug that the same host/path is not permitted in the same ingress controller despite the docs?
https://kubernetes.github.io/ingress-nginx/how-it-works/#building-the-nginx-model

If the same path for the same host is defined in more than one Ingress, the oldest rule wins.

Imho that's expected.

[airhorns]

Opened this issue #10820 to further discuss purposeful ambiguous hostnames with the same ingress class.