Ingress configured APIs started to throw 400 No Required SSL Certificate Error after upgrade from V1.10.0 to V1.12.0

[Laveshsp]

What happened:

My API ingresses started throwing 400 Bad Request No Required SSL Certificate Error at nginx level after I upgraded the controller from V1.10.0 to V1.12.0

The APIs with both mTLS and non mTLS works as expected on V1.10.0 but both are not working on V1.12.0
Below are my ingress-conf

1. Ingress Non mTLS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress-internal
spec:
  ingressClassName: nginx
  rules:
    - host: api.test.dev.edifecs.cloud
      http:
        paths:
          - path: /Path1
            pathType: Prefix
            backend:
              service:
                name: test-276-277-rt-profile-service
                port:
                  number: 9069
          - path: /Path2
            pathType: Prefix
            backend:
              service:
                name: test-270-271-rt-profile-service
                port:
                  number: 9072
          - path: /Path3
            pathType: Prefix
            backend:
              service:
                name: test-278-rt-profile-service
                port:
                  number: 9073

2. Ingress with mTLS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
      nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
      nginx.ingress.kubernetes.io/auth-tls-secret: test/ca-secret
      nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
      nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
  name: test-ingress-internal
spec:
  ingressClassName: nginx
  rules:
    - host: api.test.dev.edifecs.cloud
      http:
        paths:
          - path: /Path1
            pathType: Prefix
            backend:
              service:
                name: test-276-277-rt-profile-service
                port:
                  number: 9069
          - path: /Path2
            pathType: Prefix
            backend:
              service:
                name: test-270-271-rt-profile-service
                port:
                  number: 9072
          - path: /Path3
            pathType: Prefix
            backend:
              service:
                name: test-278-rt-profile-service
                port:
                  number: 9073
  tls:
    - hosts:
        - api.test.dev.edifecs.cloud
      secretName: internal-api-tls

What you expected to happen:

With the version upgrade to V1.12.0, the APIs should perform as before how it was working with V1.10.0 for both mTLS ingress and non mTLS ingress

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

Kubernetes version (use kubectl version): 1.30

Environment:

• Cloud provider or hardware configuration: AWS

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>

controller:
  allowSnippetAnnotations: true
  autoscaling:
    enabled: true
    maxReplicas: 2
  config:
    enable-real-ip: true
    enable-underscores-in-headers: true
    log-format-upstream: '{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName":
      "$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus":
      $upstream_status, "upstreamAddr": "$upstream_addr", "httpRequest": {"requestMethod":
      "$request_method", "requestUrl": "$host$request_uri", "status": $status,"requestSize":
      $request_length, "responseSize": $upstream_response_length, "userAgent": "$http_user_agent",
      "remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time
      s", "protocol": "$server_protocol"}}'
    use-forwarded-headers: true
    use-proxy-protocol: true
    hsts-preload: true
  extraArgs:
    default-ssl-certificate: ingress/default-tls
  metrics:
    enabled: true
  podAnnotations:
    '"prometheus.io/port"': 10254
    '"prometheus.io/scrape"': true
  replicaCount: 1
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: 3600
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: false
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
      service.beta.kubernetes.io/aws-load-balancer-scheme: internal
      service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
      service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: proxy_protocol_v2.enabled=true
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
    loadBalancerSourceRanges:
    - 10.0.0.0/8
tolerations:
- effect: NoSchedule
  key: mainnode
  operator: Exists

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Laveshsp]

I was able to fix this issue. There was one more ingress created with same hostname and different path which had mTLS annotations.
In this scenario, when the endpoint was hit with path whose conf didnt have mTLS, that too was expecting mTLS certs.

For understanding, if we have multiple ingresses configured with same hostname but different paths, won't we be able to enable/disable mTLS for each ingress conf based on annotations. Also, for knowledge, is only the mTLS annotations are shared or all annotations and how is the priority decided to pick what annotation if they are shared