OCPBUGS-43745: Add IdleCloseOnResponse field to IngressControllerSpec

[frobware]

Introduce a new knob, IdleConnectionTerminationPolicy, in the IngressController configuration to control how idle connections are handled during router reloads.

Context

In OCPBUGS-32044, the idle-close-on-response option was unconditionally added to the HAProxy confuguration to address issues with incoming HTTP requests failing during router reloads. This issue primarily affected Apache HTTPClient versions prior to 5.0, which do not gracefully handle connection resets. Adding the option ensured that idle connections were left open to handle one final request before being closed.

Historically, HAProxy 2.2 maintained idle connections during router reloads by default, allowing requests on those connections to complete even when routing configuration changes were applied. Starting with HAProxy 2.4, the default behaviour changed to close idle connections immediately during soft reloads.

To accommodate existing clients dependent on the HAProxy 2.2 behaviour, the unconditional addition of idle-close-on-response restored the previous OpenShift status quo, where customers upgrading their OpenShift clusters experienced a behaviour change due to the jump from HAProxy 2.2 to 2.6, which altered the default handling of idle connections during router reloads.

However, unconditionally enabling idle-close-on-response has now led to issues (OCPBUGS-43745) with Route backend switching. When a Route switches its service backend, requests on persistent connections could continue being routed to the previously active backend due to HAProxy handling these connections in the old process. This behaviour occurs until the connection is closed, either by a new request, the expiration of the client keep-alive, or the expiration of the HAProxy timeout http-keep-alive 300s. While this behaviour is desirable in some cases (e.g., for clients sensitive to connection resets), it can lead to temporary inconsistencies and unexpected routing behaviour during backend switching.

This PR addresses these regressions by making the behaviour configurable through a new knob.

Changes

• Add a new knob, IdleConnectionTerminationPolicy, to the IngressController configuration.
• Provide two modes:
  • Immediate: Closes idle connections immediately during router reloads.
  • Deferred: Maintains idle connections during router reloads.

Behavioural Differences

• Immediate (New Default in OpenShift 4.19+):

  • Closes idle connections immediately during router reloads.
  • Ensures immediate propagation of route changes.
  • May reset connections for clients sensitive to connection interruptions.

• Deferred (Default for backports to 4.14–4.18):

  • Keeps idle connections open until a terminating event, such as:
    • A new request on the connection (handled by the old HAProxy process).
    • Expiration of haproxy timeout http-keep-alive (300 seconds in OpenShift).
    • The client closing the connection.
  • Restores pre-2.4 HAProxy behaviour for idle connection handling.
  • May cause temporary inconsistencies during the first request on a persistent connection after a reload.
  • Can lead to resource accumulation if connections remain idle for extended periods, especially with frequent reloads.

References:

• Router change: openshift/router#639
• Ingress Controller Operator change: openshift/cluster-ingress-operator#1166

[openshift-ci]

Hello @frobware! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

[openshift-ci-robot]

@frobware: This pull request references Jira Issue OCPBUGS-43745, which is invalid:

• expected the bug to target the "4.18.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

• OCPBUGS-43745: make generate-with-container
• OCPBUGS-43745: Add IdleCloseOnResponse field to IngressControllerTuningOptions

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[frobware]

/jira refresh

[openshift-ci-robot]

@frobware: This pull request references Jira Issue OCPBUGS-43745, which is invalid:

• expected the bug to target either version "4.18." or "openshift-4.18.", but it targets "4.19.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[JoelSpeed]

I'm not sure I follow why backports would set the default to Deferred rather than Immediate, if the change in behaviour has already been made as of 4.14 (HAProxy 2.2 to 2.4), doesn't this break a change that was done a while back? What did I miss?

[frobware]

I'm not sure I follow why backports would set the default to Deferred rather than Immediate, if the change in behaviour has already been made as of 4.14 (HAProxy 2.2 to 2.4), doesn't this break a change that was done a while back? What did I miss?

The reason for defaulting to "Deferred" in backports is that OCPBUGS-32044 unconditionally made "Deferred" the default behaviour. Switching to "Immediate" in older releases would break customers by changing the behaviour in a patch release, which we need to avoid.

In hindsight, adding a knob initially would have been better, allowing customers to opt in if they wanted pre-2.4 HAProxy behaviour.

While I think "Immediate" makes the most sense as the default, using it now would break customers who rely on idle-close-on-response being enabled by default in all versions ≤ 4.18. Changing behaviour in a .z release isn’t something I want to do.

[frobware]

/test integration

[alebedev87]

/lgtm
/hold

Holding for @grzpiotrowski to have a final look.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alebedev87, frobware, JoelSpeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JoelSpeed]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[grzpiotrowski]

Everything looks good to me. I really appreciate the detailed description of the changes.

/unhold

[openshift-ci]

@frobware: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@frobware: Jira Issue OCPBUGS-43745: Some pull requests linked via external trackers have merged:

• openshift/api#2102

The following pull requests linked via external trackers have not merged:

• openshift/router#639 is open

These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-43745 has not been moved to the MODIFIED state.

In response to this:

Introduce a new knob, IdleConnectionTerminationPolicy, in the IngressController configuration to control how idle connections are handled during router reloads.

Context

In OCPBUGS-32044, the idle-close-on-response option was unconditionally added to the HAProxy confuguration to address issues with incoming HTTP requests failing during router reloads. This issue primarily affected Apache HTTPClient versions prior to 5.0, which do not gracefully handle connection resets. Adding the option ensured that idle connections were left open to handle one final request before being closed.

Historically, HAProxy 2.2 maintained idle connections during router reloads by default, allowing requests on those connections to complete even when routing configuration changes were applied. Starting with HAProxy 2.4, the default behaviour changed to close idle connections immediately during soft reloads.

To accommodate existing clients dependent on the HAProxy 2.2 behaviour, the unconditional addition of idle-close-on-response restored the previous OpenShift status quo, where customers upgrading their OpenShift clusters experienced a behaviour change due to the jump from HAProxy 2.2 to 2.6, which altered the default handling of idle connections during router reloads.

However, unconditionally enabling idle-close-on-response has now led to issues (OCPBUGS-43745) with Route backend switching. When a Route switches its service backend, requests on persistent connections could continue being routed to the previously active backend due to HAProxy handling these connections in the old process. This behaviour occurs until the connection is closed, either by a new request, the expiration of the client keep-alive, or the expiration of the HAProxy timeout http-keep-alive 300s. While this behaviour is desirable in some cases (e.g., for clients sensitive to connection resets), it can lead to temporary inconsistencies and unexpected routing behaviour during backend switching.

This PR addresses these regressions by making the behaviour configurable through a new knob.

Changes

• Add a new knob, IdleConnectionTerminationPolicy, to the IngressController configuration.
• Provide two modes:
• Immediate: Closes idle connections immediately during router reloads.
• Deferred: Maintains idle connections during router reloads.

Behavioural Differences

• Immediate (New Default in OpenShift 4.19+):

• Closes idle connections immediately during router reloads.

• Ensures immediate propagation of route changes.

• May reset connections for clients sensitive to connection interruptions.

• Deferred (Default for backports to 4.14–4.18):

• Keeps idle connections open until a terminating event, such as:

  • A new request on the connection (handled by the old HAProxy process).
  • Expiration of haproxy timeout http-keep-alive (300 seconds in OpenShift).
  • The client closing the connection.

• Restores pre-2.4 HAProxy behaviour for idle connection handling.

• May cause temporary inconsistencies during the first request on a persistent connection after a reload.

• Can lead to resource accumulation if connections remain idle for extended periods, especially with frequent reloads.

References:

• Router change: openshift/router#639
• Ingress Controller Operator change: openshift/cluster-ingress-operator#1166

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

Distgit: ose-cluster-config-api
This PR has been included in build ose-cluster-config-api-container-v4.19.0-202412181337.p0.g2731647.assembly.stream.el9.
All builds following this will include this PR.

[frobware]

/cherry-pick release-4.18

[openshift-cherrypick-robot]

@frobware: new pull request created: #2145

In response to this:

/cherry-pick release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.