perf: set max conns idle per host as docappender max request

[kruskall]

Motivation/summary

By default the http transport will use 2 as the max idle connections per host.
The docappender client will send request a single host and will have at most
MaxRequest appenders sharing the same client. To improve performance and
connection reuse we set the number of max idle conns per host to MaxRequest

Checklist

• Update CHANGELOG.asciidoc
• Update package changelog.yml (only if changes to apmpackage have been made)
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

Related issues

[mergify]

This pull request does not have a backport label. Could you fix it @kruskall? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.17 is the label to automatically backport to the 7.17 branch.
• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.

NOTE: backport-skip has been added to this pull request.

[simitt]

Why setting it to 10 here and then only later to esConfig.MaxRequests?

[kruskall]

10 is the default value

We overwrite this value with esConfig.MaxRequests after the config has been unpacked.

I've made it more explicit.

[simitt]

Other default values are set in the elasticsearch/config.go; IMO the default value should also be set there.

[kruskall]

That would change the value for every ES client which might not be ideal.

The default value of 10 only makes sense for the docappender client because it's the same as the default number of max requests handled by the docappender.

[simitt]

So in this case, shouldn't this default value then always be set via esConfig.MaxRequests in line 724? Why setting it to a hardcoded default, that easily gets outdated (as the docappender default value comes from a different source)?

[kruskall]

The hardcoded default is for when MaxRequests is 0.

Without this line, MaxIdleConnsPerHost will use a default value of 0 and the http.Transport will use the internal default (which is 2).

The docappender on the other hand will use a default value of 10, leading to the same issue we have right now (resources being underused and connections being recreated).

We need to pass the es config before creating the es client which is then passed to the docappender so we can't do this in the go-docappender library.

[simitt]

LGTM; would be good to have some actual numbers on this improvement.