Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move config reloading to beatcmd #9371

Merged
merged 6 commits into from
Oct 17, 2022
Merged

Move config reloading to beatcmd #9371

merged 6 commits into from
Oct 17, 2022

Conversation

axw
Copy link
Member

@axw axw commented Oct 15, 2022

Motivation/summary

beatcmd is now provided with a function that returns a "Runner", given some parameters: a complete configuration object containing both input and output configuration, a logger, and beat.Info. beatcmd registers with the libbeat management API for handling config changes. beatcmd takes care of stopping and starting Runners in response to config reloads.

beater is now only responsible for running a single instance of the APM Server logic with a static configuration. beater does not depend on beatcmd by design. Some of the test helper code in beater has been moved to a new package, beatertest. In a followup we should rename beate", to clarify that it is no longer beat-centric.

Checklist

- [ ] Update CHANGELOG.asciidoc
- [ ] Update package changelog.yml (only if changes to apmpackage have been made)
- [ ] Documentation has been updated

How to test these changes

Non-functional change.

Related issues

Closes #8928

@axw axw added the v8.6.0 label Oct 15, 2022
@mergify
Copy link
Contributor

mergify bot commented Oct 15, 2022

This pull request does not have a backport label. Could you fix it @axw? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-7.x is the label to automatically backport to the 7.x branch.
  • backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 15, 2022
@apmmachine
Copy link
Contributor

apmmachine commented Oct 15, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-10-17T03:10:59.701+0000

  • Duration: 29 min 10 sec

Test stats 🧪

Test Results
Failed 0
Passed 150
Skipped 0
Total 150

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate and publish the docker images.

  • /test windows : Build & tests on Windows.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

beatcmd is now provided with a function that
returns a "Runner", given some parameters:
a complete configuration object containing
both input and output configuration, a logger,
and beat.Info. beatcmd registers with the
libbeat management API for handling config
changes.

beatcmd takes care of stopping and starting
Runners in response to config reloads.

beater is now only responsible for running a
single instance of the APM Server logic with
a static configuration. beater does not
depend on beatcmd by design.

Some of the test helper code in beater has
been moved to a new package, beatertest.
@axw axw force-pushed the beatcmd-reloader branch from f47975d to 58fda31 Compare October 16, 2022 02:57
@apmmachine
Copy link
Contributor

apmmachine commented Oct 16, 2022

📚 Go benchmark report

Diff with the main branch

name                                                                                              old time/op    new time/op    delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
ModelIndexer/NoCompression-12                                                                       1.13µs ±25%    1.00µs ± 4%  -11.74%  (p=0.032 n=5+5)
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/ratelimit.ndjson-12                     17.6µs ± 7%    20.7µs ±29%  +17.77%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/minimal-service.ndjson-12               1.31µs ± 1%    1.29µs ± 0%   -0.87%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors.ndjson-12                      7.42µs ± 2%    7.27µs ± 1%   -1.94%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12       1.89µs ± 1%    1.87µs ± 1%   -0.88%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata-2.ndjson-12           493ns ± 1%     486ns ± 0%   -1.39%  (p=0.016 n=5+4)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/minimal-service.ndjson-12              951ns ± 1%     965ns ± 1%   +1.46%  (p=0.016 n=5+5)
ReadBatch/heavy.ndjson-12                                                                           3.76ms ±16%    4.70ms ±22%  +25.14%  (p=0.016 n=5+5)
ReadBatch/metadata-null-values.ndjson-12                                                            8.37µs ±28%   11.10µs ±17%  +32.54%  (p=0.032 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
ShardedWriteTransactionUncontended-12                                                                849ns ± 1%     909ns ±11%   +7.12%  (p=0.032 n=4+5)

name                                                                                              old alloc/op   new alloc/op   delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/transactions_spans_rum.ndjson-12                                                   5.55kB ± 1%    5.63kB ± 1%   +1.49%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/invalid-metadata.ndjson-12              3.25kB ± 1%    3.31kB ± 1%   +1.86%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/optional-timestamps.ndjson-12           5.05kB ± 1%    5.15kB ± 2%   +1.88%  (p=0.032 n=5+5)
ReadBatch/transactions_spans.ndjson-12                                                              22.8kB ± 0%    22.8kB ± 0%   +0.06%  (p=0.048 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64

name                                                                                              old allocs/op  new allocs/op  delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64

name                                                                                              old speed      new speed      delta
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/errors_rum.ndjson-12                                                             69.9MB/s ± 4%  85.4MB/s ±19%  +22.19%  (p=0.032 n=4+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/ratelimit.ndjson-12                    240MB/s ± 7%   208MB/s ±24%  -13.41%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/minimal-service.ndjson-12              326MB/s ± 1%   328MB/s ± 0%   +0.90%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors.ndjson-12                     855MB/s ± 2%   872MB/s ± 1%   +1.97%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12      236MB/s ± 1%   238MB/s ± 1%   +0.88%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata-2.ndjson-12         885MB/s ± 1%   897MB/s ± 0%   +1.40%  (p=0.016 n=5+4)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/minimal-service.ndjson-12            447MB/s ± 1%   441MB/s ± 1%   -1.43%  (p=0.016 n=5+5)
ReadBatch/heavy.ndjson-12                                                                          107MB/s ±16%    86MB/s ±19%  -20.03%  (p=0.016 n=5+5)
ReadBatch/metadata-null-values.ndjson-12                                                          64.9MB/s ±34%  48.1MB/s ±19%  -25.94%  (p=0.032 n=5+5)

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

@axw axw requested a review from a team October 17, 2022 00:19
@axw axw marked this pull request as ready for review October 17, 2022 00:19
Copy link
Contributor

@marclop marclop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doubtlessly an improvement, I find the reloading and creation of the beater much easier to follow and also helps that some of the test creation of was moved to its own "test' package.

@@ -283,14 +257,25 @@ func (b *Beat) Run(ctx context.Context) error {
}()
defer logp.Info("%s stopped.", b.Info.Beat)

logger := logp.NewLogger("")
if runtime.GOARCH == "386" {
logger.Warn("" +
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this empty concatenation necessary?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only necessary for my compulsion to have the proceeding strings lined up :)
I can get rid of it if you prefer?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with it, I was just curious to learn why.

logp.Info("%s started.", b.Info.Beat)
b.Manager.SetStopCallback(cancel)
return beater.Run(&b.Beat)
if err := g.Wait(); err != nil && !errors.Is(err, context.Canceled) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're calling g.Wait here and then on line 271 again. Is that intentional?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's intentional to ensure all goroutines have exited by the time the Run method returns. Not strictly necessary, but may avoid surprises in tests with code from prior tests still running. I'll add a comment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the comment will help clarify things for readers. I thought that was why, but wasn't sure.

internal/beatcmd/reloader.go Outdated Show resolved Hide resolved
internal/beater/beater.go Outdated Show resolved Hide resolved
internal/beater/beater.go Outdated Show resolved Hide resolved
@axw axw enabled auto-merge (squash) October 17, 2022 02:25
- Don't overwrite ctx in Runner.Run while it's being read
- Atomically increment "requests" in HTTP handler in test
- Allow for extra log message in test
@axw axw merged commit a824df6 into elastic:main Oct 17, 2022
@axw axw deleted the beatcmd-reloader branch October 17, 2022 04:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-skip Skip notification from the automated backport with mergify v8.6.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Refactor "beater" package
3 participants