[Agentless] healthy agent without cloudbeat logs

[orouz]

Describe the bug
on ESS we have an elastic-agent 8.16.1 (agentless) reporting healthy status but when looking into Cloudbeat logs there are none for an extended time period (weeks)

To Reproduce

1. Visit agent page on ESS long running env and check the logs for Cloudbeat

Expected behavior

• Cloudbeat logs what it's currently doing or had failed to do.

Additional context

this agent is using a policy that has assets inventory installed and is misconfigured - all 3 cloud vendors are enabled at once, which is not supported. might not be related though.

[kubasobon]

There's definitely an interval issue. Running locally with a 5m period:

No results after 10 minutes besides the initial run.

[kubasobon]

This checks out. Look at this piece of code.

The workflow looks like:

1. Start fetcher.Fetch() in a goroutine for each fetcher
2. Run a loop that forever checks for new events unless the context is cancelled

The issue is fetcher.Fetch() will only ever run once. That's it.

[kubasobon]

With the fix:

Still does not explain missing logs, but it's a start. We should see "Interval reached without events" every 10 seconds outside of fetch windows.

[kubasobon]

Fix for intervals in #2902

[kubasobon]

As for no logs.
This is all the data we see for one of the cloudbeats - 258 logs with info and warn levels.

If there truly is only 1 fetcher run, it would check out. Since Interval was reached without events is a debug level log, it could be missing if logger is set to info level.

macbook:elastic-agent-diagnostics-2025-01-05T13-09-03Z-00 cat components-actual.yaml | grep 'logging.level'
                    - logging.level=info
                    - logging.level=info
                    - logging.level=info
                    - logging.level=info
                    - logging.level=info
                    - logging.level=info

Yup, diagnostics confirm it's all set to info. So no debug logs would appear. All good.

[kubasobon]

PR merged. 8.x backport can be found here: #2905

[kubasobon]

Ok, backported to 8.x, 8.16 and 8.17 branches.