[gce_testing] Create findMatchingLogs to return all found logs and create QueryAllLogs.

[franciscovalentecastro]

Update hasMatchingLog to return all found logs and create QueryAllLogs.

Context :

• Helpful to simplify some integration tests like [confgenerator] Add file offset storage to Otel Logging receivers. ops-agent#2166, we need to able to know how many logs did a query returned.

[jefferbrecht]

Why not just leave QueryLog alone and add a new QueryLogs that returns a slice of matching entries? As long as we're iterating all of the returned logs we may as well pass them on to the caller, who can just call len(...) to get the number of matches. That would also have the benefit of not breaking the API requiring us to change all of the call sites.

[franciscovalentecastro]

Why not just leave QueryLog alone and add a new QueryLogs that returns a slice of matching entries? As long as we're iterating all of the returned logs we may as well pass them on to the caller, who can just call len(...) to get the number of matches. That would also have the benefit of not breaking the API requiring us to change all of the call sites.

Thank you for the comment @jefferbrecht ! I was going back and forth between the possibilities of how to expose the "found log count" while minimizing the API updates. The last update returns all found logs in hasMatchingLog and creates QueryAllLogs to be able to return all the found logs.

[jefferbrecht]

Nit: append here is guaranteed to result in a non-empty slice, and we set found at the same time as append, so we no longer need the found variable -- just test on len(matchingLogs) > 0. We can also eliminate the returned bool: if the query turns up empty then we'd signal "logs not found" by returning an empty slice.

[franciscovalentecastro]

Done! Renamed hasMatchingLogs to findMatchingLogs to represent more accurately the new implementation.

[jefferbrecht]

We should be able to implement QueryLog by just calling QueryAllLogs internally and returning the first element of the slice.

[franciscovalentecastro]

Done!

[jefferbrecht]

QueryAllLogs arguably does not need the "retry until at least one log is found" logic (although the "retry on retriable failure" logic should stay). IMO that's only there for "get me the first matching log ASAP" (which is just QueryLog), whereas here the caller is more likely to be looking for "get me all matching logs from the given time window" (after they've waited an appropriate amount of time for the eventual consistency of the backend).

[franciscovalentecastro]

I don't see how to remove "retry until at least one log is found" from QueryAllLogs without having some downsides.

@jefferbrecht Could you add more details in the expected solution here ?

IMO that's only there for "get me the first matching log ASAP"

I think it could also be thought as "return query result if you find any data". It has the additional value of shorter tests.

QueryAllLogs arguably does not need the "retry until at least one log is found" logic (although the "retry on retriable failure" logic should stay).

Yes, "get me all matching logs from the given time window" maybe the most likely use case of QueryAllLogs, but i don't see an alternative way of "stopping the lookup" that is more helpful for testing.

Alternatives :

• [Current] Stops querying the backend if any data is found (len(matchingLogs) > 0).
  • Shorter tests.
  • Caller is expecting to see "all found data".
• Stop when err == nil
  • Shorter tests. May result in "no data found".
  • No consideration for race conditions when log may take longer to be ingested.
• Create an expectedLogCount and stop when len(matchingLogs) >= expectedLogCount.
  • Same as the previous one, but more restrictive.
• Remove the "stop after data is found" will result in the always querying "maxAttempts".
  • The tests is long (maxAttempt * backoffTime) and if the time window is small (e.g. 1 minute), the last query could return less data after backoff retries.