How to deal with the increasing access log?

[spgyip]

We deploy a 'service pod' along with a sidecar envoy. The problem is the access log of envoy, there is no size-limit or log-rotate supported by envoy. So we have to clean out the access log when it grow too large, e.g. launch another shell/program to rename access log and reload the envoy(Just like what we do with nginx access log).

However, dealing with hundreds or thousands of 'service pod' is also a tough job I think, not graceful.
What do you suggest to do with?

Or if I don't make my scenario understandable, please reach me out.

[dnoe]

I am doing some ongoing work to ship Access Logs in proto format to a gRPC log sink which should be properly formed up into to a PR later this week. One canonical example of this is the Google Stackdriver public API, but you could develop your own gRPC service and send the logs to it. This will give you the ability to collect all access logs centrally and decide what you want to keep and for how long. I don't know whether this would meet your needs or is too much complexity but I wanted to let you know it's something that's in the works. You can see the proto format in https://github.com/lyft/envoy-api/blob/master/api/accesslog.proto

For a simple solution I'd suggest using something like the standard logrotate typically shipped by Linux distributions rather than trying to build this into Envoy itself.

[mattklein123]

per @dnoe we will likely not implement any sysadmin type functionality like this directly in envoy when well known solutions already exist (in this case logrotate).

[spgyip]

@dnoe Thanks for that. Actually, logging to remote is a workable solution, but I think it's too heavy to handle the access log situation.

I am looking forward to solution like logrotate, not invaded to envoy itself.

[mattklein123]

Closing for now as we are unlikely to build this type of ops functionality into envoy directly.

[perlun]

For people reading this who might be unfamiliar with logrotate, it's configuration files are kept in /etc/logrotate.d on a typical Debian/Ubuntu machine. man logrotate will give you some more information about how it works, including an example rotate config you can use as template for building your own, personalized configuration.

[yury-alpatov]

Can you please explain me, how to enable this functionality in already installed Istio (in OpenShift)? I tried to edit ConfigMap “istio” but it doesn’t take effect (enableEnvoyAccessLogService: true, envoyAccessLogService address).

[pat70]

per @dnoe we will likely not implement any sysadmin type functionality like this directly in envoy when well known solutions already exist (in this case logrotate).

foreversd/forever#106 (comment)
"
So, in case of create logrotate option - forever proceeds to write to rotated log file instead of main one.
If copytruncate option set - forever write to main log file, but it's not wiped when rotated and logs keep accumulate.

Does someone have a solution?
"

[mladenkralev]

We have a similar issue where we use logrotate to rotate logs in a pod. Seems strange that despite enforcing log rotation the newly created file is ignored and only the old (already rotated file) is used and growing.

Does anyone else also hit this issue?
We are using 1.17.4 envoy version and the logrotate configuration file looks similar to:
/tmp/* { rotate 3 size 1k compress delaycompress }

Edit
Adding copytruncate option from logrotate solved the problem for our team