Support for specifying log groups for subscription filters that exceed parameter value limit

[pushred]

Describe the enhancement:

The project where I am trying to use ESF currently has 176 log groups with more likely to come. Following the published docs for deploying ESF, all three of the options require the use of CloudFormation parameters to specify the ARNs for these groups. However with so many groups I quickly exceed the 4,096 byte limit of parameter values.

I'm not sure that there are any other possible ways I could provide the ARNs, given the type limitations of parameter values. Ideally I could pass an actual array/list, but CF does not appear to support this.

So given that, the enhancement I may need is a means of deploying ESF without using SAR/CloudFormation. The only options that I can see are:

1. "Sharding" ESF deployments. Multiple deploys that each handle a subset of the ARNs. This would currently require 5 stacks — not ideal.

2. Building my own CloudFormation stack. Essentially forking this project in order to bypass parameters and decomposing the stack and deploying it through some other means. Aside from the upfront cost of doing this I think it would impose a maintenance burden in staying aligned. And almost definitely unsupported.

Are there any other options?

Is option 1 the only viable option while AWS does not offer a solution?

Describe a specific use case for the enhancement or feature:

Projects with more than ~30 log groups.

[aspacca]

hello @pushred

So given that, the enhancement I may need is a means of deploying ESF without using SAR/CloudFormation

we created a script for such purpose, it is already merged in main branch: https://github.com/elastic/elastic-serverless-forwarder/blob/main/publish_lambda.sh

it requires a yaml file for configuring the ARNs and IDs, similar to the ones you provide as parameters when publishing from SAR. It will be part of the lambda-v1.6.0 release and you'll find the relevant documentation at https://www.elastic.co/guide/en/observability/current/aws-elastic-serverless-forwarder.html once we will release

[aspacca]

1. "Sharding" ESF deployments. Multiple deploys that each handle a subset of the ARNs. This would currently require 5 stacks — not ideal.

this is indeed not possible when deploying from SAR: our SAR templates include a macro, that can be referenced only by literal name. basically we cannot create multiple macros for each deployment with dynamic names, since we could not use function required to compose the dynamic name when referencing the macros. trying to create the same macro after the first deployment will result in the failure of the following deployments
it's a cloudformation limit we cannot overcome

[pushred]

@aspacca thank you for such a quick reply, and good timing with the addition of this script! I managed to run it successfully with a few tweaks:

• Updated script to call python3 vs. python as my shell alias for this doesn't work in the script. Maybe not an issue in our CI env but I haven't run Python in that environment before so not sure. Either way I can modify the script but perhaps another parameter might be warranted for this.

• Had to run with AWS_PAGER="" to suppress paged output from the aws s3api command — perhaps this is something that should be configured at the environment level but a flag on that command could avoid that.

I also had to have Docker running for SAM.

With all that out of the way I successfully deployed a stack with a single log group. However once I added all 176 groups I am now encountering this error:

Successfully packaged artifacts and wrote output template to file /tmp/publish.KxVn82uCKi/.aws-sam/build/publish/packaged.yaml.
Execute the following command to deploy the packaged template
sam deploy --template-file /tmp/publish.KxVn82uCKi/.aws-sam/build/publish/packaged.yaml --stack-name <YOUR STACK NAME>

Error: Templates with a size greater than 51,200 bytes must be deployed via an S3 Bucket. Please add the --s3-bucket parameter to your command. The local template will be copied to that S3 bucket and then deployed.

My publish config file is 19k.

When this error is raised the temporary file no longer exists so I'm not able to simply resume the process. The script may need to be revised to use the S3 bucket deploy method. I'll review making that modification for my purposes but curious if this would also potentially happen for your version.

[pushred]

There actually appears to be a bug in the script that is at least contributing to the file size issue. I commented out the trap call to examine the build artifacts and found that I have a 1.9mb publish.yaml. Inspecting the file I found that for every log group I've specified there are multiple PolicyDocument Statement instances, including a logs:DescribeLogStreams that includes all groups that have been looped through up to that point, e.g. given 3 ARNs I have output like:

- Effect: Allow
  Action: logs:DescribeLogGroups
  Resource:
  - arn:aws:logs:us-east-1:123456789101:log-group:*:*
- Effect: Allow
  Action: logs:DescribeLogStreams
  Resource:
  - arn:aws:logs:us-east-1:123456789101:log-group:/aws/lambda/log-group-a:log-stream:*
- Effect: Allow
  Action: logs:DescribeLogGroups
  Resource:
  - arn:aws:logs:us-east-1:123456789101:log-group:*:*
- Effect: Allow
  Action: logs:DescribeLogStreams
  Resource:
  - arn:aws:logs:us-east-1:123456789101:log-group:/aws/lambda/log-group-a:log-stream:*
  - arn:aws:logs:us-east-1:123456789101:log-group:/aws/lambda/log-group-b:log-stream:*
- Effect: Allow
  Action: logs:DescribeLogGroups
  Resource:
  - arn:aws:logs:us-east-1:123456789101:log-group:*:*
- Effect: Allow
  Action: logs:DescribeLogStreams
  Resource:
  - arn:aws:logs:us-east-1:123456789101:log-group:/aws/lambda/log-group-a:log-stream:*
  - arn:aws:logs:us-east-1:123456789101:log-group:/aws/lambda/log-group-b:log-stream:*
  - arn:aws:logs:us-east-1:123456789101:log-group:/aws/lambda/log-group-c:log-stream:*

If I remove all of those instances I am still exceeding the template size limit at 60k but it is still far less.

[pushred]

Well I worked around that, but hit another AWS limit that will probably force a much bigger workaround. With this many log groups my ElasticServerlessForwarderPolicy is ~22k but this exceeds a maximum policy size of 10,240 bytes. To workaround that I tried specifying a wildcard to match based on a log group name prefix, e.g.

Effect: Allow
Action: `
Resource:
- arn:aws:logs:us-east-1: 123456789101:log-group:/aws/lambda/naming-prefix-*:log-stream:*

It's unclear if that is supported but attempting to deploy I ran into either a side effect of that or another limit with failures to create AWS::Lambda::Permission with this error:

The final policy size (20702) is bigger than the limit (20480). (Service: AWSLambdaInternal;
Status Code: 400;
Error Code: PolicyLengthExceededException;    

This appears to be due to the number of CloudWatchLogs events that each add a resource-based policy statement for the function's permissions. Those combine into a single policy document that reaches this limit at ~42 functions for me. I'm not sure if there's any possible workaround for this as we of course need the triggers and CloudFormation is generating these policy statements so I don't believe I have control over their content, if it is even possible to specify partial ARNs in that with wildcards as I did for the logs:DescribeLogStreams permissions.

Are there any options remaining other than somehow deploying this fully outside of CloudFormation? Or a major refactor of our project to bring our function count way down.

[aspacca]

hi @pushred , thanks for your feedback

There actually appears to be a bug in the script that is at least contributing to the file size issue

thanks for reporting it: I missed that. I will push a fix as soon as possible
not only ARNs from the cloudwatch-logs might be affected

The script may need to be revised to use the S3 bucket deploy method

thanks again, I will think if making it the default behaviour or an option

The final policy size (20702) is bigger than the limit (20480). (Service: AWSLambdaInternal;

while can optimise the size of the policy to make it the smallest possibile for the number of given ARNs, in the end the final size can hit the limit from AWS. there's nothing we (or you) can do for this, but for deploying multiple lambdas so that the policy is split in different smaller ones

[aspacca]

To workaround that I tried specifying a wildcard to match based on a log group name prefix, e.g.

if you provide a "glob" value directly as ARN in the publish config yaml, while this might produce (just assuming, I didn't check) a smaller policy that's working properly, it will fail to attach the cloudwatch logs groups as trigger of the forwarder

[aspacca]

thanks for reporting it: I missed that. I will push a fix as soon as possible
not only ARNs from the cloudwatch-logs might be affected

I spotted the bug in an extra level of indentation

[aspacca]

thanks again, I will think if making it the default behaviour or an option

I've decided to make it the default behaviour: could you please test script from the branch with the fixes?

[aspacca]

• Updated script to call python3 vs. python as my shell alias for this doesn't work in the script. Maybe not an issue in our CI env but I haven't run Python in that environment before so not sure. Either way I can modify the script but perhaps another parameter might be warranted for this.

not changing this for the moment: we'll most likely provide instructions about the fact that python is used, and suggest to run the script in a venv, so that the binary will be available

but thanks anyway for rising the issue

[pushred]

Confirmed that with the latest script the generated publish.yaml PolicyDocument is as expected and that I'm able to deploy with 40 log groups. Thanks!

if you provide a "glob" value directly as ARN in the publish config yaml, while this might produce (just assuming, I didn't check) a smaller policy that's working properly, it will fail to attach the cloudwatch logs groups as trigger of the forwarder

Good to know.. there is no escaping the limits around this.

suggest to run the script in a venv, so that the binary will be available

Ah right, I haven't worked with Python lately but we do use venv in all of our Python projects so that should work well.

while can optimise the size of the policy to make it the smallest possibile for the number of given ARNs, in the end the final size can hit the limit from AWS. there's nothing we (or you) can do for this, but for deploying multiple lambdas so that the policy is split in different smaller ones

So based on what you mentioned earlier re: multiple deployments, my understanding is that multiple CloudFormation ESF stacks isn't possible but there could be multiple Lambdas within the deployed stack over which the log groups can be spread. Given that ESF controls the stack, is this an enhancement that could be made anytime soon?

Beyond the current project I am working on I'm also wondering about other Lambda-based services where we would like to ship logs. If everything must be routed through the same ESF stack we would need this capability regardless of how well we could decompose the project or otherwise reduce the # of functions. Multiple AWS accounts could be an option but I don't believe a viable one in our org.

[pushred]

Thinking about this more, the prospect of having dozens of Lambdas and queues to handle our quantity of log groups across multiple environments is daunting.

Our use case is probably better suited by Elastic Agent which I looked at that initially but somehow overlooked the Kibana Fleet API for enabling log group integrations programatically. Will be pursuing that for now.

[aspacca]

So based on what you mentioned earlier re: multiple deployments, my understanding is that multiple CloudFormation ESF stacks isn't possible but there could be multiple Lambdas within the deployed stack over which the log groups can be spread. Given that ESF controls the stack, is this an enhancement that could be made anytime soon?

please, let me try to rephrase what are exactly your needs:

1. you would like to be able to manage with Infrastructure as Code the "stack" required to ingest in an Elasticsearch cluster the events coming from a high number of cloudwatch logs log groups
2. you prefer to be able to have a single "definition" in the Infrastructure as Code tool that you need to use according to the solution. I mean you prefer to have a single CloudFormation template or terraform definition where to specify multiple Lambdas to deploy, vs having multiple templates/definition for each Lambda.
3. you care for having the smallest number of infra components in the "stack" that's required for achieving 1.

Did I summarise correctly?

I assume the feature provided by each between Elastic Agent and the Elastic Serverless Forwarder, are enough for you and the choice is based only on the three points above.

how Elastic Agent vs Elastic Serverless Forwarder compare for each of the points?

Point	Elastic Agent	Elastic Serverless Forwarder
1.	You have two options to run Elastic Agent: either as Fleet Managed or Standalone (you can check here the different capabilities between to two modes). Standalone mode is ready for IaC solution, but you'll lose the capabilities provided by the Fleet Managed option. To keep both Fleet Managed capabilities and IaC you have to built your own IaC tool to interact with the Fleet API	Currently you have to options to deploy the Elastic Serverless Forwarder: either form AWS SAR or directly with the publishing script. The difference between the two options is about the level of granularity configuration that you can set for a single ingestion source (in general, for the specific case of cloudwatch logs log group, at the moment this does not apply, since what can be configured is the same for both - this might change if we will support the possibility to configure the FilterPattern in the publishing script). Another difference is that, given AWS limits about the "information" you can provide in a single CloudFormation template, the publishing script can be used as a workaround to bypass those limits. SAR option is available through AWS Console, CloudFormation and Terraform. The publishing script has to be integrated according to its requirement in your existing IaC solution. In general IaC is first citizen for us in providing deployment solutions for the Elastic Serverless Forwarder and our goal is to improve the IaC experience for the users, both providing a solution for the most used IaC tools. as well as providing the more capabilities we can for every IaC tool we will support.
2.	In the case of a Standalone Elastic Agent this is probably generally achievable, according to the specific IaC solution you will adopt, even if requiring to manage more VMs due to scaling factor. In the case of a Fleet Managed Elastic Agent, you'll have to build your own IaC tool, and this will be achievable according to how you build the tool.	This is true, currently, if you don't hit the limits given by AWS. If you hit those limits the publishing script can be used as a workaround to bypass them but at the same time, this means having multiple publish config yaml files. Since you have to integrate the execution of the publishing script in your IaC tool, according to the capabilities of that tool, the publish config yaml files could be generated on the fly from a single "definition" in the tool. Referring to our effort of making IaC a first-class citizen, we are already investigating alternative IaC solutions to automatically deal with the AWS limits without requiring the users to manage the scenario. No ETA or expectation about the support for a specific IaC tool can be given at the moment.
3.	According to scaling factor related to the number of the cloudwatch logs log group you need 1 or more VM with Elastic Agent running on it. You are not forced to mange multiple VMs if not required by scaling factors, just for complying with limits on provisioning too much "information"	According to scaling factor related to the number of the cloudwatch logs log group you need 1 or more Lambda and two SQS queues for each Lambda (the continuing queue and the replay queue). You might be forced to have multiple Lambda and the two SQS queues given the limits set by AWS about the "information" you can provide to a single Lambda while provisioning it.

Finally I want to mention an aspect to consider in relation to scaling:

• AWS limits the number of concurrent execution for the Lambdas in a given region for a single account to 1000. You can request to AWS to increase this limit. I don't know it this limit can be increased indefinitely. There are others not increasable limits related to Lambda, like the size of a synchronous payload, that might impact on the infinite scaling possibility of a Lambda.
• Something similar applies to EC2 Spot/On Demand instances (https://aws.amazon.com/ec2/faqs/#EC2_On-Demand_Instance_limits). In terms of compute power scaling there is no not increasable quotas as much as I can see. Still I don't know it the existing limits can be increased indefinitely.

This is my knowledge, just looking at the information made available from AWS: if you are concerned about hitting any scaling limits in your choice, you should contact AWS in order to properly address how a solution based on Lambda and one on EC2 instances compare.

[pushred]

you would like to be able to manage with Infrastructure as Code the "stack" required to ingest in an Elasticsearch cluster the events coming from a high number of cloudwatch logs log groups

It's a preference in our org for IaC via Terraform but some exceptions have been made for projects built with the Serverless framework, which is built over CloudFormation as well. So by extension we expected ESF to be granted a similar exception.

But we also have instances of Fleet-managed Elastic Agent already in use. The agents thus far are containerized and run on the same server as the services that are sending telemetry.

you prefer to be able to have a single "definition" in the Infrastructure as Code tool that you need to use according to the solution. I mean you prefer to have a single CloudFormation template or terraform definition where to specify multiple Lambdas to deploy, vs having multiple templates/definition for each Lambda.

This would be ideal as we otherwise have to somehow otherwise handle the stack splitting.

you care for having the smallest number of infra components in the "stack" that's required for achieving 1.

Correct, but we understand the hard AWS constraints that are preventing this from happening.

---

Thanks for your comparison of Elastic Agent vs. ESF and the other notes re: scaling. This has all been helpful today and spawned some lengthier discussion on our requirements. I confess that I jumped into trying to use ESF without much review, following a colleague's earlier effort that identified it as a solution following a separate earlier POC effort using Functionbeat. After further reviewing our needs and the context of everything else running in our account we've concluded that getting logs from CloudWatch's API, as the Elastic Agent/Filebeat integration does, won't be feasible either due to hard limits on FilterLogEvents API requests.

I think we may have been misguided in gravitating to Functionbeat and ESF for a solution to ingesting Lambda logs. This was partly due to their familiar deployment model. We associated the other solutions, e.g. Filebeat as being more suited to our EC2 instances. I see the value of ESF in simpler scenarios as something that is faster to deploy and cheaper to operate.

Longer term I believe what we actually need is something like the Elastic APM AWS Lambda extension which would bypass the CloudFront layer altogether. The team behind that indeed added some support for collecting logs last year but it is still preliminary. The recently released AWS Lambda Telemetry API seems that it will further the possibilities and many of Elastic's competitors already have extension-based solutions for collecting logs, using an earlier iteration of that API.

Unfortunately the project I am working on has a timeline that requires observability to be in place before such a solution will exist. So we are going to pursue building our own extension in order to write logs to S3 and then ingest them using some other Elastic solution.

Thank you for all of your assistance and time!