[FEAT] Firehose: ProcessingConfiguration with a Lambda processor (record transformation)

[okinaka]

Service

Data Firehose

API Action / Feature

ProcessingConfiguration carrying a Lambda processor, on CreateDeliveryStream and UpdateDestination, and the transformation it implies on the delivery path.

AWS Documentation

https://docs.aws.amazon.com/firehose/latest/APIReference/API_ProcessingConfiguration.html and https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html

Why is this needed?

ProcessingConfiguration is not modelled at all today, so it is dropped by @JsonIgnoreProperties(ignoreUnknown = true). A stream configured with a transformation is accepted with a 200, the member is absent from DescribeDeliveryStream, and the records are delivered untransformed. Anyone testing a transform function against Floci gets a pass that real AWS would not give: the function is never invoked, and every record arrives in the destination exactly as it was put.

This is the same shape as DataFormatConversionConfiguration before #3235, and it is the other half of what makes a Firehose delivery stream useful for local testing. It also composes with the conversion work: on AWS the transformation runs first and the converter sees its output, so a stream with both configured exercises a path Floci cannot reproduce at all right now.

I probed the behavior against real AWS (us-west-2) to establish what the emulation has to match:

• The function receives invocationId, deliveryStreamArn, region and records, each record carrying recordId, approximateArrivalTimestamp and base64 data. recordId is a shardId-prefixed sequence token, not a UUID.
• Ok replaces the record's data, Dropped discards it with no error object, and ProcessingFailed routes it to the error output.
• A record missing from the response, an Ok without data, and a repeated recordId each have their own error code, and an unknown recordId in the response is ignored rather than reported.
• A function that throws is retried NumberOfRetries + 1 times, while a response without a records key is not retried at all.
• The error output resolves !{firehose:error-output-type} to processing-failed, and its NDJSON shape differs from the conversion one: it leads with rawData and carries errorCode, errorMessage and lambdaARN.
• NumberOfRetries is not range-checked at configuration time, despite the documented 1 to 8.

I would like to take this in two steps, as with the conversion work: one change modelling, validating and echoing the configuration, and a second one implementing the transformation itself.

Are you willing to contribute a PR?

• Yes
• No