[SVLS-8979] Add CloudFormation template for Lambda Durable Function event forwarder

[lym953]

Context

To capture TIMED_OUT and STOPPED status of Lambda durable function executions, we need to capture the status change events in EventBridge and forward them to Datadog. This involves three changes:

1. AWS client-side integrations: EventBridge rule to get the logs, Firehose to batch send the logs to Datadog. The CloudFormation template will be published to S3 so customers can install it by 1 click, in the same way as installing Datadog forwarder. (This PR)
2. Logs pipeline on Datadog side: transforms the logs (https://github.com/DataDog/integrations-internal-core/pull/3389)
3. Datadog's Lambda UI: consume the logs

This template was originally proposed in DataDog/datadog-serverless-functions#1149. Per team guidance, this repo (cloudformation-template) is the correct place to host the template, so the PR has been re-created here following this repo's conventions (a top-level aws_<name>/ directory with the template and README.md). Publishing the template to S3 will be handled separately in a different repo, so this PR contains no release script.

Architecture

This is Option 4.3 in the design doc. See the doc for why we need to capture the status change events.

Changes

• Add aws_durable_function_event_forwarder/durable_function_event_forwarder.yaml — the CloudFormation template for the AWS-side resources
• Add aws_durable_function_event_forwarder/README.md

Params of the CloudFormation template

• It supports three ways to set DD API key:
  • Plaintext
  • Secrets Manager secret ARN
  • SSM SecureString parameter name
• Datadog site, defaults to datadoghq.com
• Statuses to forward, defaults to empty, i.e. forward all statuses: RUNNING,SUCCEEDED,FAILED,TIMED_OUT,STOPPED.
• Function ARN filters. Can be either the unqualified ARN for a single function, e.g. arn:aws:lambda:us-east-2:425362996713:function:my-durable-function, or a wildcard pattern, e.g. arn:aws:lambda:us-east-2:425362996713:function:my-durable-*.
  • We support up to 5 filters. If there's a ask, we can consider adding more. However, I expect most customers to leave them empty so the stack covers all the durable functions in the region.
• Firehose buffer interval, defaults to 60 seconds. This is the interval at which events are sent to Datadog.

Next steps

• Enable releasing to Datadog Prod account

Test plan

Steps

• Upload the CloudFormation template to S3 in Datadog Serverless Sandbox by running, from the aws_durable_function_event_forwarder/ directory:

  aws-vault exec sso-serverless-sandbox-account-admin -- \
    aws s3 cp durable_function_event_forwarder.yaml \
    s3://datadog-cloudformation-template-serverless-sandbox/aws/lambda-durable-function-event-forwarder/latest.yaml \
    --content-type text/yaml

• Deploy this stack: https://us-east-2.console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/create/review?templateURL=https://datadog-cloudformation-template-serverless-sandbox.s3.amazonaws.com/aws/lambda-durable-function-event-forwarder/latest.yaml&stackName=datadog-durable-function-event-forwarder. Add a function ARN filter arn:aws:lambda:us-east-2:425362996713:function:yiming-durable-py-custom-tracer
• Invoke the function yiming-durable-py-custom-tracer

Result

After a few minutes, the logs appeared in Datadog.

Raw logs, without the new logs pipeline on Datadog side for transforming the durable execution event logs: (query)

Processed logs (with the logs pipeline added in https://github.com/DataDog/integrations-internal-core/pull/3389): (query)

[datadog-system-tests-org]

📊 dbt CICD (full report)

Impact Lineage

⚠️ Analysis failed. An error occurred while computing the impact lineage for this PR.

Error details

Error fetching impact lineage results.

Drift Detection

⚠️ Drift detection failed. An error occurred while computing prod-vs-CI tests for this PR.

Error details

Error fetching drift detection results.

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 812c224 | Docs | Give us feedback!}

[Copilot]

Pull request overview

Adds a new, customer-installable CloudFormation template (plus documentation) to capture AWS Lambda Durable Function execution status change events from EventBridge and forward them to Datadog via Kinesis Data Firehose, with failed-record backup to S3.

Changes:

• Introduces aws_durable_function_event_forwarder/durable_function_event_forwarder.yaml to provision EventBridge rule, Firehose delivery stream, IAM roles, and S3 backup bucket.
• Adds aws_durable_function_event_forwarder/README.md explaining architecture, parameters, outputs, and the raw forwarded event shape.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
aws_durable_function_event_forwarder/durable_function_event_forwarder.yaml	New CloudFormation template provisioning EventBridge → Firehose → Datadog forwarding with S3 failed-record backup.
aws_durable_function_event_forwarder/README.md	Documentation for deploying/configuring the template and understanding the forwarded payload.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.