Enable compressed payloads on ThreadStats object

[csssuf]

What does this PR do?

Enable submitting compressed payloads to the Datadog API.

Motivation

We use lambdas to scrape metrics which we can't easily collect by other means. These metric lambdas can collect large numbers of metrics quickly, and occasionally submit too many metrics within the same flush interval for the Datadog API to handle. This is similar to the issue in DataDog/datadogpy#465. The solution implemented to fix that issue was compression of payloads submitted to the DD API.

Testing Guidelines

./scripts/run_tests.sh

Additional Notes

n/a

Types of Changes

• Bug fix
• New feature
• Breaking change
• Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

• This PR's description is comprehensive
• This PR contains breaking changes that are documented in the description
• This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
• This PR impacts documentation, and it has been updated (or a ticket has been logged)
• This PR's changes are covered by the automated tests
• This PR collects user input/sensitive content into Datadog
• This PR passes the integration tests (ask a Datadog member to run the tests)

[tianchu]

Hi @csssuf, I'm closing the PR due to the lack of activity here. Feel free to reopen if this problem still persists and you would like to move forward. I also want to mention a new way (using the Lambda extension) to submit metrics directly from your Lambda functions (without using the forwarder) https://docs.datadoghq.com/serverless/datadog_lambda_library/extension/.

[jeid64]

Please reopen this issue, it has a lack of activity as the datadog maintainers didn't respond. @tianchu the PR is ready to merge.

[tianchu]

Thanks for the confirmation that this PR is still relevant! we will prioritize some testing before merging. Sorry for the long wait 🙇

[tianchu]

I could successfully reproduce the issue by submitting tons of data points within 10s, and enabling compress_payload did fix the issue. I'm going to merge this PR and release in the next version. Thanks for your contribution!

However, I do want to point out, although compress_payload does help workaround the issue, submitting a really large payload does add a noticeable overhead to the function duration. Not sure if this is the case, but if you are looping through a huge list and submitting a data point for each item, it would be way more efficient if you aggregate those data points in the memory and then submit one aggregated value at the end of the loop.

# inefficient
for i in range(1000000):
    lambda_metric(
        metric_name='my.metric',
        value=i+1,
        tags=['tag1:value1']
    )
    
# efficient
sum = 0
for i in range(1000000):
    sum = sum + i + 1
lambda_metric(
    metric_name='my.metric',
    value=sum,
    tags=['tag1:value1']
)

[csssuf]

🎉 thanks!

Not sure if this is the case, but if you are looping through a huge list and submitting a data point for each item, it would be way more efficient if you aggregate those data points in the memory and then submit one aggregated value at the end of the loop.

Unfortunately, since we're submitting several distinct metrics with a large number of unique tag sets for each metric, aggregating that way wouldn't do anything for us - we do only submit a single value per unique (metric, tagset) pair. These lambdas are also only used for scraping and submitting metrics, so a long runtime isn't a major concern for us.