Disable flushing metrics in thread

[tianchu]

What does this PR do?

Disable flushing_in_thread of threadstats by default. Previously, for synchronous metric submission (without piggybacking on the forwarder and logs), the threadstats is configured to flush the metrics both at the end of the invocation (blocking) and every 10s in a background thread (non-blocking). This PR disables the periodic background flushing.

This change is a soft breaking change, though it shouldn't result in any data loss, it does delay synchronous metric submission for long-running Lambda functions (duration > 10s). In the worst case, metrics submitted at the beginning of an invocation could be delayed for showing up in Datadog for 15 min (the max Lambda duration allowed).

For affected functions, the behavior could be re-enabled by setting env var DD_FLUSH_IN_THREAD to true. However, given the problematic nature of this flushing behavior, I would recommend submitting metrics through the new Datadog Lambda extension instead.

Datadog Forwarder relies on the datadog-lambda to flush metrics, so the datadog-lambda version needs to be bumped to get the issue fixed for the Datadog Forwarder DataDog/datadog-serverless-functions#436

Also fixed the broken and flaky integration tests.

Motivation

A bug has been recently identified (see sample logs below). If there happen to be a background flushing in progress, the end-of-invocation flushing will be skipped (because there is already an in-flight flushing). However, if the background flushing doesn't finish before the AWS freezing the execution environment, it will resume at the beginning of the next invocation. There are two problems with it. First, if AWS decides to destroy the current execution environment, this request will never complete. Second, if the next invocation only happens a few min later (e.g., a cronjob), the request will fail due to connection timeout.

In short, the background thread flushing isn't suitable for Lambda and should be disabled.

# current invocation
Flushing metrics in thread
Flush #8 sending 640 distributions
A flush is already in progress. Skipping this one.
datadog_lambda_wrapper _after() done
REPORT RequestId: d0deabde-936b-467c-aba6-7d4cebdf25c0	Duration: 843.65 ms	Billed Duration: 844 ms	Memory Size: 1024 MB	Max Memory Used: 146 MB	
END RequestId: d0deabde-936b-467c-aba6-7d4cebdf25c0

# next invocation (5 mins later)
START RequestId: faa32e6f-8cd4-4384-aac1-2a5698344e54 Version: $LATEST
Could not request POST https://api.datadoghq.com/api/v1/distribution_points: ('Connection aborted.', timeout('The write operation timed out')). Please check the network connection or try again later.

Testing Guidelines

Additional Notes

Types of Changes

• Bug fix
• New feature
• Breaking change
• Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

• This PR's description is comprehensive
• This PR contains breaking changes that are documented in the description
• This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
• This PR impacts documentation, and it has been updated (or a ticket has been logged)
• This PR's changes are covered by the automated tests
• This PR collects user input/sensitive content into Datadog
• This PR passes the integration tests (ask a Datadog member to run the tests)

[nhinsch]

Looks good, I would just suggest we de-emphasize the setting more, either be removing it completely, renaming it, and/or hiding it, to avoid confusion, since it is discouraged/potentially broken to flush metrics in the background.