Skip to content

fix(threadstats): ensure MetricsAggregator is threadsafe#370

Merged
zippolyte merged 1 commit intoDataDog:masterfrom
TheKevJames:threadsafe-metrics-aggregator
Jun 4, 2019
Merged

fix(threadstats): ensure MetricsAggregator is threadsafe#370
zippolyte merged 1 commit intoDataDog:masterfrom
TheKevJames:threadsafe-metrics-aggregator

Conversation

@TheKevJames
Copy link
Copy Markdown
Contributor

As described in #142, the MetricsAggregator is not threadsafe under Python 3. This is a regression caused by a change in how the .keys() method works under Python 2 vs Python 3: in Python 3, calling .keys() returns a view rather than a copy of a list.

The solution here is to use a reentry lock around the modifiers and accesses to self._metrics, in order to ensure that any concurrent calls to MetricsAggregator.flush() and MetricsAggregator.add_point() have no race conditions.

Note that the stack trace which this changeset fixes is:

RuntimeError: dictionary changed size during iteration
  File "datadog/threadstats/base.py", line 307, in flush
    metrics, dists = self._get_aggregate_metrics_and_dists(timestamp or time())
  File "datadog/threadstats/base.py", line 348, in _get_aggregate_metrics_and_dists
    rolled_up_metrics = self._metric_aggregator.flush(flush_time)
  File "datadog/threadstats/metrics.py", line 173, in flush
    past_intervals = [i for i in self._metrics.keys() if i < interval]
  File "datadog/threadstats/metrics.py", line 173, in <listcomp>
    past_intervals = [i for i in self._metrics.keys() if i < interval]

Copy link
Copy Markdown
Contributor

@zippolyte zippolyte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM, thanks a lot for this @TheKevJames

It should fix #365

@zippolyte zippolyte merged commit e7364bd into DataDog:master Jun 4, 2019
@edkorthof
Copy link
Copy Markdown

Thanks for the fix. But to clear -- this wasn't just unsafe in python3; in python2 it was also unsafe (but less likely to trigger); I provided a script which allows reproduction in python2.7.

dabcoder pushed a commit to dabcoder/datadogpy that referenced this pull request Oct 25, 2019
dabcoder pushed a commit to dabcoder/datadogpy that referenced this pull request Nov 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants