add aggregator #841
Merged
andrewqian2001datadog merged 25 commits intoadd-client-side-aggregationfrom Jul 16, 2024
Merged
Conversation
Merged
6 tasks
andrewqian2001datadog
added
the
changelog/Added
rayz
reviewed
Jul 11, 2024
rayz
reviewed
Jul 11, 2024
gh123man
reviewed
Jul 12, 2024
This comment was marked as resolved.
This comment was marked as resolved.
rayz
reviewed
Jul 16, 2024
gh123man
reviewed
Jul 16, 2024
6 tasks
andrewqian2001datadog
added a commit
that referenced
this pull request
Aug 12, 2024
* add aggregator (#841) * add aggregator WIP * add metric types enum * add threading to metrics.py, add unsafe_flush * fix aggregator * add tests * remove print statements * remove lock for flush unsafe function * fix removing wrong lock * remove unecessary If * Run tests on any branch * Change MetricAggregator propreties so that it has all necessary fields to send to client * change aggregator to use MetricAggregator type * fix test * fix lint * remove enums * fix lint x2 * fix lint x3 * use existing threading function * remove import * fix lint * refactor _stop_flush_thread * rename variable * remove client from aggregator * move changes to base.py to next PR --------- Co-authored-by: Brian Floersch <brian.floersch@datadoghq.com> * add flush_function parameter * refactor base.py to use the aggregator * revert test.yml change * WIP * disable buffering when aggregation is enabled * add flush_aggregated_metrics function * check whether or not _start_flush_thread is being called for buffering or aggregation before returning * WIP * finish disabling aggregation functions * move line inside if statement * fix lint * fix lint 2 * fix lint * fix renamed variable * fix unit tests, rename variables * check if thread is None * fix tests * fix tests * rerun tests * add back unicode * revert lint to forking * reverse lint changes * add aggregation to increment * TESTING * revert name back to flush_interval * test * change setter name * fix lint * change order of disable aggregation and buffering * test * change order of disable aggregation and buffering * remove setter design * set default aggregation time interval * fix lint * remove function call * fix lint * do not allow buffering when aggregation is enabled * testing for benchmarks * check if removing logs fixes test * add back logs * remove logs * rerun tests * set default aggregation flush interval to 2 seconds * remove print statement * address comments * use a single flush thread * remove unecessary lint changes, fix refactored code * add back import * fix refactored code * remove lint and comment * check lint error * fix variable naming, deleted test function * add back imports for comment variables * fix lint * test add disable_aggregating to test_fork_hooks * add parameter * revert changes * test * check for tags = None * remove test * fix comment * rename variable * add flush function --------- Co-authored-by: Brian Floersch <brian.floersch@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds the aggregator functionality for client side aggregation, it is pushing to a feature branch. This has already been implemented for the GO client.
Issue
Description of the Change
Adds a class that periodically aggregates (this is not implemented yet as it will rely on functionality from the base.py class) metrics for a given
flush_intervaland then sends the aggregated metrics to intake. A metric will be aggregated if there was another matching context sent within the same flush interval.Possible Drawbacks
A possible drawback is that the current code relies on object oriented programming (inheritance) for the
add_metricand other functions, it requires that theMetricAggregatorobjects will have the same fieldsVerification Process
Unit tests that verify the aggregator.py class matches the existing behavior in the client side aggregator for the go client.
Release Notes
No behavioral changes until the
base.pyclass is changedReview checklist (to be filled by reviewers)
changelog/label attached. If applicable it should have thebackward-incompatiblelabel attached.do-not-merge/label attached.kind/andseverity/labels attached at least.