Skip to content

Flush export queue when it reaches max_export_batch_size #1521

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
codeboten merged 2 commits into from
Feb 12, 2021
Merged

Flush export queue when it reaches max_export_batch_size #1521

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2b293d8
Flush export queue when it reaches `max_export_batch_size`
anton-ryzhov Jan 11, 2021
f1ac8a9
Merge branch 'main' into export-earlier
Feb 11, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
([#1507](https://github.com/open-telemetry/opentelemetry-python/pull/1507))
- `opentelemetry-exporter-jaeger` Updated Jaeger exporter status code tag
([#1488](https://github.com/open-telemetry/opentelemetry-python/pull/1488))
- `BatchExportSpanProcessor` flushes export queue when it reaches `max_export_batch_size`

## [0.16b1](https://github.com/open-telemetry/opentelemetry-python/releases/tag/v0.16b1) - 2020-11-26
### Added
Expand Down
2 changes: 1 addition & 1 deletion opentelemetry-sdk/src/opentelemetry/sdk/trace/export/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ def on_end(self, span: Span) -> None:

self.queue.appendleft(span)

if len(self.queue) >= self.max_queue_size // 2:
if len(self.queue) >= self.max_export_batch_size:
Copy link
Copy Markdown
Member

@toumorokoshi toumorokoshi Jan 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the change! I would wonder if it would be better to modify the exporter to flush the queue at something less than the max batch size?

I think this code was written to notify a flush well before the queue filled to handle race conditions. I haven't looked at the code super closely recently, but IIRC there is a possibility for multiple spans to end at the same time, and as a result:

  • first span would notify the flush
  • second span would be dropped, since the flush is notified, but the export_batch_size has been reached.

So maybe take something a little more cautious, like filling up to 80%, and have be the condition for both the notify and the flush itself (refactoring both to a single function call would prevent future splitting of the code).

Copy link
Copy Markdown
Contributor Author

@anton-ryzhov anton-ryzhov Jan 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Queue overflow and spans drop — that's exactly the issue I'm addressing here.

There are 2 different numbers — max_queue_size = 2048 and max_export_batch_size = 512.

Don't think it's a good idea to flush half- (or 80%-) full requests. That will only make effective_export_batch_size = 0.8 * max_export_batch_size.

Second span in your example won't be dropped if max_queue_size > max_export_batch_size, which should be true.

Maybe as extra check the code should validate that max_queue_size > 2 * max_export_batch_size or so.

Copy link
Copy Markdown
Member

@toumorokoshi toumorokoshi Jan 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it! Makes sense. Sorry about that, I misread the code the first time.

I think the checks there are there are already good (albeit maybe require more documentation).

with self.condition:
self.condition.notify()

Expand Down