Skip to content

Beat unique fix - #6

Merged
TkTech merged 9 commits into
mainfrom
beat_unique_fix
Oct 6, 2023
Merged

Beat unique fix#6
TkTech merged 9 commits into
mainfrom
beat_unique_fix

Conversation

@TkTech

@TkTech TkTech commented Oct 5, 2023

Copy link
Copy Markdown
Owner

It's possible for the celery beat scheduler to skip calling apply_async(), if it's unable to import the actual task. In these cases, it'll resort to using send_task(), see https://github.com/celery/celery/blob/main/celery/beat.py#L406. This means that our unique lock may never be checked, since we hook it into Task.apply_async().

This change adds verification of the lock at the time a task is run, while keeping the early lock as well.

This is a known issue with other similar libraries, see:

@TkTech
TkTech requested a review from dominicplouffe October 5, 2023 16:17
@TkTech TkTech self-assigned this Oct 5, 2023
@codecov

codecov Bot commented Oct 5, 2023

Copy link
Copy Markdown

Codecov Report

Attention: 15 lines in your changes are missing coverage. Please review.

Comparison is base (43727fa) 58.74% compared to head (fbbcc5a) 60.00%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main       #6      +/-   ##
==========================================
+ Coverage   58.74%   60.00%   +1.25%     
==========================================
  Files           7        7              
  Lines         286      295       +9     
==========================================
+ Hits          168      177       +9     
  Misses        118      118              
Files Coverage Δ
celery_heimdall/__init__.py 100.00% <100.00%> (ø)
celery_heimdall/config.py 100.00% <100.00%> (ø)
celery_heimdall/errors.py 100.00% <100.00%> (ø)
celery_heimdall/contrib/inspector/monitor.py 0.00% <0.00%> (ø)
celery_heimdall/contrib/inspector/cli.py 0.00% <0.00%> (ø)
celery_heimdall/task.py 93.79% <89.13%> (+0.41%) ⬆️
celery_heimdall/contrib/inspector/models.py 0.00% <0.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@TkTech
TkTech merged commit b6709f6 into main Oct 6, 2023
@jzelez

jzelez commented Jan 10, 2025

Copy link
Copy Markdown

Hey, first of all - awesome work on this package.

Secondly, I've noticed an issue with this approach - namely, when you call the decorated function directly, the lock is still acquired (while it shouldn't in this case) and it is then never released, because it doesn't use any of the task api calls/signals.

I've subclassed the HeimdallTask to handle this. Basically, bypassing the acquire lock behaviour if the task/function is called directly.

    def __call__(self, *args, **kwargs):
        if self.request.called_directly:
            return self.run(*args, **kwargs)

        return super().__call__(*args, **kwargs)

@TkTech

TkTech commented Jan 10, 2025

Copy link
Copy Markdown
Owner Author

Great catch! It may still be desirable to have the lock used when running directly, so we'll need to make this a configuration option and move cleanup into the call. We'll add this and a test for it in the refactor in #10.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants