Skip to content

fix(site_worker): remove watchmedo auto-restart causing BrokenPipeError loop#2429

Merged
Didayolo merged 2 commits into
developfrom
fix/site_worker
Jun 18, 2026
Merged

fix(site_worker): remove watchmedo auto-restart causing BrokenPipeError loop#2429
Didayolo merged 2 commits into
developfrom
fix/site_worker

Conversation

@Didayolo

@Didayolo Didayolo commented Jun 18, 2026

Copy link
Copy Markdown
Member

Description

Uploading a competition bundle leaves the "unpacking" step spinning forever. The unpack_competition Celery task is enqueued on the site-worker queue but never executed, because the site_worker container — although reported as Up by Docker — has no consumer attached to RabbitMQ.

Root cause

The previous site_worker command wrapped Celery in watchmedo so that task code edits would auto-reload:

command: ["watchmedo auto-restart -p '*.py' --recursive -- celery -A celery_config worker -B -Q site-worker -l info -n site-worker@%n --concurrency=2"]

watchmedo watches the entire mounted /app tree, which includes:

  • __pycache__/*.pyc files written by Python on every import
  • src/celerybeat-schedule* SQLite files written continuously by Celery beat itself

These files keep changing faster than Celery can finish initializing its prefork pool. Each aborted startup leaks pool worker processes and produces a BrokenPipeError in billiard. As a result no consumer ever attaches to the site-worker queue and tasks pile up indefinitely.

Fix

Remove the watchmedo wrapper from the site_worker command in docker-compose.yml:

   site_worker:
-    # This auto-reloads
-    command: ["watchmedo auto-restart -p '*.py' --recursive -- celery -A celery_config worker -B -Q site-worker -l info -n site-worker@%n --concurrency=2"]
+    command: ["celery -A celery_config worker -B -Q site-worker -l info -n site-worker@%n --concurrency=2"]
     working_dir: /app/src

Issue solved

Checklist

  • Code review by me
  • Hand tested by me
  • I'm proud of my work
  • Code review by reviewer
  • Hand tested by reviewer
  • CircleCi tests are passing
  • Ready to merge

AybH26 and others added 2 commits June 16, 2026 17:11
@Didayolo Didayolo self-assigned this Jun 18, 2026
@Didayolo Didayolo merged commit 58235d4 into develop Jun 18, 2026
1 check passed
@Didayolo Didayolo deleted the fix/site_worker branch June 18, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

site_worker stuck in restart loop — competition unpacking hangs forever

2 participants