Queue congestion (public queue)

[ihsaan-ullah]

We frequently notice that submissions get stuck in the queue. This happens with RabbitMQ when incoming messages rate is greater then outgoing messages.

How can we solve this issue?

• Can we have multiple Rabbit Queues (one for each Codabench Queue)
• Can we increase the rate limits so that we don't get this problem
• Can we have a UI to monitor the queue and interact with the queue?

[Didayolo]

TODO P1 development:

• If congestion in the queue detected (e.g. 10+ submissions in "Submitted"), add a warning and prevent new submissions.

The "Submitted" count depends on the queue of the competition (including for default queue).

Problem: sometimes it's stuck on "Submitted", even after service restart, so it may be considered congested indefinitely.

---

Suggestions:

• Similarly to Limit CPU usage on the default queue #1141, we could limit the "daily submissions limit" for competitions using the default queue
• Having more workers would be beneficial now that Codabench is widely adopted
• Should we optimize the queueing / ordering algorithm? Right now it is a simple first-in-first-out, so a single competition can block a whole queue.

---

Bug report: all queues blocked

Apparently, when one queue is congested, then all queues get stuck. @ihsaan-ullah Can we confirm the existence of this bug?

[Didayolo]

There is actually a bug, not simply overload of submissions.

When the bug happens, the workers stopped being able to communicate their status with the main server because of a 500 error.

@ObadaS Do you have some logs to complete this diagnosis?

[Didayolo]

On going: fix server time (5 minutes late)

To check:

timedatectl status

Set time:

sudo ntpdate ntp.u-psud.fr

[ObadaS]

[2024-05-28 07:52:15,862: INFO/ForkPoolWorker-155] Updating submission @ https://www.codabench.org/api/submissions/65174/ with data = {'status': 'Preparing', 'status_details': None, 'secret': ''}
[2024-05-28 07:52:16,003: INFO/ForkPoolWorker-155] Submission patch failed with status = 500, and response = 
b'<h1>Server Error (500)</h1>'
[2024-05-28 07:52:16,003: INFO/ForkPoolWorker-155] Updating submission @ https://www.codabench.org/api/submissions/65174/ with data = {'status': 'Failed', 'status_details': 'Failure updating submission data.', 'secret': ''}
[2024-05-28 07:52:16,116: INFO/ForkPoolWorker-155] Submission patch failed with status = 500, and response = 
b'<h1>Server Error (500)</h1>'
[2024-05-28 07:52:16,116: INFO/ForkPoolWorker-155] Destroying submission temp dir: /codabench/tmpuz0or1fz
[2024-05-28 07:52:16,122: ERROR/ForkPoolWorker-155] Task compute_worker_run[...] raised unexpected: SubmissionException('Failure updating submission data.')
Traceback (most recent call last):
  File "/compute_worker.py", line 115, in run_wrapper
    run.prepare()
  File "/compute_worker.py", line 751, in prepare
    self._update_status(STATUS_PREPARING)
  File "/compute_worker.py", line 359, in _update_status
    self._update_submission(data)
  File "/compute_worker.py", line 342, in _update_submission
    raise SubmissionException("Failure updating submission data.")
compute_worker.SubmissionException: Failure updating submission data.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 650, in __protected_call__
    return self.run(*args, **kwargs)
  File "/compute_worker.py", line 123, in run_wrapper
    run._update_status(STATUS_FAILED, str(e))
  File "/compute_worker.py", line 359, in _update_status
    self._update_submission(data)
  File "/compute_worker.py", line 342, in _update_submission
    raise SubmissionException("Failure updating submission data.")
compute_worker.SubmissionException: Failure updating submission data.

I removed the secret in the logs, so them being empty is normal.
Also removed the id that was in the [] after compute_worker_run

[Didayolo]

Conjointly, users reported that their custom workers stopped receiving submissions. It worked again after we cured the production server (#1455).