Submissions with exit code != 0 are marked as completed instead of failed

[ndido98]

I am hosting a private competition on Codabench (https://www.codabench.org/competitions/4188/) and we are running into an issue when the participants’ submissions fail the scoring step.

In particular, when the scoring program fails (mostly because due to an error from the participant in formatting the submission package), the submission should be marked as failed and not count towards the submission limit per day/person. However, it looks like the exit code of the scoring program is not checked and all submissions are marked as finished instead.

[Didayolo]

Hi @ndido98, how does your scoring program handle the formatting error?

To trigger the submission failure, the scoring program should raise an error.

[ndido98]

Hi @Didayolo, currently the scoring program just raises an exception, and as a consequence it returns an exit code of 1. The scores.txt file is created but it's empty.

[Didayolo]

Indeed, I am able to see the scoring error logs. I'm surprised the submissions are marked as finished.

I'm actually not sure what triggers the "Failed" status. Maybe it is because the scores.txt file exist? This requires a little investigation.

[ndido98]

From what I gathered by taking a look at the source code I found that the exit code of the scoring program is never checked:

codabench/compute_worker/compute_worker.py

Lines 878 to 881 in 7cf7a6e

	if self.is_scoring:
	self._update_status(STATUS_FINISHED)
	else:
	self._update_status(STATUS_SCORING)

Probably a submission should be marked as failed if either ingestion (if present) or scoring programs have a non-zero exit code.
The reason it fails when no scores.txt file is present is because this exception is triggered:

codabench/compute_worker/compute_worker.py

Line 903 in 7cf7a6e

raise SubmissionException("Could not find scores file, did the scoring program output it?")

Not outputting a scores.txt file (thus triggering the exception) could be a workaround for the time being, but still I believe that checking the exit code of the process should be preferred.

[Didayolo]

Interesting, that may explain the "Submission stuck in Scoring status" when compute worker crashes problem, reported in #1205.

[Didayolo]

I close this issue to keep track of it only in #1205