Problem with BEA 2019 Shared Task submissions

[mfelice]

Hi there,

I know I'm not the only one facing this problem, but I couldn't find a solution in previous threads.

I recently created two competitions:

• https://www.codabench.org/competitions/10318/ (migrated "as is" from Codalab)
• https://www.codabench.org/competitions/10319/ (same bundle, with scripts updated to work with python3 and an updated docker image)

In both cases, submissions are stuck at "Preparing submission... this may take a few moments..". Logs are completely empty.

Could you help?

Thanks.

[ObadaS]

Hello,
I see that your submissions did pass:

Seems like they failed because they exceeded the time limit of 600 seconds.
Can you try submitting something else ?
Please note that "Preparing submission... this may take a few moments.." means that the worker is downloading the image and the submission, which might take some time. The status should refresh by itself after the download is finished.

[mfelice]

Hi there,

I tried submitting a different file, and the same thing happened. It times out after 600 seconds. I'm uploading a zip file that contains a text file inside, as in Codalab. Has that changed? I believe I should be seeing some errors in the log if the evaluation script failed, but there are no messages at all. It's as if the script never gets to run.

Could there be an issue with the docker images? The one in competition 10318 is exactly the same as the one in Codalab (mfelice/bea2019st:v1.0 - 1.28GB compressed), while the one in competition 10319 is new (mfelice/bea2019st:v3.0 - 408.29 MB compressed). At least, I would expect 10318 to work - it's an exact copy of what's running in Codalab.

Can you see anything on your end? Any suggestions?

Thanks.

[ObadaS]

I had some time to look at this problem, and I think I know what is happening.

It seems like the worker is not finding the correct files to execute.

@Didayolo Any ideas on if there is anything to change in the competition itself to make it work ?

[bethard]

What is a reasonable amount of time to wait for "a few moments"? I'm having a similar problem at https://www.codabench.org/competitions/10558/ and a run there has been stuck in "Preparing submission..." state for 4 hours.

[Didayolo]

Hi @bethard,

We had a problem on our side that is now fixed. Can you confirm if the issue is still happening or not for this competition?

Thanks.

[mfelice]

Thanks for looking into this, @Didayolo. Unfortunately, the issue is still happening for competitions 10318 and 10319.

[bethard]

The problem seems to be resolved for competition 10558. Thanks @Didayolo !

[mfelice]

@Didayolo @ObadaS Could you help resolve the original issue please? Thanks.

[bruAristimunha]

the issue is happened at the eeg 2025 competition too @Didayolo and @ObadaS

[Didayolo]

Hi @mfelice,

Sorry, I realized that your problem is around for quite some time. Let's try to sort it out.

The logs suggest a path problem (EDIT: no, it's results submission, so it's normal to not find ingestion):

/codabench/[...]/ingestion_program not found, no program to execute 

I am going to download the bundle and investigate the issue.

In the meantime, did you make any progress on your side?

@bruAristimunha
If you are facing a bug on your competition, please open a github issue with as much details and logs as possible so we can provide you help.

[Didayolo]

Hi @mfelice,

I figured out the problem.

You need to zip your files (at least reference data, but maybe scoring program too) directly, without directory structure:

reference-data.zip
|  codalab.test.m2
|  metadata

Instead of:

reference-data.zip
|  reference-test
|   |  codalab.test.m2
|   |  metadata

I uploaded new version of the scoring program and reference data directly on your account, and edited the "Open Phase" task of the competition 10319.

Now the scoring program runs, and an error occurs under 5 minutes:

Traceback (most recent call last):
  File "/app/program/evaluate.py", line 70, in <module>
    raise ValueError("There are multiple files in your submission. Please submit only one file.")
ValueError: There are multiple files in your submission. Please submit only one file.

Looks like a normal functionning of the program and easy to resolve now.

[Didayolo]

The there are multiple files in your submission seems to be due to a metadata file that appears in the zip passed to the scoring program. Because the submission actually has only one file initially.

This may be due to this: #1874

Not sure what is the best solution here, maybe simply update the scoring program to not be disturbed by this.

[mfelice]

Hi @Didayolo,

Thanks for looking into this. I am surprised to learn that I should change the structure of the original bundle. How to transition from CodaLab to Codabench? says:

(...) competition bundles are back-compatible. This means that you can upload your CodaLab competition bundles into Codabench without any modifications and have them working just fine.

So I assumed that downloading the original competition bundle and re-uploading it to Codabench would work straight away. I see that's not the case.

Regarding your comment about the metadata file: that was explicitly required in Codalab -- see User_Building a Scoring Program for a Competition. Anyway, I followed your advice and deleted the metadata file from the reference data, but kept it for the scoring program, as it seems necessary to specify the command that needs to be run. I've set up a new competition with all fixes here: #10960.

After debugging for hours, it seems that the metadata file that is sneaking into the input directory (the res directory, to be precise) is a metadata file generated by Codabench. I assume this is the equivalent of the metadata file generated by Codalab with information about the submission (see Metadata passed to scoring program). On Codalab, this file was in the input directory, but now it's been moved to input/res, where the submission files reside, hence the message triggered by my code. This became clear after my script also failed to read the metadata file from its previous location later on when building a "detailed results" report. This also revealed a new issue: the new metadata file does not contain the same information as it did in Codalab, so now I'm unable to generate the same reports.

This is what the metadata file had on Codalab (from Metadata passed to scoring program):

ref: competition/2/2/data/competition/2/2/data/reference_7.zip
res: competition/2/submission/1/ebbf8284-6aeb-4a5b-b200-54e28534d769.zip
history: competition/2/8/submissions/1/2/history.txt
submitted-by: codalab
submitted-at: 2014-07-18T00:09:55+00:00
competition-submission: 2
competition-phase: 2
automatic-submission: False

This is what it has now on Codabench:

elapsedTime: null
exitCode: null
ingestionElapsedTime: null
ingestionExitCode: null

So where do I find all the previous information about a submission now?

Anyway, even after fixing these things, my submissions still hang on "Running" most of the time, and sometimes they never time out. It is becoming very difficult for me to debug without seeing any errors or warnings from Codabench. I can print debugging information in my scoring script, but sometimes it isn't even executed by Codabench. It all seems hit and miss. Not sure what else I can do from my end. Any advice will be really appreciated.

Thanks.

[Didayolo]

Hi @mfelice,

Sorry that you are encoutering so many difficulties. I am also surprised that this was the actual CodaLab structure, and that it worked on CodaLab but not on Codabench. From my experience, porting challenges from CodaLab to Codabench is farily straightforward.

As it is a simple results submission competition, we should be able to sort it out.

I am not aware of metadata file being created by Codabench, only of submissions files getting copied to the "prediction output".

EDIT: OK, it looks like, in CodaLab, there were a metadata file passed from the system to the scoring program. We still have this file but it shows different fields. Competitions using this in their scoring program therefore have retro-compatibility problems.

Anyway, even after fixing these things, my submissions still hang on "Running" most of the time, and sometimes they never time out.

This is probably due to this annoying bug that I am currently working on:
#2030

If it's the case, it means that the submissions are stuck to "Running" while they are in reality failed.

---

EDIT 2: Strangely, I can see the elapsedTime metadata field being called in a CodaLab scoring program:
https://github.com/codalab/competition-examples/blob/master/codalab/Iris/iris_competition_bundle/scoring_program/score.py

Even if this is supposedely a Codabench metadata filed. This is a bit confusing. I'll investigate this a bit more.

---

Sorry again for the inconvenience. Competition organizers rarely face so many bugs to setup / port a competition.