Migration with "Force Last" setting gives n/a scores

[ktgiahieu]

Hi,

I want to report a bug where the scores on the leaderboard returns n/a on the Final Phase of this competition: https://www.codabench.org/competitions/1285.

Although the chosen submissions are run successfully:

[ihsaan-ullah]

Can you please share more details about:

• how you got it
• steps to reproduce the bug
• competition bundle and sample submission

[Didayolo]

@ktgiahieu Have you deleted this benchmark?

[ktgiahieu]

@Didayolo My apology. I made the competition Private by accident. It should work now.

[Didayolo]

My bad, I thought that as an admin I should be able to see it even if it's private

[ihsaan-ullah]

@ktgiahieu can you upload the following from the failing submission here

[ktgiahieu]

@ihsaan-ullah Sorry for the confusion. The chosen submissions are all run successfully and produces the scores.json file, and have their scores available in the "My Submissions" tab. The error is that the leaderboard shows n/a even though the chosen submissions are run successfully.

Here are the files you asked for my chosen submission:
scoring_result.zip
prediction_result.zip

[bethard]

I'm seeing the same problem with "Force Best" on this competition: https://www.codabench.org/competitions/1481/#/results-tab.

If I switch the leaderboard from "Force Best" to "Add" and manually add submissions to the leaderboard, I get real scores instead of the "n/a"s.

FWIW, out of curiosity, I tried Force_Latest_Multiple, and with that the "n/a"s only show up on the one ranked 1st. Rank 2nd and later seem to be rendered okay.

[Didayolo]

I encountered a problem while testing auto-migration on the test server.

Here is the test competition:
https://codabench-test.lri.fr/competitions/143

Here is what I did:

• Make several submissions in development phase from 2 users ("pavao" and "testos")
• Have only 2 submissions on the leaderboard (1 for each user)
• Select "auto-migration" option, and wait for the final phase to start

When the final phase started, the two submissions got run, so the change worked (only leaderboard submissions got run).

In the logs, I can find the scores and everything, so the submissions worked well:

However, on the final phase leaderboard, the scores are N/A and the detailed results lead to 404 error:

[Didayolo]

I close this issue to merge and keep track of it here: #1540.