Storage and clean up

[Didayolo]

Implementing a storage quota and effectively managing capacity could be crucial for enhancing Codabench's scalability and ensuring long-term service availability in production.

Features

Deleting files from the MinIO

@OhMaley: Add a button, in admin analytics page, to manually starts a script removing all orphans files.

• Nothing is never erased from the MinIO. Add a button to remove all orphan files manually ([Feature] Delete orphan files #1523, [Feature] Delete orphan files #1575)
• To check: Delete files  #1319
• More orphans files to remove: Cleaning data from deleted competitions #2059 (code included in the comments)

Quota

We need to:

• Impose a storage quota to each user (e.g. 15GB, platform setting) Quota #1254
• Idea: separated quota for each joined competition
• Admins should be able to manually increase the quota of any user (on Django admin interface) Quota #1254
• Have an user interface to manage and delete submissions
  Solved by: Resource interface : Cleanup and Quota #918
• Actually, submissions can't be removed if there are part of a benchmark (which makes sense to avoid spamming submissions when there is a daily limit).
• Solution: submissions are counted on the quota of the participants, but they can remove a submission (even if it's part of a benchmark) if it's not on the leaderboard. Submission soft-delete functionality added #1738
• Be able to delete tasks when deleting a benchmark Task deletion #810
  Partially Solved by: Resource interface : Cleanup and Quota #918
• Be able to re-use past submissions Re-using resources to save time and storage space #632
• Failed submission when deleted still remains as Data (deleted from leaderboard/submission panel but can be found in submissions in resources)
• The dumps should be counted too
• Refresh the quota figure automatically when submisison/dataset is uploaded in resource interface
• Readable format for the quota in the user table in admin interface .e.g MB

Light/archive mode after competition end

Option to clean competitions when there are completed. It would for instance keep only leaderboard submissions.

Automatic cleaning

• Failed submissions, orphans datasets, useless benchmarks, etc. should be marked and cleaned automatically. Users could manually unmark the object to keep it (from the resources interface). Deleted object should be kept in a trash can for one month. This feature needs to be discussed.
• Clean up old failed uploads #302 (schedule)

Monitoring and statistics

• Fix analytics page download CSV buttons: Analytics display and download csv #649
  Solved by: analytics download fixed #988
• Improve analytics page to include storage statistics. Migrate analytics features from CodaLab.
  Solved by: Feature/storage/monitoring tool #1248
• date of storage calculation like codalab
  Solved by: Analytics snapshot date #1265

Per benchmark submission size limit

Migrate the CodaLab feature on Codabench. This may be less important once the quota feature is implemented.

• Limit the size of submissions
• Django admin interface to define the limit independantly for each benchmark

Interface

@ihsaan-ullah

Improve resources interface

The interface for managing datasets, submissions and tasks could be improved. Indeed, improving the resources interface is also part of helping participants to manage their storage space.

• Add the size of the files
  Solved by File size added in readable format in Resources -> Datasets #895
• Submission details: in which benchmark
  Solved by Show competition name and file name in Resources -> Datasets #896
• make benchmark name clickable link
  Solved by Show competition name and file name in Resources -> Datasets #896
• Submission: name it using the filename
  Solved by Show competition name and file name in Resources -> Datasets #896
• Rename datasets tab in "Datasets and programs" Rename "Datasets" into "Datasets and programs" #840,
  Solved by Rename "Datasets" tab #898
• Have a separate tab for submissions Submissions in "datasets" #596 (and show the title given by the user for datasets and programs, and other consideration in the last comment of Show competition name and file name in Resources -> Datasets #896)
  Solved by Separate Tab for submissions in Resources #899
• Public datasets should have their own tab or a "show/hide" button (see Deletion of someone else' datasets #758)
  Solved by Add option to show/hide public submissions/datasets #906
• Public datasets should not have delete button if they do not belong to us (see Deletion of someone else' datasets #758)
  Solved by Add option to show/hide public submissions/datasets #906
• submission detail, make created_by a clickable link to user profile
  Solved by Created By is now a clickable link to user profile #910
• The search bar for the tab "submissions" should search by filename
  Solved by Search submissions in resource tab by file name and competition name #917
• We need more filtering options. For instance, in the tab "submissions", we should be able to filter a specific competition.
  Solved by Search submissions in resource tab by file name and competition name #917
• UI for quota management and cleanup (A "progress bar" for the storage quota)
  Solved by Resource interface : Cleanup and Quota #918
• Delete unused tasks
  Solved by Resource interface : Cleanup and Quota #918
• Delete unused datasets and programs
  Solved by Resource interface : Cleanup and Quota #918
• Delete unused submissions
  Solved by Resource interface : Cleanup and Quota #918
• Delete failed submissions
  Solved by Resource interface : Cleanup and Quota #918
• Being able to order rows by clicking on the column name (e.g. size)
  Solved by Resource interface : Orderding/Sorting tables #919

As shown in the screenshot below, we can manage and remove submissions from the "resources" interface. However, we do not track what the benchmark on which the submission was made, so it is hard to know what we are removing.

Some issues

• celery beat schedule to clean up orphaned datasets #145
• Submission status cleanup not functioning properly #637
• Deleting database models deletes dataset from storage #552
• Unlimited dumps creation #862 (impose a limit on dumps for each benchmark)
• Add 'signals' or some such to delete items from S3 when deleted from db #163

[ihsaan-ullah]

@Didayolo

Public datasets should have their own tab

Is this only for public datasets (now datasets and programs) or also for submissions

Right now we have these tabs:

1. Submissions (shows my submissions and public submissions)
2. Datasets and Programs (shows my datasets and public datasets) Note: datasets are everything except submissions
3. Tasks

[Didayolo]

@ihsaan-ullah

I think my idea was about public datasets that you don't own. It should be either clearly indicated, either on a separated tab.

EDIT: oh and the problem is that it could become cluttered by public datasets in the future when we have many users. As submissions can also be made public, maybe different tabs is not the good way to go. I see that in the "Tasks" tab, there is a "Show public tasks" option. Maybe we need the same option for datasets and submissions, so we can show/hide the public ones.

[Didayolo]

Side remark on the resources interface, usernames could be clickable link to the profiles (when clicking on details in any tab):

In the leaderboard, the clickable link is made in this way:

<a href="{submission.slug_url}">{ submission.owner }</a>

https://github.com/codalab/codabench/blob/ce98a1f49c9233119a2997721113a668f29f60af/src/static/riot/competitions/detail/leaderboards.tag#LL59C57-L59C113

[ihsaan-ullah]

Side remark on the resources interface, usernames could be clickable link to the profiles (when clicking on details in any tab):

Added this as one todo in the interface todos list

[Didayolo]

I added the same TODO in the "job status" issue #744