diff --git a/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Django-3.md b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Django-3.md new file mode 100644 index 000000000..12452e02e --- /dev/null +++ b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Django-3.md @@ -0,0 +1,26 @@ +!!! note "After upgrading from Codabench <1.20, you will need to rebuild containers, run Django migrations and upgrade compute workers." + +1. Rebuild containers + +```sh +docker compose build +docker compose up -d +``` + +2. Run migrations and collect static + +```sh +docker compose exec django ./manage.py makemigrations +docker compose exec django ./manage.py migrate +docker compose exec django ./manage.py collectstatic --noinput +``` + +3. Upgrade compute workers + +For every compute worker associated to the instance (default queue) or to a custom queue, you need to update the worker: + +```sh + +docker compose down +docker compose up -d --pull always +``` diff --git a/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Django-4.md b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Django-4.md new file mode 100644 index 000000000..dd3cbfce6 --- /dev/null +++ b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Django-4.md @@ -0,0 +1,53 @@ +!!! note "After upgrading from Codabench <1.23.0, you will need to perform manual interventions." + + +## Django +The Django version upgrade will require all the containers to be rebuilt: +```sh +docker compose build --no-cache +docker compose up -d +``` + +Afterward, you will need to run some migrations: +```sh +docker compose exec django python manage.py migrate +``` +If it asks you to do a makemigration instead, you will have to run the following command: +```sh +docker compose exec django python manage.py makemigrations --merge +``` +You might need to modify `/app/src/apps/datasets/migrations/0014_merge_20251212_0942.py` to remove the `datagroup` changes +```python +# Generated by Django 4.2.23 on 2025-09-08 12:32 + +from django.db import migrations, models +import utils.data +import utils.storage + +class Migration(migrations.Migration): + + dependencies = [ + ("datasets", "0010_auto_20250218_1100"), + ] + + operations = [ + migrations.AlterField( + model_name="data", + name="data_file", + field=models.FileField( + blank=True, + null=True, + storage=utils.storage.PrivateStorageClass(), + upload_to=utils.data.PathWrapper("dataset"), + ), + ), + migrations.AlterField( + model_name="data", + name="id", + field=models.BigAutoField( + auto_created=True, primary_key=True, serialize=False, verbose_name="ID" + ), + ), + ] +``` +Notice that there are only two migrations in this file instead of 3. Simply delete the last one (it tries to change `datagroups` even after `datagroups` was deleted, which is why it fails) diff --git a/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Docker-py.md b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Docker-py.md new file mode 100644 index 000000000..da822bbde --- /dev/null +++ b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Docker-py.md @@ -0,0 +1,20 @@ +!!! note "After upgrading from Codabench <1.22.0, you will need to rebuild containers, run Django migrations and upgrade compute workers." + + +## Main Instance +Some of the changes will require a migration and `collectstatic` commands to be run: +```sh +docker compose build && docker compose up -d +docker compose exec django python manage.py migrate +docker compose exec django python manage.py collectstatic --no-input +``` + + +There is a new environment variable for the contact email: +``` +CONTACT_EMAIL=info@codabench.org +``` +Make sure to add it to your `.env` file before launching the containers + +## Compute Workers +Major compute workers changes will require updating the Compute Worker images for both Docker and Podman. Podman workers will also need Podman 5.4 minimum to work on the host diff --git a/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Hide-output.md b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Hide-output.md new file mode 100644 index 000000000..0e6b456f0 --- /dev/null +++ b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Hide-output.md @@ -0,0 +1,5 @@ +!!! note "After upgrading from Codabench <1.19, you will need to perform a Django migration ([#1838](https://github.com/codalab/codabench/pull/1838), [#1851](https://github.com/codalab/codabench/pull/1851))" + +```bash +docker compose exec django ./manage.py migrate +``` \ No newline at end of file diff --git a/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Minio-image.md b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Minio-image.md new file mode 100644 index 000000000..953a17435 --- /dev/null +++ b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/Minio-image.md @@ -0,0 +1,33 @@ +!!! note "After upgrading from Codabench <1.21.1, you will need to migrate MinIO, rebuild containers, run Django migrations and upgrade compute workers." + + +### 1. MinIO migration (depending on setup) +If you are running MinIO locally (defined by `.env` and `docker-compose.yml`), you may require to follow the following MinIO upgrade instructions, to duplicate and mirror your buckets and convert them to the new format: +https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-migrate-fs-gateway.html + +### 2. Rebuild all containers +```sh +docker compose build && docker compose up -d +``` + +### 3. Django migration + +```sh +docker compose exec django ./manage.py migrate +``` + +### 4. Collect static files + +```sh +docker compose exec django ./manage.py collectstatic --no-input +``` + +### 5. Upgrade compute workers + +For every compute worker associated to the instance (default queue) or to a custom queue, you need to update the worker: + +```sh + +docker compose down +docker compose up -d --pull always +``` diff --git a/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/index.md b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/index.md index aa9c8a658..926adb6cf 100644 --- a/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/index.md +++ b/documentation/docs/Developers_and_Administrators/Upgrading_Codabench/index.md @@ -22,4 +22,9 @@ You can find here various manual intervention needed depending on which version - [Submissions and Participants Counts](Submissions-and-Participants-Counts.md) (version < 1.14.0) - [Homepage counters](Homepage-counters.md) (version < 1.15.0) - [User removal](User-removal.md) (version < 1.17.0) -- [Database size Fix](Database-size-fixes.md) (version < 1.18.0) \ No newline at end of file +- [Database size Fix](Database-size-fixes.md) (version < 1.18.0) +- [Hide output](Hide-output.md) (version < 1.19.0) +- [Django 3.0](Django-3.md) (version < 1.20.0) +- [Minio image](Minio-image.md) (version < 1.21.1) +- [Docker-py](Docker-py.md) (version < 1.22.0) +- [Django 4.2](Django-4.md) (version < 1.23.0) diff --git a/documentation/docs/Organizers/Benchmark_Creation/Advanced-Tutorial.md b/documentation/docs/Organizers/Benchmark_Creation/Advanced-Tutorial.md index 2ed3d9227..2bcc8a4b7 100644 --- a/documentation/docs/Organizers/Benchmark_Creation/Advanced-Tutorial.md +++ b/documentation/docs/Organizers/Benchmark_Creation/Advanced-Tutorial.md @@ -110,26 +110,7 @@ After the bundle has been uploaded, you will see the screenshot shown below. ![image](../../_attachments/102434028-e182bb00-404e-11eb-9de8-002e4589db66_17528513099222646.png) - -## Benchmark Examples - -Example bundles for code & dataset competition can be found here: - -https://github.com/codalab/competition-examples/tree/master/codabench - -### Iris - -[Iris Codabench Bundle](https://github.com/codalab/competition-examples/tree/master/codabench/iris) is a simple benchmark involving two phases, code submission and results submission. - -### AutoWSL - -Two versions of the [Automated Weakly Supervised Learning Benchmark](https://github.com/codalab/competition-examples/tree/master/codabench/autowsl): -- [Code submission benchmark](https://github.com/codalab/competition-examples/tree/master/codabench/autowsl/code_submission) -- [Dataset submission benchmark](https://github.com/codalab/competition-examples/tree/master/codabench/autowsl/dataset_submission) - -### Mini-AutoML - -[Mini-AutoML Bundle](https://github.com/codalab/competition-examples/tree/master/codabench/mini-automl) is a benchmark template for Codabench, featuring code submission to multiple datasets (tasks). +!!! tip "You can find benchmark templates to start your own project in [Benchmark Examples](Benchmark-Examples.md) page." ## How do I set up submission comments for multiple submissions? diff --git a/documentation/docs/Organizers/Benchmark_Creation/Benchmark-Examples.md b/documentation/docs/Organizers/Benchmark_Creation/Benchmark-Examples.md new file mode 100644 index 000000000..6b8c9fe6c --- /dev/null +++ b/documentation/docs/Organizers/Benchmark_Creation/Benchmark-Examples.md @@ -0,0 +1,30 @@ +Example bundles can be found here: + +[https://github.com/codalab/competition-examples/tree/master/codabench](https://github.com/codalab/competition-examples/tree/master/codabench) + +More details are given below, to help you select the bundle that suits your own requirements. You can easily clone a template and customize it to match your project. + +### Iris + +[Iris Codabench Bundle](https://github.com/codalab/competition-examples/tree/master/codabench/iris) is a simple benchmark involving two phases, code submission and results submission. + +### Classify Wheat Seeds + +We propose three versions of the [Classify Wheat Seeds](https://github.com/codalab/competition-examples/tree/master/codabench/wheat_seeds): +- [Result submission bundle](https://github.com/codalab/competition-examples/tree/master/codabench/wheat_seeds/results_submission_bundle), with simple submission of predictions +- [Code submission bundle](https://github.com/codalab/competition-examples/tree/master/codabench/wheat_seeds/code_submission_bundle), with submission of Python algorithm +- [Ingestion during scoring bundle](https://github.com/codalab/competition-examples/tree/master/codabench/wheat_seeds/ingestion_during_scoring_bundle), where ingestion and scoring run in parallel + +### Mini-AutoML + +[Mini-AutoML Bundle](https://github.com/codalab/competition-examples/tree/master/codabench/mini-automl) is a benchmark template for Codabench, featuring code submission to multiple datasets (tasks). + +### AutoWSL + +You can find two versions of the [Automated Weakly Supervised Learning Benchmark](https://github.com/codalab/competition-examples/tree/master/codabench/autowsl): +- [Code submission benchmark](https://github.com/codalab/competition-examples/tree/master/codabench/autowsl/code_submission) +- [Dataset submission benchmark](https://github.com/codalab/competition-examples/tree/master/codabench/autowsl/dataset_submission) + +### GPU test + +[GPU test bundle](https://github.com/codalab/competition-examples/tree/master/codabench/example_GPU) is an example bundle to test if GPUs are available or not. It serves as testing compute workers and does **not** contain any problem to solve. \ No newline at end of file diff --git a/documentation/docs/Organizers/Benchmark_Creation/Competition-docker-image.md b/documentation/docs/Organizers/Benchmark_Creation/Competition-docker-image.md index a33dd244e..d34cc87f0 100644 --- a/documentation/docs/Organizers/Benchmark_Creation/Competition-docker-image.md +++ b/documentation/docs/Organizers/Benchmark_Creation/Competition-docker-image.md @@ -3,7 +3,7 @@ The competition docker image defines the docker environment in which the submiss ## Default competition docker image The default competition docker image is `codalab/codalab-legacy:py37`. -More information here: https://github.com/codalab/codalab-dockers +More information and base images are available here: https://github.com/codalab/codalab-dockers ## Set up another image @@ -14,23 +14,56 @@ You can select another docker image: - In the editor field "Competition Docker image" as shown in the following screenshot: ![](_attachments/b8124291-92ea-4d1d-b743-00fd7a35c313_17534366792118.jpg) -## Building an image - If the default image does not suit your needs (missing libraries, etc.), you can either: -- Select an existing image from DockerHub +- Select any existing image from DockerHub - Create your own image from scratch -- Create a custom image based on the Codalab image. (more information below) +- Create a custom image based on the CodaLab image (more information below) + +## Custom image based on CodaLab image If you wish to create a custom image based on the Codalab image, you can follow the steps below: -1) Install Docker -2) Sign up to DockerHub -3) `docker run -itd -u root codalab/codalab-legacy:py39 /bin/bash` -4) Use `docker ps` to find running container id -5) Now run `docker exec -it -u root bash` -6) Install anything you want at the docker container shell (`apt-get install`, `pip install`, etc.) -7) Exit the shell with `exit` -8) `docker commit username/image:tag` -9) `docker login` -10) `docker push username/image:tag` \ No newline at end of file +#### Preliminary steps + +1) Install Docker +2) Sign up to [DockerHub](https://hub.docker.com/) + +#### Method 1: update the image from a container + +1) Start a container using the base image: +```sh +docker run -itd -u root codalab/codalab-legacy:py39 /bin/bash +``` +2) Identify the running container ID using `docker ps` +3) Enter inside the container: +```sh +docker exec -it -u root bash +``` +4) Install anything you want at the docker container shell (`apt-get install`, `pip install`, etc.) +5) Exit the shell with `exit` +6) Push the new version to your DockerHub account: +```sh +docker commit username/image:tag +docker login +docker push username/image:tag +``` + +#### Method 2: update the Dockerfile and re-build + +1) Download the Dockerfile of the base image: +https://github.com/codalab/codalab-dockers/blob/master/legacy-py39/Dockerfile + +2) Edit the file to include any library or program you need + +3) Build the image +```sh +docker build -t username/image:tag . +``` + +4) Push it to your DockerHub account + +```sh +docker login +docker push username/image:tag +``` \ No newline at end of file diff --git a/documentation/docs/Organizers/Benchmark_Creation/Getting-started-with-Codabench.md b/documentation/docs/Organizers/Benchmark_Creation/Getting-started-with-Codabench.md index a0025ca21..181f87366 100644 --- a/documentation/docs/Organizers/Benchmark_Creation/Getting-started-with-Codabench.md +++ b/documentation/docs/Organizers/Benchmark_Creation/Getting-started-with-Codabench.md @@ -8,6 +8,8 @@ This getting started tutorial shows a **simple example** of how to create a comp - Download the [sample competition bundle](https://github.com/codalab/competition-examples/blob/master/codabench/wheat_seeds/code_submission_bundle.zip) and the [sample submission](https://github.com/codalab/competition-examples/blob/master/codabench/wheat_seeds/sample_code_submission.zip). - Do not unzip them. +!!! tip "You can find benchmark templates to start your own project in [Benchmark Examples](Benchmark-Examples.md) page." + ## Create a competition - From the front page of [Codabench](https://codabench.org), in the top menu, go to the **Benchmark > Management** - Click the green **Upload** button at the top right. diff --git a/documentation/mkdocs.yml b/documentation/mkdocs.yml index ef1b9f24e..b452cf2fb 100644 --- a/documentation/mkdocs.yml +++ b/documentation/mkdocs.yml @@ -117,12 +117,12 @@ nav: - Competition Creation: Organizers/Benchmark_Creation/Competition-Creation.md - Competition Creation Form: Organizers/Benchmark_Creation/Competition-Creation-Form.md - Competition Creation Bundle: Organizers/Benchmark_Creation/Competition-Creation-Bundle.md - - Competition YAML Structure: Organizers/Benchmark_Creation/Competition-Bundle-Structure.md + - Competition Bundle Structure: Organizers/Benchmark_Creation/Competition-Bundle-Structure.md - YAML Structure: Organizers/Benchmark_Creation/Yaml-Structure.md - Competition Docker Image: Organizers/Benchmark_Creation/Competition-docker-image.md - Dataset Competition Creation and participate instruction: Organizers/Benchmark_Creation/Dataset-competition-creation-and-participate-instruction.md - Leaderboard Features: Organizers/Benchmark_Creation/Leaderboard-Functionality.md - - Competition Examples: https://github.com/codalab/competition-examples/tree/master/codabench + - Competition Examples: Organizers/Benchmark_Creation/Benchmark-Examples.md - Public Tasks and Tasks Sharing: Organizers/Benchmark_Creation/Public-Tasks-and-Tasks-Sharing.md - Detailed Results and Visualization: Organizers/Benchmark_Creation/Detailed-Results-and-Visualizations.md - Running a Benchmarks: @@ -161,6 +161,11 @@ nav: - Homepage Counters (version < 1.15.0): Developers_and_Administrators/Upgrading_Codabench/Homepage-counters.md - User Removal (version < 1.17.0): Developers_and_Administrators/Upgrading_Codabench/User-removal.md - Database size fix (version < 1.18.0): Developers_and_Administrators/Upgrading_Codabench/Database-size-fixes.md + - Hide Output update (version < 1.19.0): Developers_and_Administrators/Upgrading_Codabench/Hide-output.md + - Django 3 Upgrades (version < 1.20.0): Developers_and_Administrators/Upgrading_Codabench/Django-3.md + - Minio Image Upgrade (version < 1.21.0): Developers_and_Administrators/Upgrading_Codabench/Minio-image.md + - Docker-Py (version < 1.22.0): Developers_and_Administrators/Upgrading_Codabench/Docker-py.md + - Django 4 Upgrades (version < 1.23.0): Developers_and_Administrators/Upgrading_Codabench/Django-4.md - Newsletters Archive: - 2024: Newsletters_Archive/CodaLab-in-2024.md - 2025: Newsletters_Archive/CodaLab-in-2025.md