Quota

[ihsaan-ullah]

Linked to #1251

Description

This PR adds the storage quota per user feature.

It includes the following:

• a new 'quota' field in the User model so each user has its own quota. It can, therefore, changed per a user basis
• a global default quota variable defined in the settings of the app
• the users' used space and quota is displayed on the resource page
• checks has been added on every data upload (submissions, programs and bundles)
  • the front send the the file size among other metadata to the back
  • the back checks if it is ok to upload the file based on the remaining user quota
  • if not ok an error is sent back to the frontend and is displayed to the user
• using the django admin interface you can update the quota per user

Issues this PR resolves

Part of #713

For testing

To fake the size of a file you can overwrite the line 168 of the src/static/js/ours/client.js file

Checklist

• Code review by me
• Hand tested by me
• I'm proud of my work
• Code review by reviewer
• Hand tested by reviewer
• CircleCi tests are passing
• Ready to merge

[ihsaan-ullah]

My tests

It looks like we don't have the quota branch on codabench. I have cloned Tristan's repo and chekedout the quota branch to do the test.

Admin Interface

• ✅ Quota field is visible in the user table on admin interface
  .

• ✅ Changing the value in this field is reflected in the resource interface

• ✅ default value is set to 15GB

Resource Interface

• ✅ total quota and used quota is visible in the resource interface

• ✅ Changing the user quota from admin interface is reflected in the resource interface

• ❌ uploading new dataset/submission/program from resource interface does not refresh the quota to show updated used quota

Limit check

• ✅ Uploading submission when the limit is reached gives error

• ✅ Uploading dataset/program when the limit is reached gives error

• ✅ Uploading competition when the limit is reached gives error

Manual validation (https://github.com/codalab/codabench/wiki/manual-validation)

• Create an user account and login
• Create a competition
• Create a queue
• Upload a submission
• Check that the submission was processed (results, visualization tab, leaderboard)
• Change/recover password
• Delete user
  Attempted to delete user from admin interface. Failed with this error:

django.db.utils.IntegrityError: insert or update on table "tasks_task" violates foreign key constraint "tasks_task_created_by_id_1345568a_fk_profiles_user_id"
DETAIL:  Key (created_by_id)=(2) is not present in table "profiles_user".

• Delete submission
• Delete queue
• Delete competition
• Admin page
• Look at the logs for any problematic messages (docker-compose logs -f)

Suggestions:

1. Refresh the interface automatically when submisison/dataset is uploaded in resource interface
2. Readable format for the quota in the user table .e.g MB

[Didayolo]

The last point is to add some documentation, especially in "admin procedure" (documentation about how to change the quota of users).

https://github.com/codalab/codabench/wiki/Administrator-procedures

[ihsaan-ullah]

The last point is to add some documentation, especially in "admin procedure" (documentation about how to change the quota of users).

https://github.com/codalab/codabench/wiki/Administrator-procedures

Updated:
https://github.com/codalab/codabench/wiki/Administrator-procedures#user-quota-management

[ihsaan-ullah]

There are some migration problems. I will check it in detail and try to fix

[Didayolo]

@ihsaan-ullah Maybe it can be easier to first fix develop in a separate pull request?

We are not sure if the migration problems come from the last PR (monitoring), this one, or the combination of the two.

[ihsaan-ullah]

@OhMaley

One automated test is failing because of the new addition of file_size

This is the test:

codabench/src/apps/api/tests/test_datasets.py

Line 16 in 0efcceb

def test_dataset_api_checks_duplicate_names_for_same_user(self):

This is the line where it fails:

codabench/src/apps/api/views/datasets.py

Line 85 in 0efcceb

file_size = float(request.data['file_size'])