Skip to content

Access data in worker #881

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Didayolo merged 2 commits into from
May 22, 2023
Merged

Access data in worker #881

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2f8dbf8
Mount /codabench/data to /app/data
Didayolo May 17, 2023
9ea0571
Add ":ro" for read-only access
Didayolo May 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions compute_worker/compute_worker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@


# Setup base directories used by all submissions
HOST_DIRECTORY = os.environ.get("HOST_DIRECTORY", "/tmp/codabench/") # note: we need to pass this directory to
# docker compose so it knows where to store things!
BASE_DIR = "/codabench/"
# note: we need to pass this directory to docker-compose so it knows where to store things!
HOST_DIRECTORY = os.environ.get("HOST_DIRECTORY", "/tmp/codabench/")
BASE_DIR = "/codabench/" # base directory inside the container
CACHE_DIR = os.path.join(BASE_DIR, "cache")
MAX_CACHE_DIR_SIZE_GB = float(os.environ.get('MAX_CACHE_DIR_SIZE_GB', 10))

Expand Down Expand Up @@ -182,6 +182,7 @@ def __init__(self, run_args):
self.bundle_dir = os.path.join(self.root_dir, "bundles")
self.input_dir = os.path.join(self.root_dir, "input")
self.output_dir = os.path.join(self.root_dir, "output")
self.data_dir = os.path.join(HOST_DIRECTORY, "data") # absolute path to data in the host
self.logs = {}

# Details for submission
Expand Down Expand Up @@ -522,6 +523,7 @@ async def _run_program_directory(self, program_dir, kind, can_be_output=False):
# Set the volumes
'-v', f'{self._get_host_path(program_dir)}:/app/program',
'-v', f'{self._get_host_path(self.output_dir)}:/app/output',
'-v', f'{self.data_dir}:/app/data:ro',

# Start in the right directory
'-w', '/app/program',
Expand Down