Fix #4497: Make worker and statsd optional in development setup

[ilyasd3]

Summary

• Updated docker-compose.yml to use profiles for worker and statsd-exporter
• Updated README.md to document how to start optional containers using profiles
• Ensures a lightweight development setup with only necessary containers running

Changes Introduced

• The worker and statsd-exporter containers are now optional in development.
• The README.md has been updated to explain this.

Testing

• Verified that docker-compose up only starts required services.
• Verified that worker starts correctly with --profile worker.
• Verified that statsd-exporter starts correctly with --profile statsd.
• Checked that both services work when running together.

Bug Number

Fixes #4497

[RishabhJain2018]

@ilyasd3 Please add a video showing all three commands are working as expected :)

[ilyasd3]

@RishabhJain2018 I need to make some changes to other files to get the commands to work as expected. For example, I encountered an Exit Code 100 error from apt-get install -yq google-chrome-stable in docker/dev/nodejs/Dockerfile due to ARM64 architecture incompatibility. To fix this, I need to update the Dockerfile. How should I proceed?

[RishabhJain2018]

I wonder why you need to do it? The current docker compose up works as expected, right? If not, which OS are you on?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.30%. Comparing base (96968d6) to head (2ea4f52).
Report is 1122 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4498      +/-   ##
==========================================
- Coverage   72.93%   69.30%   -3.63%     
==========================================
  Files          83       20      -63     
  Lines        5368     3574    -1794     
==========================================
- Hits         3915     2477    -1438     
+ Misses       1453     1097     -356     

see 64 files with indirect coverage changes

see 64 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dab3f1e...2ea4f52. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ilyasd3]

This is a video of the error I am encountering. I am on macOS Sequoia 15.3.2.

https://youtu.be/fVFfRMfM6NQ

[lazyperson1020]

@RishabhJain2018 this macos issue is persistent. I faced it too. Can you raise issue for this?

[RishabhJain2018]

Hey @lazyperson1020 , Thanks for confirming this. Can one of you guys work to fix this?

[lazyperson1020]

@ilyasd3 let's work together? Actually I was able to build the project on mac after bypassing some of the things.

[ilyasd3]

@ilyasd3 let's work together? Actually I was able to build the project on mac after bypassing some of the things.

Sounds good! Keep me updated with your progress and let me know what I can do.

[Zahed-Riyaz]

Hey @ilyasd3 @lazyperson1020 regarding the error you guys are encountering... apt-get install -yq google-chrome-stable worked for me by simply changing the architecture to : FROM --platform=linux/amd64.

It basically instructs Docker to pull from a compatible architecture rather than defaulting to one that suits the host machine. I've built EvalAI on my system using Mac-OS

[ilyasd3]

Hey @ilyasd3 @lazyperson1020 regarding the error you guys are encountering... apt-get install -yq google-chrome-stable worked for me by simply changing the architecture to : FROM --platform=linux/amd64.

It basically instructs Docker to pull from a compatible architecture rather than defaulting to one that suits the host machine. I've built EvalAI on my system using Mac-OS

Yeah, I had done that actually. I was just wondering if that is something we should create a pull request for as well? Maybe @RishabhJain2018 can advise.

[lazyperson1020]

@Zahed-Riyaz WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64" (line 1) does this reduce performance? I think changing the dockerfiles to detect different architectures would be better?

[Zahed-Riyaz]

@lazyperson1020 so far for me it hasn't really impacted reproducing any issues or running EvalAI, I haven't really looked into it but it's been working fine ignoring the warning

[RishabhJain2018]

Ah..I see..you guys are using M1/M2 chips Mac.

[lazyperson1020]

@Zahed-Riyaz I encountered some issues like
92.76 E: /etc/ca-certificates/update.d/jks-keystore exited with code 1. 92.76 done. 92.84 Processing triggers for libc-bin (2.28-10) ... 93.43 Errors were encountered while processing: 93.43 ca-certificates-java 93.43 openjdk-11-jre-headless:amd64 93.43 openjdk-11-jre:amd64 93.43 openjdk-11-jdk-headless:amd64 93.43 default-jre 93.43 default-jdk-headless 93.43 openjdk-11-jdk:amd64 93.43 default-jdk 93.56 E: Sub-process /usr/bin/dpkg returned an error code (1) while building. Did it happen to you too?

[RishabhJain2018]

Can you guys try this Dockerfile for the nodejs docker container and verify if it works correctly in dev environment?

# Base stage with common dependencies
FROM node:14.20.0 AS base
WORKDIR /code

# Add dependencies
ADD ./package.json /code
ADD ./bower.json /code
ADD ./gulpfile.js /code
ADD ./.eslintrc /code
ADD ./karma.conf.js /code

# Install Prerequisites
RUN npm install -g bower gulp gulp-cli
RUN npm link gulp
RUN npm cache clean -f
RUN npm install
RUN npm install -g karma-cli
RUN npm install -g qs
RUN bower install --allow-root

# AMD64-specific stage
FROM base AS amd64
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
RUN apt-get update && apt-get install -yq google-chrome-stable libxss1

# ARM64-specific stage
FROM base AS arm64
# Using newer keyring method for better compatibility
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | gpg --dearmor > /usr/share/keyrings/google-chrome.gpg
# Add a message for users on ARM64
RUN echo "Chrome installation skipped on ARM64 architecture" && \
    # Create a placeholder chrome executable that exits gracefully
    echo '#!/bin/bash' > /usr/bin/google-chrome && \
    echo 'echo "Warning: Running on ARM64 architecture. Chrome is not available."' >> /usr/bin/google-chrome && \
    echo 'echo "For full Chrome functionality, rebuild with: docker build --platform=linux/amd64 ."' >> /usr/bin/google-chrome && \
    echo 'exit 0' >> /usr/bin/google-chrome && \
    chmod +x /usr/bin/google-chrome && \
    apt-get update && apt-get install -yq libxss1

# Final stage - automatically selects the right architecture
FROM ${TARGETARCH:-amd64}

# Set common command and expose port
CMD ["gulp", "dev:runserver"]
EXPOSE 8888

[RishabhJain2018]

So, it would be nice if --

• Someone with Apple M1/M2/M3 Mac tries it. -- @ilyasd3 on M3 Mac.
• Someone with Linux tries it. -- @ilyasd3 on AWS Ubuntu Server 22.04 LTS (64-bit x86)
• Someone with Apple Intel Mac tries it. -- @ilyasd3 on Intel Mac.

Please let me know if it works and also add where did you try building it.

[ilyasd3]

I just tried it on my M3 MacBook which is still based on the ARM64 architecture as the M1 and M2 chips, and it worked for me.

[RishabhJain2018]

If it works on all three, please raise a new PR with the Dockerfile change only for nodejs :)

[lazyperson1020]

@RishabhJain2018 works fine on my mac m1 now.

[Zahed-Riyaz]

It works! I faced issues with installing worker.txt libraries but it goes smoothly when I make worker optional as suggested in #4497.

[ilyasd3]

@RishabhJain2018 Is there any other PR I can raise for anything else?

[RishabhJain2018]

Hey @Zahed-Riyaz , Which OS did you try it on?

[RishabhJain2018]

@RishabhJain2018 Is there any other PR I can raise for anything else?

@ilyasd3 Please pick from the issues list while we are iterating on this PR.

[Zahed-Riyaz]

@RishabhJain2018 I ran it on M1 chip

[RishabhJain2018]

Can you guys try it on linux and apple intel mac too before we merge this PR to check compatibility of the Dockerfile.

[ilyasd3]

I just tested it on Apple Intel MacBook and it worked. To test on Linux, can I do that in a cloud environment like AWS?

[RishabhJain2018]

To test on Linux, can I do that in a cloud environment like AWS?

Yes, please do!

[ilyasd3]

I just tested it on an AWS Ubuntu Server 22.04 LTS (64-bit x86) EC2 instance and it worked.

[RishabhJain2018]

Awesome, thanks!

[RishabhJain2018]

Thanks for checking that. Now, we need a new PR with the updated Dockerfile of nodejs (only in dev mode), and then we can merge this PR after that. If you can do that then I'll ask @Zahed-Riyaz to not open that PR.

[RishabhJain2018]

LGTM :)

[RishabhJain2018]

LGTM :)

[ilyasd3]

@RishabhJain2018 I'm really happy to have my PR merged and would love to contribute further to CloudCV as part of GSoC 2025. I think CloudCV’s mission of making AI research more reproducible and accessible is incredibly inspiring, and I’d love to support the project in any way I can. My background in AWS and Python aligns well with the skills required, and I’d really appreciate any guidance on how to get more involved!

[RishabhJain2018]

Hey @ilyasd3 , Please pick any other issue to work on!