Build rocm vLLM image

[edamamez]

Lamini v0.6.5 --> ran into wheel issue
Lamini v0.6.5 with cherrypicked commit: vllm-project/vllm#12172
and vLLM v0.6.5 with cherrypicked commit

8.190 /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_pytree.py:185: FutureWarning: optree is installed but the version is too old to support PyTorch Dynamo in C++ pytree. C++ pytree support is disabled. Please consider upgrading optree using `python3 -m pip install --upgrade 'optree>=0.13.0'`.
8.190   warnings.warn(
9.359 No ROCm runtime is found, using ROCM_HOME='/opt/rocm'
9.424 /vllm-workspace/libs/flash-attention/setup.py:95: UserWarning: flash_attn was requested, but nvcc was not found.  Are you sure your environment has nvcc available?  If you're installing within a container from https://hub.docker.com/r/pytorch/pytorch, only images whose names contain 'devel' will provide nvcc.
9.424   warnings.warn(
9.425 Traceback (most recent call last):
9.425   File "/vllm-workspace/libs/flash-attention/setup.py", line 179, in <module>
9.425
9.425
9.425 torch.__version__  = 2.6.0+cu124
9.425
9.425
9.426     CUDAExtension(
9.426   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1130, in CUDAExtension
9.426     library_dirs += library_paths(device_type="cuda")
9.426   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1264, in library_paths
9.426     if (not os.path.exists(_join_cuda_home(lib_dir)) and
9.426   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 2525, in _join_cuda_home
9.426     raise OSError('CUDA_HOME environment variable is not set. '
9.426 OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
------
Dockerfile.rocm:82
--------------------
  81 |     # Build ROCm flash-attention wheel if `BUILD_FA = 1`
  82 | >>> RUN --mount=type=cache,target=${CCACHE_DIR} \
  83 | >>>     if [ "$BUILD_FA" = "1" ]; then \
  84 | >>>         mkdir -p libs \
  85 | >>>         && cd libs \
  86 | >>>         && git clone https://github.com/ROCm/flash-attention.git \
  87 | >>>         && cd flash-attention \
  88 | >>>         && git checkout "${FA_BRANCH}" \
  89 | >>>         && git submodule update --init \
  90 | >>>         && GPU_ARCHS="${FA_GFX_ARCHS}" python3 setup.py bdist_wheel --dist-dir=/install; \
  91 | >>>     # Create an empty directory otherwise as later build stages expect one
  92 | >>>     else mkdir -p /install; \
  93 | >>>     fi
  94 |
--------------------
ERROR: failed to solve: process "/bin/sh -c if [ \"$BUILD_FA\" = \"1\" ]; then         mkdir -p libs         && cd libs         && git clone https://github.com/ROCm/flash-attention.git         && cd flash-attention         && git checkout \"${FA_BRANCH}\"         && git submodule update --init         && GPU_ARCHS=\"${FA_GFX_ARCHS}\" python3 setup.py bdist_wheel --dist-dir=/install;     else mkdir -p /install;     fi" did not complete successfully: exit code: 1

[edamamez]

Slightly different error running locally but CUDA_HOME issue is the same (above was run on llama@coresite-sv7-106-amd-mi250-26)

59.40 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
63.68 /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_pytree.py:185: FutureWarning: optree is installed but the version is too old to support PyTorch Dynamo in C++ pytree. C++ pytree support is disabled. Please consider upgrading optree using `python3 -m pip install --upgrade 'optree>=0.13.0'`.
63.68   warnings.warn(
65.42 No ROCm runtime is found, using ROCM_HOME='/opt/rocm'
66.16 Traceback (most recent call last):
66.16   File "/vllm-workspace/setup.py", line 602, in <module>
66.17     version=get_vllm_version(),
66.17   File "/vllm-workspace/setup.py", line 471, in get_vllm_version
66.17     cuda_version = str(get_nvcc_cuda_version())
66.17   File "/vllm-workspace/setup.py", line 428, in get_nvcc_cuda_version
66.17     assert CUDA_HOME is not None, "CUDA_HOME is not set"
66.17 AssertionError: CUDA_HOME is not set
------

 2 warnings found (use docker --debug to expand):
 - InvalidBaseImagePlatform: Base image rocm/pytorch:rocm6.2_ubuntu20.04_py3.9_pytorch_release_2.3.0 was pulled with platform "linux/amd64", expected "linux/arm64" for current build (line 18)
 - UndefinedVar: Usage of undefined variable '$CPLUS_INCLUDE_PATH' (line 63)
Dockerfile.rocm:136
--------------------
 135 |
 136 | >>> RUN --mount=type=cache,target=${CCACHE_DIR} \
 137 | >>>     --mount=type=bind,source=.git,target=.git \
 138 | >>>     --mount=type=cache,target=/root/.cache/pip \
 139 | >>>     python3 -m pip install -Ur requirements-rocm.txt \
 140 | >>>     && python3 setup.py clean --all \
 141 | >>>     && python3 setup.py develop
 142 |
--------------------
ERROR: failed to solve: process "/bin/sh -c python3 -m pip install -Ur requirements-rocm.txt     && python3 setup.py clean --all     && python3 setup.py develop" did not complete successfully: exit code: 1

[edamamez]

Trying vLLM v0.7.2

It worked!! Created lamini v0.7.2 and trying to build that now: https://github.com/lamini-ai/vllm/tree/lamini-v0.7.2/lamini

Praise be it built: https://hub.docker.com/repository/docker/powerml/inference-engine-amd/tags/v0.7.2/sha256-eeb854e9e63b007d1fccff04a37ec1123aa516c5af7d20f9d96c4dc74ea13f7b

[edamamez]

Pleaseeee

  Normal  Pulling    104s  kubelet            Pulling image "powerml/inference-engine-amd:v0.7.2"