I try to compile TE on a slurmcluster because containers aren't fully supported (MPI issues).
My setup is like this:
module load cuda/12.4.1
module load cmake/3.23.1
module load git/2.35.2
module load gcc/12.1.0
module load cudnn/9.1.0.70-12.x
source $WORK/venvs/megatron/bin/activate
python -m pip install --force-reinstall setuptools==69.5.1.
python -m pip install nltk sentencepiece einops mpmath packaging numpy ninja wheel
pip3 install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu124
pip install wheel
MAX_JOBS=4 pip install flash-attn==2.4.2. --no-build-isolation
pip install git+https://github.com/NVIDIA/TransformerEngine.git@stable
export CXXFLAGS=-isystem\ $CUDNN_ROOT/include
pip install git+https://github.com/NVIDIA/TransformerEngine.git@main #or stable doesn't matter
All the variables echo well. I can build megatron-lm and apex in this environment, no problem. But not TE.
Error:
conda/envs/megatron/lib/python3.10/site-packages/torch/include/ATen/cudnn/cudnn-wrapper.h:3:10: fatal error: cudnn.h: No such file or directory
3 | #include <cudnn.h>
| ^~~~~~~~~
I try to compile TE on a slurmcluster because containers aren't fully supported (MPI issues).
My setup is like this:
All the variables echo well. I can build megatron-lm and apex in this environment, no problem. But not TE.
Error: