Error Running with GPU: no kernel image is available for execution/operation received an exception

[alecpwills]

Summary

I've built deepmd-kit from source using cmake -LA -D TENSORFLOW_ROOT=/gpfs/projects/FernandezGroup/Alec/progs/tensorflow/install -D CMAKE_INSTALL_PREFIX=./install -D USE_CUDA_TOOLKIT=TRUE -D CUDA_TOOLKIT_ROOT_DIR=/gpfs/software/cuda/11.3/ -D LAMMPS_VERSION_NUMBER=20221103 -D LAMMPS_SOURCE_ROOT=/gpfs/projects/FernandezGroup/Alec/progs/lammps -D CUDA_NVCC_FLAGS="-arch=sm_37" .. and make -j4
I've built LAMMPS and Tensorflow specifying the same compute compatibility.
Everything finishes building fine, but trying to run using a model in LAMMPS generates an error.

DeePMD-kit Version

2.1.5

TensorFlow Version

2.10

Python Version, CUDA Version, GCC Version, LAMMPS Version, etc

Python: 3.9.1
CUDA: 11.3
GCC: 8.1.0
LAMMPS: 2022.11.03

Details

I'm using deepmd-kit in plugin mode, and running my lammps script generates the following output/error:

lmp -in SYS.IN

No protocol specified
LAMMPS (3 Nov 2022)
OMP_NUM_THREADS environment is not set. Defaulting to 1 thread. (src/comm.cpp:98)
  using 1 OpenMP thread(s) per MPI task
Reading data file ...
  orthogonal box = (0 0 0) to (14.373 14.373 14.373)
  1 by 1 by 1 MPI processor grid
  reading atoms ...
  290 atoms
  scanning bonds ...
  2 = max bonds/atom
  scanning angles ...
  1 = max angles/atom
  reading bonds ...
  192 bonds
  reading angles ...
  96 angles
Finding 1-2 1-3 1-4 neighbors ...
  special bond factors lj:    0        0        0       
  special bond factors coul:  0        0        0       
     2 = max # of 1-2 neighbors
     1 = max # of 1-3 neighbors
     1 = max # of 1-4 neighbors
     3 = max # of special neighbors
  special bonds CPU = 0.001 seconds
  read_data CPU = 0.026 seconds
288 atoms in group gT4P05
Loading plugin: deepmd pair style v2.1.5-dirty by Han Wang
Loading plugin: compute deeptensor/atom v2.1.5-dirty by Han Wang
Loading plugin: fix dplr v2.1.5-dirty by Han Wang
Loading plugin: kspace pppm/dplr v2.1.5-dirty by Han Wang
Summary of lammps deepmd module ...
  >>> Info of deepmd-kit:
  installed to:       /gpfs/projects/FernandezGroup/Alec/progs/deepmd-kit/source/build/install
  source:             v2.1.5-dirty
  source branch:       master
  source commit:      6e3d4a6
  source commit at:   2022-09-23 16:10:28 +0800
  surpport model ver.:1.1 
  build float prec:   double
  build variant:      cuda
  build with tf inc:  /gpfs/projects/FernandezGroup/Alec/progs/tensorflow/install/include;/gpfs/projects/FernandezGroup/Alec/progs/tensorflow/install/include
  build with tf lib:  /gpfs/projects/FernandezGroup/Alec/progs/tensorflow/install/lib/libtensorflow_cc.so;/gpfs/projects/FernandezGroup/Alec/progs/tensorflow/install/lib/libtensorflow_framework.so
  set tf intra_op_parallelism_threads: 0
  set tf inter_op_parallelism_threads: 0
  >>> Info of lammps module:
  use deepmd-kit at:  /gpfs/projects/FernandezGroup/Alec/progs/deepmd-kit/source/build/install
  source:             v2.1.5-dirty
  source branch:      master
  source commit:      6e3d4a6
  source commit at:   2022-09-23 16:10:28 +0800
  build float prec:   double
  build with tf inc:  /gpfs/projects/FernandezGroup/Alec/progs/tensorflow/install/include;/gpfs/projects/FernandezGroup/Alec/progs/tensorflow/install/include
  build with tf lib:  /gpfs/projects/FernandezGroup/Alec/progs/tensorflow/install/lib/libtensorflow_cc.so;/gpfs/projects/FernandezGroup/Alec/progs/tensorflow/install/lib/libtensorflow_framework.so
DeePMD-kit WARNING: Environmental variable TF_INTRA_OP_PARALLELISM_THREADS is not set. Tune TF_INTRA_OP_PARALLELISM_THREADS for the best performance.
DeePMD-kit WARNING: Environmental variable TF_INTER_OP_PARALLELISM_THREADS is not set. Tune TF_INTER_OP_PARALLELISM_THREADS for the best performance.
DeePMD-kit WARNING: Environmental variable OMP_NUM_THREADS is not set. Tune OMP_NUM_THREADS for the best performance.
2022-12-03 19:41:39.407049: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-03 19:41:44.909494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1614] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 10297 MB memory:  -> device: 0, name: NVIDIA Tesla K80, pci bus id: 0000:04:00.0, compute capability: 3.7
2022-12-03 19:41:44.912819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1614] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 10297 MB memory:  -> device: 1, name: NVIDIA Tesla K80, pci bus id: 0000:05:00.0, compute capability: 3.7
2022-12-03 19:41:44.915659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1614] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 10297 MB memory:  -> device: 2, name: NVIDIA Tesla K80, pci bus id: 0000:08:00.0, compute capability: 3.7
2022-12-03 19:41:44.918391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1614] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 10297 MB memory:  -> device: 3, name: NVIDIA Tesla K80, pci bus id: 0000:09:00.0, compute capability: 3.7
2022-12-03 19:41:44.921097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1614] Created device /job:localhost/replica:0/task:0/device:GPU:4 with 10297 MB memory:  -> device: 4, name: NVIDIA Tesla K80, pci bus id: 0000:83:00.0, compute capability: 3.7
2022-12-03 19:41:44.923821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1614] Created device /job:localhost/replica:0/task:0/device:GPU:5 with 10297 MB memory:  -> device: 5, name: NVIDIA Tesla K80, pci bus id: 0000:84:00.0, compute capability: 3.7
2022-12-03 19:41:44.926586: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1614] Created device /job:localhost/replica:0/task:0/device:GPU:6 with 10297 MB memory:  -> device: 6, name: NVIDIA Tesla K80, pci bus id: 0000:87:00.0, compute capability: 3.7
2022-12-03 19:41:44.929221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1614] Created device /job:localhost/replica:0/task:0/device:GPU:7 with 10297 MB memory:  -> device: 7, name: NVIDIA Tesla K80, pci bus id: 0000:88:00.0, compute capability: 3.7
2022-12-03 19:41:45.423126: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:357] MLIR V1 optimization pass is not enabled
  >>> Info of model(s):
  using   1 model(s): all_v03.pb 
  rcut in model:      7.15
  ntypes in model:    4

CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE

Your simulation uses code contributions which should be cited:
- USER-DEEPMD package:
The log file lists these citations in BibTeX format.

CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE

Generated 0 of 6 mixed pair_coeff terms from geometric mixing rule
WARNING: Using fix colvars with minimization (src/COLVARS/fix_colvars.cpp:392)
Neighbor list info ...
  update: every = 1 steps, delay = 0 steps, check = yes
  max neighbors/atom: 2000, page size: 100000
  master list distance cutoff = 9.15
  ghost atom cutoff = 9.15
  binsize = 4.575, bins = 4 4 4
  1 neighbor lists, perpetual/occasional/extra = 1 0 0
  (1) pair deepmd, perpetual
      attributes: full, newton on
      pair build: full/bin
      stencil: full/bin/3d
      bin: standard
Setting up cg style minimization ...
  Unit style    : metal
  Current step  : 0
cuda assert: no kernel image is available for execution on the device /gpfs/projects/FernandezGroup/Alec/progs/deepmd-kit/source/lib/src/cuda/prod_env_mat.cu 555
2022-12-03 19:41:53.950507: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at custom_op.cc:17 : INTERNAL: Operation received an exception: DeePMD-kit Error: CUDA Assert, in file /gpfs/projects/FernandezGroup/Alec/progs/deepmd-kit/source/op/custom_op.cc:17
INTERNAL: 2 root error(s) found.
  (0) INTERNAL: Operation received an exception: DeePMD-kit Error: CUDA Assert, in file /gpfs/projects/FernandezGroup/Alec/progs/deepmd-kit/source/op/custom_op.cc:17
	 [[{{node ProdEnvMatA}}]]
	 [[o_atom_virial/_31]]
  (1) INTERNAL: Operation received an exception: DeePMD-kit Error: CUDA Assert, in file /gpfs/projects/FernandezGroup/Alec/progs/deepmd-kit/source/op/custom_op.cc:17
	 [[{{node ProdEnvMatA}}]]
0 successful operations.
0 derived errors ignored.
ERROR: DeePMD-kit Error: TensorFlow Error: INTERNAL: 2 root error(s) found.
  (0) INTERNAL: Operation received an exception: DeePMD-kit Error: CUDA Assert, in file /gpfs/projects/FernandezGroup/Alec/progs/deepmd-kit/source/op/custom_op.cc:17
	 [[{{node ProdEnvMatA}}]]
	 [[o_atom_virial/_31]]
  (1) INTERNAL: Operation received an exception: DeePMD-kit Error: CUDA Assert, in file /gpfs/projects/FernandezGroup/Alec/progs/deepmd-kit/source/op/custom_op.cc:17
	 [[{{node ProdEnvMatA}}]]
0 successful operations.
0 derived errors ignored. (/gpfs/projects/FernandezGroup/Alec/progs/deepmd-kit/source/lmp/pair_deepmd.cpp:390)
Last command: minimize 1.0e-7 1.0e-7 10000 100000

I've built LAMMPS, Tensorflow, and deepmd using the GPU nodes that would actually be used during production runs, so I'm not sure why it's saying it can't find the kernel image.

Any help/insight would be greatly appreciated.

[njzjz]

CUDA_NVCC_FLAGS is hard-coded here. At this time, you need to modify these lines.

deepmd-kit/source/lib/src/cuda/CMakeLists.txt

Lines 32 to 43 in 6e3d4a6

	elseif (${CUDA_VERSION_MAJOR} STREQUAL "11" AND ${CUDA_VERSION_MINOR} GREATER "0")
	# nvcc flags
	set(CUDA_NVCC_FLAGS -gencode arch=compute_52,code=sm_52; # Tesla M40, Tesla M40, Quadro M6000...
	-gencode arch=compute_53,code=sm_53;
	-gencode arch=compute_60,code=sm_60; # Pascal – GP100/Tesla P100 – DGX-1 (Generic Pascal)
	-gencode arch=compute_61,code=sm_61; # Pascal - GTX 1080, GTX 1070, GTX 1060, GTX 1050, GTX 1030, Titan Xp, Tesla P40, Tesla P4, Discrete GPU on the NVIDIA Drive PX2
	-gencode arch=compute_70,code=sm_70; # Volta - GV100/Tesla V100, GTX 1180 (GV104)
	-gencode arch=compute_75,code=sm_75; # Turing - RTX 2080, Titan RTX, Quadro R8000
	-gencode arch=compute_80,code=sm_80; # Anpere - A100
	-gencode arch=compute_86,code=sm_86; # Anpere - RTX 3090
	-O3; -Xcompiler -fPIC;
	)

It should be reasonable to skip these lines if CUDA_NVCC_FLAGS has been set.

[alecpwills]

Thanks for pointing that out! I added my compute arch to those and it works now!

[njzjz]

#2168 allows -D CUDA_NVCC_FLAGS="-arch=sm_37" passed.