Out Of Memory Error using Python interface

[LiangWenshuo1118]

Summary

I am performing Monte Carlo simulation using a compressed DP model to predict energies in a python code (see attach files). The task will only be completed if the MC step is small. Usually the task will be killed, due to lack of memory. The memory of the machine is 188GB.

dp-mc.zip

DeePMD-kit Version

2.1.5

TensorFlow Version

2.9.0

Python Version, CUDA Version, GCC Version, LAMMPS Version, etc

python 3.10.4 cuda 11.6

Details

2022-11-30 21:29:09.378812: W tensorflow/core/common_runtime/bfc_allocator.cc:491] ****************************************************************************************************
2022-11-30 21:29:09.378843: W tensorflow/core/framework/op_kernel.cc:1722] OP_REQUIRES failed at constant_op.cc:82 : RESOURCE_EXHAUSTED: OOM when allocating tensor of shape [2340,600] and type double
0
1
2
3
4
5
6
7
Traceback (most recent call last):
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1377, in _do_call
return fn(*args)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1360, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1453, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [2340,600] and type double
[[{{node load/Const}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/utils/sess.py", line 21, in run_sess
return sess.run(*args, **kwargs)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 967, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1190, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1370, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1396, in _do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:

Detected at node 'load/Const' defined at (most recent call last):
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 109, in
x = mcmc(logp, x, mc_steps)
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 82, in mcmc
x, logp, num_accepts = step(i, (x, logp, num_accepts))
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 67, in step
logp_proposal = logp_fn(x_proposal)
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 19, in logp
dp = DeepPot('graph.pb')
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 88, in init
DeepEval.init(
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/infer/deep_eval.py", line 40, in init
self.graph = self._load_graph(
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/infer/deep_eval.py", line 171, in _load_graph
tf.import_graph_def(
Node: 'load/Const'
OOM when allocating tensor of shape [2340,600] and type double
[[{{node load/Const}}]]

Original stack trace for 'load/Const':
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 109, in
x = mcmc(logp, x, mc_steps)
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 82, in mcmc
x, logp, num_accepts = step(i, (x, logp, num_accepts))
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 67, in step
logp_proposal = logp_fn(x_proposal)
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 19, in logp
dp = DeepPot('graph.pb')
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 88, in init
DeepEval.init(
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/infer/deep_eval.py", line 40, in init
self.graph = self._load_graph(
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/infer/deep_eval.py", line 171, in _load_graph
tf.import_graph_def(
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/util/deprecation.py", line 561, in new_func
return func(*args, **kwargs)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/framework/importer.py", line 403, in import_graph_def
return _import_graph_def_internal(
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/framework/importer.py", line 516, in _import_graph_def_internal
_ProcessNewOps(graph)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/framework/importer.py", line 247, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 3904, in _add_new_tf_operations
new_ops = [
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 3905, in
self._create_op_from_tf_operation(c_op, compute_device=compute_devices)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 3787, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 2133, in init
self._traceback = tf_stack.extract_stack_for_node(self._c_op)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/utils/batch_size.py", line 66, in execute
n_batch, result = callable(max(self.current_batch_size // natoms, 1), start_index)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/utils/batch_size.py", line 108, in execute_with_batch_size
return (end_index - start_index), callable(
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 378, in _eval_inner
v_out = run_sess(self.sess, t_out, feed_dict = feed_dict_test)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/utils/sess.py", line 39, in run_sess
raise OutOfMemoryError(MESSAGE) from e
deepmd.utils.errors.OutOfMemoryError: Your memory may be not enough, thus an error has been raised above. You need to take the following actions:

1. Check if the network size of the model is too large.
2. Check if the batch size of training or testing is too large. You can set the training batch size to auto.
3. Check if the number of atoms is too large.
4. Check if another program is using the same GPU by execuating nvidia-smi. The usage of GPUs is controlled by CUDA_VISIBLE_DEVICES environment variable (current value: None).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 109, in
x = mcmc(logp, x, mc_steps)
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 82, in mcmc
x, logp, num_accepts = step(i, (x, logp, num_accepts))
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 67, in step
logp_proposal = logp_fn(x_proposal)
File "/home/input_lbg-1559-5350005/dp-rdf.py", line 24, in logp
energies,forces,virials = dp.eval(coord, cell, atype)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 264, in eval
output = self._eval_func(self._eval_inner, numb_test, natoms)(coords, cells, atom_types, fparam = fparam, aparam = aparam, atomic = atomic, efield = efield)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 197, in eval_func
return self.auto_batch_size.execute_all(inner_func, numb_test, natoms, *args, **kwargs)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/utils/batch_size.py", line 116, in execute_all
n_batch, result = self.execute(execute_with_batch_size, index, natoms)
File "/opt/deepmd-kit-2.1.5/lib/python3.10/site-packages/deepmd/utils/batch_size.py", line 74, in execute
raise OutOfMemoryError("The callable still throws an out-of-memory (OOM) error even when batch size is 1!") from e
deepmd.utils.errors.OutOfMemoryError: The callable still throws an out-of-memory (OOM) error even when batch size is 1!

[njzjz]

Avoid loading the same graph multiple times. At this time, TensorFlow never releases the memory before the process shuts down.

[LiangWenshuo1118]

Nice advice, this should be written into the manual. Need me to do it?

[njzjz]

Yes, please.