Please answer the following questions for yourself before submitting an issue.
2021-02-05 14:26:00.620416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-02-05 14:26:03.557625: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2021-02-05 14:26:03.790381: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0002:00:00.0 name: Tesla M60 computeCapability: 5.2
coreClock: 1.1775GHz coreCount: 16 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 149.31GiB/s
2021-02-05 14:26:03.796912: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-02-05 14:26:03.805410: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-02-05 14:26:03.813387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-02-05 14:26:03.817933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-02-05 14:26:03.827291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-02-05 14:26:03.834728: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-02-05 14:26:03.850086: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-02-05 14:26:03.857292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2021-02-05 14:26:03.860948: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2021-02-05 14:26:03.875535: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x80ec5ff940 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-02-05 14:26:03.880482: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-02-05 14:26:03.885168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0002:00:00.0 name: Tesla M60 computeCapability: 5.2
coreClock: 1.1775GHz coreCount: 16 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 149.31GiB/s
2021-02-05 14:26:03.891972: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-02-05 14:26:03.895720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-02-05 14:26:03.899501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-02-05 14:26:03.904606: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-02-05 14:26:03.908422: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-02-05 14:26:03.912343: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-02-05 14:26:03.916413: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-02-05 14:26:03.924000: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2021-02-05 14:26:04.700603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength
1 edge matrix:
2021-02-05 14:26:04.704509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2021-02-05 14:26:04.706737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2021-02-05 14:26:04.726496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7048 MB memory) -> physical GPU (device: 0, name: Tesla M60, pci bus id: 0002:00:00.0, compute capability: 5.2)
2021-02-05 14:26:04.736932: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x810d9b6950 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-02-05 14:26:04.741982: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla M60, Compute Capability 5.2
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0205 14:26:04.749317 8188 mirrored_strategy.py:500] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: None
I0205 14:26:04.755326 8188 config_util.py:552] Maybe overwriting train_steps: None
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0205 14:26:04.755326 8188 config_util.py:552] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Reading unweighted datasets: ['E:/DS_2020_Wildlife/Multi_Class_Classification/Tensorflow/workspace/annotations/train.record']
I0205 14:26:04.923312 8188 dataset_builder.py:163] Reading unweighted datasets: ['E:/DS_2020_Wildlife/Multi_Class_Classification/Tensorflow/workspace/annotations/train.record']
INFO:tensorflow:Reading record datasets for input file: ['E:/DS_2020_Wildlife/Multi_Class_Classification/Tensorflow/workspace/annotations/train.record']
I0205 14:26:04.926311 8188 dataset_builder.py:80] Reading record datasets for input file: ['E:/DS_2020_Wildlife/Multi_Class_Classification/Tensorflow/workspace/annotations/train.record']
INFO:tensorflow:Number of filenames to read: 1
I0205 14:26:04.926311 8188 dataset_builder.py:81] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0205 14:26:04.926311 8188 dataset_builder.py:87] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py:101: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
W0205 14:26:04.928313 8188 deprecation.py:317] From E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py:101: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
WARNING:tensorflow:From E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W0205 14:26:04.952313 8188 deprecation.py:317] From E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is
deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
Traceback (most recent call last):
File "model_main_tf2.py", line 115, in <module>
tf.compat.v1.app.run()
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\platform\app.py", line 40,
in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\absl\app.py", line 300, in run
_run_main(main, args)
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\absl\app.py", line 251, in _run_main
sys.exit(main(argv))
File "model_main_tf2.py", line 106, in main
model_lib_v2.train_loop(
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\model_lib_v2.py", line 569,
in train_loop
load_fine_tune_checkpoint(detection_model,
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\model_lib_v2.py", line 352,
in load_fine_tune_checkpoint
features, labels = iter(input_dataset).next()
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 858, in __iter__
iterators, element_spec = _create_iterators_per_worker_with_input_context(
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 1401, in _create_iterators_per_worker_with_input_context
dataset = dataset_fn(ctx)
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\model_lib_v2.py", line 521,
in train_dataset_fn
train_input = inputs.train_input(
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\inputs.py", line 893, in train_input
dataset = INPUT_BUILDER_UTIL_MAP['dataset_build'](
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py", line 251, in build
dataset = dataset_map_fn(dataset, decoder.decode, batch_size,
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py", line 236, in dataset_map_fn
dataset = dataset.map_with_legacy_function(
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\util\deprecation.py", line
324, in new_func
return func(*args, **kwargs)
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 2402, in map_with_legacy_function
ParallelMapDataset(
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 4016, in __init__
self._map_func = StructuredFunctionWrapper(
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 3196, in __init__
self._function.add_to_graph(ops.get_default_graph())
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\framework\function.py", line 544, in add_to_graph
self._create_definition_if_needed()
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\framework\function.py", line 376, in _create_definition_if_needed
self._create_definition_if_needed_impl()
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\framework\function.py", line 398, in _create_definition_if_needed_impl
temp_graph = func_graph_from_py_func(
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\framework\function.py", line 969, in func_graph_from_py_func
outputs = func(*func_graph.inputs)
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 3188, in wrapper_fn
ret = _wrapper_helper(*args)
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 3156, in _wrapper_helper
ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 265, in wrapper
raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:
E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\data_decoders\tf_example_decoder.py:524 default_groundtruth_weights *
[tf.shape(tensor_dict[fields.InputDataFields.groundtruth_boxes])[0]],
E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\ops\array_ops.py:2967 ones **
output = _constant_if_small(one, shape, dtype, name)
E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\ops\array_ops.py:2662 _constant_if_small
if np.prod(shape) < 1000:
<__array_function__ internals>:5 prod
E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\numpy\core\fromnumeric.py:3030 prod
return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out,
E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\numpy\core\fromnumeric.py:87 _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\framework\ops.py:748 __array__
raise NotImplementedError("Cannot convert a symbolic Tensor ({}) to a numpy"
NotImplementedError: Cannot convert a symbolic Tensor (strided_slice:0) to a numpy array.
Prerequisites
Please answer the following questions for yourself before submitting an issue.
1. The entire URL of the file you are using
https://github.com/tensorflow/models/blob/master/research/object_detection/model_main_tf2.py
2. Describe the bug
I am trying to train object detection model for custom data using tutorial on link. I tested the for environment faults using https://github.com/tensorflow/models/blob/master/research/object_detection/builders/model_builder_tf2_test.py. All test passed. But when I put it for training the models gives out error.
Logs:
6. System information