"Not found: No algorithm worked!" error for convolutional layer during search

[robin-p-schmitt]

My config looks as follows: https://gist.github.com/robin-p-schmitt/fe9880b8ff3cd1a4c7201626776fdaab. The relevant part seems to be this:

"source": {"class": "eval", "eval": <...specaug...>},
"source0": {"class": "split_dims", "axis": "F", "dims": (-1, 1), "from": "source"},  # (T,40,1)

"conv0": {
  "class": "conv", "from": "source0",
  "padding": "same", "filter_size": (3, 3),
  "n_out": 32, "activation": None, "with_bias": True},  # (T,40,32)

It works for both train as well as search mode when I am using /u/merboldt/setups/2020-01-08--rnnt-rna/crnn as my RETURNN version. However, when I am using the most recent RETURNN version (/u/schmitt/src/returnn), only train mode seems to work. When I try to run search with the current RETURNN version I get the following error: https://gist.github.com/robin-p-schmitt/5f89cd01ed4c5b74d9aebee256a4707c.

The main error seems to be:

2021-09-30 12:52:25.768174: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at conv_ops.cc:1
115 : Not found: No algorithm worked!
TensorFlow exception: 2 root error(s) found.
  (0) Not found: No algorithm worked!
         [[node conv0/convolution (defined at u/schmitt/src/returnn/returnn/tf/layers/basic.py:4061) ]]
         [[output/rec/while/Switch_14/_553]]
  (1) Not found: No algorithm worked!
         [[node conv0/convolution (defined at u/schmitt/src/returnn/returnn/tf/layers/basic.py:4061) ]]
0 successful operations.

The log for the relevant layer is:

layer root/'source' output: Data{'source_output', [B,T|'time'[B],F|F'feature:data'(40)]}
layer root/'source0' output: Data{'source0_output', [B,T|'time'[B],F|F'feature:data'(40),'source0_split_dims1'(1)]}
layer root/'conv0' output: Data{'conv0_output', [B,T|'time'[B],'source0_split_dims1'(1),F|F'conv0:channel'(32)]}
layer root/'conv0p' output: Data{'conv0p_output', [B,T|'time'[B],'conv0p:pool:s1'(1),F|F'conv0:channel'(32)]}

I am using Python 3.8.0 with TF 2.3 and CUDA 10.1.

[robin-p-schmitt]

@albertz

[albertz]

Can you fix the formatting?

• Remove the escape codes.
• Also, put the whole config in <details> (or however that works here on GitHub), or otherwise upload it to a Gist or so, and then only directly show the relevant conv layer dict. Probably this:

  "source": {"class": "eval", "eval": <...specaug...>},
  "source0": {"class": "split_dims", "axis": "F", "dims": (-1, 1), "from": "source"},  # (T,40,1)
  
  "conv0": {
    "class": "conv", "from": "source0",
    "padding": "same", "filter_size": (3, 3),
    "n_out": 32, "activation": None, "with_bias": True},  # (T,40,32)
  

• Same for the log including the error. Put it in <details> or upload it to a Gist. Only the main error is really relevant to be posted directly. And maybe the log for the relevant layer output:

  layer root/'source' output: Data{'source_output', [B,T|'time'[B],F|F'feature:data'(40)]}
  layer root/'source0' output: Data{'source0_output', [B,T|'time'[B],F|F'feature:data'(40),'source0_split_dims1'(1)]}
  layer root/'conv0' output: Data{'conv0_output', [B,T|'time'[B],'source0_split_dims1'(1),F|F'conv0:channel'(32)]}
  layer root/'conv0p' output: Data{'conv0p_output', [B,T|'time'[B],'conv0p:pool:s1'(1),F|F'conv0:channel'(32)]}
  

[albertz]

The RETURNN version is not printed correctly (RETURNN starting up, version 1.0.0+unknown). Can you fix that? I assume latest RETURNN? Please mention that in the initial description.

Also explicitly mention the TF. Looks like Python 3.8.0 with TF 2.3.

Also mention the CUDA version. I assume CUDA 10.1 from the log.

[albertz]

Oh fuck, I just notice another bug: SplitDimsLayer (source0 here) marks the wrong feature dim axis. Thus ConvLayer has the wrong shape. (That's why explicitly specifying the shape via out_shape or so as in #706 would be helpful...) I reported this in #704. It's probably unrelated.

[albertz]

So, on the error itself, there are many similar errors when you search for it, e.g.:

• NotFoundError: No algorithm worked! when using Conv2D tensorflow/tensorflow#43174
• OP_REQUIRES failed at conv_ops.cc:1106 : Not found: No algorithm worked! tensorflow/tensorflow#45044
• OP_REQUIRES failed at conv_ops.cc:1106 : Not found: No algorithm worked! tensorflow/tensorflow#48117
• https://stackoverflow.com/questions/62904830/keras-no-algorithm-worked
• https://stackoverflow.com/questions/64605457/meaning-of-notfounderror-no-algorithm-worked
• https://forums.developer.nvidia.com/t/conv-ops-not-found-no-algorithm-worked/167905
• https://fantashit.com/op-requires-failed-at-conv-ops-cc-1115/

[robin-p-schmitt]

The RETURNN version is not printed correctly (RETURNN starting up, version 1.0.0+unknown). Can you fix that? I assume latest RETURNN? Please mention that in the initial description.

I will try to fix this on the weekend.

[albertz]

Some of these report that the GPU has too less memory and this causes such an error.
Can you try with a smaller batch size?

[robin-p-schmitt]

Some of these report that the GPU has too less memory and this causes such an error. Can you try with a smaller batch size?

Yes, I will also do this over the weekend. If this and the other links don't help, I will also try to create a minimal config which reproduces the error.

[tbscode]

I get this also on some of my setups, which where working before.
Only the ones using conv-layers and only in recog.

[albertz]

Maybe #704 is actually relevant. This bug probably did not appear for too long, only since recently (I need to check since when exactly). Although it is somewhat strange that this could cause such an error.

I anyway would also test with a smaller batch size.

[albertz]

I fixed #704 via #705 now. So also try with current master again (and the original batch size).

But in any case, also test with smaller batch size (before testing with current master). I want to see if this already solves it.