Is it possible to set time for running each notebook cell with nbval?

[marimeireles]

Hello nbval folks, first of all thanks a lot for your plugin! :)
We've decided to use it on ipycytoscape but we found a problem.
Currently we're having the following error on MacOS:

Notebook cell execution failed
Cell 2: Timeout of 5 seconds exceeded waiting for output.

The way libs like nbconvert are dealing with this is allowing the user to pass a flag that tells jupyter to wait an X amount of time for the cell finish loading with flags like this one: --ExecutePreprocessor.timeout=600 , docs.

Is it possible to do something similar on nbval?

Thanks a lot, please let me know if I can help somehow :)

[vidartf]

So my understanding of this error is that it is slightly different than the nbconvert case. In this case:

1. The cell execute request is sent.
2. The execution is processed.
3. The execute_reply message is received.
4. All further messages are awaited with a hard-coded timeout of 5 seconds per message receipt (until an execution_state: idle message is received).

For steps 1-3, a different timeout is used (which is configurable, via --nbval-cell-timeout which defaults to 2000 seconds).

I'm not entirely sure what is going on for the messages (after execution has finished) to take so long to send/process.

We could certainly make this configurable, but I just want to make sure it is the right fix. For now, could you try increasing this value in your local copy to A: see that it actually fixes the problem, and secondarily B: give some indication of how long is needed?

nbval/nbval/plugin.py

Line 336 in 3597e0b

self.output_timeout = 5

[Borda]

I was checking the --nbval-cell-timeout and it seems to have no effect, even with setting value to 300 I get the same error

Notebook cell execution failed
Cell X: Timeout of 5 seconds exceeded waiting for output.

[jhlegarreta]

According to the JupyterLab folks, timeout is not standard metadata, and it is being ignored:
jupyterlab/jupyterlab#12018

See specifically jupyterlab/jupyterlab#12018 (comment)

Is there any plan to fix it or support it?

We're interested in continuing to use nbval within a CI context due to its ability to check the outputs stored in our notebooks. Many of the cells in our notebooks have large execution times, and thus we are stuck due to the timeout flag being ignored.

[mikemhenry]

@jhlegarreta Does --nbval-cell-timeout=3000 fix your issue? (2000 is the default (in seconds) so you might need more time depending on how long it takes)

[jhlegarreta]

@mikemhenry thanks for the suggestion. Here is the attempt:
carpentries-incubator/SDC-BIDS-dMRI#255

I'll keep an eye on it to see if the flag is being taken into account.

Edit: CI's are failing due to an unrelated issue. It will be some time before I get a chance to have a look at this.

[julian-belina]

@mikemhenryUnfortunately, this does not resolve the matter for me. Have you made any progress on this? I am using nbval 0.11.0.

[mikemhenry]

What isn't working? Do you still get a timeout error or something else?

[julian-belina]

Yes, exactly. It tried to increase incrementally to --nbval-cell-timeout=900000, but still got timeout errors after a couple of minutes.

[fmigneault]

I am often seeing notebook runs failing with the same error:

Notebook cell execution failed
Cell X: Timeout of 5 seconds exceeded waiting for output.

How is the default 2000s actually applied? Because it does not seem to be the case when no timeout is applied explicitly.