Support for code-cell to read code from a file

[mmcky]

Is your feature request related to a problem? Please describe.

Within a large lecture series it can be a pretty common pattern to re-use code in different documents. In the past we have used literalinclude to bring in code from a file. As sphinx parses code eventually in the literal_block method it works pretty well.

The issue with myst_nb/jupyter-book is this no longer works when building in this context as the literalinclude is not brought in until sphinx is called. This means the code won't get executed at the jupyter_cache level as the literalinclude is contained in the markdown block when converting using jupytext. The code will be visible in the end HTML but not executed prior to building with sphinx.

Describe the solution you'd like

A solution to this would be to add :file: option to code-cell directives for including code that is meant to be executed.

```{code-cell} python3
:file: _code/code.py
```

Typically file type attributes are usually arguments in sphinx but I think it would be better in this case to retain the simplicity of optional language argument.

The more typical sphinx approach would be

```{code-cell} _code/code.py
```

Describe alternatives you've considered

None -- changing the behaviour of literalinclude to be parsed as code is not a viable solution given out jupytext works.

Additional context

This is currently an issue for migrating the python.quantecon.org series as literalincludes are used pretty extensively in 6 lectures. We could copy the code across these lectures but would prefer not to do that to assist with maintaining the code and consistency across lectures that use the same code.

[mmcky]

One issue with the proposal of adding :file: as an option (rather than an argument) is that it becomes harder to support the usual options associated with, for example selecting lines etc. from the file, as the relationship is a bit less clear

https://www.sphinx-doc.org/en/1.5/markup/code.html#directive-literalinclude

[choldgraf]

I think a feature like this would certainly be useful - though I suspect it might get some pushback due to the fact that it would break the "notebook <--> source file" parity that we have thus far enforced fairly heavily. So I think it's something we'd need to discuss w/ the team

[mmcky]

Yeah -- not sure how to fully solve the "notebook <--> source file" issue.

One possibility is to keep that in tact (which I think is very important to keep two way roundtrips) and add filters/transform concepts to jupytext so jupytext could:

1. Convert notebook myst <--> ipynb
2. Apply filters / transforms to support things like or parsing code-cell with a file reference to bring in files

Alternatively perhaps it would be better to add pre-transforms to myst_nb so jupytext remains the same as a two-way conversion tool and we apply filters and transforms prior to calling jupytext.

[mmcky]

Another alternative would be to enable myst_parser to interpret code-cell with a file reference (or introduce code-file directive) and read the contents of the file directly on parsing.

[mmcky]

@chrisjsewell @choldgraf what do you think of a myst_nb loopback / update option that could update source notebooks once the myst nodes have been parsed by Sphinx and the context is understood for issues like this (and the exercise execution issue).

Perhaps it could work like:

1. Set option mystnb_update_jupytercache = True
2. delay usual jupyter-cache run to execute source files
3. Add a transform to generate a set of source notebooks to unpack any nested code-cell contexts (literalinclude, code-cell read from file feature, exercise execution). Updated notebooks wouldn't update the original sources but some intermediate temp object (such as <sourcedir>/.jupyter-cache/sphinx).
4. run jupyter-cache -> sphinx -> output pipeline.

That initial run could be pretty useful and would be a very flexible approach. It would also seem relatively easy to debug independently of the usual build pipeline and wouldn't interfere with jupytext.

Note: the loop would only focus on execution. We wouldn't build building notebooks from sphinx.ast

[chrisjsewell]

To put it simply; if the file can not be (fully) run by vanilla Jupyter then it is not a Jupyter Notebook and should not have the .ipynb extension.
We already have functionality to "pre-convert" custom files extensions to notebooks (https://myst-nb.readthedocs.io/en/latest/examples/custom-formats.html#custom-notebook-formats), this new format should use that

[mmcky]

To put it simply; if the file can not be (fully) run by vanilla Jupyter then it is not a Jupyter Notebook and should not have the .ipynb extension.

Perhaps we can bring in code from files using %load in a code-cell and leverage an ipython magic.

```{code-cell} ipython
%load <path>/filename.py
```

I tried this but it doesn't seem that the notebooks are executed in the same <path> as the content.

[chrisjsewell]

I tried this but it doesn't seem that the notebooks are executed in the same as the content.

It definitely is, execution_in_temp which is set to False by default was added ages ago (6346d39)

The CWD is the same as the folder of the notebook

[mmcky]

@chrisjsewell what happens though when you're using md as the source. Don't they get converted and saved in _build/jupyter_execute?

Ah I think I know what is happening. The %load will alter the input code cell (not the output of the code-cell) -- and I suspect jupyter-cache isn't setup for that kind of change right? The cell also needs to run twice (1 x to run magic and 1 x to execute code)

[mmcky]

Maybe we can leverage nbconvert preprocessors/filters to enable cell level transformations before executing the notebook -- through jupyter-cache. If I understand the stack correctly -- myst_nb handles the conversion to notebook via jupytext before calling jupyter-cache to execute. Is that right @chrisjsewell ?

If that is true -- then perhaps we can add a preprocessor level -- using nbconvert in myst_nb to achieve this.

[choldgraf]

hmmm - I don't believe that we depend on nbconvert currently, do we? I feel like that'd be a heavy-ish dependency to add given that it also duplicates much of the functionality of jupytext, though I could be wrong there.

@mmcky as a start couldn't you add a step to your build process that would:

1. Grab some text in a .py file (or whatever)
2. Search all detected files for cells with a particular tag or other piece of metadata
3. Update the content of those cells with your shared text
4. Then move forward with building the site

Then updating the shared content would be a matter of:

1. Updating the shared file
2. Running a build
3. Committing the updated content files to the repo