Python interpreter not respecting QUARTO_PYTHON

[Robinlovelace]

Bug description

I've set QUARTO_PYTHON as follows in a Docker container:

ENV QUARTO_PYTHON=/opt/venv/bin/python

Source: https://github.com/geocompx/docker/blob/master/rocker-rpy/Dockerfile

And this is visible in the image but when I try to run Python code interactively in the resulting Devcontainer, a different Python version is used.

Steps to reproduce

Click here to open a devcontainer based on the setup described (requires GitHub account). Try to run some Python code, you will see something like this:

Actual behavior

Error shown above.

Expected behavior

It should use the /opt/venv/bin/python version

Context: geocompx/docker#107

Your environment

• IDE: VSCode
• OS: Ubunt-based container

[juliasilge]

Hello there, @Robinlovelace! This here is the repository for the Visual Editor and the Quarto extension for VSCode and Positron.

We've got this guidance in the bug report template:

• If you're reporting an issue with the VS Code extension or the Visual Editor, you are in the right place! 🎉
• If you're reporting an issue with Quarto CLI, please visit https://github.com/quarto-dev/quarto-cli
• If you're reporting an issue inside RStudio, please visit https://github.com/rstudio/rstudio
• If you're reporting an issue inside Positron, please visit https://github.com/posit-dev/positron
• If you want to ask for help with using Quarto, please use the Q&A GitHub Discussions.

Do you think this is about the behavior in the VS Code extension? Does the document render correctly and it's only a problem with interactive use in VS Code?

[Robinlovelace]

If you're reporting an issue with the VS Code extension or the Visual Editor, you are in the right place! 🎉

I think so, yes.

[Robinlovelace]

Do you think this is about the behavior in the VS Code extension? Does the document render correctly and it's only a problem with interactive use in VS Code?

Yes and yes, as illustrated below.

quarto preview

Which generates:

Preparing to preview
[ 1/39] dstp.qmd
[ 2/39] d2/example.qmd
...

I guess there is some code in the extension which decides which version of Python to use and for some reason for the interactive environment that appears on the left-hand side after entering Ctrl+Enter in the code chunk, it's not following the CLI and prioritising the value of QUARTO_PYTHON in this case.

[cderv]

@Robinlovelace My understanding is that the Quarto extension will preview in a terminal by setting QUARTO_PYTHON to the python version set with the python extension

quarto/apps/vscode/src/core/terminal.ts

Lines 40 to 44 in 2349d21

	// QUARTO_PYTHON
	const python = await activePythonInterpreter(uri);
	if (python) {
	env.QUARTO_PYTHON = python;
	}

quarto/apps/vscode/src/core/python.ts

Lines 26 to 49 in 2349d21

	export async function activePythonInterpreter(uri?: Uri) {
	const workspaceFolder = activeWorkspaceFolder(uri);
	const pyExtension = extensions.getExtension("ms-python.python");
	if (pyExtension) {
	if (!pyExtension.isActive) {
	await pyExtension.activate();
	}
	const execDetails = pyExtension.exports.settings.getExecutionDetails(
	workspaceFolder?.uri
	);
	if (Array.isArray(execDetails?.execCommand)) {
	let python = execDetails.execCommand[0] as string;
	if (!path.isAbsolute(python)) {
	const path = which.sync(python, { nothrow: true });
	if (path) {
	python = path;
	}
	}
	return python;
	}
	}
	}

So the QUARTO_PYTHON env var is overridden in the terminal pane open by Preview.

So VSCODE settings would need to be "python.defaultInterpreterPath": "/opt/venv/bin/python"

I guess the one currently selecting in the VSCODE project is not the one you want.

I wonder if we should respect QUARTO_PYTHON if set already, before overriding it... 🤔
But AFAIK this was done to be sure to use the same version as the one active with Python extension.

Hope it helps

[Robinlovelace]

So VSCODE settings would need to be "python.defaultInterpreterPath": "/opt/venv/bin/python"

Totally makes sense, many thanks for detailed diagnosis with reference to the code.

Follow-up question: any ideas of the best place to set this globally in the Dockerfile?

[Robinlovelace]

I wonder if we should respect QUARTO_PYTHON if set already, before overriding it... 🤔

That is a good question. My sense is that QUARTO_PYTHON should indeed override "python.defaultInterpreterPath": "/opt/venv/bin/python". Especially because python.defaultInterpreterPath seems to get set without the user doing anything, I mean, where is it set in the fairly minimal set-up I've got? It's just bog standard VS Code-based devcontainer. In VS Code default settings I guess (but not at all sure).

[Robinlovelace]

I've double checked the devcontainer settings and there is no trace of anything that could be interpretted as wanting to override QUARTO_PYTHON: https://github.com/itsleeds/tds/blob/main/.devcontainer/devcontainer.json Quarto CLI should always use same Python version as Quarto interactive shell is a good principle.

[cderv]

In VS Code default settings I guess (but not at all sure).

Yes this can be set in VSCODE settings. For now, I believe your VSCODE session in devcontainer does select a default interpreter which is not what you want. So I guess you could do

	"customizations": {
	    "vscode": {
	      "extensions": [
	        "reditorsupport.r",
	        "GitHub.copilot-chat",
	        "quarto.quarto",
	        "ms-python.python",
	        "ms-toolsai.jupyter"
	      ],
	      "settings": {
	        "python.defaultInterpreterPath": "/opt/venv/bin/python"
	      }
	    }
	}

Documentation for this python extensions settings (https://code.visualstudio.com/docs/python/settings-reference) says it could be set to a ${workspaceFolder}/.venv for example, so that you can easily work on a project using Python virtual environment.

I don't exactly know how the VSCODE python extension choose its python version. Maybe some env var can also drive that.

[Robinlovelace]

Thanks for the suggestion @cderv, sounds promising, although ideally there would be a global setting for any interactive shell created by Quarto's UI, would that setting also work to set the correct version for Positron, for example?

[cderv]

Click here to open a devcontainer based on the setup described (requires GitHub account). Try to run some Python code, you will see something like this

So I checked your devcontainer in Codespace

Terminal seems to know python is the one you want

root@codespaces-9a1317:/workspaces/tds# echo $QUARTO_PYTHON
/opt/venv/bin/python
root@codespaces-9a1317:/workspaces/tds# which python
/opt/venv/bin/python
root@codespaces-9a1317:/workspaces/tds# which python3
/opt/venv/bin/python3

And Quarto can render

root@codespaces-9a1317:/workspaces/tds# quarto check jupyter
Quarto 1.7.24
[✓] Checking Python 3 installation....OK
      Version: 3.12.3
      Path: /opt/venv/bin/python
      Jupyter: 5.8.1
      Kernels: python3

[✓] Checking Jupyter engine render....OK

When I do the interactive part you mentionned I get the same error, though notice the selected interpreter

This is /usr/bin/python, so it seem different but it points to the same

root@codespaces-9a1317:/workspaces/tds# ls -lisa /usr/bin/python
987248 0 lrwxrwxrwx 1 root root 20 Sep  3 12:49 /usr/bin/python -> /opt/venv/bin/python

The errors says that ipykernel is missing. The /opt/venv/bin/python does have access to it while using the link does not.

root@codespaces-9a1317:/workspaces/tds# /opt/venv/bin/python -m pip list | grep -i ipykernel
ipykernel                 6.30.1
root@codespaces-9a1317:/workspaces/tds# /usr/bin/python -m pip list | grep -i ipykernel

So they are using different environment indeed

$ /usr/bin/python -c "import sys, pprint; print('exe=', sys.executable); pprint.pprint(sys.path)"
exe= /usr/bin/python
['',
'/usr/lib/python312.zip',
'/usr/lib/python3.12',
'/usr/lib/python3.12/lib-dynload',
'/usr/local/lib/python3.12/dist-packages',
'/usr/lib/python3/dist-packages']
$ /opt/venv/bin/python -c "import sys, pprint; print('exe=', sys.executable); pprint.pprint(sys.path)"
exe= /opt/venv/bin/python
['',
'/usr/lib/python312.zip',
'/usr/lib/python3.12',
'/usr/lib/python3.12/lib-dynload',
'/opt/venv/lib/python3.12/site-packages']

So this is all about the VSCODE Python extension defaulting to /usr/bin/python and not your venv version. You would need to configure that to make it work.

I don't think this is Quarto version (QUARTO_PYTHON) that should drive the python extension interpreter selection in VSCODE.

However, what I do understand is that you are executing inline cell in a .qmd document, and you would expect the QUARTO_PYTHON version to be used, and thus modified the interpreter used in VSCODE for execution, is that right ?

[Robinlovelace]

Many thanks for investigating this further, greatly appreciated. In answer to your question:

However, what I do understand is that you are executing inline cell in a .qmd document, and you would expect the QUARTO_PYTHON version to be used

Correct.

in answer to your related question:

and thus modified the interpreter used in VSCODE for execution, is that right ?

I'm not sure what you mean here, sorry, please could you clarify the question?

All the changes I've made can be found in the Dockerfile, I did create a symbolic link and am wondering if that could be causing issues.

[cderv]

I'm not sure what you mean here, sorry, please could you clarify the question?

The problem you face is that

• VSCODE is using /usr/bin/python by default. Not sure why as I would have expect python to be looked in PATH, and /opt/venv/bin/python to be found. Anyhow, VSCODE is not using this interpreter by default.
• "Run cell" does leverage the VSCODE python extension and so it will use whatever interpreter is configured. Your default is not the one you want.
• This interpreter logic happens without any Quarto knowledge. So my interrogation is
  • Could we detect that 'Run Cell' opens the interpreter from Quarto context ?
  • If so could we check QUARTO_PYTHON env var, and thus change the interpreter path to use to this one ?

However, "python.defaultInterpreterPath" is the setting to use it seems. And having your venv in your project would be easier. But I do thing you made it global for some reason. I usually use uv these days and it does create environment local to project - so I am biased.

I did create a symbolic link and am wondering if that could be causing issues.

This could cause issue maybe. But I don't know if VSCODE would use your venv by default without changing the VSCODE setting.

[Robinlovelace]

Update here: I'm trying to tackle the issue by modifying the Dockerfile. Instead of running install_python.sh in Rocker, which is set to be removed rocker-org/rocker-versioned2#944 I'm trying the following:

RUN apt-get update && apt-get install -y --no-install-recommends \
    python3-dev \
    python3-pip \
    python3-venv \
    libpython3-dev \
    swig && \
    rm -rf /var/lib/apt/lists/*

# Create a virtual environment for Python at /opt/venv
# This isolates Python packages from the system Python
RUN python3 -m venv /opt/venv

# Upgrade pip within the virtual environment to ensure we have the latest version
RUN /opt/venv/bin/python -m pip install --no-cache-dir --upgrade pip

# Set permissions for the virtual environment to allow non-root users to install packages
RUN chown -R root:staff /opt/venv && \
    chmod -R g+ws /opt/venv

# Install reticulate R package to enable R-Python interoperability
RUN R -e "install.packages('reticulate', repos='https://cloud.r-project.org/')"

# Prefer the venv python and put its bin on PATH so reticulate picks it up
ENV RETICULATE_PYTHON=/opt/venv/bin/python
ENV PATH=/opt/venv/bin:${PATH}

# Override the python symlink to use the venv Python (not system Python)
# This ensures that when 'python' is called, it uses the venv version
RUN ln -sf /opt/venv/bin/python /usr/bin/python
# RUN ln -sf /opt/venv/bin/python /bin/python

# Set version of Python that Quarto will use
ENV QUARTO_PYTHON=/opt/venv/bin/python

Note sure if that is the right approach so any comments/guidance welcome 🙏