Extension loads forever when there are a lot of conda environments (remote SSH)

[matmel]

Environment data

• VS Code version: 1.43.2 (user setup)
• Extension version (available under the Extensions sidebar):
  • local:
    • Remote development 0.20.0
    • Remote SSH 0.51.0
• OS and version: Windows 10 Enterprise 1709 16299.1686
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.7 (from conda default channel)
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
• Relevant/affected Python packages and their versions: Not relevant from my understanding
• Relevant/affected Python-related VS Code extensions and their versions: Python extension 2020.3.69010
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Jedi (python.JediEnabled = True)
• Value of the python.languageServer setting: Microsoft

Expected behaviour

Editing python files is snappy.

Actual behaviour

Python extension loading... message runs indefinitely.

Steps to reproduce:

Open any folder or multi root workspace on a CentOS 7 machine and this behavior is shown.

Logs

Output of Remote - SSH

[12:48:20.332] Log Level: 1
[12:48:20.379] remote-ssh@0.51.0
[12:48:20.379] win32 x64
[12:48:20.382] SSH Resolver called for "ssh-remote+poisux803", attempt 1
[12:48:20.383] SSH Resolver called for host: poisux803
[12:48:20.383] Setting up SSH remote "poisux803"
[12:48:20.421] Using commit id "0ba0ca52957102ca3527cf479571617f0de6ed50" and quality "stable" for server
[12:48:20.423] Install and start server if needed
[12:48:20.430] Checking ssh with "ssh -V"
[12:48:24.138] > OpenSSH_7.9p1, OpenSSL 1.1.1a  20 Nov 2018
[12:48:24.141] Running script with connection command: ssh -T -D 62632 poisux803 bash
[12:48:24.146] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[12:48:25.386] "install" terminal received data: "�[0m�[0K"
[12:48:25.387] Got some output, clearing connection timeout
[12:48:27.655] "install" terminal received data: "e56d19b5f442: running�[0K�[?25l
Acquiring lock on /import/poi/home/poimat04/.vscode-server/bin/0ba0ca52957102ca3527cf479571617f0de6ed50/vscode-remote-lock.poimat04.0ba0ca52957102ca3527cf479571617f0de6ed50�[0K
\ln /import/poi/home/poimat04/.vscode-server/bin/0ba0ca52957102ca3527cf479571617f0de6ed50/vscode-remote-lock.poimat04.0ba0ca52957102ca3527cf479571617f0de6ed50.target /import/poi/home/poimat04/.vscode-server/bin/0ba0ca�[0K5
2957102ca3527cf479571617f0de6ed50/vscode-remote-lock.poimat04.0ba0ca52957102ca3527cf479571617f0de6ed50�[0K
�[0K�[?25h"
[12:48:27.721] "install" terminal received data: "Found existing installation at /import/poi/home/poimat04/.vscode-server/bin/0ba0ca52957102ca3527cf479571617f0de6ed50...�[0K�[?25l
�[0K�[?25h"
[12:48:27.740] "install" terminal received data: "Found running server...�[0K�[?25l
�[0K
*�[0K
* Reminder: You may only use this software with Visual Studio family products,�[0K
* as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)�[0K
*�[0K
�[0K
�[0K�[?25h"
[12:48:27.776] "install" terminal received data: "Checking server status on port 37978 with wget�[0K�[?25l
e56d19b5f442: start�[0K
sshAuthSock====�[0K
agentPort==37978==�[0K
osReleaseId==centos==�[0K
arch==x86_64==�[0K
webUiAccessToken====�[0K
tmpDir==/run/user/10029==�[0K
platform==linux==�[0K
e56d19b5f442: end�[0K
�[0K�[?25h"
[12:48:27.776] Received install output: 
sshAuthSock====
agentPort==37978==
osReleaseId==centos==
arch==x86_64==
webUiAccessToken====
tmpDir==/run/user/10029==
platform==linux==

[12:48:27.777] Remote server is listening on port 37978
[12:48:27.777] Parsed server configuration: {"agentPort":37978,"osReleaseId":"centos","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","tmpDir":"/run/user/10029","platform":"linux"}
[12:48:27.780] Starting forwarding server. localPort 62659 -> socksPort 62632 -> remotePort 37978
[12:48:27.781] Forwarding server listening on 62659
[12:48:27.781] Waiting for ssh tunnel to be ready
[12:48:27.784] Tunneled remote port 37978 to local port 62659
[12:48:27.784] Resolved "ssh-remote+poisux803" to "127.0.0.1:62659"
[12:48:27.786] [Forwarding server 62659] Got connection 0
[12:48:27.796] TELEMETRY: {"eventName":"resolver","properties":{"outcome":"success","osReleaseId":"centos","arch":"x86_64","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","remoteInConfigFile":"1","gotUnrecognizedPrompt":"0","dynamicForwarding":"1","localServer":"0"},"measures":{"resolveAttempts":1,"retries":1}}
[12:48:27.798] ------




[12:48:27.806] [Forwarding server 62659] Got connection 1
[12:48:27.807] [Forwarding server 62659] Got connection 2
[12:48:29.015] [Forwarding server 62659] Got connection 3
[12:48:29.017] [Forwarding server 62659] Got connection 4
[12:48:29.019] [Forwarding server 62659] Got connection 5

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

I can see several instances of those errors:

Error: Trying to add a disposable to a DisposableStore that has already been disposed of. The added object will be leaked!
    at l.add (lifecycle.ts:134)
    at e._register (lifecycle.ts:164)
    at e.addDisposable (terminalInstance.ts:343)
    at m._initInstanceListeners (terminalTab.ts:272)
    at new m (terminalTab.ts:246)
    at h._createInstance (instantiationService.ts:117)
    at h.createInstance (instantiationService.ts:82)
    at T._showBackgroundTerminal (terminalService.ts:597)
    at T.setActiveInstance (terminalService.ts:316)
    at p.$show (mainThreadTerminalService.ts:108)
    at g._doInvokeHandler (rpcProtocol.ts:398)
    at g._invokeHandler (rpcProtocol.ts:383)
    at g._receiveRequest (rpcProtocol.ts:299)
    at g._receiveOneMessage (rpcProtocol.ts:235)
    at rpcProtocol.ts:101
    at c.fire (event.ts:587)
    at v.fire (ipc.net.ts:453)
    at t.PersistentProtocol._receiveMessage (ipc.net.ts:733)
    at ipc.net.ts:592
    at c.fire (event.ts:587)
    at g.acceptChunk (ipc.net.ts:239)
    at ipc.net.ts:200
    at Socket.t (ipc.net.ts:28)
    at Socket.emit (events.js:203)
    at addChunk (_stream_readable.js:295)
    at readableAddChunk (_stream_readable.js:276)
    at Socket.Readable.push (_stream_readable.js:210)
    at TCP.onStreamRead (internal/stream_base_commons.js:166)
add @ lifecycle.ts:134
_register @ lifecycle.ts:164
addDisposable @ terminalInstance.ts:343
_initInstanceListeners @ terminalTab.ts:272
m @ terminalTab.ts:246
_createInstance @ instantiationService.ts:117
createInstance @ instantiationService.ts:82
_showBackgroundTerminal @ terminalService.ts:597
setActiveInstance @ terminalService.ts:316
$show @ mainThreadTerminalService.ts:108
_doInvokeHandler @ rpcProtocol.ts:398
_invokeHandler @ rpcProtocol.ts:383
_receiveRequest @ rpcProtocol.ts:299
_receiveOneMessage @ rpcProtocol.ts:235
(anonymous) @ rpcProtocol.ts:101
fire @ event.ts:587
fire @ ipc.net.ts:453
_receiveMessage @ ipc.net.ts:733
(anonymous) @ ipc.net.ts:592
fire @ event.ts:587
acceptChunk @ ipc.net.ts:239
(anonymous) @ ipc.net.ts:200
t @ ipc.net.ts:28
emit @ events.js:203
addChunk @ _stream_readable.js:295
readableAddChunk @ _stream_readable.js:276
Readable.push @ _stream_readable.js:210
onStreamRead @ internal/stream_base_commons.js:166

I can see this also:

console.ts:137 [Extension Host] Error Python Extension: 2020-03-30 12:53:07: stderr jediProxy Error (stderr) completion.py:589: DeprecationWarning: Deprecated since version 0.16.0. Use Script(...).get_names instead.  all_scopes=True,
t.log @ console.ts:137
$logExtensionHostMessage @ mainThreadConsole.ts:39
_doInvokeHandler @ rpcProtocol.ts:398
_invokeHandler @ rpcProtocol.ts:383
_receiveRequest @ rpcProtocol.ts:299
_receiveOneMessage @ rpcProtocol.ts:226
(anonymous) @ rpcProtocol.ts:101
fire @ event.ts:587
fire @ ipc.net.ts:453
_receiveMessage @ ipc.net.ts:733
(anonymous) @ ipc.net.ts:592
fire @ event.ts:587
acceptChunk @ ipc.net.ts:239
(anonymous) @ ipc.net.ts:200
t @ ipc.net.ts:28
emit @ events.js:203
addChunk @ _stream_readable.js:295
readableAddChunk @ _stream_readable.js:276
Readable.push @ _stream_readable.js:210
onStreamRead @ internal/stream_base_commons.js:166

I see also a huge amount of messages saying that the extension is searching for interpreters

[Extension Host] Info Python Extension: 2020-03-30 12:52:37: Get Interpreters in CacheableLocatorService, Class name = m, completed in 24ms, has a truthy return value, Arg 1: <Uri:/import/projects/cfd/Matthieu/Workspace/oplib/oplib/taskmanager/OPRunManager.py>, Return Value: []

My sysadmin on the server host told me that when doing a trace on the SSH traffic there was 200k messages in a few minutes and my VPN client says that there are GB of data exchanged which surprises us.

[kimadeline]

Hi Matthieu Melot (@matmel),

Thank you for reaching out. I have a couple of questions for you so that we can try narrowing down this issue:

• When you say that you use conda, you mean on the remote machine?
• If you open a non-Python file on your remote (so the Python extension doesn't get activated), do you still get an important volume of messages?
• Can you reproduce this issue when creating a folder and adding a python file to it, instead of opening existing projects?
• What is the output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)?

Thanks!

[matmel]

• When you say that you use conda, you mean on the remote machine?

Yes. I also use it on the local machine but I think it is irrelevant here.

• If you open a non-Python file on your remote (so the Python extension doesn't get activated), do you still get an important volume of messages?

No the volume of messages is much much reduced in that case.

• Can you reproduce this issue when creating a folder and adding a python file to it, instead of opening existing projects?

Yes, same problem in a single empty folder workspace with a one liner python script:

$ cat foo.py
from pathlib import Path

• What is the output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)?

Empty in the case of my big workspace. In the small one of the previous test:

User belongs to experiment group 'ShowPlayIcon - start'
User belongs to experiment group 'ShowExtensionSurveyPrompt - enabled'
User belongs to experiment group 'DebugAdapterFactory - experiment'
User belongs to experiment group 'PtvsdWheels37 - experiment'
User belongs to experiment group 'Reload - experiment'
User belongs to experiment group 'UseTerminalToGetActivatedEnvVars - control'
User belongs to experiment group 'AA_testing - control'
User belongs to experiment group 'WebHostNotebook - control'
User belongs to experiment group 'CollectLSRequestTiming - control'
User belongs to experiment group 'CollectNodeLSRequestTiming - experiment'
> pyenv root
> python3.7 -c "import sys;print(sys.executable)"
> python3.6 -c "import sys;print(sys.executable)"
> python3 -c "import sys;print(sys.executable)"
> python2 -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> /import/common/conda/bin/conda info --json
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"

and it is frozen here.

I can add that the amount of network messages seems similar to the big workspace. Just to give you an idea, here is the output of a tcpdump filtered with my laptop ip and ssh traffic only (I have no other ssh traffic on this host except for vscode)

I also tried to downgrade the python extension on the host 2020.1.xxxxxx, because this problem seems new. Unfortunately it doesn't help. Maybe it is due to the recent upgrades of the remote ssh extension or because we are soliciting the VPN more in those troubled times?

[matmel]

Just a random thought. I read the python extension changelog, and I've seen this:

Functional tests using real jupyter can take 30-90 seconds each. Most of this time is searching for interpreters. Cache the interpreter search.

I understand this is not my situation here (I don't use jupyter nor do I try to launch your test suite), but for your information,

/import/common/conda/bin/conda info --json

lists 79 environments when I tried it myself interactively. Most of them correspond to "released" tools and created with

conda create -p path/to/released/tool/directory/v.x.y.z

[DonJayamanne]

Does VSCode load properly When the the python extension is disabled.

Functional tests using real jupyter can take 30-90 seconds each. Most of this time is searching for interpreters. Cache the interpreter search.

Thanks however this only applies when running tests (part of CI).

[matmel]

Does VSCode load properly When the the python extension is disabled.

Yes as fast as usual. And I can edit easily files afterwards. Whereas when I am trying to even save a python file when the extension is loading indefinitely, the actual save can happen minutes later.

[EntilZha]

For what its worth, I ran into this issue yesterday, but don't know the best way to provide information to help debug/fix. I'm willing to provide more information if you let me know what would help

Anecdotally, I noticed when I was editing/saving a jupyter file that I would get a network flood (I have network IO shown on my top bar). Locally I'm running arch linux with most recent version of VS Code, remotely I am on a redhat system. The Redhat system has base python, a python that I compiled with linux brew, and anaconda python that has ~10 virtual environments.

[matmel]

As Pedro Rodriguez (@EntilZha) said, is there anything we can do to help you debug this? I am thinking about the messages that the client and the server exchange? One commonality Pedro Rodriguez (@EntilZha) and I seem to have, is that the remote system is a RedHat / CentOS machine.

[kimadeline]

Since Pedro Rodriguez (@EntilZha) said that the increased network load was happening when saving a jupyter file, the root causes of your issues might be different.

There are a couple of things either of you could try so we can narrow down the issue:

• See if you can reproduce this issue on a remote machine with a different OS;
• Downgrading the VS Code remote extension (you said that downgrading the Python extension didn't help);
• Disabling telemetry;
• The messages being exchanged, as you suggested. doesn't seem possible