Setting python.experiments.optInto gives confusing hover information

[ldorigo]

Issue Type: Bug

This is a minor issue, but when opening settings.json, the "python.experiments.optInto" has the following hover text:

"List of experiment to opt into. If empty, user is assigned the default experiment groups. See https://github.com/microsoft/vscode-python/wiki/Experiments for more details."

That page (https://github.com/microsoft/vscode-python/wiki/Experiments) has a large title indicating that it's deprecated.

Overal, it's confusing whether it's possible to opt in/out of experiments, and if experiments set with that setting are deprecated, that hover message should probably be updated to show this information.

Extension version: 2021.2.636928669
VS Code version: Code 1.54.1 (f30a9b73e8ffc278e71575118b6bf568f04587c8, 2021-03-04T22:38:50.094Z)
OS version: Linux x64 5.10.19-1-lts

System Info

Item	Value
CPUs	AMD Ryzen 7 3700X 8-Core Processor (16 x 4226)
GPU Status	2d_canvas: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: disabled_off opengl: enabled_on protected_video_decode: unavailable_off rasterization: disabled_software skia_renderer: enabled_on video_decode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled
Load (avg)	1, 2, 2
Memory (System)	31.36GB (2.52GB free)
Process Argv	--verbose --no-sandbox --crash-reporter-id 66ea3ec9-05eb-4edd-8d35-34bbe6c1aba1
Screen Reader	no
VM	0%
DESKTOP_SESSION	/usr/local/share/xsessions/plasma-i3
XDG_CURRENT_DESKTOP	KDE
XDG_SESSION_DESKTOP	KDE-i3
XDG_SESSION_TYPE	x11

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383cf:30185419
pythonvspyt700cf:30270857
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstry244:30256636
pythonvsdeb440:30248342
pythonvsded773:30248341
pythonvspyt875:30259475
dockersubset:30270708
vspre833:30267464

[ldorigo]

Alternatively, or additionally, the page at https://github.com/microsoft/vscode-python/wiki/Experiments should make it clear what the current way is for users to opt in/out of experiments

[luabud]

Thanks for bringing this up, Luca Dorigo (@ldorigo)! I agree that the current link is confusing. I'm considering either removing that line, or linking to this following page: https://github.com/microsoft/vscode-python/wiki/AB-Experiments.

The problem is that we change the experiments often enough that it would be hard to manually maintain the list of the active experiments. The idea is that the auto completions in the settings.json file should help with knowing which values are accepted or not.

[ldorigo]

The new page you linked is definitely better. I'm not sure I understand your second paragraph - do you mean that it's hard to update the wiki, or to update the autocompletions? I'm not sure how experiments are implemented, but I guess that when they are setup, there's probably some metadata attached to them (what they are testing, what the changes are). I think it would be nice to have a bit more transparency for the end user - to know which experiments are enabled/available, what they do, etc.

[luabud]

I meant it's tricky to update the wiki as it would have to be a manual sourcing process (we have experiments that are run cross-teams, so there isn't necessarily one person with the whole information to be able to update it).

I like your idea though, I'm marking this issue as "needs decision" so I can talk to the team about it 😊 thanks!

[luabud]

I talked to the team the overall thought was that cost to get the information of the experiments to expose it is quite high, whether it's programmatically or not, and we have many other higher priority bugs and features to address first. But I'm submitting a PR to update the link to this wiki page instead, to try to help clarify the confusion a bit, at least.