Enable extensions.json

[colin-grant-work]

What it does

Fixes #8982 by adding support for extensions.json to the preference system.
Also fixes #9034.
Also fixes #8070 (for the @recommended case).

I'd like to see a real solution to #9255, but this PR just creates overrides to be used with this package that adds some extra rules for determining whether a preference should be saved in extensions.json or settings.json.

Here are the features of VSCode's extensions.json system that I'm aware of. Feel free to mention or add any I've missed:

• extensions.json recognized by preference system.
  • supports recommendations and anti-recommendations (at the moment Theia doesn't have default recommendations that need to be overridden, but downstream adopters might)
• Recommended widget appears in plugin view.
  • Installed plugins don't appear in the Recommended widget.
• Toast when recommended extensions detected in the workspace.
  • Preference to ignore the toast. (The VSCode preference has the same extensions prefix as the name of the file, which will be a bit of a challenge for our preference providers to handle, since they assume that a section name (extensions, tasks, etc.) should not be used as a prefix in the settings object.)
• Support for @recommended query in Plugin View search bar
  • Command to show the recommended plugins.
  • With distinction between workspace and default recommendations

How to test

1. Open a workspace (single or multiroot).
2. Create an extensions.json file in a .theia or .vscode folder under the/a root.
3. Add some extensions inside the recommendations array of the following structure: {"recommendations": []}
4. Observe that the schema warns if your extensions don't follow the format <alphanumeric or - >.<alphanumeric or - > (with - barred from first position in both alphanumeric groups)
5. Open the plugins view.
6. Observe that a widget with the heading Recommended is present. (May require resetting workbench layout.)
7. Observe that if you add any real plugins to the JSON, they appear in that widget, unless they're already installed (you can check in the Installed widget.
8. Of the plugins you've added to recommendations, add a few to a new array labeled unwantedRecommendations (leaving them in recommendations, as well). Observe that any listed under unwantedRecommendations do not appear in the Recommended widget.
9. You should be able to add an extensions.json file to each root of a multi-root workspace, and the Recommended widget should aggregate and display (all recommendations) - (all anti-recommendations). That is, an anti-recommendation anywhere will override a recommendation anywhere, even if they aren't the same folder.
10. It should be possible to add an extensions object with the same schema to the settings object of a workspace file (after Enable Workspace-Scoped Tasks #8917, it will be possible to put extensions outside the settings object.)
11. Open the preferences UI and modify the value of extensions.ignoreRecommendations. Observe that the value is set in a settings.json file, not an extensions.json file.

Review checklist

• as an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• as a reviewer, I agree to behave in accordance with the review guidelines

[colin-grant-work]

@Kreyren, I should have some time to circle back to this shortly. Would you be willing to let me know which features listed above (or not listed above) you'd like to see in the initial version of this functionality?

[colin-grant-work]

@vince-fugnitto, since you've been active in this area, I wonder which of the features mentioned above - or others not mentioned - you'd like to see implemented as part of this PR. I tried reaching out to the original issue creator, but haven't gotten any response.

[vince-fugnitto]

@vince-fugnitto, since you've been active in this area, I wonder which of the features mentioned above - or others not mentioned - you'd like to see implemented as part of this PR. I tried reaching out to the original issue creator, but haven't gotten any response.

@colin-grant-work I think what you’ve listed is already great, maybe only thing we could add would be the toast which is the main way I ever installed recommended extensions (although this is subjective). I’d also be fine if it is omitted in the initial pull-request :)

[colin-grant-work]

A lot of code has been rewritten here. The substantive change is that after trying the files matching the section name (e.g. extensions.json if the preference starts with 'extensions'), we fall back to settings.json if the change has been rejected. This allows the provider for extensions.json to reject changes that contain the prefix extensions but are not in the schema for that file (e.g. extensions.ignoreRecommendations).

There is also a stylistic change in the setup of the final iterators. The code on master uses an array containing functions that either return a provider or add another function to the end of the array. The conditions for returning the provider are increasingly broad with each subsequent function, so it was effectively a prioritization mechanism that avoided a sort call.

The new code follows a similar principle, but instead of gradually adding to the same array, it creates a number of prioritized buckets, guaranteeing that we only consider each candidate once after it has been prioritized, and, I hope, clarifying the precedence order of the various conditions.

[colin-grant-work]

This is also rewritten to allow us to fall back to settings.json if the change is rejected by section providers (e.g. extensions.json). There isn't any real change to the other operations.

[colin-grant-work]

@vince-fugnitto, in the end, to address the extensions section name vs. extensions section prefix, I've created overrides for the preference providers that are sensitive to the schema for the preferences file. It required a little bit of change elsewhere (described in my comments above), but it doesn't require a major reorganization of the preference system or deprecation of existing preference setting mechanisms.

[colin-grant-work]

@vince-fugnitto, would you be willing to take a look at this for me? I think you've tested it out previously, and I've since implemented a solution to the preference naming issue, so I think it's in working order.

[vince-fugnitto]

I verified the changes under different use-cases, and it looks very good.
I have a few improvements in mind:

1. Possibly re-ordering the tree parts in the extensions-view - we might want to keep recommendations the highest (so they are discoverable and align with vscode), while builtins should be last.

2. Minor: Should we collapse 'recommended' - should the recommended tree-part be collapsed if no recommendations are present?

3. More user-friendly problem marker for invalid extensions - when an invalid format is present in the extensions.json for an extension, we display a message, albeit not so user friendly compared to vscode:

theia:

String does not match the pattern of "^\w[\w-]+\.\w[\w-]+$".

vscode:

Expected format '${publisher}.${name}'. Example: 'vscode.csharp'.

4. I've had some issues where the recommended stops displaying in the browser after a reload, I suspect it might have to do with cors issues I reported in the past: (eclipse/openvsx#289).

5. improvement (can be omitted initially): toast when an extension cannot be found (at the moment there is an error log regarding refreshing):

theia:

root ERROR [dbaumer.vscode-eslint]: failed to refresh, reason: Error: Extension with id dbaumer.vscode-eslint not found at https://open-vsx.org/api
    at VSXRegistryAPI.<anonymous> (file:///home/evinfug/workspaces/theia/examples/electron/lib/71.bundle.js:4049:31)
    at step (file:///home/evinfug/workspaces/theia/examples/electron/lib/71.bundle.js:3908:23)
    at Object.next (file:///home/evinfug/workspaces/theia/examples/electron/lib/71.bundle.js:3889:53)
    at fulfilled (file:///home/evinfug/workspaces/theia/examples/electron/lib/71.bundle.js:3880:58)

[colin-grant-work]

@vince-fugnitto, thanks for the review. Mostly sounds great, and I've done what could be done easily. Let me know what of the rest you definitely want to see in this PR.

1. Possibly re-ordering the tree parts in the extensions-view - we might want to keep recommendations the highest (so they are discoverable and align with vscode), while builtins should be last.

👍 , done.

2. Minor: Should we collapse 'recommended' - should the recommended tree-part be collapsed if no recommendations are present?

Maybe, but the timing and interaction with state restoration are awkward. The preferences won't settle for a while, but we could wait until the same time that we show the toast and then collapse it if we haven't found any recommendations? Might upset state-restoration purists who deliberately opened it in the last session, and leaving it open because we think there are going to be plugins to populate it with could wind up failing if the plugins just don't load as you describe below. Happy to go either way.

3. More user-friendly problem marker for invalid extensions - when an invalid format is present in the extensions.json for an extension, we display a message, albeit not so user friendly compared to vscode:

👍, done.

1. I've had some issues where the recommended stops displaying in the browser after a reload, I suspect it might have to do with cors issues I reported in the past: (eclipse/openvsx#289).
2. improvement (can be omitted initially): toast when an extension cannot be found (at the moment there is an error log regarding refreshing):

Seems like we could use better error handling in the loading process more generally? Most of the time, when I do a search, the progress bar just keeps going forever, so there's also something odd going on there. I'm willing to take a look at smoothing things out, but I might be inclined to do it in a separate PR, if it's about interacting with the VSX registry generally, rather than the functionality here.

[vince-fugnitto]

I verified the functionality and confirmed the following 👍

• the extensions.json file is recognized when under .theia.
• the extensions.json file is recognized when under .vscode.
• the extensions.json schema correctly identifies when the id for the extension is not formatted correctly following ({publisher}.{name})
• users are prompted with a user-friendly warning when they have an invalid extension id.
• extensions from the extensions.json are correctly displayed, and placed under the recommended tree-view part.
• extensions under recommended are responsive to changes made under extensions.json (additions, deletions, adding under unwantedRecommendations).
• unwantedRecommendations correctly have precedence over recommendations.
• the show recommended action correctly opens the extensions-view with the @recommended prefix.
• the show recommended extensions command correctly opens the extensions-view with the @recommended prefix.
• extensions.ignoreRecommendations set to true will ignore recommendations and hide the notification on startup.
• extensions.ignoreRecommendations set to false will prompt users if they want to install recommended extensions on startup.
• the notification successfully installs extensions when selecting install.
• in a multi-root workspace, the individual extensions.json (potentially under each workspace root) will successfully aggregate all recommended extensions.

[vince-fugnitto]

The changes work well for me 👍 The feedback I had last iteration and discussed with you offline regarding issues when reading from .theia/extensions.json has been resolved along with some minor UI updates.