Program 'winget.exe' failed to run: The file cannot be accessed by the system.At line:1 char:1 [Running Virtual Machine via SSH]

[ranavale]

Brief description of your issue

I created a VM and I installed winget using latest release on its github page, then I connected to my VM from VSCode SSH remote connection. I am trying to run the winget from my local remote-VSCode (connected to VM via SSH).
Program 'winget.exe' failed to run: The file cannot be accessed by the system. At line:1 char:1 [Running Virtual Machine via SSH]

Steps to reproduce

run comment "winget"

Expected behavior

winget to work
This is output when I login to my VM and run winget

Actual behavior

Program 'winget.exe' failed to run: The file cannot be accessed by the system. At line:1 char:1 [Running Virtual Machine via SSH]

Environment

Winget Package

Windows Package Manager v1.0.11692
Copyright (c) Microsoft Corporation. All rights reserved.

Windows specifications:

Edition: Windows 10 Enterprise for Virtual Desktops
Version: 21H1
OS Build: 19043.1237

App Installer

Version: 1.12.11692.0

[jedieaston]

This is a similar issue to #995.

If it helps the team narrow it down, it doesn't just happen with winget.exe, but with any App Execution Alias (Windows Terminal registers wt.exe and it doesn't work either). It happens over PowerShell Remoting and SSH (via the Windows built-in OpenSSH Server), and it happens even if winget works completely fine on the local system from a shell. It also happens if you are running Windows inside a docker container and attempting to execute winget via docker exec.

This is probably not a winget problem then, but @denelon can you ask whoever manages the App Execution Aliases stuff to look into it? It basically blocks using winget remotely unless you get creative.

Proofsies:

PS E:\> Enter-LabPSSession wingettest

[wingettest]: PS C:\Users\Administrator\Documents> whoami 
wingettest\administrator

[wingettest]: PS C:\Users\Administrator\Documents> winget --info
Program 'winget.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> python3
Program 'python3.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> python
Program 'python.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> wt 
Program 'wt.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> # Aw, beans!

[JohnMcPMS]

@jedieaston , my guess is that this is because PowerShell is attempting to read the file before launching it. The way that AEA is implemented in the OS makes the shells not work if they attempt to read it, since it isn't the actual file (nor any kind of file really).

There are a few solutions:

1. We change a few things about the winget package so that the actual binary is launchable directly. As I recall, there was some work done in more recent Windows builds to enable that. That would possibly enable a workaround that an Invoke-WinGet PowerShell function could be written to handle. All speculative on my part; only helps those who seek out the solution but I imagine this type of remoting falls into "people who seek out solutions" territory anyway. Only works on "newer" (TBD) OS.
2. PowerShell (and docker, and anyone else) updates to not attempt to read from the "executable" or at least handle a failure to do so by simply executing it. Requires multiple teams to do work; will keep coming up.
3. The OS is updated to allow the file to be read through the AEA. Only works in some future timeframe, but would hopefully solve the issue for everything. I think might actually be possible (I was party to the original implementation but am no longer on the owning team). Requires getting that work approved, and it is not a bug level change.

Our changes for 1 are actually pretty easy, but again it would only (potentially) make something like this work:

C:\Program Files\WindowsApps\Microsoft.DesktopAppInstaller_1.16.12581.0_x64__8wekyb3d8bbwe> winget.exe install foo

And not the default path winget command.

[jedieaston]

Obviously 3 would be wonderful, but if we have to settle for 1 that’d still be good enough to get winget working for now in these scenarios.

By TBD, do you mean that the OS changes necessary for 1 aren’t in stable yet? The only thing I know of that rings a bell with that is Invoke-CommandInDesktopPackage and that doesn’t seem to work either.

[JohnMcPMS]

Agreed on 3, I think there is a case for it but if it is going to take tens of hours of work then there needs to be a justification that meets the bar. Need to ensure that it is actually the problem first, then we can put it on the backlog. But even if it was a go today, I don't think we would see it released for at least a year and then it wouldn't be serviced back to older Windows versions. We need the short term solution too.

By TBD I meant more "to be looked up because I remember seeing the PR but I have no idea when it was released, or if it even has been yet". This was work specifically to enable creating the process directly on the package file in certain instances. Normally packaged processes need to be launched by special API to enable all of packagy goodness (don't @ me), but this provided the ability to CreateProcess against the file and have it invoke the special API for you.

ah, I see everyone is busy with App Execution Aliases , but that's not the issue.
the issue is MSIX packaged apps. MSIX packages can't be run over SSH or WinRM , even worse one user can't launch the app as another user. MSIX doesn't support per machine class registration.

[jedieaston]

But if I SSH as myself into the system, the MSIX apps I've installed as myself work. The AEAs don't though.

TL;DR : win32 OpenSSH port might want to implement AEAs/NTFS reparse points handling mechanism. I believe, @JohnMcPMS you as a MS employee, your words/winget teams words will have more weight to the win32-openssh team than ours. people even have pointed out where the fix might be needed : https://github.com/PowerShell/openssh-portable/blob/0b73c4636d38f0c69424721d52d0e7752db99c81/contrib/win32/win32compat/fileio.c#L1124

Does it ? 🤔
No MSIX (or Appx) packages regardless of installation medium (loose appxmanifest.xml installation or through AppInstaller installation or directly through Store Installation) run over SSH, here on my machine.

here's a screenshot of me from a non-elevated shell sshing to myself and trying to launch a MSIX package over ssh (eg: winget)

crap, let me dig a little

Update: same result regardless of specified locations or AEAs. It turns out AEA is the MSIX Problem, which could be anything among Symbolik Link/Junction/Mount Point/Reparse Point. is indeed a Special NTFS Reparse Point. specified path(%program files%\windowsapps\CrapFolder\crap.exe launches fine over SSH)

[Masamune3210]

Here you are trying to use a AEA, proving the point. Winget should work if you drill down into the correct package location and find the specific exe

Here you are trying to use a AEA

yeah, that's right ~\appdata\local\microsoft\windowsapps is the AEA folder.

[Masamune3210]

Oooh i see, I didnt realize that AEA's were plopped in a specific folder, figured they were just namespace links or some other magic. Forgive the comment

[jedieaston]

@denelon Do you know how winget is executed during the Installation Validation step of the winget-pkgs pipeline? Presumably it isn't via PowerShell Remoting or SSH due to this bug.

[JohnMcPMS]

@mshaikhcool , do you have an issue that I could comment on? I don't really understand why these remoting shell type operations need to read the target "exe", but I can certainly comment on what could be done in this case. I'm not sure that the internal data of the AEA reparse point are public, but the type certainly is (IO_REPARSE_TAG_APPEXECLINK).

@jedieaston , @chuckferring would know for sure but I think it is part of a script run on the machine as remoting would not be allowed for security reasons.