Single-File: Implement statically linked apphost

[swaroop-sridhar]

In order to support single-file apps, implement the "static-host" host-builds as described here, publish them to be consumed by the SDK.

[swaroop-sridhar]

Adding notes from an offline discussion:

@vitek-karas wrote:
What is the current plan to build the apphost for self-contained single-exe? Specifically the hosting layers:

• We know we need at least some of the hosting code (parsing of .runtimeconfig.json, probably also parsing .deps.json in some cases)
• We know we don’t need all of the code (framework resolution, most of deps processing, lot of the customizations and backward compat stuff)
• We should not have a fork of that code – that will be hell to maintain
• The code lives in a different place than coreclr

@swaroop-sridhar wrote:
In our previous discussion about building the super-host (for self-contained builds), we concluded that:

• In order to support upstream scenarios (ex: AssemblyLoadContext), we’ll start with statically linking all existing host components (apphost, hostfxr, hostpolicy)
• The checks/processes involved should be optimized by removing unnecessary checks (likely for all self-contained builds).
• The deps.json parsing etc code will still be available, they may simply not be run in the startup path.

This is noted in the design here: https://github.com/dotnet/designs/blob/master/accepted/2020/single-file/design.md#host-builds (although I realized not in so many words).

On Linux, we’ll additionally link the CoreCLR components to this host.

What is the plan to ship this new apphost? My understanding is that we plan to have two apphosts:

• The existing one which will be able to handle framework-dependent normal mode, self-contained normal mode and framework-dependent single-exe mode
• The new “superhost” with runtime and other native libs linked in which will handle self-contained single-exe mode.

Yes, for any given target, we’ll have two hosts:
• Windows/Osx: AppHost and StaticHost (host components linked)
• Linux: AppHost and SuperHost (host and runtime linked).

The main difference between apphost and superhost is that currently apphost is tiny and has no real versioning problems. On the other hand the superhost is very tied to a specific version of the runtime/corelib/corefx.
Currently apphost ships as a separate NuGet package which the SDK downloads as needed – I don’t know what are the versioning requirements today – what the SDK uses to decide which version of that package to use.

For a given TFM and RID, the SDK expects a unique apphost package, from which it obtains the apphost to use:
https://github.com/dotnet/sdk/blob/611fcdd2ce0b29abae9967ca8e425db1a4def176/src/Tasks/Microsoft.NET.Build.Tasks/ResolveAppHosts.cs#L71-L90

Will the existing infra work for superhost? Do we want to ship it in the same package as apphost or do we want it to be part of the runtime pack instead? What’s the versioning story?

I think it’s better to publish the SuperHost as its own package -- that’s only downloaded for SingleFilePublish scenarios, again based on TFM, RID, PublishSingleFile, and Self-Contained properties.

The StaticHost fits well with runtime.<arch>.microsoft.netcore.dotnetapphost (along with apphost, comhost, ijwhost, etc), since it only consists of host components. However, we may still want to ship it as a separate pack, because:

• StaticHost is a temporary artifact, till super-host is built for Windows, so staying close to superhost scenario.
• We currently don’t have plans to use static-host for all self-contained builds.
• When using the static-host, in addition to packaging these hosts, we’ll also need more information about required native binaries. Here’s the proposed list of files that are published on Windows: coreclr.dll, clrjit.dll, clrcompression.dll, mscordaccore.dll. This is a small subset of all files SDK currently obtains from runtime.<arch>.microsoft.netcore.app package. So, we need to do one of:
- Annotate FrameworkList in the runtime pack with an additional XML element to indicate the files that should be included in the singe-file scenario.
- Ship the necessary components in a separate package.

A proposal is to build the runtime.<arch>.microsoft.netcore.superhost packs as follows:
Windows/Osx

runtime.win-x64.microsoft.netcore.superhost\<version>\runtimes\win-x64\native\
    superhost.exe (even though this is just hostcomponents)
    coreclr.dll, 
    clrjit.dll, 
    clrcompression.dll, 
    mscordaccore.dll.

Linux

runtime.linux-x64.microsoft.netcore.superhost\<version>\runtimes\win-x64\native\
superhost.exe 

For self-contained single file builds (only), the SDK obtains native components from runtime.<arch>.microsoft.netcore.superhost and managed components from runtime.<arch>.microsoft.netcore.app.

[swaroop-sridhar]

CC: @dsplaisted @wli3

[wli3]

So that means publish self contained vs non self contained and Windows VS Linux will have different host. 1 code path now spitted to 4 code path, including how the package flows.

If so, we also need to double the apphost pack count in core-sdk. We used to have 8 apphost pack, and now it will be 16?

All of them can be done, but we need to watch out the infra load, and we may need more time to design it. Also, can they be stored in the same package? apphost and superhost? May reduce some of the issues.

[jkotas]

Also, can they be stored in the same package

I agree that it would make sense to have both hosts in the same package.

I do not think we should be calling this superhost. It does not say anything about what it is. This can be described netcoreapp single file host. I would call it netcoreapp host or single file host, but it would be a good idea for others to chime in.

[swaroop-sridhar]

So that means publish self contained vs non self contained and Windows VS Linux will have different host. 1 code path now spitted to 4 code path, including how the package flows.

@wli3 the change will be for self-contained single-file publish vs all other paths.
I am hoping this will be two paths instead of 4:

• PublishSingleFile self-contained will obtain native components and Apphost from a different pack
• All other configurations will proceed as normal
  This is regardless of whether the hosts live in the same pack or two.

If so, we also need to double the apphost pack count in core-sdk. We used to have 8 apphost pack, and now it will be 16?
All of them can be done, but we need to watch out the infra load, and we may need more time to design it. Also, can they be stored in the same package? apphost and superhost? May reduce some of the issues.

I agree that it would make sense to have both hosts in the same package.

We can configure the new hosts to be set within the existing host packages if that is simpler
This involves:

• Building the host packages versioned based on changes in the Runtime.
• Configuring the SDK to choose the correct host from the same package.
• Annotating the RuntimeList in the runtime pack with an additional attributes to identify the files that should be included in the singe-file scenario.

I do think we should be calling this superhost. It does not say anything about what it is. This can be described netcoreapp single file host. I would call it netcoreapp host or single file host, but it would be a good idea for others to chime in.

I agree. @samsp-msft can you please suggest the proper names for this host?