Support single-file distribution

[morganbr]

This issue tracks progress on the .NET Core 3.0 single-file distribution feature.
Here's the design doc and staging plan for the feature.

[mattwarren]

Out of interest, how does this initiative compare to CoreRT? They seem like similar efforts?

Is it related to 'possibly native user code', i.e. it this will still allow code to be JIT-compiled, not just AOT?

Also, I assume that the runtime components ('Native code (runtime, host, native portions of the framework..') will be the ones from the CoreCLR repo?

[morganbr]

You're asking great questions, but since this is still early in design, I don't have great answers yet.

Out of interest, how does this initiative compare to CoreRT? They seem like similar efforts?

There would likely be somewhat similar outcomes (a single file), but the design may have different performance characteristics or features that do/don't work. For example, a possible design could be to essentially concatenate all of the files in a .NET Core self-contained application into a single file. That's 10s of MB and might start more slowly, but on the other hand, it would allow the full capabilities of CoreCLR, including loading plugins, reflection emit and advanced diagnostics. CoreRT could be considered the other end of the spectrum -- it's single-digit MB and has a very fast startup time, but by not having a JIT, it can't load plugins or use reflection emit and build time is slower than most .NET devs are used to. It currently has a few other limitations that could get better over time, but might not be better by .NET Core 3.0 (possibly requiring annotations for reflection, missing some interop scenarios, limited diagnostics on Linux). There are also ideas somewhere between the two. If folks have tradeoffs they'd like to make/avoid, we'd be curious to hear about them.

Is it related to 'possibly native user code', i.e. it this will still allow code to be JIT-compiled, not just AOT?

By "native user code," I meant that your app might have some C++ native code (either written by you or a 3rd-party component). There might be limits on what we can do with that code -- if it's compiled into a .dll, the only way to run it is off of disk; if it's a .lib, it might be possible to link it in, but that brings in other complications.

Also, I assume that the runtime components ('Native code (runtime, host, native portions of the framework..') will be the ones from the CoreCLR repo?

Based on everything above, we'll figure out which repos are involved. "Native portions of the framework" would include CoreFX native files like ClrCompression and the Unix PAL.

[ayende]

A single file distribution in this manner, even if has slightly slower startup time, can be invaluable for ease of deployment. I would much rather have the ability to have the full power than be forced to give up some of that.

Some scenarios that are of interest to us. How would this work in terms of cross platform?
I assume we'll have a separate "file" per platform?

With regards to native code, how would I be able to choose different native components based on the platform?

[TheBlueSky]

Some scenarios that are of interest to us. How would this work in terms of cross platform?
I assume we'll have a separate "file" per platform?
With regards to native code, how would I be able to choose different native components based on the platform?

@ayende, I'm quoting from @morganbr comment:

a possible design could be to essentially concatenate all of the files in a .NET Core self-contained application into a single file.

The current cross-platform story for self-contained applications is creating a deployment package per platform that you'd like to target, because you ship the application with the runtime, which is a platform-specific.

[mattwarren]

@morganbr I appreciate you taking to time to provide such a detailed answer

I'll be interested to see where the design goes, this is a really interesting initiative

[morganbr]

I have a few questions for folks who'd like to use single-file. Your answers will help us narrow our options:

1. What kind of app would you be likely to use it with? (e.g. WPF on Windows? ASP.NET in a Linux Docker container? Something else?)
2. Does your app include (non-.NET) C++/native code?
3. Would your app load plugins or other external dlls that you didn't originally include in your app build?
4. Are you willing to rebuild and redistribute your app to incorporate security fixes?
5. Would you use it if your app started 200-500 ms more slowly? What about 5 seconds?
6. What's the largest size you'd consider acceptable for your app? 5 MB? 10? 20? 50? 75? 100?
7. Would you accept a longer release build time to optimize size and/or startup time? What's the longest you'd accept? 15 seconds? 30 seconds? 1 minute? 5 minutes?
8. Would you be willing to do extra work if it would cut the size of your app in half?

[tpetrina]

1. Console/UI app on all platforms.
2. Maybe as a third party component.
3. Possibly yes.
4. Yes, especially if there is a simple ClickOnce-like system.
5. Some initial slowdown can be tolerated. Can point 3 help with that?
6. Depends on assets. Hello world should have size on the order of MB.
7. Doesn't matter if it is just production.
8. Like whitelisting reflection stuff? Yes.

[TheBlueSky]

@morganbr, do you think that these questions are better asked to a broader audience; i.e., broader that people who know about this GitHub issue?

[benaadams]

For example, a possible design could be to essentially concatenate all of the files in a .NET Core self-contained application into a single file.

Looking at compressing it; or using a compressed file system in the file?

[morganbr]

@tpetrina, thanks! Point 3 covers a couple of design angles:

1. Tree shaking doesn't go well with loading plugins that the tree shaker hasn't seen since it could eliminate code the plugin relies on.
2. CoreRT doesn't currently have a way to load plugins
   Point 5 is more about whether we'd optimize for size or startup time (and how much)
   Point 8, yes I was mostly thinking about reflection stuff

@TheBlueSky, we've contacted other folks as well, but it helps to get input from the passionate folks in the GitHub community.

@benaadams, compression is on the table, but I'm currently thinking of it as orthogonal to the overall design. Light experimentation suggests zipping may get about 50% size reduction at the cost of several seconds of startup time (and build time). To me, that's a radical enough trade-off that if we do it, it should be optional.

[Suchiman]

@morganbr several seconds of startup time when using compression? I find that hard to believe when considering that UPX claims decompression speeds of

~10 MB/sec on an ancient Pentium 133, ~200 MB/sec on an Athlon XP 2000+.

[ayende]

@morganbr, for me the answers are:

1. Service (console app running Kestrel, basically). Running as Windows Service / Linux Daemon or in docker.
2. Yes
3. Yes, typically managed assemblies using AssemblyContext.LoadFrom. These are provided by the end user.
4. Yes, that is expected. In fact, we already bundle the entire framework anyway, so no change from that perspective.
5. As a service, we don't care that much for the startup time. 5 seconds would be reasonable.
6. 75MB is probably the limit. A lot depends on the actual compressed size, since all packages are delivered compressed.
7. For release builds, longer (even much longer) build times are acceptable.
8. Yes, absolutely. Size doesn't matter that much, but smaller is better.

Something that I didn't see mentioned and is very important is the debuggability of this.
I hope that this isn't going to mangle stack traces, and we would want to be able to include pdb files or some sort of debugging symbols.

[ayende]

About compression, take into account the fact that in nearly all cases, the actual delivery mechanism is already compressed.
For example, nuget packages.
Users are also pretty well versed in unzipping things, so that isn't much of an issue.
I think you can do compression on the side.

[morganbr]

Thanks, @ayende! You're right that I should have called out debuggability. I think there are only a few minor ways debugging could be affected:

1. It might not be possible to use Edit and Continue on a single-file (due to needing a way to rebuild and reload the original assembly)
2. The single-file build might produce a PDB or some other files that are required for debugging beyond those that came with your assemblies.
3. If CoreRT is used, it may have some debugging features that get filled in over time (especially on Linux/Mac).

When you say "include pdb files", do you want those inside the single file or just the ability to generate them and hang onto them in case you need to debug the single-file build?