[Single-File] DllImport fails to find native library

[elinor-fung]

.NET SDK 5.0.100-rc.2.20473.8

1. Include a native library in a .NET project's output:

   <ItemGroup>
     <None Include="...">
       <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
     </None>
   </ItemGroup>

2. Add a P/Invoke:

   static void Main(string[] args)
   {
       NativeLib.HelloWorld();
   }
   
   [DllImport("NativeLib")]
   private static extern void HelloWorld();

3. Publish as single file: dotnet publish -p:PublishSingleFile=true ...
   • with different combinations of --self-contained and IncludeNativeLibrariesForSelfExtract
4. Depending on publish configuration, application fails with DllNotFoundException
   • --self-contained true -p:IncludeNativeLibrariesForSelfExtract=false (default)
     • Linux: fails
     • Windows: works - directory with the single-file executable ends up on native search path because the native runtime bits are there
   • --self-contained false -p:IncludeNativeLibrariesForSelfExtract=false
     • Linux: fails
     • Windows: works - default library load logic will look next to the running executable
   • --self-contained true -p:IncludeNativeLibrariesForSelfExtract=true
     • Linux: fails
     • Windows: works - extraction directory ends up on native search path because the native runtime bits are extracted there
   • --self-contained false -p:IncludeNativeLibrariesForSelfExtract=true
     • Linux: fails
     • Windows: fails

This works with 3.1 single-file.
Setting IncludeAllContentForSelfExtract=true (with the fix in #42435) works.

cc @vitek-karas

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[elinor-fung]

Repro project: DllImportTest.zip

Double-checked the single-file design doc - it calls out adding the extraction directory to NATIVE_DLL_SEARCH_DIRECTORIES, but it looks like that wasn't done. That would address the IncludeNativeLibrariesForSelfExtract=true case, but the non-extract case would still need to be handled.

We use the path of the assembly defining the P/Invoke to look next to it by default (unless the user specified a different DllImportSearchPath), but in single-file that path is empty now. Maybe the unmanaged library loading logic needs to be updated to look next to the bundle if the assembly defining the P/Invoke is in the bundle.

I guess we could also add the app's directory to NATIVE_DLL_SEARCH_DIRECTORIES if it is a bundle, but that seems like a bit of a hammer. Actually, doing it in the extraction case seems like a bit of a hammer too, since it takes precedence over other locations (like next to the assembly defining the P/Invoke)

[vitek-karas]

Adding the extraction path to the NATIVE_DLL_SEARCH_DIRECTORIES is I think a good solution. I understand it's not exactly the same as if runtime looks there on its own, but it's close enough. Additionally, I don't see how else we would tell the runtime where to look. We would have to introduce yet another runtime property - which would be weird as it would effectively duplicate the existing one with just a tiny difference in behavior. This is mostly because on its own runtime has no way to tell the path to the extraction directory unless it itself is loaded from there (or some managed assemblies are from there).

I also think that if the app has dependency on a native library which exists in two different locations, both of which runtime knows about, then it's a problem with the app - .NET historically doesn't handle native library versioning in any way. I understand that this would be a "change of behavior", but I think it's OK in this case.

As for finding the native libraries next to the bundle - I'm kind of split on this one:

• On one hand I think it would also be OK to send the bundle path in the NATIVE_DLL_SEARCH_DIRECTORIES, but I do agree that it's sort of a "big hammer".
• I could also see us handling this in the runtime - when we try to look for native dependency on an assembly which is in the bundle, we would look next to the bundle - this makes sense - but requires a runtime change.

So my preferred solution would be to tweak runtime to recognize bundled assemblies and look next to the bundle in that case and for the host to add the extraction directory to NATIVE_DLL_SEARCH_DIRECTORIES (pretty much always it extracts anything).

[vitek-karas]

Note that on Windows we pass the bundle location into NATIVE_DLL_SEARCH_DIRECTORIES - because that's where the native dependencies are located per .deps.json resolution in hostpolicy - for example clrcompression.dll. Not doing that would be rather complicated. So we might as well add it there always in single-file case.