SDK: Address no-network inner-build restore

[jviau]

The functions SDK for dotnet isolated performs an inner-restore/build/publish to discover and collect WebJobs extensions required to support worker extensions. This works fine for most cases; however, it is a headache when using secure/compliant builds. These secure builds often have network access only during restore phase and our design requires it during build phase. The end result is build failures due to no network access for the inner-build.

After having synced with the dotnet, msbuild, and nuget teams it is not feasible for us to participate in the restore target. We do not have enough information at restore time and rely on targets that are a part of build. The best option we came up with is to provide a tool, probably as part of func cli, which can generate this inner WorkerExtensions.csproj and have it checked in as part of source control. Customers can then include this project as part of their restore phase. This will be a tool only for customers with this strict network requirements.

Work Needed

1. Updated SDK build targets to recognize and use a pre-existing WorkerExtensions.csproj
2. Create a tool to generate said WorkerExtensions.csproj and to point the function worker csproj to this generate project.

Workaround

To workaround this issue today, you can avoid requiring network for restore by pre-restoring the necessary packages ahead of time.

1. Build function app locally
2. Copy WorkerExtension.csproj into your solution and ensure it is part of your builds restore phase
   • Can find the file path from the build logs. Or if using 1.17.3-preview* SDK, the project will be in intermediate output directory (obj) of your function app
   • If using central package management, replace Version with VersionOverride

The goal is to just have those set of PackageReference from WorkerExtension.csproj be restored ahead of time, so those packages are cached locally. This way the inner-build's restore will be 100% cache hits and never access the network.

[morrisonbrett]

Can you please post an example WorkerExtensions.csproj file here that I can add to my solution to work around this issue?

[jviau]

@morrisonbrett that won't work as we have to update our SDK build targets to recognize and use that pre-existing WorkerExtension.csproj

[morrisonbrett]

I'm just looking for a workaround until this is fixed properly. I tried adding a project named WorkerExtension.csproj to my solution, added all the "InProcess" AzureFunctions Nuget packages and even a Unit Test but it didn't work. My main projects keep pulling in their own WorkerExtension.csproj. Not only that, I can't seem to exclude it from my dotnet test and coverage run, so I've had to bump my coverage threshold down to accommodate it.

[jviau]

@morrisonbrett yes the function app will keep generating using its own WorkerExtension.csproj. There is no logic in the targets currently to detect a pre-existing one.

WorkerExtensionStartupCodeExecutor is not related to this project. That is a source-generated file, there are ways to exclude specific files/types from code-coverage. Not sure why this generated file is not automatically excluded.

https://learn.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis?view=vs-2022

[morrisonbrett]

Thanks @jviau. I just tried and it did not work. I already have a coverlet.runsettings file in my project, so I added the <CodeCoverage><ModulePaths> section to it per the article referenced above. Same result, the coverage run is still including it.

Updated .runsettings file:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat Code Coverage">
        <Configuration>
          <Format>cobertura,opencover,lcov</Format>          
          <Exclude>[xunit*]\*</Exclude>
          <SingleHit>false</SingleHit>
          <UseSourceLink>true</UseSourceLink>
          <IncludeTestAssembly>false</IncludeTestAssembly>
          <SkipAutoProps>true</SkipAutoProps>
          <CodeCoverage>
            <ModulePaths>
              <Exclude>
                <ModulePath>*.WorkerExtensionStartupCodeExecutor</ModulePath>
              </Exclude>
            </ModulePaths>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

[EdwinOtten]

Hi there! I ran into problems because of this "inner-build restore", I'm sharing this info because it might save other people some time debugging.

The issue occured when a team member added the Microsoft.Azure.Functions.Worker.Extensions.DurableTask package to our project. Everything worked on their machine, but when I pulled their changes and attempted to build our project, it tried to restore WorkerExtensions.csproj. Which failed to resolve Microsoft.Azure.WebJobs.Extensions.DurableTask . Resulting in the build output below:

Determining projects to restore...
C:\Users\MyUser\AppData\Local\Temp\gt02a21z.5m5\WorkerExtensions.csproj : error NU1100: Unable to resolve 'Microsoft.Azure.WebJobs.Extensions.DurableTask (>= 2.13.0)' for 'net6.0'.
Failed to restore C:\Users\MyUser\AppData\Local\Temp\gt02a21z.5m5\WorkerExtensions.csproj (in 1.24 sec).
Done building project "WorkerExtensions.csproj" -- FAILED.
========== Build: 3 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

My workaround was:

1. Open the WorkerExtensions.csproj mentioned in the build output in Visual Studio
2. Right-click on the project and open the NuGet Package Manager
3. Look at the Package source, in my case there was none (this was the culprit)
   
4. Add NuGet v3 package source: https://api.nuget.org/v3/index.json
   
5. Right-click on the solution and click Restore packages
6. Now you should be able to build your project

So the problem for me was that WorkerExtensions.csproj in my local AppData didn't use the NuGet v3 package feed. I was under the impression that Visual Studio would be configured to use that by one by default. But apparently not?
I'm curious why I experienced this issue, but my team members using Rider did not.

[jviau]

@EdwinOtten the WorkerExtensions.csproj will not inherit Visual Studios restore settings. VS restore settings only apply to projects listed directly in the solution file, this project is not part of that.

Do you have a global nuget.config? Check %appdata%\NuGet\NuGet.Config and %ProgramFiles(x86)%\NuGet\Config. This might be removing nuget v3 feed.

[EdwinOtten]

@jviau thanks for clarifying! 👍 I checked both:

• %appdata%\NuGet\NuGet.Config adds 2 sources (Microsoft Visual Studio Offline Packages and NuGet V3). The latter wasn't there when I first experienced the issue, but was added when I added it via the Package Manager GUI in VS2022.
• %ProgramFiles(x86)%\NuGet\Config adds 1 source (Microsoft Visual Studio Offline Packages)

FYI: I'm running Visual Studio 2022 Professional.

[alayala-MSFT]

Hi @jviau. In this thread you mention that the workaround in the meantime is to create a dummy project as mentioned here.

One piece of data I'd like to add is that this can workaround can still cause issues with staying compliant if the organization requires upgrades of newer packages that the WorkerExtensions.csproj project references.

For example, the below is a WorkerExtensions.csproj that is generated when using the latest (5.2.0) Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues.

This will generate <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage.Queues" Version="5.1.3" /> below which transitively relies on Azure.Identity >= 1.7.0.

The restore will take this version of Azure.Identity which is known to have a high severity vulnerability: https://www.nuget.org/packages/Azure.Identity/1.7.0

However, we cannot manually override this package for WorkerExtensions.csproj to use a newer version, so we're forced to use a version with a vulnerability.

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <TargetFramework>net6.0</TargetFramework>
        <LangVersion>preview</LangVersion>
        <Configuration>Release</Configuration>
        <AssemblyName>Microsoft.Azure.Functions.Worker.Extensions</AssemblyName>
        <RootNamespace>Microsoft.Azure.Functions.Worker.Extensions</RootNamespace>
        <MajorMinorProductVersion>1.0</MajorMinorProductVersion>
        <Version>$(MajorMinorProductVersion).0</Version>
        <AssemblyVersion>$(MajorMinorProductVersion).0.0</AssemblyVersion>
        <FileVersion>$(Version)</FileVersion>
        <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    </PropertyGroup>
    <ItemGroup>
        <PackageReference Include="Microsoft.NETCore.Targets" Version="3.0.0" PrivateAssets="all" />
        <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.2.0" />
        <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage.Queues" Version="5.1.3" />

    </ItemGroup>
</Project>