[Bug]: Downloading AL Language extension consistently fails, blocking app compilation

[DF1229]

AL-Go version

9.0

Describe the issue

Not entirely sure if this is even AL-Go related, but I don't know where else to post this. Apologies if this is the wrong place.

We recently made the switch to self-hosted runners, which went pretty smoothly, up until now.
We have various apps, in various repositories. One "monolith" in its own repo, and some modules split into various apps also in their own repo.
The modules repo is fine, everything compiles without issue. The monolith repo is running into an issue where the AL Language extension download times out after the local container has been made.

Both repositories use the exact same settings, aside from the obvious differences in repo name and project name settings.

Expected behavior

Downloading the AL Language extension should succeed without issue

Steps to reproduce

Run any workflow that builds the app

Additional context (logs, screenshots, etc.)

Succesful compilation log extract

     _____                      _ _ _
    / ____|                    (_) (_)
   | |     ___  _ __ ___  _ __  _| |_ _ __   __ _    __ _ _ __  _ __  ___
   | |    / _ \| '_ ` _ \| '_ \| | | | '_ \ / _` |  / _` | '_ \| '_ \/ __|
   | |____ (_) | | | | | | |_) | | | | | | | (_| | | (_| | |_) | |_) \__ \
    \_____\___/|_| |_| |_| .__/|_|_|_|_| |_|\__, |  \__,_| .__/| .__/|___/
                         | |                 __/ |       | |   | |
                         |_|                |___/        |_|   |_|
  
  Using Version 28.0.103.2
  Copying dependencies from D:\runners\al-go-2\_work\AppSource\AppSource\Build\Core (W1)\.buildartifacts\Dependencies to D:\runners\al-go-2\_work\AppSource\AppSource\Build\Core (W1)\.packages
  Creating AppSourceCop.json for validation
  AppSourceCop.json content:
  {
      "mandatoryAffixes":  [
                               "OKE"
                           ]
  }
  
  Compiling D:\runners\al-go-2\_work\AppSource\AppSource\Apps\One-IT Core Module\Main App
  Copying DLLs from C:\Program Files\Microsoft Dynamics NAV\280\Service to assemblyProbingPath
  Removing dotnet Framework Assemblies
...
...

Failed compilation log extract

     _____                      _ _ _
    / ____|                    (_) (_)
   | |     ___  _ __ ___  _ __  _| |_ _ __   __ _    __ _ _ __  _ __  ___
   | |    / _ \| '_ ` _ \| '_ \| | | | '_ \ / _` |  / _` | '_ \| '_ \/ __|
   | |____ (_) | | | | | | |_) | | | | | | | (_| | | (_| | |_) | |_) \__ \
    \_____\___/|_| |_| |_| .__/|_|_|_|_| |_|\__, |  \__,_| .__/| .__/|___/
                         | |                 __/ |       | |   | |
                         |_|                |___/        |_|   |_|
  
  Using Version 27.7.449.2
  Copying dependencies from D:\runners\al-go-1\_work\OKE\OKE\.buildartifacts\Dependencies to D:\runners\al-go-1\_work\OKE\OKE\.packages
  Copying previous apps to packages folder
  Downloading AL Language Extension to D:\BcContainerHelper\alLanguageExtension\17.0.2273547.zip
  Error downloading..., retrying in 4 seconds...
  Error downloading..., retrying in 8 seconds...
  Error downloading..., retrying in 16 seconds...
  Error downloading..., retrying in 32 seconds...
Files in build artifacts folder:
Removing container
  Get Event Log from container
  Getting event log for bc26102462516
  Removing Session bc26102462516
  Removing container bc26102462516
  Removing entries from hosts
  Removing bc26102462516 from container hosts file
  Removing bc26102462516-* from container hosts file
  Removing Desktop shortcuts
  Removing D:\BcContainerHelper\Extensions\bc26102462516
Run-AlPipeline Telemetry Correlation Id: d2e9d5c2-6095-44ed-9ce6-8e5e6b4119dd
Test-BcContainer Telemetry Correlation Id: 3697e9a2-dfba-4b5b-b2e1-ff1df7a6d7dc
Enabling Microsoft telemetry...
Enabling partner telemetry...
Error: Unexpected error when running action. Error Message: Timeout downloading file, StackTrace: at Download-File, C:\ProgramData\BcContainerHelper\6.1.14\BcContainerHelper\Common\Download-File.ps1: line 83 <- at DownloadLatestAlLanguageExtension, C:\ProgramData\BcContainerHelper\6.1.14\BcContainerHelper\HelperFunctions.ps1: line 1322 <- at RunAlTool, C:\ProgramData\BcContainerHelper\6.1.14\BcContainerHelper\HelperFunctions.ps1: line 1339 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.14\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 2336 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.14\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 2334 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.14\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1911 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.14\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1906 <- at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.1.14\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1437 <- at Run-AlPipeline, C:\ProgramData\BcContainerHelper\6.1.14\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1422 <- at <ScriptBlock>, D:\runners\al-go-1\_work\_actions\microsoft\AL-Go-Actions\v9.0\RunPipeline\RunPipeline.ps1: line 491 <- at <ScriptBlock>, D:\runners\al-go-1\_work\_temp\80654844-6de0-4378-b24d-278160014eef.ps1: line 3 <- at <ScriptBlock>, D:\runners\al-go-1\_work\_actions\microsoft\AL-Go-Actions\v9.0\Invoke-AlGoAction.ps1: line 21 <- at <ScriptBlock>, D:\runners\al-go-1\_work\_temp\80654844-6de0-4378-b24d-278160014eef.ps1: line 2 <- at <ScriptBlock>, <No file>: line 1
Error: Process completed with exit code 1.

[aholstrup1]

it's hard to say what's going wrong from these logs alone. One thing I notice is that the successful log doesn't have this in it:

Copying previous apps to packages folder
Downloading AL Language Extension to D:\BcContainerHelper\alLanguageExtension\17.0.2273547.zip

So it looks like the successful workflow isn't hitting that part of the code that downloads the AL Language Extension. A couple of questions that might help narrow it down:

• Are the workflows failing consistently on this? Or is it a transient thing?
• Did this used to work before you set up your self-hosted runners? Does it work if you try running the workflow on a GitHub runner?

To rule out any network issues on your self-hosted runners, could you try logging into your runner and executing this:

Import-Module BcContainerHelper
$url = Get-LatestAlLanguageExtensionUrl
Download-File -sourceUrl $url -destinationFile "$env:TEMP\al.vsix" -Description "AL Language Extension"

[DF1229]

Thanks for your reply Alexander! I fear this might be a networking issue, but currently can't 100% confirm this as I am on the same network as the server hosting the runners, until later today.

• The failures are consistent, but only for one of our repositories.
• This used to work just fine on Github runners, and when switching back to them also works fine again.
• Downloading with the provided script fails with the same time-out

Do you have any idea why the download would only happen for one repository? All our builds are currently running on a cached 28.1.49838.50586 w1 version.

[aholstrup1]

I think it could be due to one of two reasons:

1. Your monolith repository is the only repository that has a (GitHub) release created with the "Create Release" workflow.
2. Your monolith repository is the only repository that has skipUpgrade set to false

I'm guessing it's probably 1 but I can't know for sure without seeing more of the logs :)

You're hitting this part of the code because AL-Go/BCContainerHelper recognizes that there are some "previous apps" i.e. some baseline apps from a previous release that it needs to run upgrade tests against. Somewhere in that part of the code, it tries to download the latest vsix.

You might be able to unblock your repo by setting skipUpgrade to true. Obviously, this would disable some of the validation that runs on your workflows but at least you might be able to get a successful build in that case.

[DF1229]

Interesting, I hadn't even considered that because I know both have releases created by AL-Go 😅

Both repos contain actively published and updated AppSource extensions with previous versions, so I don't see why one would skip this step compared to the other.
I'll just assume this is a networking issue and attempt the skipUpgrade remedy for now, and re-open if we discover this is a bigger underlying issue.

Thanks for the support!

[DF1229]

Found the underlying issue, turns out we had also renamed the AL-Go projects in the non-monolith repository. Locating the previous releases worked just fine, but retrieving the assets returned nothing since the assets still used the old project names, this quietly disabled upgrade testing.

Failed downloads of the AL Language extension are definitely a networking issue, but I am kind of surprised to learn that these are not downloaded from the container. The documentation for the vsixFile setting states:

Determines which version of the AL Language Extension to use for building the apps. This can be:
default to use the AL Language Extension which ships with the Business Central version you are building for

While not explicitly stated, I had assumed that "which ships with the Business Central version you are building for" means it is downloaded from the container.

Is always downloading from external sources not unnecessary overhead?

[aholstrup1]

Compilation will run using the bits from the vsix that ships with the artifact or whatever the vsixFile setting is pointing to. This behavior of downloading the latest vsix is only used one place in Run-ALPipeline as far as I can see. In that case all it is used for is to get the package manifest.

I tend to agree with you though; It should just use the vsix from the artifacts if available. I'm not aware of any reason it wouldn't just use the same vsix file as we do for compilation so it's probably a design flaw / bug in BCContainerHelper that causes this download.

[DF1229]

Gotcha, any chance we can get this addressed? I realize it's not exactly high priority 😄

[aholstrup1]

We'll see what we can do but yeah... We do have a few other things we'd like to look at first 🫤