NullReferenceException calling Microsoft.IdentityModel.Clients.ActiveDirectory.CryptographyHelper.GetCryptoProviderForSha256

[tjrobinson]

On 2018-02-05 we started getting exceptions with our existing Azure Functions when attempting to read from Key Vault. Our code hadn't changed, and it started failing across multiple Azure Functions in our Azure environments.

The Azure Function is precompiled from Visual Studio 2017, targetting .NET 4.6.2.

We're using Microsoft.IdentityModel.Clients.ActiveDirectory 3.14.1.

This could be an issue with the Azure Functions host switching to .NET 4.7: pnp/PnP-Sites-Core#1405

We confirmed that the certificate wass available, key vault itself works, the key is there, etc. All of this was working up until this happened. It only affects our Azure Functions, not Web Apps using the same code/technique to load secrets from Key Vault.

We resolved the issue by switching to Managed Service Identity (https://medium.com/statuscode/getting-key-vault-secrets-in-azure-functions-37620fd20a0b).

2018-02-05T17:28:39.273 A ScriptHost error has occurred
2018-02-05T17:28:39.273 System.NullReferenceException : Object reference not set to an instance of an object.
   at Microsoft.IdentityModel.Clients.ActiveDirectory.CryptographyHelper.GetCryptoProviderForSha256(RSACryptoServiceProvider rsaProvider)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.CryptographyHelper.SignWithCertificate(String message,X509Certificate2 certificate)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.ClientAssertionCertificate.Sign(String message)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.JsonWebToken.Sign(IClientAssertionCertificate credential)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.ClientKey.AddToParameters(IDictionary`2 parameters)
   at async Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.SendTokenRequestAsync()
   at async Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.RunAsync()
   at async Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenForClientCommonAsync(String resource,ClientKey clientKey)
   at async Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenAsync(??)
   at async http://Empactis.HRCM.Data.Store .Cosmos.Factory.CertificateHelper.GetAccessToken(String authority,String resource,String scope)
   at async http://Microsoft.Azure .KeyVault.KeyVaultCredential.PreAuthenticate(Uri url)
   at async http://Microsoft.Azure .KeyVault.KeyVaultCredential.ProcessHttpRequestAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at async http://Microsoft.Azure .KeyVault.KeyVaultClient.GetSecretWithHttpMessagesAsync(String vaultBaseUrl,String secretName,String secretVersion,Dictionary`2 customHeaders,CancellationToken cancellationToken)
   at async http://Microsoft.Azure .KeyVault.KeyVaultClientExtensions.GetSecretAsync(??)
   at http://Empactis.HRCM.Data.Store .Cosmos.Factory.KeyVaultSecretFinder.Find(KeyVaultSecretFinderParams keyVaultSecretFinderParams)
   at http://Empactis.HRCM.Data.Store .Cosmos.Factory.DocumentClientFactory.CreateDocumentClient(ConsistencyLevel consistencyLevel)
   at Empactis.HRCM.AzureFunctions.Common.ExtensionConfigProvider.BuildEntityStore(ISecretService secretService)
   at Empactis.HRCM.AzureFunctions.Common.ExtensionConfigProvider.RegisterServices(IServiceCollection services)
   at Empactis.HRCM.AzureFunctions.Common.ExtensionConfigProvider.Initialize(ExtensionConfigContext context)
   at http://Microsoft.Azure.WebJobs.Host .Executors.JobHostConfigurationExtensions.InvokeExtensionConfigProviders(ExtensionConfigContext context)
   at http://Microsoft.Azure.WebJobs.Host .Executors.JobHostConfigurationExtensions.CreateStaticServices(JobHostConfiguration config)
   at http://Microsoft.Azure .WebJobs.JobHost.InitializeServices()
   at http://Microsoft.Azure .WebJobs.Script.Utility.CreateMetadataProvider(JobHost host) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Utility.cs : 362
   at http://Microsoft.Azure .WebJobs.Script.ScriptHost.Initialize() at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 465
   at http://Microsoft.Azure .WebJobs.Script.ScriptHostManager.RunAndBlock(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHostManager.cs : 181
2018-02-05T17:28:39.273 Stopping Host

[joeizy]

Also experiencing the issue

[davidebbo]

This was fixed a while back, but we forgot to close it.

[tjrobinson]

@davidebbo Thanks. Do you have a link to the fix or know which specific version it was fixed in?

[davidebbo]

This was the fix: Azure/azure-functions-host@7addbee.

But note that the fix applies to running on Azure & IIS, but apparently there are still issues when running locally using the Functions tools (where the web.config does not apply). Is that what you're seeing?

[Raffledoocious]

I am seeing this locally debugging an Azure function on Visual Studio 15.6.5, targeting .NET 4.6.1 and 1.x of the Azure Function runtime. I saw this was fixed but has a version with this fix been released? If this is new I can remake as a new issue.

I can run the same code in a console application and it pulls down key vault secrets fine.

Azure Functions and Web Jobs Tools version: 15.0.40322.0
Microsoft AADL version: 2.28.4

at Microsoft.IdentityModel.Clients.ActiveDirectory.CryptographyHelper.GetCryptoProviderForSha256(RSACryptoServiceProvider rsaProvider)\r\n   
at Microsoft.IdentityModel.Clients.ActiveDirectory.CryptographyHelper.SignWithCertificate(String message, X509Certificate2 x509Certificate)\r\n   
at Microsoft.IdentityModel.Clients.ActiveDirectory.JsonWebToken.Sign(ClientAssertionCertificate credential)\r\n   
at Microsoft.IdentityModel.Clients.ActiveDirectory.RequestParameters.AddClientKey(ClientKey clientKey)\r\n   
at Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.<SendTokenRequestAsync>d__9.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   
at Microsoft.IdentityModel.Clients.ActiveDirectory.AcquireTokenHandlerBase.<RunAsync>d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.<AcquireTokenForClientCommonAsync>d__69.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.<AcquireTokenAsync>d__1d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n  
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\r\n   
at SkyKick.Bcl.Configuration.KeyVault.AuthenticationCallbacks.ClientCertificateAuthenticationCallback.<AuthenticationCallback>d__5.MoveNext() in C:\\Source\\SkyKick.BaseClassLibrary\\SkyKick.Bcl.Configuration.KeyVault\\AuthenticationCallbacks\\ClientCertificateAuthenticationCallback.cs:line 79\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.KeyVault.KeyVaultCredential.<PostAuthenticate>d__4.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   
at Microsoft.Azure.KeyVault.KeyVaultCredential.<ProcessHttpRequestAsync>d__9.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   
at Microsoft.Azure.KeyVault.Internal.SecretOperations.<ListAsync>d__1e.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.KeyVault.KeyVaultClient.<>c__DisplayClassf2.<<GetSecretsAsync>b__f1>d__f4.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.KeyVault.KeyVaultClient.<Do>d__11b`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n  
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.KeyVault.KeyVaultClient.<GetSecretsAsync>d__f8.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\r\n   
at SkyKick.Bcl.Configuration.KeyVault.Abstractions.KeyVaultClient.<GetSecretsAsync>d__2.MoveNext() in C:\\Source\\SkyKick.BaseClassLibrary\\SkyKick.Bcl.Configuration.KeyVault\\Abstractions\\KeyVaultClient.cs:line 62\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at SkyKick.Bcl.Configuration.KeyVault.KeyVaultConfigurationProvider.<LoadAsync>d__5.MoveNext() in C:\\Source\\SkyKick.BaseClassLibrary\\SkyKick.Bcl.Configuration.KeyVault\\KeyVaultConfigurationProvider.cs:line 88" | string

[davidebbo]

@Raffledoocious the situation is that the issue was fixed on Azure, but can still happen when running locally. I think it's best if you open a separate issue so we don't mix the two cases (you can post link from here).

[Raffledoocious]

Thanks @davidebbo, I may do that. I have narrowed the issue down to using version AADL 2.28.4 within an Azure function. It works fine in a console application. If I upgrade to 3.19.2 I can get a secret value from an Azure function running locally.

It'd be nice to be able to use 2.28.4 from within an Azure function but the release is over a year old at this point and I can move forward with 3.19.2.

[ThomasGuttesen]

I was having this issue too locally with VS2017 . But I found out was using the .cer version which does not have the private key embedded. Importing the .p12 of the certificate solved my problem. Maybe the error could be more specific?