RevocationStatusUnknown unable to get certificate CRL

[joch0a]

Description
We are running a .NET Core 3.1 application on Kestrel inside a Linux Docker container.
This is what 'uname -a' gives:
Linux 2a628053d7bd 4.19.76-linuxkit dotnet/aspnetcore#1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 GNU/Linux

We are invoking an API endpoint using client certificates and we are getting the following error from Microsoft.AspNetCore.Authentication.Certificate.CertificateAuthenticationHandler:

[01:11:58 WRN] Certificate validation failed, subject was CN=****, OU=****, O=****, L=****, S=CA, C=US.
RevocationStatusUnknown unable to get certificate CRL
[01:11:58 DBG] MTLS authentication failed, error: Client certificate failed validation..
[01:11:58 INF] HTTP POST /connect/mtls/token responded 403 in 63.3324 ms

Our application is using the following certificate validation code

services.AddAuthentication()
    .AddCertificate("x509", options =>
    {
          options.RevocationMode = X509RevocationMode.Online;
    });

The root and the intermediate CAs are successfully being installed in Linux using the following command update-ca-certificates and despite the error, the CRLs are also being successfully downloaded into the ~/.dotnet/corefx/cryptography/crls/ folder.

The container obviously has access to the internet (We tested this by manually downloading the DigiCert CRLs using curl).
The same code works fine using IIS.

Could you provide more information on how the CertificateAuthenticationHandler retrieves CRLs and validates client certificates?

[davidfowl]

@blowdart @bartonjs

[chrismcv]

I've had two issues this week relating to expiring CRLs as well. (Having never previously had any)

When using a ServerValidationCallback I could see the error: OfflineRevocation CRL has expired.

When I deleted from ~/.dotnet/corefx/cryptography/crls/ my issue was immediately sorted.

Can I ask:

1. How long are crls cached for?
2. In a docker context, would they be part of the image, or do they relate to container lifetime?
3. Should I be cronning an rm for the ~/.dotnet/corefx/cryptography/crls/ to workaround?

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq
Notify danmosemsft if you want to be subscribed.

[bartonjs]

I hate to ask this question, but: Are you sure you're running 3.1?

For .NET Core 3.0 and earlier versions we had an assumption that each CA used one and only one URL for their CRLs, and DigiCert proved us wrong by having an arbitrary split to a different CRL for different issuance slices from the same CA. For 3.1 we already changed it to use both the CA DN and the URL in the cache key (aka file name), so two different CDPs result in two different files.

This should have been fixed by dotnet/corefx#39430. If you can figure out how you got into a bad state (which seems like it would be "build a chain for this certificate, then build a chain for this other one"), that'd be useful for figuring things out. Or maybe DigiCert had briefly published a CRL to the wrong URL and we ended up caching that badness on your machine.

How long are crls cached for?

Until we load one and the nextUpdate value was in the past.

In a docker context, would they be part of the image, or do they relate to container lifetime?

It would use whatever the persistence model is for the ~/.dotnet/corefx/cryptography/crls/ directory. I believe that's "container lifetime" by default?

Should I be cronning an rm for the ~/.dotnet/corefx/cryptography/crls/ to workaround?

If you're consistently getting into this state then hopefully we can get a repro and figure out what to change in our layer. But you can certainly periodically wipe out the cache if you want.

[chrismcv]

@bartonjs - thanks!

I hate to ask this question, but: Are you sure you're running 3.1?

One of my issues was against a 2.2 container, which has now been upgraded to 3.1. The other was against a 3.1 container, build within a week, and the container lifetime was 2 hours.

I'll keep an eye on it and see how frequently it happens. Could be a fluke that there were 2 in one week.

[joch0a]

Yes, I can confirm we are running ASP .NET Core 3.1. This is what out dockerfile looks like

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
/*Docker steps*/
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
/*Docker steps*/
FROM build AS publish
/*Docker steps*/
FROM base AS final
/*Docker steps*/
ENTRYPOINT ["dotnet", "app.dll"]

We are using DigiCert and we see two CRL files under ~/.dotnet/corefx/cryptography/crls/: 5474e8e2.2d0546da.crl and b60aa296.bb2394b8.crl
This problem occurs every time, so it's easy to reproduce. In other words, all requests that include client certificates fail due to the server being unable to get the certificate CRL. I'm not sure I understand the "figure out how you got into a bad state" part. We've been given root and intermediate CAs by DigiCert, and are using a valid certificate signed by the intermediate CA. The CRLs are correctly being downloaded into the previous directory. Not sure how to determine where the error comes from.

Thanks for the reply!

[bartonjs]

Sorry, I missed the reply notification.

If this is repeatable, can you share a minimal repro? I assume it requires two different chain builds for two different certificates; so the repro would need to involve those two (or however many) certificates.

If the certificates can't be easily shared, can you try it on a .NET 5 preview build and see if we've already fixed the problem via the various chain processor enhancements we have already done in 5?