Use CRL by default instead of OCSP on android

[stormshield-gt]

Hi, I have a certificate signed by let's encrypt, which only has a CRL extension but not an OCSP one, as for the let's encrypt new policy.

On the rustls-platform-verifier documentation on Android, from my understanding, it says that if there is not a stapled OCSP response, an OCSP extension or a CRL extension must be present.

But I've the following error:

WARN rustls_platform_verifier::verification::android: certificate was revoked: java.security.cert.CertPathValidatorException: Certificate does not specify OCSP responder    
ERROR rustls_platform_verifier::verification::android: failed to verify TLS certificate: invalid peer certificate: Revoked 

I use the latest version of the crate, 0.6.0.

This MR propose to enable CRL by default on Android

[ctz]

I guess my question on this is: reading the documentation it seems quite clear that CRLs are the fallback from OCSP, but the observed behaviour contradicts that. The described behaviour seems acceptable, so can we work out a way to have that behaviour rather whatever is happening here?

If the "fallback" thing doesn't actually work as described, maybe we can try two verifications with NO_FALLBACK and do the fallback ourselves?

[iliabylich]

I have exactly the same issue with my server that uses certificate from Let's Encrypt (certificate is returned by the server but then it gets immediately rejected with an error Certificate does not specify OCSP responder, which is true, openssl returns OCSP response: no response sent).

The patches fixes it for me. Thanks @stormshield-gt !

[phatblat]

This is the exact same change we made to the CertificateVerifier to work around these errors we saw in our apps.

[djc]

@complexspaces would be awesome if you could take a look at this and comment with your thoughts!

[andrew-signal]

I would actually strongly recommend against merging this PR in its current form, because it actually disables certificate revocation checking in practice in a few cases, and people who are using this library are likely to actively think they are getting that protection when they are not.

First, by setting the PKIXRevocationChecker.Option.NO_FALLBACK in combination with the PKIXRevocationChecker.Option.PREFER_CRLS option, you actually indicate that if the CRL does not indicate a revocation, you do not want to check the OSCP at all. The combination of these two options maps in the internals of the RevocationChecker to a ONLY_CRLS mode. In practice, this should not cause an issue, because they should be fed from the same source of truth, but it's worth calling out, and important for understanding the next more severe problem.

The second problem is that in practice, the CRL check is almost always failing in practice, so in combination with the PKIXRevocationChecker.Option.SOFT_FAIL option and the above issue, you're not actually getting any revocation coverage. The problem here is that since roughly Android M, Android has blocked insecure HTTP traffic by default, and by convention the CRLs are hosted via HTTP, not HTTPS. So, when the RevocationChecker tries to fetch the CRL, it fails with a java.security.cert.CertPathValidatorException: Unable to determine revocation status due to network error, which traces back to a java.io.IOException: Cleartext HTTP traffic to <RCL domain> not permitted.

If you unblock HTTP traffic for the CRL fetch, the current configuration works fine, and actually performs a revocation check as expected. This fix only patches over that the revocation check is actually silently failing completely.

I hope this is helpful!

• Andrew

[complexspaces]

As some immediate feedback, nonwithstanding the newly posted information about Android's internal behavior, this part would need to be gated behind if (!BuildConfig.TEST) in order to support our tests that validate stapled OCSP data.

Despite OCSP going on its way out I don't think the tests are removable yet because its stable-ish behavior and I don't see a reason to completely drop them yet.

[complexspaces]

@andrew-signal Thank you very much for the feedback here, I wouldn't have guessed that was the current platform behavior with any likelihood.

The fact that the CRL fetches are blocked by default is problematic for a library, but we might be able to solve the issue. According to the Android docs manifests from libraries will be merged (with lowest priority) into the end application's. Given that, we might be able to modify our manifest to include the allowance and have it automatically pulled into app builds. I'm not really a Gradle expert, so I'd appreciate feedback if you see a reason this wouldn't work:

<application
    android:usesCleartextTraffic="true"
/>

... you actually indicate that if the CRL does not indicate a revocation, you do not want to check the OSCP at all.

IIUC the situation correctly, I think that this is OK behavior to keep. Since OCSP and soft failing is a bit of a nothingburger, I don't think trying to fetch an OCSP revocation if a CRL (from a platform trusted CA, too) would have a security benefit and might just hurt performance.

Can I ask how you ran into this issue originally though? On the surface it sounds like Signal or another Android app you work with run into this and workaround it by adding usesCleartextTraffic="true" to the app manifests?

[andrew-signal]

The main concern with that solution is that Android's HTTP allow/block setting is on a per-process basis, so including that manifest flag will allow HTTP traffic to any domain for the entire app for any app that pulls in this library. To a certain extent, given the Android security model here, and how the CRL system works, that's somewhat inevitable. Consumers who don't like this behavior would have the ability to change it back. Still, that may upset some consumers, given the security focus here. The alternatives I see are:

Alternative A) To go ahead and create a custom network security policy only whitelisting the CRL domains for the root CAs, but that would be a maintenance nightmare.
Alternative B) To mirror what the major web browsers are doing, and use something like CRLite to manage CRLs. This is not ideal, because this is not the "platform" behavior, but also the default platform behavior on Android is to ignore CRL revocation altogether, so.

It's also worth noting that even these alternatives are only partial solutions, and would not avoid the problem completely - users are still free to add CAs to their trust stores, and in that case, the CRLs for those CAs would not be included in the network security policy whitelist or the CRLite, so you're back at square one re: arbitrary HTTP to arbitrary network endpoints.

I think in an ideal world, Android's built-in system libraries that are effectively upstream of this project would include something like CRLite for us to depend on to handle this properly. If this is the way the ecosystem is moving, it'd be great if Android's libraries handled it well for us, and so all applications could benefit. However, that seems like a non-starter in practice if for no other reason than that billions of Android devices are already deployed without such support built-in.

I don't really have anything to recommend in this situation right now. We ran into this issue as we are testing using rustls-platform-verifier as a dependency for more of our network stack, and ran into issues with the new Let's Encrypt change on Android like everyone else, so we wanted to run it down fully to find the most secure way to handle this problem. At this stage, we're still evaluating solutions. We have not deployed any changes or workarounds yet, because we don't actually use this on Android in production yet.

[complexspaces]

... Consumers who don't like this behavior would have the ability to change it back. Still, that may upset some consumers, given the security focus here...

If we go that route: since this provider is explicitly for Rust-based networking/HTTP stacks we should definitely advise people to use https_only client restrictions in their apps as a mitigation factor / middle ground between overriding the policy and wanting some revocation data. Out of the current options discussed here, I do lean towards it being the most favorable.

I primarily want to ensure this wait doesn't sit so long figuring out the approach that it becomes a huge problem for users of the library.

I think in an ideal world, Android's built-in system libraries that are effectively upstream of this project would include something like CRLite for us to depend on to handle this properly. If this is the way the ecosystem is moving, it'd be great if Android's libraries handled it well for us, and so all applications could benefit. However, that seems like a non-starter in practice if for no other reason than that billions of Android devices are already deployed without such support built-in.

I would generally agree with this. At times its very frustrating how weird of a line Android has drawn between "the platform does it for you" and "do it yourself", often leaving little-to-none room to fill in the gaps well yourself.

I'm curious about the possibility of this happening given the decline of certificate lifetimes in general vs the need for revocation. But since Android often has massive delays due to backwards compat and wide user needs, them implementing CRLite in the platform may still be worthwhile if it takes less years then making certificates extremely short lived.

so we wanted to run it down fully to find the most secure way to handle this problem. At this stage, we're still evaluating solutions. We have not deployed any changes or workarounds yet, because we don't actually use this on Android in production yet.

Got it, thanks for the clarification. We didn't notice this at 1P because all of our services are hosted behind AWS's load balancers with AWS-provisioned certificates. I'm grateful to LetsEncrypt and the people who reported this issue. But the security expertise of the folks over at Signal is always appreciated :)

[banasiak]

Had a chat with @complexspaces today and the route we ended up taking with the 1P app for Android was to add a Network Security Config to the manifest. As @andrew-signal pointed out, Android's network stack blocks clear-text traffic by default and this library uses Android's network stack to download and verify the CRL which by convention is served via HTTP instead of HTTPS.

There's a couple ways this can be configured to either allow or forbid clear-text traffic and I've provided examples of each for further discussion.

Option 1: Allow clear-text traffic by default, and require encryption for certain domains:

<?xml version="1.0" encoding="utf-8"?>
<network-security-config 
    xmlns:tools="http://schemas.android.com/tools">
    <base-config cleartextTrafficPermitted="true"
        tools:ignore="InsecureBaseConfiguration"/>
    <domain-config cleartextTrafficPermitted="false">
        <domain includeSubdomains="true">1password.com</domain>
        <domain includeSubdomains="true">1password.ca</domain>
        <domain includeSubdomains="true">1password.eu</domain>
    </domain-config>
</network-security-config>

Option 2: Forbid clear-text traffic by default, and make exceptions for certain domains:

<?xml version="1.0" encoding="utf-8"?>
<network-security-config >
    <base-config cleartextTrafficPermitted="false"/>
    <domain-config cleartextTrafficPermitted="true">
        <domain includeSubdomains="true">c.lencr.org</domain>
        <domain includeSubdomains="true">some-other-crl-domain</domain>
        <domain includeSubdomains="true">this-is-clearly-not-going-to-scale</domain>
    </domain-config>
</network-security-config>

This isn't a great option, but it's an option. If ya'll are interested, I'd be happy to open a PR with Option 2 allowing cleartext traffic for the c.lencr.org domain so their CRL can be downloaded on Android.

[complexspaces]

FYI 042330d added a test only workaround for this so CI using LetsEncrypt certificates can pass again. I found that the reason it returned was just CertPathValidatorException.BasicReason.UNSPECIFIED which isn't super helpful.

[complexspaces]

Per some discussion in the rustls Discord channel last week we are thinking the most feasible path forward here will be a manifest containing all well-known CRL distribution endpoints.

We are looking at using the ccadb data to gather a list of all CAs (roots and intermediates that are allowed to issue certificates) that public hosts could have certificates from and using this data to autogenerate an appropriate library library manifest for the distributed .aar library. We would specify the manifest in such a way Android Studio should automatically merge it with the main app one if the app developers haven't explicitly blocked that. Once that's done I'd need to solve #115 so that list is actually distributed to all current users (I am aware of ~2 parties who just grab the Java classes and discard the rest of the library, which would be problematic with the policy manifest becoming required for correct behavior).