System.Security.Cryptography.Tests outerloop tests broken

[ManickaP]

Multiple outerloop tests broken with:

System.Security.Cryptography.X509Certificates.Tests.RevocationTests.DynamicRevocationTests.RevokeEndEntityWithInvalidRevocationSignature(pkiOptions: AllRevocation) [FAIL]
      System.AggregateException : One or more errors occurred. (Assert.Equal() Failure: Collections differ
                                   ↓ (pos 2)
      Expected: [NoError, NoError, RevocationStatusUnknown | OfflineRevocation]
      Actual:   [NoError, NoError, Revoked]
                                   ↑ (pos 2)) (Assert.Equal() Failure: Collections differ
                                   ↓ (pos 2)
      Expected: [NoError, NoError, RevocationStatusUnknown | OfflineRevocation]
      Actual:   [NoError, NoError, Revoked]
                                   ↑ (pos 2)) (Assert.Equal() Failure: Collections differ
                                   ↓ (pos 2)
...

Based on Kusto data, this seems to be broken for quite a while (seeing failures as far as April).

Example log: https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-117207-merge-a7140fac46094fc59f/System.Security.Cryptography.Tests/1/console.66205cf5.log?helixlogtype=result

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[bartonjs]

Offhand, it looks like OSSL 3.5 has decided to treat "it says revoked, but the signature is invalid" as "so it's revoked, I guess". But more investigation is required.

[jeffhandley]

Marking as 10.0.0 for completing the investigation to verify the likely change in behavior from OpenSSL, which would result in changing the test's assertion.

[vcsjones]

I started looking at this. It's not new in OpenSSL 3.5. It happened somewhere between 3.0 and 3.5, we just jumped from 3.0 to 3.5.

I have an automated bisect going against OpenSSL between 3.0 and 3.5 to find the commit that introduced the change so we can understand what the intention was. Will report back.

[vcsjones]

Okay. This is an interesting one. I can also reproduce it against OpenSSL 3.0.

A bisect of the runtime shows it started in 00b0050. /cc @bartonjs

On e231b75:

On 00b0050:

More interestingly, I cannot reproduce any of the failures if I run any of the theories individually. I suspect something is getting memoized when it shouldn't, or some cache key that was introduced in that commit is not taking something in to account.