der: preservation of order in sets during reencode

[metthal]

Hi. Recently I ran into this PKCS7/CMS structure which contains what seems like non-canonical signed/authenticated attributes. It was failing signature verification in Rust code and after closer look, it seems like reencode of the signed/authenticated attributes returns a different DER encoding than OpenSSL for example. This results in different digest being calculated and even though everything is correctly verified, the hashes won't match.

Here's the PKCS7:

-----BEGIN PKCS7-----
MIIOtgYJKoZIhvcNAQcCoIIOpzCCDqMCAQMxDzANBglghkgBZQMEAgEFADB3Bgsq
hkiG9w0BCRABBKBoBGYwZAIBAQYJYIZIAYb9bAcBMDEwDQYJYIZIAWUDBAIBBQAE
II0q38EcQ5R9Xqa36BQprPBCmTC+YP1wxBwm6OfFsXruAhAAyFUUJYnDeJ5/1abK
npRXGA8yMDE2MDMyODE4MjEwNVqgggvBMIIGiDCCBXCgAwIBAgIQAs5ClFkCpPPA
QLD/d5PRTzANBgkqhkiG9w0BAQsFADByMQswCQYDVQQGEwJVUzEVMBMGA1UEChMM
RGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cuZGlnaWNlcnQuY29tMTEwLwYDVQQD
EyhEaWdpQ2VydCBTSEEyIEFzc3VyZWQgSUQgVGltZXN0YW1waW5nIENBMB4XDTE1
MTIyNDAwMDAwMFoXDTI1MDEwNzAwMDAwMFowUjELMAkGA1UEBhMCVVMxFzAVBgNV
BAoTDkRpZ2lDZXJ0LCBJbmMuMSowKAYDVQQDEyFEaWdpQ2VydCBTSEEyIFRpbWVz
dGFtcCBSZXNwb25kZXIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDK
3JuzEc2uIHplciw1L945+SgkkdGU5aDUHl5oDDgot3i9np/x8qXeEMygOAEBqXoX
sICBLbDBhAIW/RB27EcsTW6WKNBlBBFdeMgXH4xOh1cdIevsHOfMfzT44a8rrGau
Cik37dYFj7wN3GuZLhyErnDDarcuWEsRx6ZQOvsWG/AGOfCaizghD2JeKh2F6PeH
yv3HF9iuWCdHFkZ08Dt5+cGocyi2+QzfxYrxzsCc+MtFH34bKvz73h6l6WPdKn+8
l0IiE/BUC9BKXEsIHuz/5163R9ui31O+GZ5HzYLj1ng2JAYsQj4G5AZvvFzHT9si
1eXIqp6tF8jP9dPW5HaBAgMBAAGjggM4MIIDNDAOBgNVHQ8BAf8EBAMCB4AwDAYD
VR0TAQH/BAIwADAWBgNVHSUBAf8EDDAKBggrBgEFBQcDCDCCAb8GA1UdIASCAbYw
ggGyMIIBoQYJYIZIAYb9bAcBMIIBkjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cu
ZGlnaWNlcnQuY29tL0NQUzCCAWQGCCsGAQUFBwICMIIBVh6CAVIAQQBuAHkAIAB1
AHMAZQAgAG8AZgAgAHQAaABpAHMAIABDAGUAcgB0AGkAZgBpAGMAYQB0AGUAIABj
AG8AbgBzAHQAaQB0AHUAdABlAHMAIABhAGMAYwBlAHAAdABhAG4AYwBlACAAbwBm
ACAAdABoAGUAIABEAGkAZwBpAEMAZQByAHQAIABDAFAALwBDAFAAUwAgAGEAbgBk
ACAAdABoAGUAIABSAGUAbAB5AGkAbgBnACAAUABhAHIAdAB5ACAAQQBnAHIAZQBl
AG0AZQBuAHQAIAB3AGgAaQBjAGgAIABsAGkAbQBpAHQAIABsAGkAYQBiAGkAbABp
AHQAeQAgAGEAbgBkACAAYQByAGUAIABpAG4AYwBvAHIAcABvAHIAYQB0AGUAZAAg
AGgAZQByAGUAaQBuACAAYgB5ACAAcgBlAGYAZQByAGUAbgBjAGUALjALBglghkgB
hv1sAxUwHwYDVR0jBBgwFoAU9LbhIB3+Ka7S5GGlsqIlssgXNW4wHQYDVR0OBBYE
FCR52VgCnqj486lPziFWQ4nmo1SiMHEGA1UdHwRqMGgwMqAwoC6GLGh0dHA6Ly9j
cmwzLmRpZ2ljZXJ0LmNvbS9zaGEyLWFzc3VyZWQtdHMuY3JsMDKgMKAuhixodHRw
Oi8vY3JsNC5kaWdpY2VydC5jb20vc2hhMi1hc3N1cmVkLXRzLmNybDCBhQYIKwYB
BQUHAQEEeTB3MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20w
TwYIKwYBBQUHMAKGQ2h0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2Vy
dFNIQTJBc3N1cmVkSURUaW1lc3RhbXBpbmdDQS5jcnQwDQYJKoZIhvcNAQELBQAD
ggEBALT8t75EH5RG003hoiSZ2Xocn2w8amGrNc18t0FphyX8J5XaL1scebkfMCai
GEnQYpsGyZqsB20Gz6SAhnEVry6xQOrRg6w1tKb6YYEyu+pDnPdV6WNVMmum1OHF
JzZ8VTLQW3wQr+9iYEDYyz/itixcqrwbiLsVi8k3yBgcnDuAJqgPV1/+z+5lkJUB
bshewVvAlftVToXb9oSedQg8Zn0Fm4karF6zDEu1ZkEXCjUXHCJfdjzxLPIH8QOx
SXKgRTsmm7/h2ozINzNnNNKRFE+8RHNx8zQtp7doyJqxtiNlChogD3qRw5MaeHz4
N8vvN4b9a36ybPPjcs0i6xI6HqIwggUxMIIEGaADAgECAhAKoSXW1jIbfkHkBdo2
l8IVMA0GCSqGSIb3DQEBCwUAMGUxCzAJBgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdp
Q2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5jb20xJDAiBgNVBAMTG0Rp
Z2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNjAxMDcxMjAwMDBaFw0zMTAx
MDcxMjAwMDBaMHIxCzAJBgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMx
GTAXBgNVBAsTEHd3dy5kaWdpY2VydC5jb20xMTAvBgNVBAMTKERpZ2lDZXJ0IFNI
QTIgQXNzdXJlZCBJRCBUaW1lc3RhbXBpbmcgQ0EwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC90DLuS82Pf92puoKZxTlUKFe2I0rEDgdFM1EQfdD5fU1o
fue2oPSNs4jkl79jIZCYvxO8V9PD4X4I1moUADj3Lh477sym9jJZ/l9lP+Cb6+NG
RwYaVX4LJ37AovWg4N4iPw7/fpX786O6Ij4YrBHk8JkDbTuFfAnT7l3ImgtU46gJ
cWvgzyIQD3XPcXJOCq3fQDpct1HhoXkUxk0kIzBdvOw8YGqsLwfM/fDqR9mIUF79
Zm5WYScpiYRR5oLnRlD9lCosp+R1PrqYD4R/nzEU1q3V8mTLex4F0IQZchfxFwbv
Pc3WTe8GQv2iUypPhR3EHTyvz9qsEPXdrKzpVv+TAgMBAAGjggHOMIIByjAdBgNV
HQ4EFgQU9LbhIB3+Ka7S5GGlsqIlssgXNW4wHwYDVR0jBBgwFoAUReuir/SSy4Ix
LVGLp6chnfNtyA8wEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYw
EwYDVR0lBAwwCgYIKwYBBQUHAwgweQYIKwYBBQUHAQEEbTBrMCQGCCsGAQUFBzAB
hhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20wQwYIKwYBBQUHMAKGN2h0dHA6Ly9j
YWNlcnRzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEFzc3VyZWRJRFJvb3RDQS5jcnQw
gYEGA1UdHwR6MHgwOqA4oDaGNGh0dHA6Ly9jcmw0LmRpZ2ljZXJ0LmNvbS9EaWdp
Q2VydEFzc3VyZWRJRFJvb3RDQS5jcmwwOqA4oDaGNGh0dHA6Ly9jcmwzLmRpZ2lj
ZXJ0LmNvbS9EaWdpQ2VydEFzc3VyZWRJRFJvb3RDQS5jcmwwUAYDVR0gBEkwRzA4
BgpghkgBhv1sAAIEMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LmRpZ2ljZXJ0
LmNvbS9DUFMwCwYJYIZIAYb9bAcBMA0GCSqGSIb3DQEBCwUAA4IBAQBxlRLpUYdW
ac3v3dp8qmN6s3jPBjdAhO9LhL/KzwMC/cWnww4gQiyvd/MrHwwhWiq3BTQdaq6Z
+CeiZr8JqmDfdqQ6kw/4stHYfBli6F6CJR7Euhx7LCHi1lssFDVDBGiy23UC4HLH
mNY8ZOUfSBAYX4k4YU1iRiSHY4yRUiyvKYnleB/WCxSlgNcSR3CzddWThZN+tpJn
+1Nhiaj1a5bA9FhpDXzIAbG5KHW3mWOFIoxhynmUfln8jA/jb7UBJrZspe6HUSHk
WGCbugwtK22ixH67xCUrRwIIfEmuE7bhfEJCKMYYVs9BNLZmXbZ0e/VWMyIvIjay
S6JKldj1po5SMYICTTCCAkkCAQEwgYYwcjELMAkGA1UEBhMCVVMxFTATBgNVBAoT
DERpZ2lDZXJ0IEluYzEZMBcGA1UECxMQd3d3LmRpZ2ljZXJ0LmNvbTExMC8GA1UE
AxMoRGlnaUNlcnQgU0hBMiBBc3N1cmVkIElEIFRpbWVzdGFtcGluZyBDQQIQAs5C
lFkCpPPAQLD/d5PRTzANBglghkgBZQMEAgEFAKCBmDAaBgkqhkiG9w0BCQMxDQYL
KoZIhvcNAQkQAQQwHAYJKoZIhvcNAQkFMQ8XDTE2MDMyODE4MjEwNVowLwYJKoZI
hvcNAQkEMSIEIALRfHgBo/1U0NtasppsMkFq1GGUR6zdDguTsWvQR25jMCsGCyqG
SIb3DQEJEAIMMRwwGjAYMBYEFMY29N2ofO49gmO/miUUtFM0aNdeMA0GCSqGSIb3
DQEBAQUABIIBAJT4B7VgkVo2iUBC0loKNOy/s5F8TA4KAt4nAYwRbFwPHx2HylG9
Zah2PT0JrF4wG7TfvF7rVenG3wCnJY9J+YZVOqNTSZUKrZGwtrpLDUoB5O4Lok7P
WHakBQkVvrRWUhdRmK/0PIXuUzfDjYq7xJq06WVd/OUlwK43+TSa6pWjtbm+cNiB
CG8exiWVh1Osba6sGHeYsLJnfuvRzc2YmmQTKX0L2h1irZv2f8YjAR6UMuN5FO2p
glg+ttCzSwBLwVE+u74k7ovMMMIYJAFCpANcEe1FsmvVfbFXN72JCksdkdJM870s
bBZGJZsE6wvM7sLLApbgzRsesql3KX5j4GM=
-----END PKCS7-----

Code used to extract the signed attributes from this on the Rust side:

let mut reader = SliceReader::new(raw_der).unwrap();
let cms_content = ContentInfo::decode(&mut reader).unwrap();
let signed_data = cms_content.content.decode_as::<SignedData>().unwrap();
let signer_info = signed_data.signer_infos.0.get(0).unwrap();
let attrs = &signer_info.signed_attrs;

let mut reencode = Vec::new();
let _ = attrs.encode_to_vec(&mut reencode).unwrap();

Here's what I get from your crate:

$ openssl asn1parse -inform der -in rust.der -i                                                                                                                                                                   23.269ms 2024-04-19 14:17:37
    0:d=0  hl=3 l= 152 cons: SET               
    3:d=1  hl=2 l=  26 cons:  SEQUENCE          
    5:d=2  hl=2 l=   9 prim:   OBJECT            :contentType
   16:d=2  hl=2 l=  13 cons:   SET               
   18:d=3  hl=2 l=  11 prim:    OBJECT            :id-smime-ct-TSTInfo
   31:d=1  hl=2 l=  28 cons:  SEQUENCE          
   33:d=2  hl=2 l=   9 prim:   OBJECT            :signingTime
   44:d=2  hl=2 l=  15 cons:   SET               
   46:d=3  hl=2 l=  13 prim:    UTCTIME           :160328182105Z
   61:d=1  hl=2 l=  43 cons:  SEQUENCE          
   63:d=2  hl=2 l=  11 prim:   OBJECT            :id-smime-aa-signingCertificate
   76:d=2  hl=2 l=  28 cons:   SET               
   78:d=3  hl=2 l=  26 cons:    SEQUENCE          
   80:d=4  hl=2 l=  24 cons:     SEQUENCE          
   82:d=5  hl=2 l=  22 cons:      SEQUENCE          
   84:d=6  hl=2 l=  20 prim:       OCTET STRING      [HEX DUMP]:C636F4DDA87CEE3D8263BF9A2514B4533468D75E
  106:d=1  hl=2 l=  47 cons:  SEQUENCE          
  108:d=2  hl=2 l=   9 prim:   OBJECT            :messageDigest
  119:d=2  hl=2 l=  34 cons:   SET               
  121:d=3  hl=2 l=  32 prim:    OCTET STRING      [HEX DUMP]:02D17C7801A3FD54D0DB5AB29A6C32416AD4619447ACDD0E0B93B16BD0476E63

Here's what I get from OpenSSL:

    0:d=0  hl=3 l= 152 cons: SET               
    3:d=1  hl=2 l=  26 cons:  SEQUENCE          
    5:d=2  hl=2 l=   9 prim:   OBJECT            :contentType
   16:d=2  hl=2 l=  13 cons:   SET               
   18:d=3  hl=2 l=  11 prim:    OBJECT            :id-smime-ct-TSTInfo
   31:d=1  hl=2 l=  28 cons:  SEQUENCE          
   33:d=2  hl=2 l=   9 prim:   OBJECT            :signingTime
   44:d=2  hl=2 l=  15 cons:   SET               
   46:d=3  hl=2 l=  13 prim:    UTCTIME           :160328182105Z
   61:d=1  hl=2 l=  47 cons:  SEQUENCE          
   63:d=2  hl=2 l=   9 prim:   OBJECT            :messageDigest
   74:d=2  hl=2 l=  34 cons:   SET               
   76:d=3  hl=2 l=  32 prim:    OCTET STRING      [HEX DUMP]:02D17C7801A3FD54D0DB5AB29A6C32416AD4619447ACDD0E0B93B16BD0476E63
  110:d=1  hl=2 l=  43 cons:  SEQUENCE          
  112:d=2  hl=2 l=  11 prim:   OBJECT            :id-smime-aa-signingCertificate
  125:d=2  hl=2 l=  28 cons:   SET               
  127:d=3  hl=2 l=  26 cons:    SEQUENCE          
  129:d=4  hl=2 l=  24 cons:     SEQUENCE          
  131:d=5  hl=2 l=  22 cons:      SEQUENCE          
  133:d=6  hl=2 l=  20 prim:       OCTET STRING      [HEX DUMP]:C636F4DDA87CEE3D8263BF9A2514B4533468D75E

Here's the base64 encoded DER of what OpenSSL gives me (considering I use PKCS7_ATTR_VERIFY, see below for more info):

MYGYMBoGCSqGSIb3DQEJAzENBgsqhkiG9w0BCRABBDAcBgkqhkiG9w0BCQUxDxcNMTYwMzI4MTgy
MTA1WjAvBgkqhkiG9w0BCQQxIgQgAtF8eAGj/VTQ21qymmwyQWrUYZRHrN0OC5Oxa9BHbmMwKwYL
KoZIhvcNAQkQAgwxHDAaMBgwFgQUxjb03ah87j2CY7+aJRS0UzRo114=

Seems to be related to ordering of sets in the canonical form which is mentioned in your codebase and also OpenSSL with their PKCS7_ATTR_VERIFY which was specially created for this purpose.

I am just considering what are my options here because if the sets are ordered during decode, then I won't really get back the original order. So the very first question would be whether it's even somehow possible to use this crate for my use-case? If not, would you be interested in maybe supporting also non-canonical form? I'm not sure what it entails all together other than ordering, but I wouldn't have problem contributing to this if it helped me to obtain exactly what I want. Thanks!

[tarcieri]

In DER, SET OF requires an ordered encoding. From ITU X.690:

11.6 Set-of components
The encodings of the component values of a set-of value shall appear in ascending order, the encodings being compared as octet strings with the shorter components being padded at their trailing end with 0-octets.

However, the schema uses SET, not SET OF.

It seems the problem is the cms crate is using the der::SetOf* types to model SET (rather than SET OF) which does not stipulate a sorted encoding.

cc @carl-wallace

[carl-wallace]

I’m not sure what you are getting at with SET vs SET OF but the case here is SET OF in any case: SignedAttributes ::= SET SIZE (1..MAX) OF Attribute The CMS RFC is very clear on this: “SignedAttributes MUST be DER encoded, even if the rest of the structure is BER encoded.” The problem is that the generator signed a BER encoding and the verifier is using a DER encoding for verification (with the length field is what establishes the ordering here). The comment in the OpenSSL codebase describes the hack they used to work around this (i.e., they are re-encoding the elements as a SEQUENCE then changing the tag to be SET at verification time). Something like that would likely work here too (depending on if the order returned from the decoder is as received, which I am not certain of offhand). Where I’ve run into stuff like this, I’ve used the defer decoding capability to avoid the need to re-encode entirely. That likely would work here but would require variants of SignedData, SignerInfos and SignerInfo to reference the defer decoding. There is an example that uses defer decoding in the x509_cert crate test cases: https://github.com/RustCrypto/formats/blob/1cb066a80b361e1b5cd3bafd1e84217d7a352cf6/x509-cert/tests/certificate.rs#L62. A somewhat less contrived example is here (to defer decoding the TBSCertificate field of a Certificate): https://github.com/carl-wallace/rust-pki/blob/2dbd11e1fd6f21079e700ce60ee3ec27856a1b02/certval/src/validator/pdv_certificate.rs#L237. Then use the original structures when generating and the structures that support defer decoding when verifying. From: Tony Arcieri ***@***.***> Reply-To: RustCrypto/formats ***@***.***> Date: Friday, April 19, 2024 at 8:50 AM To: RustCrypto/formats ***@***.***> Cc: Carl Wallace ***@***.***>, Mention ***@***.***> Subject: Re: [RustCrypto/formats] [Question] Preservation of order in sets during reencode (Issue #1386) In DER, SET OF requires an ordered encoding. From ITU X.690: 11.6 Set-of components The encodings of the component values of a set-of value shall appear in ascending order, the encodings being compared as octet strings with the shorter components being padded at their trailing end with 0-octets. However, the schema uses SET, not SET OF. It seems the problem is the cms crate is using the der::SetOf* types to model SET (rather than SET OF) which does not stipulate a sorted encoding. cc @carl-wallace — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[tarcieri]

Aah, my bad, that is a really wacky hack on OpenSSL's part.

[metthal]

Thank you. I'll try to look into the deferred decoding as having access to the original data would most probably solve the problem. The only thing that I see as a difference is that original data starts as a081... while OpenSSL gives me 3181... but I guess that has to do with the BER encoding. I'll just have to read through the standards to actually know the exact difference. This helped me a lot.

And btw, the encoded data is not under my control and I have no idea how it was created. It's taken from real, in the wild, examples so it's not like it's just something I purposefully created but it seems like there are encoders with these issues out there.

[carl-wallace]

The tag difference is due to IMPLICIT tagging. I forgot about that in my reply. The Rust code I have for verifying SignedData is just reencoding SignedAttributes (but I control both sides in that case). I looked back at some C code that was more generic and it defer decodes then fixes the tag. There’s probably not much else to be done. Either re-encode in order preserving way (as OpenSSL did) and change the tag from SEQUENCE to SET or defer decode and change the tag from the IMPLICIT[0] to SET. From: Marek Milkovič ***@***.***> Reply-To: RustCrypto/formats ***@***.***> Date: Friday, April 19, 2024 at 10:54 AM To: RustCrypto/formats ***@***.***> Cc: Carl Wallace ***@***.***>, Mention ***@***.***> Subject: Re: [RustCrypto/formats] [Question] Preservation of order in sets during reencode (Issue #1386) Thank you. I'll try to look into the deferred decoding as having access to the original data would most probably solve the problem. The only thing that I see as a difference is that original data starts as a081... while OpenSSL gives me 3181... but I guess that has to do with the BER encoding. I'll just have to read through the standards to actually know the exact difference. This helped me a lot. And btw, the encoded data is not under my control and I have no idea how it was created. It's taken real, in the wild, examples so it's not like it's just something I purposefully created but it seems like there are encoder with these issues out there. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[tarcieri]

We now have EncodingRules::Ber although it only applies to decoding, not encoding.

We could potentially add BER encoding support which doesn't eagerly sort the fields.