der: replace O(n^2) SetOf sort with O(n log n) sort#2345
Merged
tarcieri merged 2 commits intoJun 5, 2026
Conversation
der_sort was a hand-rolled insertion sort. On reverse-sorted input it performs n(n-1)/2 comparisons and swaps, so a crafted SET OF turns a single decode into quadratic-time work. SetOfVec::decode_value runs this on every DER decode, which makes it a denial-of-service vector on any SetOfVec parsed from untrusted DER (for example RelativeDistinguishedName in x509-cert, and DigestAlgorithmIdentifiers / CertificateSet / SignerInfos in cms). A ~246 KB reverse-sorted SET OF with 18k elements took roughly 11 seconds to decode. Switch to slice::sort_unstable_by, an in-place O(n log n) introsort that is available in core, so it still works on heapless no_std targets where the stable slice::sort_by (which needs alloc) is not. The first der_cmp error is captured and returned after the sort, preserving the existing error-bubbling behavior. Sorting unstably does not change the result: two elements that compare Equal under DerOrd have identical DER encodings, so their relative order does not affect the serialized output. Duplicates are still preserved (no deduplication), matching the decoder behavior after #2272. Adds regression tests that cross-check the new ordering against the original insertion sort on randomized and adversarial inputs, assert the DER SET OF ordering invariant, verify duplicate preservation, and decode a large reverse-sorted SET OF end to end. Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
CI under -D warnings flagged four lints in the tests added by this branch, plus a rustfmt diff. All are in test code; the der_sort fix itself is unchanged. - cast_possible_truncation (u64 -> u16 in the LCG helper): mask the low 16 bits explicitly instead of a lossy as-cast, so the truncation is intentional and obvious. - cast_possible_truncation (usize -> u8 in the DER length encoder): use u8::try_from with an expect documenting the in-range invariant (short-form length < 0x80; significant length-byte count <= 8). - doc_markdown: add backticks around DoS in a doc comment. - rustfmt: wrap the long LCG expression onto multiple lines. Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
tarcieri
approved these changes
Jun 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2319
Problem
der_sort()inder/src/asn1/set_of.rsis a hand-rolled insertion sort that runs on everySetOfVec/ heaplessSetOfdecode in DER mode. It is O(n^2), and reverse-sorted input forcesn(n-1)/2comparisons and swaps. A crafted reverse-sortedSET OF(about 18k elements in 246 KB) makes a singleSetOfVec::decode_valuetake roughly 11 seconds, a denial-of-service vector.This is reachable from any
SetOfVec<T>decoded from untrusted DER, includingRelativeDistinguishedNamein x509-cert (Certificate::from_der) andDigestAlgorithmIdentifiers/CertificateSet/SignerInfosin cms.Fix
Replace the insertion sort with
slice::sort_unstable_by(in-place O(n log n) introsort), capturing the firstder_cmperror and returning it after the sort so error bubbling is preserved.A few correctness points worth calling out:
no_std:sort_unstable_byis provided bycore, so it compiles under#![no_std]with noalloc. The stablesort_bywould requireallocand break the heapless no-alloc build, so unstable is the right choice here. Verified the heapless feature combination compiles and tests pass.EqualunderDerOrdhave byte-identical DER encodings, so an unstable sort cannot change the serialized output. This is cross-checked in a test against a reference copy of the old insertion sort on both randomized and adversarial inputs.der_sortnever deduplicated, and that behavior is unchanged (covered by a test).Before / after
Decoding a reverse-sorted
SET OF INTEGER:Tests
Added
der_sort_matches_reference_ordering(old insertion sort vs new sort produce identical ordering on randomized and adversarial inputs),setofvec_decodes_large_reverse_sorted_der, andder_sort_preserves_duplicates. All green across default,alloc,--all-features, and--no-default-features --features heapless.cargo clippy -p der --all-featuresis clean. x509-cert and cms build against the change.Out of scope
The issue also notes the
SetOfVec(sorts) vsSetOfRef(rejects out-of-order) behavior difference, which a maintainer already commented on; that is a separate behavior question and is not touched here. I also did not add a hard element-count cap, since the algorithmic fix is the clean behavior-preserving change and a cap would reject currently-accepted inputs (a policy call for maintainers).