Support RFC 9802 LMS and XMSS in X.509 certificate and CSR generation

[Frauschi]

Extends wolfCrypt's certificate generation to issue HSS/LMS and XMSS/XMSS^MT certificates and PKCS#10 requests, building on the existing RFC 9802 verification support so the library can now both produce and consume stateful hash-based-signature certificates.

Changes

• New key-type selectors LMS_TYPE / XMSS_TYPE / XMSSMT_TYPE for wc_MakeCert_ex, wc_SignCert_ex, and wc_MakeCertReq_ex.
• New SPKI encoders wc_LmsKey_PublicKeyToDer and wc_XmssKey_PublicKeyToDer to emit RFC 9802 SubjectPublicKeyInfo.
• Runtime signature-buffer sizing and sigType/key consistency checks for the new algorithms.
• Generation is ASN.1-template only (WOLFSSL_ASN_TEMPLATE), matching where the verification path already lives. Non-template builds reject these key types with ALGO_ID_E.

Tests

• New in-process tests generate self-signed LMS/XMSS roots, CSRs, and a CA→ECC-leaf chain, then verify each through the parsing/verification path.
• These replace the previously checked-in (patched Bouncy Castle) DER fixtures; only the stock RFC 9802-aligned LMS interop anchor is retained.

No change to default builds — the feature is gated behind the existing LMS/XMSS and certificate-generation options.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m3

• FLASH: .text +4 B (+0.0%, 120,909 B / 262,144 B, total: 46% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +384 B (+0.1%, 766,060 B / 1,048,576 B, total: 73% used)

gcc-arm-cortex-m4-pkcs7

• FLASH: .text +256 B (+0.1%, 210,929 B / 262,144 B, total: 80% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +320 B (+0.1%, 277,208 B / 1,048,576 B, total: 26% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text +192 B (+0.1%, 322,296 B / 1,048,576 B, total: 31% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +320 B (+0.1%, 277,784 B / 1,048,576 B, total: 26% used)

linuxkm-pie

• Data: __patchable_function_entries +56 B (+0.2%, 24,232 B)

linuxkm-standard

• Data: __patchable_function_entries +32 B (+0.1%, 45,904 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-tls13
• stm32-sim-stm32h753

[Copilot]

Pull request overview

This PR extends wolfCrypt’s X.509 certificate and PKCS#10 CSR generation pipeline to produce RFC 9802 HSS/LMS and XMSS/XMSS^MT artifacts (matching the existing verification support), including new key-type selectors, SPKI encoders, and runtime signature sizing so large hash-based signatures are handled correctly. It also updates tests to generate certificates/CSRs in-process instead of relying on committed DER fixtures (keeping a single LMS interop anchor).

Changes:

• Add new certificate/CSR key selectors (LMS_TYPE, XMSS_TYPE, XMSSMT_TYPE) and internal key-type routing for cert/CSR generation.
• Add RFC 9802 SPKI encoders for LMS and XMSS, plus runtime signature-buffer sizing and sigType/key-family consistency checks to prevent mismatched OIDs.
• Update memory sizing constants to separate “classic” signature buffers from PQC-sized buffers; replace most committed LMS/XMSS fixtures with in-process generation + round-trip verification tests.

Reviewed changes

Copilot reviewed 16 out of 29 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
wolfssl/wolfcrypt/wc_xmss.h	Refactors XmssKey typedef to allow forward-declare and adds wc_XmssKey_PublicKeyToDer() API declaration.
wolfssl/wolfcrypt/wc_lms.h	Refactors LmsKey typedef to allow forward-declare and adds wc_LmsKey_PublicKeyToDer() API declaration.
wolfssl/wolfcrypt/types.h	Introduces MAX_ENCODED_CLASSIC_SIG_SZ vs MAX_ENCODED_SIG_SZ and adds WOLFSSL_MAX_SIG_SZ for fixed-buffer builds.
wolfssl/wolfcrypt/settings.h	Enables asym key import/export when LMS/XMSS signing support is compiled in (non-verify-only).
wolfssl/wolfcrypt/asn.h	Uses classic-tier signature-copy sizing for RSA/DSA verify paths and adds internal key enums for LMS/XMSS.
wolfssl/wolfcrypt/asn_public.h	Adds forward declarations for LmsKey/XmssKey and exposes new cert/CSR key selector values.
wolfcrypt/src/signature.c	Switches fixed RSA plain-text buffer sizing to classic-tier max when ASN isn’t available.
wolfcrypt/src/asn.c	Implements LMS/XMSS SPKI encoders, adds runtime signature sizing + sigType/key checks, and wires LMS/XMSS into cert/CSR generation (ASN template path).
wolfcrypt/src/asn_orig.c	Explicitly rejects LMS/XMSS cert/CSR generation on the non-template ASN path (template-only support).
tests/api/test_lms_xmss.h	Registers new RFC 9802 LMS/XMSS X.509 generation tests.
tests/api/test_lms_xmss.c	Reworks RFC 9802 tests to generate certs/CSRs/chains in-process and verify via existing parse/verify path; removes most fixture dependence.
src/pk_rsa.c	Uses classic-tier max buffer sizing for RSA DigestInfo encoding.
src/internal.c	Removes fixed ceiling when copying parsed signatures into X509 objects so large LMS/XMSS signatures are retained.
certs/xmss/include.am	Removes XMSS fixture distribution list (fixtures no longer used).
certs/lms/include.am	Keeps only the stock BC LMS interop anchor fixture in distribution list.
certs/include.am	Stops including the XMSS fixture include file.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Frauschi]

Jenkins retest this please

[dgarske]

Skoll Code Review

Scan type: reviewOverall recommendation: COMMENT
Findings: 5 total — 5 posted, 0 skipped
5 finding(s) posted as inline comments (see file-level comments below)

Posted findings

• [Medium] CheckSigTypeForKey now rejects the wc_InitCert default sigType for non-RSA keys — wolfcrypt/src/asn.c:29956-29964
• [Low] NO_MALLOC sign paths pass MAX_ENCODED_CLASSIC_SIG_SZ capacity against a WOLFSSL_MAX_SIG_SZ buffer — wolfcrypt/src/asn.c:30360-30370, 36899-36912
• [Low] XMSS test key-persistence callbacks use raw stdio instead of XFILE abstraction — tests/api/test_lms_xmss.c:921-944
• [Low] Uninitialized derSz in rfc9802_gen_roundtrip — tests/api/test_lms_xmss.c:766-790
• [Info] OID-occurrence count assertion assumes the HBS OID never appears inside the signature value — tests/api/test_lms_xmss.c:660-676, 1027-1033

Review generated by Skoll

[dgarske]

🟠 [Medium] CheckSigTypeForKey now rejects the wc_InitCert default sigType for non-RSA keys

SignCert, wc_MakeSigWithBitStr, wc_SignCRL_ex, and wc_SignCert_cb now call CheckSigTypeForKey, which returns ALGO_ID_E when sType does not match the signing key's algorithm family. wc_InitCert/wc_InitCert_ex default cert->sigType to CTC_SHA256wRSA (confirmed at asn.c:26216 / wc_InitCert_ex). Any existing caller that signs with a non-RSA key via wc_SignCert/wc_SignCert_ex but forgets to set cert.sigType previously succeeded (emitting a cert whose signatureAlgorithm OID contradicted the key) and will now fail with ALGO_ID_E. This is a beneficial hardening and all new tests set sigType explicitly, but it is a behavior change that could surface as a 'regression' for buggy third-party callers.

Fix: Confirm all in-tree cert/CSR/CRL generation tests and examples set cert.sigType for non-RSA keys (wolfCrypt internal tests do), and call out the new strict sigType/key validation in the release notes so downstream callers that relied on the previously-permissive behavior update accordingly.

[dgarske]

🔵 [Low] NO_MALLOC sign paths pass MAX_ENCODED_CLASSIC_SIG_SZ capacity against a WOLFSSL_MAX_SIG_SZ buffer

Under WOLFSSL_NO_MALLOC, certSignCtx->sig is the fixed array byte sig[WOLFSSL_MAX_SIG_SZ] (types.h:2491). wc_SignCert_cb and wc_SignCRL_ex pass MAX_ENCODED_CLASSIC_SIG_SZ as the buffer capacity to MakeSignatureCb/MakeSignature. With the default WOLFSSL_MAX_SIG_SZ == MAX_ENCODED_SIG_SZ >= MAX_ENCODED_CLASSIC_SIG_SZ this is safe. However, the new WOLFSSL_MAX_SIG_SZ override is documented (types.h:2470) as a way to 'shrink builds that only sign with classic/compact algorithms'; if a user sets it below MAX_ENCODED_CLASSIC_SIG_SZ, the claimed capacity would exceed the real fixed buffer and a full-modulus RSA signature could overflow it. The coupling is implicit.

Fix: Either clamp the capacity passed to MakeSignature*/MakeSignatureCb to min(MAX_ENCODED_CLASSIC_SIG_SZ, WOLFSSL_MAX_SIG_SZ) under WOLFSSL_NO_MALLOC, or document that WOLFSSL_MAX_SIG_SZ must not be reduced below MAX_ENCODED_CLASSIC_SIG_SZ (the max RSA modulus).

[dgarske]

🔵 [Low] XMSS test key-persistence callbacks use raw stdio instead of XFILE abstraction

The new XMSS generation tests define key persistence callbacks using raw fopen/fwrite/fread/fclose and FILE*, whereas the same file's rfc9802_load_file helper uses the XFILE/XFOPEN/XFREAD/XFCLOSE abstraction. Mixing raw stdio is inconsistent with the file's own conventions and will not honor platforms that remap file I/O through the X-macros.

Fix: Use XFILE/XFOPEN/XFWRITE/XFREAD/XFCLOSE for consistency with rfc9802_load_file and portability across NO_FILESYSTEM-style platform remaps.

[dgarske]

🔵 [Low] Uninitialized derSz in rfc9802_gen_roundtrip

int derSz; is declared uninitialized. It is only read inside ExpectIntEQ(rfc9802_gen_verify_selfsigned(der, derSz), ...) blocks that the Expect framework short-circuits once a prior expectation fails, so in practice it is never read while uninitialized. Still, caSz/leafSz in the sibling rfc9802_gen_chain are initialized to 0; initializing derSz too avoids a possible -Wmaybe-uninitialized under -Werror and matches the surrounding convention.

Fix: Initialize derSz to 0 for consistency and to silence potential uninitialized-use warnings.

[dgarske]

⚪ [Info] OID-occurrence count assertion assumes the HBS OID never appears inside the signature value

rfc9802_gen_xmss_oid_tamper asserts exactly 3 occurrences of the 8-byte 1.3.6.1.5.5.7.6.\<lastByte> OID body in the generated DER (TBS-sig, SPKI-key, outer-sig). Because the XMSS signatureValue is large and freshly random per run, the same 8-byte sequence could in principle appear inside the signature, making ExpectIntEQ(n, 3) fail. The probability is ~2^-64 per cert (effectively never), so this is informational only.

Fix: Optionally bound the scan to the TBS/algId region (before the signatureValue BIT STRING) to make the count structurally exact rather than statistically exact. Not required given the negligible probability.