fix: EVP_DigestSignUpdate/VerifyUpdate accept size_t, not unsigned int

[MarkAtwood]

Root Cause

Three defects, all in wolfcrypt/src/evp.c and wolfssl/openssl/evp.h:

1. wolfSSL_EVP_DigestSignUpdate was declared with unsigned int cnt — unlike OpenSSL's size_t cnt, breaking source-level API compatibility and FFI bindings that pass size_t.
2. wolfSSL_EVP_DigestVerifyUpdate had the right size_t cnt declaration but the implementation cast it to unsigned int before calling the internal helper, silently truncating any count larger than UINT_MAX on 64-bit platforms.
3. The internal helper wolfssl_evp_digest_pk_update accepted unsigned int, which forced the truncating cast in powerPC changes #2.

Why It Was Missed

DigestSignUpdate and DigestVerifyUpdate were added at different times. The Verify side was updated to match OpenSSL's size_t in the public declaration but the internal helper (which served both) was never updated, leaving the cast in place. The Sign side declaration was never updated at all. Because both functions compile without warning (implicit narrowing is legal in C), the type mismatch survived code review and CI undetected.

Fix

• wolfssl_evp_digest_pk_update: unsigned int → size_t; add overflow guard before narrowing to word32 for wc_HmacUpdate
• wolfSSL_EVP_DigestSignUpdate declaration (evp.h) and definition: unsigned int → size_t
• wolfSSL_EVP_DigestVerifyUpdate implementation: remove the (unsigned int) cast

Test

New test test_wolfSSL_EVP_DigestSign_size_t_cnt in tests/api/test_evp_pkey.c:

• Passes size_t msgSz (not unsigned int) to both DigestSignUpdate and DigestVerifyUpdate
• Verifies output against RFC 4231 §4.2 HMAC-SHA256 test vector (independent oracle)
• On 64-bit platforms, verifies that cnt > UINT_MAX returns WOLFSSL_FAILURE instead of truncating

Refs: ZD-21734

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10365

Scan targets checked: wolfcrypt-bugs, wolfcrypt-src

No new issues found in the changed files. ✅

[Copilot]

Pull request overview

Aligns wolfSSL’s OpenSSL-compat EVP DigestSign/DigestVerify update APIs with OpenSSL’s size_t count type to restore source-level compatibility (including FFI bindings) and prevent silent truncation on 64-bit platforms.

Changes:

• Update wolfSSL_EVP_DigestSignUpdate’s public declaration/definition to accept size_t cnt.
• Change the shared internal helper to accept size_t and add an overflow guard before narrowing for wc_HmacUpdate.
• Add a regression test covering size_t usage and the oversized-count failure behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
wolfssl/openssl/evp.h	Changes wolfSSL_EVP_DigestSignUpdate(..., cnt) to size_t for OpenSSL API parity.
wolfcrypt/src/evp.c	Updates helper + removes narrowing cast; adds overflow guard for HMAC update path.
tests/api/test_evp_pkey.h	Registers new EVP PKEY test in declarations list.
tests/api/test_evp_pkey.c	Adds regression test validating size_t usage and overflow handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m0plus

• FLASH: .text +476 B (+0.8%, 63,163 B / 262,144 B, total: 24% used)

gcc-arm-cortex-m3

• FLASH: .text +440 B (+0.4%, 120,573 B / 262,144 B, total: 46% used)

gcc-arm-cortex-m4

• FLASH: .text +448 B (+0.2%, 197,934 B / 262,144 B, total: 76% used)

gcc-arm-cortex-m4-baremetal

• FLASH: .text +448 B (+0.7%, 65,739 B / 262,144 B, total: 25% used)

gcc-arm-cortex-m4-crypto-only

• FLASH: .text +448 B (+0.3%, 173,038 B / 262,144 B, total: 66% used)

gcc-arm-cortex-m4-dtls13

• FLASH: .text +448 B (+0.3%, 178,584 B / 1,048,576 B, total: 17% used)

gcc-arm-cortex-m4-min-ecc

• FLASH: .text +448 B (+0.7%, 60,717 B / 262,144 B, total: 23% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .rodata +544 B, .text +640 B (+0.2%, 764,308 B / 1,048,576 B, total: 73% used)

gcc-arm-cortex-m4-pkcs7

• FLASH: .text +448 B (+0.2%, 210,417 B / 262,144 B, total: 80% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +384 B (+0.1%, 276,120 B / 1,048,576 B, total: 26% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text +448 B (+0.1%, 321,592 B / 1,048,576 B, total: 31% used)

gcc-arm-cortex-m4-sp-math

• FLASH: .text +448 B (+0.7%, 60,717 B / 262,144 B, total: 23% used)

gcc-arm-cortex-m4-tls12

• FLASH: .text +384 B (+0.3%, 121,293 B / 262,144 B, total: 46% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +448 B (+0.2%, 233,440 B / 262,144 B, total: 89% used)

gcc-arm-cortex-m7

• FLASH: .text +448 B (+0.2%, 197,934 B / 262,144 B, total: 76% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +384 B (+0.1%, 276,696 B / 1,048,576 B, total: 26% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +448 B (+0.2%, 233,440 B / 262,144 B, total: 89% used)

linuxkm-pie

• Data: __patchable_function_entries +48 B (+0.2%, 24,176 B)

linuxkm-standard

• Data: __patchable_function_entries +32 B (+0.1%, 45,880 B)

stm32-sim-stm32h753

• FLASH: .text +704 B (+0.4%, 181,428 B / 2,097,152 B, total: 9% used)

[dgarske]

@MarkAtwood please resolve merge conflicts.