sp_sqrmod: zero modulus rejected or accepted depending on output aliasing

[MarkAtwood]

Version

master (a103f5a, 2026-08)

Description

mp_sqrmod() disagrees with itself about a zero modulus. Whether it returns an error depends on where the caller points the result:

mp_sqrmod(a, m, r)  a=0, m=0, r distinct  ->  MP_VAL
mp_sqrmod(a, m, m)  a=0, m=0              ->  MP_OKAY, result 0

In wolfcrypt/src/sp_int.c, sp_sqrmod() sends the r != m case through sp_sqr() then sp_mod(), and sp_mod() rejects a zero modulus. The r == m case calls _sp_sqrmod(), which short-circuits on sp_iszero(a) and returns MP_OKAY without ever looking at m.

A zero modulus is undefined either way, but the two paths should agree.

Found via OSS-Fuzz 513887571 (cryptofuzz BignumCalc SqrMod).

Reproduction steps

#include <wolfssl/options.h>
#include <wolfssl/wolfcrypt/settings.h>
#include <wolfssl/wolfcrypt/integer.h>
#include <stdio.h>

int main(void) {
    mp_int a, m, r;
    mp_init_multi(&a, &m, &r, NULL, NULL, NULL);
    mp_set_int(&a, 0);
    mp_set_int(&m, 0);
    printf("r != m: %d\n", mp_sqrmod(&a, &m, &r));
    mp_set_int(&m, 0);
    printf("r == m: %d\n", mp_sqrmod(&a, &m, &m));
    return 0;
}

Built against ./configure --enable-static --disable-shared.

Relevant log output

r != m: -98
r == m: 0

[MarkAtwood]

mp_mulmod() has the same defect, via the same mechanism. _sp_mulmod_tmp() short-circuits on sp_iszero(a) || sp_iszero(b) and stores zero without ever inspecting m, exactly as _sp_sqrmod() does on sp_iszero(a).

Both are also documented-contract violations, not just a disagreement between two paths. sp_sqrmod() documents "MP_VAL when a, m or r is NULL; or m is 0", and sp_mulmod() documents "MP_VAL when a, b, m or r is NULL; m is 0". The r == m paths return MP_OKAY for a zero modulus.

Reproduced on current master (e5b3fb1), built with ./configure --enable-static --disable-shared:

mp_sqrmod  r != m -> -98
mp_sqrmod  r == m -> 0
mp_mulmod  r != m -> -98
mp_mulmod  r == m -> 0

The mp_mulmod case uses a = 0, b = 5, m = 0.

One note for whoever picks this up: mp_test() is gated behind WOLFSSL_PUBLIC_MP, so a default testwolfcrypt run skips the entire mp suite and any regression test added to mp_test_mulmod_sqrmod() passes vacuously. Building with CFLAGS=-DWOLFSSL_PUBLIC_MP is needed to actually exercise it.

I have a fix that checks the modulus inside the zero-operand branch of both functions, which keeps the check off the exptmod hot path where sp_sqrmod/sp_mulmod are called per bit. It extends the existing aliasing test in mp_test_mulmod_sqrmod() to cover a zero modulus in both arrangements; the test fails before the change and passes after. Happy to open a PR.

[MarkAtwood]

Correction to my previous comment. I said mp_test() being gated behind WOLFSSL_PUBLIC_MP means a regression test added to mp_test_mulmod_sqrmod() "passes vacuously" and implied it would not run in CI. The CI half of that is wrong.

WOLFSSL_PUBLIC_MP is set in upstream CI:

• .github/workflows/cmake.yml builds with -DWOLFSSL_SP_MATH_ALL:BOOL=yes (line 73) and -DWOLFSSL_PUBLIC_MP:BOOL=yes (line 82), then runs ctest (line 87).
• .github/configs/os-check-linux.json passes CPPFLAGS=-DWOLFSSL_PUBLIC_MP (lines 43, 54), and other entries use examples/configs/user_settings_all.h, which defines both WOLFSSL_PUBLIC_MP and WOLFSSL_SP_MATH_ALL.
• It is also picked up transitively through --enable-all, via the eccsi path in configure.ac.

So a test added there does run upstream. What is true, and what actually caught me, is narrower: a plain ./configure && make check locally does not define WOLFSSL_PUBLIC_MP, so testwolfcrypt skips the entire mp suite and exits 0 without printing an mp test line. Anyone verifying a wolfcrypt math fix locally should build with CFLAGS=-DWOLFSSL_PUBLIC_MP and confirm the test actually fails before the fix.

One further note on scope, from tracing the other math backends:

• tfm.c (fp_mulmod, fp_sqrmod) and integer.c (mp_mulmod, mp_sqrmod) do not have this defect. They have no zero-operand short-circuit and always funnel through fp_div / mp_div, which reject a zero divisor before writing output. Both already return MP_VAL in either aliasing.
• wolfcrypt/src/port/maxim/max3266x.c does still have it. hw_mulmod (around line 2497) and hw_sqrmod (around line 2593) short-circuit on a zero operand with mp_zero(result); return 0; and never inspect mod, and wolfmath.h redirects mp_mulmod/mp_sqrmod to them when WOLFSSL_USE_HW_MP is defined. I do not have MAX3266X hardware to test against, so I am reporting it rather than changing it.