Report a failed PEM decode as WS_PARSE_E, not WS_BAD_FILE_E

[yosuke-wolfssl]

Problem

WS_BAD_FILE_E (-1019) meant two unrelated things: "I could not read the file"
(ReadFileIntoBuffer()) and "the PEM body was malformed" (DoPemCert() in
src/ssh.c, wolfSSH_ProcessBuffer() in src/internal.c).

So wolfSSH_ReadCert_buffer() — a public buffer entry point that opens no file
— answered -1019 for a PEM whose base64 body will not decode. A caller could
not tell "the bytes you gave me are not a certificate" from "your file would
not read" without knowing which form it handed in. WS_PARSE_E (-1005) is
already the answer for malformed input everywhere else in this reader,
including the DER arm two functions away.

Inherited style rather than a new mistake: wolfSSH_ProcessBuffer() has done
this for years and DoPemCert() matched its neighbour when PR #1140 added it.

Fix (src/ssh.c)

"The PEM body would not decode" now answers WS_PARSE_E from every entry
point, for certificate, CA and private-key buffers alike. WS_BAD_FILE_E is
left meaning only file I/O.

Site	Was	Is
DoPemCert(), src/ssh.c	WS_BAD_FILE_E	WS_PARSE_E
wolfSSH_ProcessBuffer() PEM arm, src/internal.c	WS_BAD_FILE_E	WS_PARSE_E

Measured against the built library, before → after:

ReadCert_buffer  [bad body]     -1019 -> -1005   ReadCert_file    [NULL name]    -1019 -> -1019
ReadCert_buffer  [header only]  -1019 -> -1005   ReadCert_file    [missing file] -1019 -> -1019
UseCert_buffer   [bad body]     -1019 -> -1005   UseCert_file     [missing file] -1019 -> -1019
UseCert_buffer   [header only]  -1019 -> -1005   AddRootCert_file [missing file] -1019 -> -1019
AddRootCert_buf  [bad body]     -1019 -> -1005
UsePrivateKey_bf [bad body]     -1019 -> -1005

wolfssh/ssh.h gained a note naming the split. No caller in the tree
discriminates on these codes; the only == WS_BAD_FILE_E comparison is in
src/wolfsftp.c and never reaches these paths.

Tests (tests/api.c)

This path had no coverage at all, so all six assertions are new pins rather
than moved ones — added to test_wolfSSH_ReadCert_buffer,
test_wolfSSH_CTX_UseCert_buffer, test_wolfSSH_CTX_AddRootCert_file (the
BUFTYPE_CA case, as no _buffer test exists) and
test_wolfSSH_CTX_UsePrivateKey_buffer_pem. The existing half-PEM case stays
pinned at WS_BAD_FILETYPE_E so sniff-rejection and decode-rejection remain
distinguishable.

Verification

• make check: 11 passed, 0 failed, 1 skipped (network-dependent external.test).
• Negative control: the new assertions fail at -1019 before the fix.
• gcc-13 -Werror clean across 6 configs, certs-on and certs-off.

[Copilot]

Pull request overview

This PR refines wolfSSH’s certificate/key loading error semantics so malformed PEM bodies are consistently reported as WS_PARSE_E (parse failure) rather than WS_BAD_FILE_E (file I/O failure), especially for buffer-based APIs that never touch the filesystem.

Changes:

• Change PEM decode failure returns from WS_BAD_FILE_E to WS_PARSE_E in the PEM certificate decode path (DoPemCert) and the generic buffer processing PEM arm (wolfSSH_ProcessBuffer).
• Document the intended split of WS_BAD_FILE_E (file read problems only) vs WS_PARSE_E (malformed but recognized content) in the public header.
• Add negative tests to pin the corrected return codes for malformed PEM certificate/key bodies across several APIs.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
wolfssh/ssh.h	Documents the intended separation between file I/O failures and decode/parse failures for cert reading APIs.
src/ssh.c	Updates the certificate PEM decode failure to return WS_PARSE_E and clarifies the debug log message.
src/internal.c	Updates generic PEM decode failure handling in wolfSSH_ProcessBuffer() to return WS_PARSE_E.
tests/api.c	Adds regression tests ensuring malformed PEM bodies return WS_PARSE_E for buffer-based cert/key loading APIs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #1151

Scan targets checked: wolfssh-bugs, wolfssh-src

Findings: 4
4 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

🔵 [Low] New out-param assertions in test_wolfSSH_ReadCert_buffer pass vacuously · Weak or missing assertions

out, outSz, outType, outTypeSz and flavor are already NULL/0/UNKNOWN entering these calls, cleared by the preceding half-PEM rejection at line 870. The five checks would pass even if wolfSSH_ReadCert_buffer left every out param untouched, so they pin nothing.

Fix: Poison the out params with sentinels before each call, as the existing block at lines 934-946 does.

[wolfSSL-Fenrir-bot]

⚪ [Info] New ssh.h note misstates when WS_BAD_FILE_E is returned · Incorrect error handling

The note says WS_BAD_FILE_E comes from the _file entry point "only when the file itself will not read", but wolfSSH_ReadCert_file also returns it for a NULL name (src/ssh.c:2753) with no file access — behaviour pinned at tests/api.c:990.

Fix: Extend the note to say a NULL file name also answers WS_BAD_FILE_E.

[wolfSSL-Fenrir-bot]

🔵 [Low] New ret = 0 rejection is unpinned on the private-key arm · Missing edge-case coverage on a function the PR also changed

The PR widens wolfSSH_ProcessBuffer's PEM guard from ret < 0 to ret <= 0 (src/internal.c:2610), newly rejecting a zero-length decode. Only zeroLenPemCert pins that for BUFTYPE_CERT/BUFTYPE_CA; badPemKey exercises only the negative-return path, so reverting the guard on the BUFTYPE_PRIVKEY arm still passes and lets derSz == 0 reach IdentifyAsn1Key.

Fix: Add a zero-length-body PEM private key fixture (a -----BEGIN PRIVATE KEY----- block under one base64 group) asserting WS_PARSE_E.

[wolfSSL-Fenrir-bot]

⚪ [Info] New ssh.h return-code contract is contradicted by pinned behaviour · SSH protocol violations

The added comment states WS_BAD_FILE_E comes from the _file entry point "only when the file itself will not read", but wolfSSH_ReadCert_file also returns it for a NULL name (src/ssh.c:2754, pinned at tests/api.c:990). It also omits WS_INVALID_ALGO_ID and WS_UNIMPLEMENTED_E, which the buffer form returns for decodable certificate content (tests/api.c:864, 958).

Fix: Note that a NULL name also yields WS_BAD_FILE_E and that WS_INVALID_ALGO_ID / WS_UNIMPLEMENTED_E are possible for unsupported key types.