feat(agents): use certifi as CA fallback - #1917
Conversation
🦋 Changeset detectedLatest commit: 0bcdc31 The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| return true; | ||
| } | ||
|
|
||
| return systemCaFiles.some(pathIsFile) || systemCaDirs.some(pathIsDirectory); |
There was a problem hiding this comment.
🔴 Certificate fallback skipped on minimal Docker images that have empty certificate directories
The trust-store detection treats an empty directory as proof that certificates exist (systemCaDirs.some(pathIsDirectory) at agents/src/certificates.ts:50), so the certifi fallback never activates on systems that need it.
Impact: On minimal Linux containers (e.g., Debian-slim or Alpine without ca-certificates), the fallback silently does nothing, leaving TLS errors unresolved — exactly the scenario this feature targets.
Directory-existence check matches empty dirs created by OpenSSL packages
On Debian-based or Alpine images, the openssl package creates /etc/ssl/certs/ as part of its directory structure, but the ca-certificates package is what actually populates it with certificate files. In minimal Docker images that omit ca-certificates, /etc/ssl/certs/ exists as an empty directory.
hasSystemTrustStore() at agents/src/certificates.ts:45-51 checks:
return systemCaFiles.some(pathIsFile) || systemCaDirs.some(pathIsDirectory);
The file checks (systemCaFiles.some(pathIsFile)) correctly fail because no cert bundle files like ca-certificates.crt exist. But the directory check (systemCaDirs.some(pathIsDirectory)) matches the empty /etc/ssl/certs/ directory, making the function return true.
This causes setDefaultCertEnv() at agents/src/certificates.ts:54-62 to return early without setting the certifi fallback.
For comparison, Go's crypto/x509 also lists these directories, but it attempts to load certificates from them rather than using directory existence as a boolean "has trust store" signal. The semantic difference is significant: an existing-but-empty directory means "no certificates" when loading, but means "has trust store" in this boolean check.
The file checks alone (without the directory checks) would correctly detect the absence of certificates.
Prompt for agents
The hasSystemTrustStore() function in agents/src/certificates.ts:50 uses systemCaDirs.some(pathIsDirectory) to check for system CA directories, but this matches empty directories that contain no actual certificate files. On minimal Docker images (e.g., Debian-slim or Alpine without ca-certificates), /etc/ssl/certs/ can exist as an empty directory created by the openssl package.
Two approaches to fix:
1. Remove the systemCaDirs check entirely, relying only on systemCaFiles.some(pathIsFile). The file checks already cover all major distros and are more reliable since they verify actual cert bundle files exist.
2. Enhance pathIsDirectory to verify the directory is non-empty by checking for at least one .pem or .crt file inside, e.g., using readdirSync and filtering by extension.
Approach 1 is simpler and sufficient. The directory-based check was likely ported from Go's crypto/x509, but Go uses those directories to load certificates (empty = no certs loaded), whereas here it's used as a boolean signal.
Was this helpful? React with 👍 or 👎 to provide feedback.
| "@opentelemetry/sdk-trace-node": "^1.28.0", | ||
| "@opentelemetry/semantic-conventions": "^1.28.0", | ||
| "@types/pidusage": "^2.0.5", | ||
| "certifi": "^14.5.15", |
There was a problem hiding this comment.
🚩 The certifi package (v14.5.15) was last published in 2018
The certifi npm package at version ^14.5.15 was last published to npm on 2018-03-23, making its CA certificate bundle approximately 8 years old. Root CA certificates are periodically rotated, and some CAs from 2018 may have expired or been distrusted since then. While this is a dependency choice rather than a code bug, it could mean the fallback bundle doesn't include newer root CAs, potentially causing TLS verification failures for services using recent certificate chains. The Python certifi package is actively maintained with frequent updates; the Node.js port appears abandoned. Consider whether an alternative like node-forge's bundled certs or a more maintained package would be more appropriate.
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
SSL_CERT_FILEbefore worker child processes spawn so RTC/native TLS code inherits the fallbackNODE_EXTRA_CA_CERTSfor forked Node.js subprocesses when it is not already configuredTesting
corepack pnpm@9.7.0 --filter @livekit/agents buildcorepack pnpm@9.7.0 --filter @livekit/agents lint(passes with existing warnings)Ported from livekit/agents#6231
Original PR description
LLM client uses httpx, which falls back to certifi's CA bundle, while STT/TTS plugins go through a shared aiohttp session that relied solely on the host's system trust store. On hosts without a resolvable system store (minimal containers, distroless images missing ca-certificates), aiohttp-based TLS would fail while the LLM client kept working.
The shared TCPConnector now builds its SSL context to: (1) honor
SSL_CERT_FILE/SSL_CERT_DIRenv overrides, (2) prefer the system trust store, and (3) fall back to certifi only when no system store is resolvable on disk. This gives consistent TLS trust roots across LLM, STT, and TTS without overriding custom system CAs when a store is present.The fix is also applied to the worker and job process so they can both use
certifias a fallback