feat(deploy): stream deploy_component as multipart/form-data

[kriszyp]

Summary

• Switches deploy_component from a CBOR-encoded Buffer payload to a chunked multipart/form-data body so components can exceed the Node.js 2 GB Buffer cap.
• The CLI streams the tar+gzip directly from streamPackagedDirectory() into the request, never materializing the full archive in memory.
• The operations API registers a multipart content-type parser that hands the route handler a Readable for the file part, which Application.extractApplication pipes straight into gunzip-maybe + tar-fs.extract — no in-memory hop on the server either.

Why

First slice of #524: support payload-based deploys larger than 2 GB. Next slices are SSE-based progress visibility (#526) and the per-peer HTTPS relay for replication (also in #524).

Where to look

• server/serverHelpers/multipartParser.ts — busboy-driven parser registered as a Fastify content-type parser for multipart/form-data. Calls done(body) as soon as the file part starts (not when it finishes) so the route handler can pump data through extraction with normal backpressure. Field-then-file ordering is a contract documented in the file header; out-of-order late fields are warned and ignored (see the inline comment for why we can't usefully propagate them — by the time we see them, done has already fired and the file stream has typically ended).
• bin/multipartBuilder.ts — small CLI-side body builder, also used by the round-trip test. Fields are JSON-serialized on the wire when non-string so the server-side decoder can reverse it for booleans/numbers/objects.
• bin/cliOperations.js — PREPARE_OPERATION.deploy_component no longer sets cborEncode; it builds a streamPackagedDirectory() and the dispatch code now constructs a multipart body for any _multipart request. TRANSPORT_ONLY_FIELDS is the new allowlist that keeps CLI-internal fields (target, credentials, etc.) out of the wire body.
• utility/common_utils.js — httpRequest now .pipe()s a Readable body when given one; otherwise unchanged.
• components/Application.ts — payload?: Buffer | string | Readable now. The Readable branch is used directly; Buffer/base64 string branches are unchanged.
• package-lock.json — adds busboy + @types/busboy. The lockfile also picks up some incidental cleanup that npm install did: @harperfast/integration-testing is upgraded to the version already specified in package.json's caret range, an unused node-unix-socket entry is removed, and some peer: true flags are reconciled. Worth a quick scan to confirm those are all benign.

Backward compatibility

• The server still accepts application/json and application/cbor bodies for deploy_component (and every other operation), so external clients that built against the old wire format are unaffected.
• packageDirectory() (buffered) is retained alongside the new streamPackagedDirectory() so any other caller depending on a Buffer return value keeps working.

Test plan

• npx mocha unitTests/server/serverHelpers/multipartParser.test.js — 8/8 pass (field decoding, file part streaming, done-before-EOF, unexpected file field rejection, late field ignored, empty body, missing Content-Type, stream error propagation).
• npx mocha unitTests/bin/multipartBuilder.test.js — 6/6 pass (ordering, JSON-stringify of non-strings, undefined skipped, file-less form, unique boundaries, CR/LF + quote sanitization).
• npx mocha unitTests/components/packageComponent.test.js — 2/2 pass (full round-trip: streamPackagedDirectory → multipart body → server parser → gunzip → tar extract; plus buffered/streaming size parity).
• npx mocha unitTests/server/fastifyRoutes/operations.test.js — 21/21 pass (existing deployComponent force-flag tests, unaffected).
• End-to-end deploy of a large component against a live Harper instance — not exercised locally; needs the integration test infra.

Follow-ups (not in this PR)

• Improved deployment visibility #526: SSE-based progress reporting (phase/install/replicate events). The route handler will need to wire a progress emitter through deployComponent — the multipart parser already exposes the byte stream so upload-progress tracking can plug in at the parser layer too.
• Support for deployments larger than 2GB #524 (replication slice): origin spools the multipart file part to a temp tarball on disk, mints a short-lived JWT via create_authentication_tokens, and streams a direct HTTPS deploy to each peer with per-peer retry semantics. Out of scope here.

🤖 Generated with Claude Code

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​types/​busboy@​1.5.4
	busboy@​1.6.0

View full report

[claude]

Reviewed; no blockers found.

[Ethan-Arrowood]

amazing. much need!