Chore
Describe the chore
We run edge-runtime at scale in a self-hosted setting, using the default per_request supervisor policy. For worker lifecycle management, we already have a working implementation locally and have verified it is viable. We'd like to confirm whether this implementation can be adopted upstream, and whether the interface needs adjustment before adoption.
Both improvements point to the same underlying issue: in per_request mode, the count and held environment state of in-process workers lack explicit management. Each is described below with the current behavior and our implemented approach.
Improvement 1 — Process-level worker cap.
Current behavior: Worker concurrency is limited only at the per-function level. ActiveWorkerRegistry (crates/base/src/worker/pool.rs:156) is created per service_path, each holding a Semaphore sized to max_parallelism (pool.rs:164-169). With N distinct function paths, up to N × max_parallelism workers can coexist, with no process-level ceiling. In a self-hosted deployment where the set of functions is open-ended, concurrent requests spanning enough distinct functions drive the live worker count up until the process is OOM-killed.
Our implemented approach:
- One process-global Semaphore with permit count = the global worker capacity ceiling
- User worker creation queues in two stages: acquire the function-level permit first (unchanged), then the global permit
- On global-permit timeout, fail via the existing request_wait_timeout_ms
- Defaults to unlimited, preserving current behavior; the ceiling is opt-in via a CLI flag, set by the deployer based on their machine
This approach has been verified in our local self-hosted environment: under high-concurrency, multi-function load the process is no longer OOM-killed, and creation requests beyond the ceiling queue and time out as expected instead of memory running unbounded.
Improvement 2 — Secret revision refresh.
Current behavior: A worker's env_vars is a one-time snapshot taken at creation — the main worker reads std::env::vars() (worker_surface_creation.rs:645), and user workers receive theirs from the main worker via op_user_worker_create (ext/workers/lib.rs:208→248), stored in WorkerContextInitOpts and never refreshed afterward. The docs state secrets are "available immediately" after update with no redeploy, but this relies on workers being short-lived and passively reclaimed (mark_idle at pool.rs:212, then try_cleanup_idle_workers). Under per_request at high concurrency, a function's workers keep being reused before they can be reclaimed, so they keep serving with stale secrets.
Our implemented approach (the main worker notifies the WorkerPool to advance the revision, and the WorkerPool retires stale workers):
- Snapshot the revision at worker creation. When a user worker is created, the secret revision at that moment (env_revision) is stored in UserWorkerRuntimeOpts and UserWorkerProfile as the worker's revision snapshot (immutable thereafter).
- Re-check on build completion. Worker construction is asynchronous, so the env_revision carried at creation may go stale during the build. After builder.build() completes, the worker re-reads the pool's current minimum valid revision minimum_env_revision and compares; if already stale, it is not registered into active_workers (avoiding a worker that is stale from birth still getting dispatched).
- The main worker notifies the WorkerPool after a secret update. The runtime exposes the op_advance_env_revision(revision) op (TS side: EdgeRuntime.UserWorker.advanceEnvRevision(revision)). After the main worker completes a secret update, it calls this to send the new revision to the WorkerPool. On receiving AdvanceEnvRevision(revision), the WorkerPool invokes advance_env_revision.
- The WorkerPool advances the minimum valid revision and retires stale workers. advance_env_revision uses fetch_max to push minimum_env_revision to the new value (monotonically increasing), then walks active_workers and removes any worker whose revision is below the new minimum from the dispatch table (revoking eligibility for new requests, without terminating immediately — to avoid interrupting in-flight requests), and notifies the corresponding worker's supervisor via env_revision_tx that it is now stale.
- The supervisor terminates the stale worker once in-flight requests finish. On receiving the stale notification, the supervisor sets the is_env_stale flag and waits until all of that worker's in-flight requests have completed (demand == req_ack_count), then goes through the unified termination flow. After termination, the worker's function-level and global permits are released, and the next request builds a fresh worker reading the latest secret.
Key design trade-offs:
- Retirement and termination are separate. Retiring (revoking dispatch eligibility from active_workers) and terminating (releasing the worker process) are two steps, separated by "wait for in-flight requests to finish." retire() cannot be reused here — it releases permits immediately and triggers the exit flow, which would interrupt in-flight requests.
- The trigger lives on the main-worker side. The runtime only provides the op and the pool-side handling; "when a secret update is considered complete, and which revision to pass to advanceEnvRevision" is decided by the main worker (user function code). In our local build, the main worker calls it after a secret update succeeds.
This approach has been verified locally: after a secret update, workers holding the old secret stop accepting new requests and are terminated once in-flight requests finish; new requests build fresh workers that read the new secret, no longer dependent on the timing of passive reclamation.
(Note: under per_worker, workers are long-lived and secrets never refresh at all — a heavier form of the same issue. We consider that acceptable by shortening worker lifetime and tolerating brief inconsistency, so it's not the focus here.)
Additional context
Two notes:
- Both approaches above are already implemented and verified viable in our local self-hosted environment — this is not a design-stage proposal. If the direction is agreed, we're happy to refactor the existing implementation to an aligned interface and open a PR.
- Our implementation is based on v1.70.3; before opening a PR we'll rebase onto the latest main and resolve any conflicts.
We'd like to confirm:
- Can these two improvements, and our existing implementation approach, be adopted?
- Is a secret-revision mechanism already planned, or being explored in another direction? If so, we'd like to avoid duplicating it.
Chore
Describe the chore
We run edge-runtime at scale in a self-hosted setting, using the default per_request supervisor policy. For worker lifecycle management, we already have a working implementation locally and have verified it is viable. We'd like to confirm whether this implementation can be adopted upstream, and whether the interface needs adjustment before adoption.
Both improvements point to the same underlying issue: in per_request mode, the count and held environment state of in-process workers lack explicit management. Each is described below with the current behavior and our implemented approach.
Improvement 1 — Process-level worker cap.
Current behavior: Worker concurrency is limited only at the per-function level. ActiveWorkerRegistry (crates/base/src/worker/pool.rs:156) is created per service_path, each holding a Semaphore sized to max_parallelism (pool.rs:164-169). With N distinct function paths, up to N × max_parallelism workers can coexist, with no process-level ceiling. In a self-hosted deployment where the set of functions is open-ended, concurrent requests spanning enough distinct functions drive the live worker count up until the process is OOM-killed.
Our implemented approach:
This approach has been verified in our local self-hosted environment: under high-concurrency, multi-function load the process is no longer OOM-killed, and creation requests beyond the ceiling queue and time out as expected instead of memory running unbounded.
Improvement 2 — Secret revision refresh.
Current behavior: A worker's env_vars is a one-time snapshot taken at creation — the main worker reads std::env::vars() (worker_surface_creation.rs:645), and user workers receive theirs from the main worker via op_user_worker_create (ext/workers/lib.rs:208→248), stored in WorkerContextInitOpts and never refreshed afterward. The docs state secrets are "available immediately" after update with no redeploy, but this relies on workers being short-lived and passively reclaimed (mark_idle at pool.rs:212, then try_cleanup_idle_workers). Under per_request at high concurrency, a function's workers keep being reused before they can be reclaimed, so they keep serving with stale secrets.
Our implemented approach (the main worker notifies the WorkerPool to advance the revision, and the WorkerPool retires stale workers):
Key design trade-offs:
This approach has been verified locally: after a secret update, workers holding the old secret stop accepting new requests and are terminated once in-flight requests finish; new requests build fresh workers that read the new secret, no longer dependent on the timing of passive reclamation.
(Note: under per_worker, workers are long-lived and secrets never refresh at all — a heavier form of the same issue. We consider that acceptable by shortening worker lifetime and tolerating brief inconsistency, so it's not the focus here.)
Additional context
Two notes:
We'd like to confirm: