hooks enabled: an exporter exiting via onFailure: exit reports itself as Available to the controller

[evakhoni]

exporter with beforehook fail exit, reports itself as available before closing, so the controller keeps reporting it like this for a while.
also, an attempt to lease it in this state is causing a client to acquire a lease and freeze.

 uv run jmp shell --selector hooks=both --duration 30m 
[02/17/26 19:05:40] INFO     Acquiring lease 019c6c90-935b-70d3-9c0f-79388873f4a3 for selector hooks=both for duration 0:30:00 [jumpstarter.client.lease]
                    INFO     Waiting for ready connection at /run/user/1000/jumpstarter-9rs3mgl_/socket [jumpstarter.client.lease]
                    INFO     Status changed: None -> BEFORE_LEASE_HOOK_FAILED (version=3) [jumpstarter.client.status_monitor]
                    INFO     Releasing Lease 019c6c90-935b-70d3-9c0f-79388873f4a3 [jumpstarter.client.lease]
Error: beforeLease hook failed (on_failure=exit, shutting down): Hook failed with exit code 127


 uv run jmp admin get exporter --namespace jumpstarter-lab
NAME             STATUS     ENDPOINT                               DEVICES  AGE  
hooks-test-both  Available  grpc.jumpstarter.10.0.0.1.nip.io:8082  3        6d21h
hooks-test-none  Offline    grpc.jumpstarter.10.0.0.1.nip.io:8082  0        6d21h


 uv run jmp shell --selector hooks=both --duration 30m 
[02/17/26 19:06:00] INFO     Acquiring lease 019c6c90-e1be-75b2-8891-477cced1ef4c for selector hooks=both for duration 0:30:00 [jumpstarter.client.lease]
                    INFO     Waiting for ready connection at /run/user/1000/jumpstarter-ymsxey97/socket [jumpstarter.client.lease]
^CSIGINT pressed, terminating
[02/17/26 19:06:05] INFO     Releasing Lease 019c6c90-e1be-75b2-8891-477cced1ef4c [jumpstarter.client.lease]

exporter with afterLease fail exit, disconnects the client with connection lost error, and similarly reports as available at the controller

 uv run jmp shell --selector hooks=both --duration 30m 
[02/17/26 19:09:29] INFO     Acquiring lease 019c6c94-12a6-716a-ab0b-ee9994e4bd28 for selector hooks=both for duration 0:30:00 [jumpstarter.client.lease]
                    INFO     Waiting for ready connection at /run/user/1000/jumpstarter-gsahhm8v/socket [jumpstarter.client.lease]
                    INFO     Status changed: None -> LEASE_READY (version=3) [jumpstarter.client.status_monitor]
python ⚡hooks-test-both ➤ 
[02/17/26 19:09:32] INFO     Running afterLease hook (Ctrl+C to skip)... [jumpstarter_cli.shell]
                    INFO     Starting hook subprocess... [exporter:afterLease]
                    INFO     Creating PTY pair... [exporter:afterLease]
                    INFO     PTY created: parent_fd=16, child_fd=17 [exporter:afterLease]
                    INFO     Executing inline script (interpreter: /bin/sh) [exporter:afterLease]
                    INFO     Spawning subprocess with command: ['/bin/sh', '-c', 'echo "exit failure prelease"\nexit 127\n'] [exporter:afterLease]
                    INFO     Subprocess spawned with PID 2728537 [exporter:afterLease]
                    INFO     Starting PTY output reader and process waiter (timeout=60) [exporter:afterLease]
                    INFO     Task group created, starting tasks... [exporter:afterLease]
                    INFO     Waiting for subprocess to complete... [exporter:afterLease]
                    INFO     exit failure prelease [exporter:afterLease]
                    INFO     Subprocess completed with code: 127 [exporter:afterLease]
[02/17/26 19:09:52] INFO     Connection lost (UNAVAILABLE), signaling waiters [jumpstarter.client.status_monitor]
Error: Ready for commands

[mangelajo]

Is this already fixed by #349 ?

[ambient-code]

Investigation: Issue #245 - Exporter with onFailure=exit reports Available before shutting down

Status: Partially Fixed by PR #349

PR #349 (commit d6cad89) directly addressed this issue. The fix adds an OFFLINE status report before the shutdown call in the run_before_lease_hook() method (hooks.py):

# Before PR #349: only reported BEFORE_LEASE_HOOK_FAILED, then called shutdown()
# After PR #349: reports BEFORE_LEASE_HOOK_FAILED, then OFFLINE, then calls shutdown()
await report_status(
    ExporterStatus.OFFLINE,
    "Exporter shutting down due to beforeLease hook failure",
)
shutdown(exit_code=1, wait_for_lease_exit=True, should_unregister=True)

The same fix was applied to run_after_lease_hook() for the afterLease case.

Additionally, PR #426 (commit c562cf2) added a filterOutNotReadyExporters check in the controller's lease assignment pipeline, which prevents exporters in non-ready states (including hook states) from being assigned new leases.

Remaining Concern: Race Window

There is still a small race window between the BEFORE_LEASE_HOOK_FAILED status report and the OFFLINE status report. If the controller checks exporter status in this window, it could theoretically see the BEFORE_LEASE_HOOK_FAILED status. However, PR #426's server-side filtering mitigates this since BEFORE_LEASE_HOOK_FAILED is not in the "ready" set.

The _request_lease_release() method (exporter.py line 363-404) also has a guard: when _stop_requested is true and a shutdown is pending, the _cleanup_after_lease method (line 617-624) skips reporting AVAILABLE status:

if not self._stop_requested:
    await self._report_status(ExporterStatus.AVAILABLE, "Available for new lease")
else:
    logger.debug("Exporter is shutting down, skipping AVAILABLE status report")

And the run_after_lease_hook finally block (hooks.py line 722-723) guards lease release when shutdown was called:

if request_lease_release and not shutdown_called:
    await request_lease_release()

Verdict

This issue should be resolved by the combination of PR #349 and PR #426. The exporter now:

1. Reports OFFLINE before shutdown on onFailure=exit (PR Fix exporter state machine: stuck exporters on lease-end during hooks #349)
2. Skips AVAILABLE status when shutdown is pending (exporter.py line 617-624)
3. The controller won't assign leases to non-ready exporters regardless (PR fix: prevent lease assignment to non-ready exporters #426)

Client Freeze Sub-Issue

The reported client freeze when leasing an exporter in the transient AVAILABLE state is also mitigated: the controller-side filtering (PR #426) prevents the lease assignment entirely. If a client somehow does get a lease on a dying exporter, the exporter's shutdown and unregistration will cause the client to see UNAVAILABLE errors, which the StatusMonitor handles by setting connection_lost and exiting.

Recommend closing this issue if PR #349 and #426 are confirmed deployed, or adding a focused e2e test for the beforeLease onFailure=exit -> no AVAILABLE status scenario to prevent regression.

Relevant Files

• /workspace/repos/jumpstarter/python/packages/jumpstarter/jumpstarter/exporter/hooks.py - OFFLINE status before shutdown (PR Fix exporter state machine: stuck exporters on lease-end during hooks #349)
• /workspace/repos/jumpstarter/python/packages/jumpstarter/jumpstarter/exporter/exporter.py - _cleanup_after_lease skips AVAILABLE when stopping
• controller/internal/controller/lease_controller.go - filterOutNotReadyExporters (PR fix: prevent lease assignment to non-ready exporters #426)