feat(serve): surface a coarse startup phase while a service comes up

[michaelroy-amd]

Stacked on #106 (EAI-7350). Review/merge #106 first; this PR's base is fix/dash-instance-status, so its own diff is only the startup-phase work.

Summary

While a managed service is coming up, the dashboard showed only a generic STARTING label (or a spinner) with no indication of what it was doing — pulling weights, loading them onto the device, or warming up. A slow first launch was therefore indistinguishable from a hang. This PR surfaces a coarse startup phase (Downloading / Loading / Warmup) end to end.

Root Cause

The serve supervisor (apps/rocmd) already redirects the engine's stdout/stderr into the per-service log file, but wait_for_service_ready only polled a healthcheck RPC and never inspected that output. ManagedServiceRecord had no field to carry a startup stage, and the dashboard's InstanceStatus::Starting was a unit variant with nowhere to put one.

Changes

• rocm-core: add ManagedServiceRecord.startup_phase: Option<String> (#[serde(default)] so older on-disk records still deserialize), cleared back to None once the service reaches ready.
• rocmd: wait_for_service_ready now tails the service log while polling and reports each phase transition, which the caller persists to the record. Parsing is split into small pure, unit-tested helpers:
  • last_cr_segment collapses \r progress redraws (pip/tqdm/HF), the same rule the dashboard job console uses.
  • classify_startup_phase matches the common vLLM / llama.cpp / Hugging Face startup vocabulary into a downloading/loading/warmup token.
• rocm-dash-core: add a fieldless StartupPhase enum and change InstanceStatus::Starting to Starting { phase: Option<StartupPhase> }. StartupPhase is Copy, so InstanceStatus stays Copy — the only ripple is pattern shape (Starting { .. }), not a Copy-removal cascade. A new InstanceStatus::label() centralizes the status text (READY / DOWNLOADING / LOADING / WARMUP / …).
• rocm-dash-daemon (registry): map the record's startup_phase onto Starting { phase } for starting/recovering records; Docker discovery and the scrape-success promotion keep phase: None.
• rocm-dash-tui: status_meta/status_role and the services-manager list now render via label(), so the phase shows on instance cards, the detail pane, and the services list. Note: serving.rs is a static verb menu with no per-instance status rendering, so the live phase display lives in instances.rs + services_manager.rs (the actual instance-status surfaces).

Scope / Notes

• Phase detection is best-effort log scraping (no structured engine signal exists today); an unrecognized phase token degrades gracefully to a plain STARTING.
• The Starting serde wire shape changes from the bare string "starting" to an externally-tagged object; this only affects the in-version daemon socket/replay stream, and the round trip is covered by a new test.

Test Plan

• cargo build (workspace)
• cargo clippy --all-targets --all-features -- -D warnings (workspace, clean)
• cargo test -p rocm-core -p rocm-dash-core -p rocm-dash-collectors -p rocm-dash-daemon — green, incl. new tests: instance_status_label_surfaces_startup_phase, instance_status_starting_round_trips_through_serde, startup_phase_from_token_round_trips_labels, discovered_from_record_surfaces_startup_phase_while_starting
• cargo test -p rocmd --lib — 109 green, incl. last_cr_segment_keeps_final_progress_redraw, classify_startup_phase_maps_engine_vocabulary, classify_startup_phase_emits_only_dashboard_known_tokens, read_new_log_phase_advances_and_tracks_latest
• cargo test -p rocm-dash-tui -- --test-threads=1 — 543 + 16 + 5 green

Relates to EAI-7355

[rominf]

LGTM. The startup-phase enum migration is complete (every Starting construction site updated; compiler-enforced exhaustiveness), the cross-crate serde contract is pinned by a real ManagedServiceRecord→ServiceRecord round-trip test, the PROTOCOL_VERSION 1→2 bump is correct with warn-and-continue on mismatch, unknown/forward tokens keep a running record serving (never downgraded to a non-serving Starting), and #[serde(default)] keeps old on-disk records readable. Right layering: phase derived in the supervisor, mapped once in the registry, labeled once in InstanceStatus::label. Log truncation on restart avoids stale-phase pickup.

Low-severity notes:

1. read_new_log_phase advances the read cursor by the bytes read regardless of line boundaries, so a phase line split across two ~200ms polls can be missed (cosmetic — the next matching line re-triggers; a seek-to-last-newline guard would fix).
2. The PR adds a record.write() per phase transition on a non-atomic writer (pre-existing fs::write pattern; negligible given ≤3 transitions and a best-effort reader).

Approving.

[volen-silo]

🔴 Automated review · pr-review-watcher · 7e1d04e

Summary

• Change: Surfaces a coarse startup phase (DOWNLOADING / LOADING / WARMUP) end to end — rocmd tails the serve log while polling for readiness, persists the phase onto ManagedServiceRecord, and the dashboard renders it on the Starting status.
• Overall assessment: Approve (with minor suggestions). No blocking issues. The core producer→consumer state machine is correct, the cross-crate serde contract is pinned by a real round-trip test, forward/foreign phase tokens degrade gracefully, and the full workspace test suite passes. This is a clean, unusually well-tested change.
• Scope note: Reviewed against base origin/fix/dash-instance-status (this PR is stacked on fix(dash): give instances a real Ready status instead of a Running catch-all #106); the diff is only the startup-phase work.

Remaining Concerns

Blocking

None.

Non-blocking suggestions

1. Partial-line log reads can drop a one-shot phase transition — apps/rocmd/src/lib.rs read_new_log_phase. The cursor advances past every byte read each 200 ms poll, including an as-yet-unterminated trailing line. If a poll lands mid-write and splits a phase-bearing line (e.g. "Downloading shards: 100%\n" → "Downloadin" + "g shards: 100%\n"), neither fragment matches and the transition is lost, because the cursor has already moved past the first half. Impact is low (the mechanism is documented best-effort, progress lines usually repeat, and a later stage supersedes a missed one), but a phase that appears only once could be missed. Consider only advancing *pos past the last \n and carrying the incomplete trailing fragment across polls.

2. Old replay recordings silently drop Starting entries — crates/rocm-dash-tui/src/replay.rs read_entries. .ndjson session recordings written under PROTOCOL_VERSION 1 encode InstanceStatus::Starting as the bare string "starting". With the new externally-tagged object shape, each such line now fails serde_json::from_str and is warn!-logged and skipped — so replaying an old recording silently vanishes every historical Starting-state entry with no user-facing signal. The daemon socket path is guarded by the PROTOCOL_VERSION bump (correct), but the on-disk replay file has no version gate or migration. This is a local developer/debug feature (degrades, doesn't crash), so it's non-blocking — but worth a note in the PR, or a one-line version check on replay load.

3. Test gaps (all non-blocking):

   • instance_status_starting_round_trips_through_serde (metrics.rs) asserts round-trip equality but never pins the literal wire string, so it would not catch an accidental removal of #[serde(rename_all = "lowercase")] that breaks compatibility with anything matching the literal tag. A single assert_eq! on the serialized JSON ({"starting":{"phase":"warmup"}}) would lock the contract.
   • read_new_log_phase has an explicit truncation-reset branch (len < *pos) with no test exercising it.
   • No regression test for the partial-line case in suggestion Let Lemonade auto-select its llama.cpp backend #1 (would pin the fix if adopted).

4. managed_service_record_startup_phase_survives_cross_crate_serde hardcodes /tmp/rocm-xcrate-test (registry.rs) instead of the PID-namespaced test_dir() helper used elsewhere. It's safe today because the test never touches disk (it only serializes/deserializes in memory), but it's a latent collision trap if someone later adds a .write(). Prefer test_dir()/tempfile for consistency.

Tradeoffs (deliberate, no action required)

• Old TUI client vs new daemon: version mismatch is warn-and-continue (client.rs:139, pre-existing, not introduced here). This PR is the first to make the wire shape genuinely incompatible, so an old client against a new daemon now connects, warns, then fails to deserialize the first Starting-carrying event and enters a reconnect/backoff loop. The PROTOCOL_VERSION bump is the right lever; enforcing it (hard-fail on mismatch) would be a separate, broader change.
• running + startup_phase overlap → Starting: mapping a running record with a recognized phase to non-serving Starting{phase} means a downloading/loading instance correctly drops out of is_serving() surfaces (dock "Running Services", launcher, home counts). This is the intended fix (a downloading model was previously miscounted as serving), verified to have no unintended fallout, but it is a visible behavior change worth confirming was deliberate.

Positive signals

• The registry.rs doc comment explaining the running + startup_phase overlap was cross-checked against the actual producer (supervise_service) and is accurate to the exact line ordering and clearing point — rare precision.
• Forward-compat handling (an unrecognized/foreign phase token keeps a running record serving rather than downgrading it) is implemented and tested (discovered_running_with_unknown_phase_stays_serving) — a real robustness win for producer/consumer version skew.
• The cross-crate contract that rocm-core (String tokens) can't type-share with rocm-dash-core (StartupPhase enum) is pinned two ways: classify_startup_phase_emits_only_dashboard_known_tokens on the producer side and managed_service_record_startup_phase_survives_cross_crate_serde on the consumer side (via a dev-only path dependency, no shipped-dependency or cycle risk).
• Switching services_manager.rs from format!("{:?}", i.status) to .label() is a genuine fix, not cosmetics: {:?} on the new struct variant would render "Starting { phase: None }" in the UI.

Deployment Notes

• Wire/protocol break: PROTOCOL_VERSION 1 → 2. InstanceStatus::Starting changes from the string "starting" to {"starting":{"phase":...}} on the daemon socket/replay stream. Correctly versioned for the live socket; old on-disk replay recordings degrade silently (see suggestion Enable native-certs for ureq across all crates #2).
• On-disk records: ManagedServiceRecord.startup_phase uses #[serde(default)], so older records deserialize fine (verified JSON storage, not positional).
• No new third-party dependencies (the added rocm-core is a workspace-local dev-dependency).

---

Automated review — plain comment only, no approval/merge. The producer-side log-scrape partial-line and replay backward-compat items are the two most worth a maintainer's glance; neither blocks merge.