server : reset fit_params_target to baseline on model (re)load

[liminfei-amd]

Overview

llama-server with --mmproj (and/or a draft/MTP model) inflates its free-memory
target every time it resumes from sleep. With --sleep-idle-seconds, each wake adds
the mmproj/draft size again, so fit_params_target grows without bound.

Root cause: handle_sleeping_state(false) re-calls load_model(params_base), and
load_model() adds the mmproj size (fit_params_target[i] += size) and the draft/MTP
size (fit_params_target[i] += bytes) to params_base.fit_params_target. That target
is never reset to its first-load baseline, so the additions compound across wakes.

Fix: snapshot the per-device fit_params_target on the first load and restore it before
the mmproj/draft sizes are added, so the first load and every resume accumulate from the
same clean baseline.

Fixes #24475.

Additional information

The same += pattern applies to the draft/MTP branch, so this also covers the draft case
the reporter suspected.

Validation (host-side server logic, hardware-independent):

• Builds cleanly (cmake --build ... --target llama-server, recompiles server-context.cpp).
• A faithful harness of the load_model target lifecycle reproduces the bug and the fix:
  with the reporter's ~1290 MiB mmproj + a 500 MiB draft, the stock target grows
  2814 -> 4604 -> 6394 -> 8184 MiB across three wakes, while the patched target stays at
  2814 MiB on every wake.

I currently have two other open PRs (#24237 HIP runtime, #24450 Vulkan build tooling); this one
touches a different area (server). Happy to serialize and wait if you'd prefer I keep fewer open at once.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES — AI was used in an assistive capacity only (locating the
  relevant code paths and drafting wording). I authored and reviewed the change, understand
  it fully, and can explain every line; the root-cause analysis and validation are mine. No
  AI-written code was submitted without manual review.

[ggml-gh-bot]

Hi @liminfei-amd, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 3 open PRs.

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.