feat(telemetry): wire metrics plugins to generation_batch_* hooks

[ajbozarth]

Issue

Fixes #1182

Description

PR #1181 introduced generation_batch_{pre_call,post_call,error} hooks for
the raw path; the tracing plugin subscribed but the metrics plugins did
not, so generate_from_raw calls produced zero metrics. This wires the
four affected metrics plugins to the corresponding batch hooks so that
token, latency, cost, and error metrics are recorded for raw-path
generations the same as for the chat path.

Source-side changes (mellea/telemetry/metrics_plugins.py):

• TokenMetricsPlugin now also subscribes to generation_batch_post_call
  and reads payload.usage.
• LatencyMetricsPlugin now also subscribes to generation_batch_post_call
  and records duration from payload.latency_ms. The batch path is
  non-streaming, so TTFB does not apply.
• ErrorMetricsPlugin now also subscribes to generation_batch_error and
  classifies the exception from payload.exception.
• CostMetricsPlugin now also subscribes to generation_batch_post_call
  and computes cost from payload.usage / payload.model /
  payload.provider.

The per-MOT handlers are unchanged. Class docstrings updated to reflect
the additional hook subscription.

Doc: removed the now-stale note in
docs/docs/observability/metrics.md claiming generate_from_raw does not
record token metrics.

Test-side changes:

• test/telemetry/test_metrics_plugins.py: 13 new unit tests mirroring the
  existing per-MOT coverage for token, latency, cost, and error batch
  handlers (success, missing fields, fallback to "unknown", cache token
  forwarding for cost, unknown-model skip).
• test/telemetry/test_metrics_backend.py: one e2e test that drives a real
  Ollama backend through generate_from_raw and asserts token and latency
  metrics are recorded.

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code was added
• Ensure existing tests and github automation passes

$ uv run pytest test/telemetry/ -m "not qualitative and not e2e"
223 passed, 19 deselected
$ uv run pytest test/telemetry/test_metrics_backend.py::test_ollama_generate_from_raw_metrics_integration
1 passed

Attribution

• AI coding assistants used

[akihikokuroda]

CostMetricsPlugin (lines 144–187)

Current implementation:
@hook("generation_post_call", mode=PluginMode.FIRE_AND_FORGET)
async def record_cost_metrics(self, payload: GenerationPostCallPayload, ...):
gen = payload.model_output.generation
if gen.usage is None:
return

model = gen.model or "unknown"
provider = gen.provider or "unknown"
details = gen.usage.get("prompt_tokens_details")
cached_tokens = (
    details.get("cached_tokens") if isinstance(details, dict) else 0
) or 0
cache_creation = gen.usage.get("cache_creation_input_tokens") or 0
prompt_tokens = gen.usage.get("prompt_tokens") or 0
cost = compute_cost(
    model=model,
    provider=gen.provider,  # ⚠️ BUG: should be `provider` not `gen.provider`
    prompt_tokens=prompt_tokens,
    completion_tokens=gen.usage.get("completion_tokens"),
    cached_tokens=cached_tokens,
    cache_creation_tokens=cache_creation,
)
if cost is not None:
    record_cost(cost=cost, model=model, provider=provider)

🐛 BUG IDENTIFIED (line 179):
cost = compute_cost(
model=model,
provider=gen.provider, # ❌ Should be provider (the fallback "unknown")
...
)

When gen.provider is None, this passes None to compute_cost, but:

• Line 170: provider = gen.provider or "unknown" (correctly gets "unknown")
• Line 179: provider=gen.provider (ignores the fallback!)
• Line 186: record_cost(..., provider=provider) (records with "unknown")

This means compute_cost receives None while record_cost receives "unknown" — inconsistent, and compute_cost may fail if it expects a string.

Fix needed:
cost = compute_cost(
model=model,
provider=provider, # Use the fallback value
prompt_tokens=prompt_tokens,
...
)

PR's batch handler (lines 257–289) repeats this bug:
cost = compute_cost(
model=model,
provider=payload.provider, # ❌ Same issue — should use the fallback
...
)

[planetf1]

Thanks for the clean fix — wiring all four plugins symmetrically with the tracing plugin is exactly right, and the test coverage is thorough. Two small notes below.

[planetf1]

The unit tests call the handlers directly, bypassing hook dispatch — the thing this PR is actually wiring. The generation_batch_error path has no end-to-end coverage. Could we add an e2e/ollama test driving generate_from_raw against a nonexistent model and asserting mellea.llm.errors == 1? Happy to track as a follow-up if easier.

[ajbozarth]

Could we add an e2e/ollama test driving generate_from_raw

done in a03e99e

This means compute_cost receives None while record_cost receives "unknown" — inconsistent, and compute_cost may fail if it expects a string.

@akihikokuroda this is intentional, compute_cost accepts str | None and passes it through to litellm as custom_llm_provider=provider or None, where None triggers model-name-based provider inference. Sending "unknown" would break pricing lookup. The raw value goes to compute_cost (resolution); the labeled value goes to record_cost (attribution).

[akihikokuroda]

LGTM