feat: add error counter metrics categorized by semantic type (#465)

AGENTS.md

@@ -89,7 +89,7 @@ mkdir -p .bob && ln -s ../.agents/skills .bob/skills
 - Prefer primitives over classes
 - **Friendly Dependency Errors**: Wraps optional backend imports in `try/except ImportError` with a helpful message (e.g., "Please pip install mellea[hf]"). See `mellea/stdlib/session.py` for examples.
 - **CLI command docstrings**: Typer command functions in `cli/` follow an enriched convention with `Prerequisites:` and `See Also:` sections — these feed the auto-generated CLI reference page. See [`docs/docs/guide/CONTRIBUTING.md`](docs/docs/guide/CONTRIBUTING.md) for the full pattern. Regenerate after changes: `uv run poe clidocs`. Test the generator: `uv run pytest tooling/docs-autogen/test_cli_reference.py -v`. Full pipeline docs: [`tooling/docs-autogen/README.md`](tooling/docs-autogen/README.md).
-- **Backend telemetry fields**: All backends must populate `mot.usage` (dict with `prompt_tokens`, `completion_tokens`, `total_tokens`), `mot.model` (str), and `mot.provider` (str) in their `post_processing()` method. `mot.streaming` (bool) and `mot.ttfb_ms` (float | None) are set automatically in `astream()` — backends do not need to set them. Metrics are automatically recorded by `TokenMetricsPlugin` and `LatencyMetricsPlugin` — don't add manual `record_token_usage_metrics()` or `record_request_duration()` calls.
+- **Backend telemetry fields**: All backends must populate `mot.usage` (dict with `prompt_tokens`, `completion_tokens`, `total_tokens`), `mot.model` (str), and `mot.provider` (str) in their `post_processing()` method. `mot.streaming` (bool) and `mot.ttfb_ms` (float | None) are set automatically in `astream()` — backends do not need to set them. Metrics are automatically recorded by `TokenMetricsPlugin`, `LatencyMetricsPlugin`, and `ErrorMetricsPlugin` — don't add manual `record_token_usage_metrics()`, `record_request_duration()`, or `record_error()` calls.
 
 ## 6. Commits & Hooks
 [Angular format](https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit): `feat:`, `fix:`, `docs:`, `test:`, `refactor:`, `release:`

docs/docs/evaluation-and-observability/logging.md

@@ -182,5 +182,5 @@ Set either `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`.
   telemetry features and configuration.
 - [Tracing](../evaluation-and-observability/tracing) — distributed traces
   with Gen-AI semantic conventions.
-- [Metrics](../evaluation-and-observability/metrics) — token usage metrics,
-  exporters, and custom instruments.
+- [Metrics](../evaluation-and-observability/metrics) — metrics, exporters,
+  and custom instruments.

docs/docs/evaluation-and-observability/metrics.md

@@ -1,15 +1,14 @@
 ---
 title: "Metrics"
-description: "Collect token usage and latency metrics, and instrument your own code with OpenTelemetry counters, histograms, and up-down counters."
+description: "Automatically collect LLM metrics and instrument your own code with OpenTelemetry counters, histograms, and up-down counters."
 # diataxis: how-to
 ---
 
 **Prerequisites:** [Telemetry](../evaluation-and-observability/telemetry)
 introduces the environment variables and telemetry architecture. This page
 covers metrics collection in detail.
 
-Mellea automatically tracks token consumption and request latency across all
-backends using OpenTelemetry metrics. Metrics follow the
+Mellea automatically records LLM metrics across all backends using OpenTelemetry. Metrics follow the
 [Gen-AI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
 for standardized observability. The metrics API also lets you create your own
 counters, histograms, and up-down counters for application-level instrumentation.
@@ -29,17 +28,16 @@ You also need at least one exporter configured — see
 ## Token usage metrics
 
 Mellea records token consumption automatically after each LLM call completes.
-No code changes are required. The `TokenMetricsPlugin` auto-registers when
-`MELLEA_METRICS_ENABLED=true` and records metrics via the plugin hook system.
+No code changes are required.
 
-### Built-in metrics
+### Token instruments
 
 | Metric Name | Type | Unit | Description |
 | ----------- | ---- | ---- | ----------- |
 | `mellea.llm.tokens.input` | Counter | `tokens` | Total input/prompt tokens processed |
 | `mellea.llm.tokens.output` | Counter | `tokens` | Total output/completion tokens generated |
 
-### Metric attributes
+### Token attributes
 
 All token metrics include these attributes following Gen-AI semantic conventions:
 
@@ -61,7 +59,7 @@ All token metrics include these attributes following Gen-AI semantic conventions
 > **Note:** Token usage metrics are only tracked for `generate_from_context`
 > requests. `generate_from_raw` calls do not record token metrics.
 
-### When metrics are recorded
+### Token recording timing
 
 Token metrics are recorded **after the full response is received**, not
 incrementally during streaming:
@@ -85,9 +83,7 @@ await mot.avalue()
 ## Latency histograms
 
 Mellea tracks request duration and time-to-first-token (TTFB) automatically
-after each LLM call. The `LatencyMetricsPlugin` auto-registers when
-`MELLEA_METRICS_ENABLED=true` alongside `TokenMetricsPlugin`. No code changes
-are required.
+after each LLM call. No code changes are required.
 
 ### Latency instruments
 
@@ -133,6 +129,50 @@ with start_session() as m:
         print(f"Time to first token: {result.ttfb_ms:.1f} ms")
 ```
 
+## Error metrics
+
+Mellea records LLM errors automatically after each failed backend call. No
+code changes are required. Errors are classified into semantic categories for
+consistent filtering across providers.
+
+### Error counter
+
+| Metric Name | Type | Unit | Description |
+| ----------- | ---- | ---- | ----------- |
+| `mellea.llm.errors` | Counter | `{error}` | Total LLM errors categorized by type |
+
+### Error attributes
+
+All error metrics include these attributes:
+
+| Attribute | Description | Example Values |
+| --------- | ----------- | -------------- |
+| `error_type` | Semantic error category (mellea-specific) | `rate_limit`, `timeout`, `auth`, `content_policy`, `invalid_request`, `transport_error`, `server_error`, `unknown` |
+| `gen_ai.provider.name` | Backend provider name | `openai`, `ollama`, `watsonx`, `litellm`, `huggingface` |
+| `gen_ai.request.model` | Model identifier | `gpt-4`, `llama3.2:7b`, `granite-3.1-8b-instruct` |
+| `error.type` | Python exception class name (standard OTel) | `RateLimitError`, `TimeoutError`, `AuthenticationError` |
+
+### Error type categories
+
+The `error_type` attribute maps exceptions to human-friendly semantic labels:
+
+| Category | Description | Matched exceptions |
+| -------- | ----------- | ------------------ |
+| `rate_limit` | Request throttled by provider | `openai.RateLimitError`, class names containing `ratelimit` |
+| `timeout` | Request or connection timed out | `TimeoutError`, `openai.APITimeoutError`, class names containing `timeout` |
+| `auth` | Authentication or authorization failure | `openai.AuthenticationError`, `openai.PermissionDeniedError`, class names containing `auth` |
+| `content_policy` | Request rejected by content moderation | `openai.BadRequestError` with `code="content_policy_violation"`, class names containing `content_policy` |
+| `invalid_request` | Malformed or unsupported request | `openai.BadRequestError` (non-content-policy) |
+| `transport_error` | Network or connection failure | `ConnectionError`, `openai.APIConnectionError`, class names containing `connection`/`transport` |
+| `server_error` | Provider-side internal error | `openai.InternalServerError`, class names containing `server` |
+| `unknown` | Unrecognized exception type | Any exception not matched above |
+
+### When errors are recorded
+
+Error metrics are recorded when a backend raises an exception during generation,
+after the request has been dispatched to the provider. Construction-time errors
+(e.g. missing API key) are not captured by the error counter.
+
 ## Metrics export configuration
 
 Mellea supports multiple metrics exporters that can be used independently or
@@ -369,8 +409,7 @@ streaming mode was used.
 ## Performance
 
 - **Zero overhead when disabled**: When `MELLEA_METRICS_ENABLED=false` (default),
-  neither `TokenMetricsPlugin` nor `LatencyMetricsPlugin` is registered and all
-  instrument calls are no-ops.
+  no auto-registered metrics plugins are active and all instrument calls are no-ops.
 - **Minimal overhead when enabled**: Counter increments and histogram recordings
   are extremely fast (~nanoseconds per operation).
 - **Async export**: Metrics are batched and exported asynchronously (default:

docs/docs/evaluation-and-observability/telemetry.md

@@ -116,14 +116,15 @@ exporter configuration (Jaeger, Grafana Tempo, etc.), and debugging guidance.
 
 ## Metrics
 
-Mellea automatically tracks token consumption and request latency across all
-backends using OpenTelemetry metrics. No code changes are required — two
-plugins hook into the generation pipeline and record metrics automatically:
+Mellea automatically records the following metrics across all backends using
+OpenTelemetry. No code changes are required:
 
-- **`TokenMetricsPlugin`** — records `mellea.llm.tokens.input` and
-  `mellea.llm.tokens.output` counters after each LLM call.
-- **`LatencyMetricsPlugin`** — records `mellea.llm.request.duration` (every
-  request) and `mellea.llm.ttfb` (streaming requests only) histograms.
+- **Token counters** — `mellea.llm.tokens.input` and `mellea.llm.tokens.output`
+  after each LLM call.
+- **Latency histograms** — `mellea.llm.request.duration` (every request) and
+  `mellea.llm.ttfb` (streaming requests only).
+- **Error counter** — `mellea.llm.errors` on each failed backend call,
+  classified by semantic error type.
 
 The metrics API also exposes `create_counter`, `create_histogram`, and
 `create_up_down_counter` for instrumenting your own application code.
@@ -134,9 +135,9 @@ Mellea supports three exporters that can run simultaneously:
 - **OTLP** — export to production observability platforms
 - **Prometheus** — register with `prometheus_client` for scraping
 
-See [Metrics](../evaluation-and-observability/metrics) for token usage and
-latency details, backend support matrix, exporter setup, custom instruments,
-and troubleshooting.
+See [Metrics](../evaluation-and-observability/metrics) for the full list of
+metrics, backend support matrix, exporter setup, custom instruments, and
+troubleshooting.
 
 ## Logging
 
@@ -156,8 +157,8 @@ configuration, OTLP log export setup, and programmatic access via
 
 - [Tracing](../evaluation-and-observability/tracing) — distributed traces
   with Gen-AI semantic conventions.
-- [Metrics](../evaluation-and-observability/metrics) — token usage metrics,
-  exporters, and custom instruments.
+- [Metrics](../evaluation-and-observability/metrics) — metrics, exporters,
+  and custom instruments.
 - [Logging](../evaluation-and-observability/logging) — console logging and OTLP
   log export.
 - [Evaluate with LLM-as-a-Judge](../evaluation-and-observability/evaluate-with-llm-as-a-judge) —

docs/docs/evaluation-and-observability/tracing.md

@@ -242,8 +242,8 @@ import mellea  # noqa: E402
 
 - [Telemetry](../evaluation-and-observability/telemetry) — overview of all
   telemetry features and configuration.
-- [Metrics](../evaluation-and-observability/metrics) — token usage metrics,
-  exporters, and custom instruments.
+- [Metrics](../evaluation-and-observability/metrics) — metrics, exporters,
+  and custom instruments.
 - [Logging](../evaluation-and-observability/logging) — console logging and OTLP
   log export.
 - [Evaluate with LLM-as-a-Judge](../evaluation-and-observability/evaluate-with-llm-as-a-judge) —

mellea/backends/huggingface.py

@@ -330,6 +330,16 @@ def __init__(
         self._generation_lock = threading.Lock()
         """Used to force generation requests to be non-concurrent. Necessary for preventing issues with adapters."""
 
+    def _get_hf_model_id(self) -> str:
+        """Return the HuggingFace model name as a string.
+
+        Returns the ``hf_model_name`` attribute when a ``ModelIdentifier`` is
+        provided, otherwise casts ``model_id`` to ``str``.
+        """
+        if hasattr(self.model_id, "hf_model_name"):
+            return str(self.model_id.hf_model_name)  # type: ignore
+        return str(self.model_id)
+
     def _make_dc_cache(self, toks, **model_options):
         dc = DynamicCache()
         with torch.no_grad():
@@ -607,6 +617,10 @@ async def granite_formatters_processing(
             seed=seed,
         )
 
+        # Set model/provider early so they are available in the error path
+        output.model = self._get_hf_model_id()
+        output.provider = "huggingface"
+
         try:
             # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine.
             # We can also support synchronous calls by adding a flag and changing this ._generate function.
@@ -855,6 +869,10 @@ async def _generate_from_context_with_kv_cache(
                 seed=seed,
             )
 
+            # Set model/provider early so they are available in the error path
+            output.model = self._get_hf_model_id()
+            output.provider = "huggingface"
+
             try:
                 # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine.
                 # We can also support synchronous calls by adding a flag and changing this ._generate function.
@@ -1000,6 +1018,10 @@ async def _generate_from_context_standard(
                 seed=seed,
             )
 
+            # Set model/provider early so they are available in the error path
+            output.model = self._get_hf_model_id()
+            output.provider = "huggingface"
+
             try:
                 # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine.
                 # We can also support synchronous calls by adding a flag and changing this ._generate function.
@@ -1153,10 +1175,7 @@ class used during generation, if any.
             }
 
         # Populate model and provider metadata
-        if hasattr(self.model_id, "hf_model_name"):
-            mot.model = str(self.model_id.hf_model_name)  # type: ignore
-        else:
-            mot.model = str(self.model_id)
+        mot.model = self._get_hf_model_id()
         mot.provider = "huggingface"
 
         # Record tracing if span exists

mellea/backends/litellm.py

@@ -372,6 +372,10 @@ async def _generate_from_chat_context_standard(
             _format=_format,
         )
 
+        # Set model/provider early so they are available in the error path
+        output.model = str(self.model_id)
+        output.provider = "litellm"
+
         try:
             # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine.
             # We can also support synchronous calls by adding a flag and changing this ._generate function.

mellea/backends/ollama.py

@@ -417,6 +417,10 @@ async def generate_from_chat_context(
             _format=_format,
         )
 
+        # Set model/provider early so they are available in the error path
+        output.model = self._get_ollama_model_id()
+        output.provider = "ollama"
+
         try:
             # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine.
             # We can also support synchronous calls by adding a flag and changing this ._generate function.
@@ -705,11 +709,7 @@ async def post_processing(
             }
 
         # Populate model and provider metadata
-        mot.model = (
-            self.model_id.ollama_name
-            if isinstance(self.model_id, ModelIdentifier)
-            else str(self.model_id)
-        )
+        mot.model = str(self._get_ollama_model_id())
         mot.provider = "ollama"
 
         # Record telemetry and close span now that response is available

mellea/backends/openai.py

@@ -559,6 +559,10 @@ async def _generate_from_chat_context_standard(
             _format=_format,
         )
 
+        # Set model/provider early so they are available in the error path
+        output.model = self._model_id
+        output.provider = "openai"
+
         try:
             # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine.
             # We can also support synchronous calls by adding a flag and changing this ._generate function.

mellea/backends/watsonx.py

@@ -446,6 +446,10 @@ async def generate_from_chat_context(
             _format=_format,
         )
 
+        # Set model/provider early so they are available in the error path
+        output.model = str(self._get_watsonx_model_id())
+        output.provider = "watsonx"
+
         try:
             # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine.
             # We can also support synchronous calls by adding a flag and changing this ._generate function.
@@ -581,11 +585,7 @@ async def post_processing(
             mot.usage = usage
 
         # Populate model and provider metadata
-        mot.model = (
-            self.model_id.watsonx_name
-            if isinstance(self.model_id, ModelIdentifier)
-            else str(self.model_id)
-        )
+        mot.model = str(self._get_watsonx_model_id())
         mot.provider = "watsonx"
 
         # Record tracing if span exists

mellea/core/base.py

@@ -510,6 +510,16 @@ async def astream(self) -> str:
                 set_span_error(span, chunks[-1])
                 end_backend_span(span)
                 del self._meta["_telemetry_span"]
+
+            # Fire generation_error hook (FIRE_AND_FORGET — does not block the raise)
+            if has_plugins(HookType.GENERATION_ERROR):
+                from ..plugins.hooks.generation import GenerationErrorPayload
+
+                err_payload = GenerationErrorPayload(
+                    exception=chunks[-1], model_output=self
+                )
+                await invoke_hook(HookType.GENERATION_ERROR, err_payload)
+
             raise chunks[-1]
 
         for chunk in chunks:

mellea/plugins/hooks/generation.py

@@ -46,3 +46,20 @@ class GenerationPostCallPayload(MelleaBasePayload):
     prompt: str | list[dict[str, Any]] = ""
     model_output: Any = None
     latency_ms: float = 0.0
+
+
+class GenerationErrorPayload(MelleaBasePayload):
+    """Payload for ``generation_error`` — fires when the LLM backend raises an exception.
+
+    This hook fires inside ``ModelOutputThunk.astream`` just before the exception
+    is re-raised, giving plugins a chance to observe (but not suppress) the error.
+
+    Attributes:
+        exception: The exception raised by the backend.
+        model_output: The ``ModelOutputThunk`` at the time of the error. ``model``
+            and ``provider`` are set when the backend set them early (before the
+            async task); otherwise they are ``None``.
+    """
+
+    exception: BaseException
+    model_output: Any = None

mellea/plugins/types.py

@@ -48,6 +48,7 @@ class HookType(StrEnum):
     # Generation Pipeline
     GENERATION_PRE_CALL = "generation_pre_call"
     GENERATION_POST_CALL = "generation_post_call"
+    GENERATION_ERROR = "generation_error"
 
     # Validation
     VALIDATION_PRE_CHECK = "validation_pre_check"
@@ -80,6 +81,7 @@ def _build_hook_registry() -> dict[str, tuple[type, type]]:
         ComponentPreExecutePayload,
     )
     from mellea.plugins.hooks.generation import (
+        GenerationErrorPayload,
         GenerationPostCallPayload,
         GenerationPreCallPayload,
     )
@@ -120,6 +122,7 @@ def _build_hook_registry() -> dict[str, tuple[type, type]]:
         # Generation Pipeline
         HookType.GENERATION_PRE_CALL.value: (GenerationPreCallPayload, PluginResult),
         HookType.GENERATION_POST_CALL.value: (GenerationPostCallPayload, PluginResult),
+        HookType.GENERATION_ERROR.value: (GenerationErrorPayload, PluginResult),
         # Validation
         HookType.VALIDATION_PRE_CHECK.value: (ValidationPreCheckPayload, PluginResult),
         HookType.VALIDATION_POST_CHECK.value: (