diff --git a/AGENTS.md b/AGENTS.md index 1dc299113..8e4b25cec 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -89,7 +89,7 @@ mkdir -p .bob && ln -s ../.agents/skills .bob/skills - Prefer primitives over classes - **Friendly Dependency Errors**: Wraps optional backend imports in `try/except ImportError` with a helpful message (e.g., "Please pip install mellea[hf]"). See `mellea/stdlib/session.py` for examples. - **CLI command docstrings**: Typer command functions in `cli/` follow an enriched convention with `Prerequisites:` and `See Also:` sections — these feed the auto-generated CLI reference page. See [`docs/docs/guide/CONTRIBUTING.md`](docs/docs/guide/CONTRIBUTING.md) for the full pattern. Regenerate after changes: `uv run poe clidocs`. Test the generator: `uv run pytest tooling/docs-autogen/test_cli_reference.py -v`. Full pipeline docs: [`tooling/docs-autogen/README.md`](tooling/docs-autogen/README.md). -- **Backend telemetry fields**: All backends must populate `mot.usage` (dict with `prompt_tokens`, `completion_tokens`, `total_tokens`), `mot.model` (str), and `mot.provider` (str) in their `post_processing()` method. `mot.streaming` (bool) and `mot.ttfb_ms` (float | None) are set automatically in `astream()` — backends do not need to set them. Metrics are automatically recorded by `TokenMetricsPlugin` and `LatencyMetricsPlugin` — don't add manual `record_token_usage_metrics()` or `record_request_duration()` calls. +- **Backend telemetry fields**: All backends must populate `mot.usage` (dict with `prompt_tokens`, `completion_tokens`, `total_tokens`), `mot.model` (str), and `mot.provider` (str) in their `post_processing()` method. `mot.streaming` (bool) and `mot.ttfb_ms` (float | None) are set automatically in `astream()` — backends do not need to set them. Metrics are automatically recorded by `TokenMetricsPlugin`, `LatencyMetricsPlugin`, and `ErrorMetricsPlugin` — don't add manual `record_token_usage_metrics()`, `record_request_duration()`, or `record_error()` calls. ## 6. Commits & Hooks [Angular format](https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit): `feat:`, `fix:`, `docs:`, `test:`, `refactor:`, `release:` diff --git a/docs/docs/evaluation-and-observability/logging.md b/docs/docs/evaluation-and-observability/logging.md index 908c43928..c211a2960 100644 --- a/docs/docs/evaluation-and-observability/logging.md +++ b/docs/docs/evaluation-and-observability/logging.md @@ -182,5 +182,5 @@ Set either `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`. telemetry features and configuration. - [Tracing](../evaluation-and-observability/tracing) — distributed traces with Gen-AI semantic conventions. -- [Metrics](../evaluation-and-observability/metrics) — token usage metrics, - exporters, and custom instruments. +- [Metrics](../evaluation-and-observability/metrics) — metrics, exporters, + and custom instruments. diff --git a/docs/docs/evaluation-and-observability/metrics.md b/docs/docs/evaluation-and-observability/metrics.md index c5139546b..ac87d3661 100644 --- a/docs/docs/evaluation-and-observability/metrics.md +++ b/docs/docs/evaluation-and-observability/metrics.md @@ -1,6 +1,6 @@ --- title: "Metrics" -description: "Collect token usage and latency metrics, and instrument your own code with OpenTelemetry counters, histograms, and up-down counters." +description: "Automatically collect LLM metrics and instrument your own code with OpenTelemetry counters, histograms, and up-down counters." # diataxis: how-to --- @@ -8,8 +8,7 @@ description: "Collect token usage and latency metrics, and instrument your own c introduces the environment variables and telemetry architecture. This page covers metrics collection in detail. -Mellea automatically tracks token consumption and request latency across all -backends using OpenTelemetry metrics. Metrics follow the +Mellea automatically records LLM metrics across all backends using OpenTelemetry. Metrics follow the [Gen-AI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) for standardized observability. The metrics API also lets you create your own counters, histograms, and up-down counters for application-level instrumentation. @@ -29,17 +28,16 @@ You also need at least one exporter configured — see ## Token usage metrics Mellea records token consumption automatically after each LLM call completes. -No code changes are required. The `TokenMetricsPlugin` auto-registers when -`MELLEA_METRICS_ENABLED=true` and records metrics via the plugin hook system. +No code changes are required. -### Built-in metrics +### Token instruments | Metric Name | Type | Unit | Description | | ----------- | ---- | ---- | ----------- | | `mellea.llm.tokens.input` | Counter | `tokens` | Total input/prompt tokens processed | | `mellea.llm.tokens.output` | Counter | `tokens` | Total output/completion tokens generated | -### Metric attributes +### Token attributes All token metrics include these attributes following Gen-AI semantic conventions: @@ -61,7 +59,7 @@ All token metrics include these attributes following Gen-AI semantic conventions > **Note:** Token usage metrics are only tracked for `generate_from_context` > requests. `generate_from_raw` calls do not record token metrics. -### When metrics are recorded +### Token recording timing Token metrics are recorded **after the full response is received**, not incrementally during streaming: @@ -85,9 +83,7 @@ await mot.avalue() ## Latency histograms Mellea tracks request duration and time-to-first-token (TTFB) automatically -after each LLM call. The `LatencyMetricsPlugin` auto-registers when -`MELLEA_METRICS_ENABLED=true` alongside `TokenMetricsPlugin`. No code changes -are required. +after each LLM call. No code changes are required. ### Latency instruments @@ -133,6 +129,50 @@ with start_session() as m: print(f"Time to first token: {result.ttfb_ms:.1f} ms") ``` +## Error metrics + +Mellea records LLM errors automatically after each failed backend call. No +code changes are required. Errors are classified into semantic categories for +consistent filtering across providers. + +### Error counter + +| Metric Name | Type | Unit | Description | +| ----------- | ---- | ---- | ----------- | +| `mellea.llm.errors` | Counter | `{error}` | Total LLM errors categorized by type | + +### Error attributes + +All error metrics include these attributes: + +| Attribute | Description | Example Values | +| --------- | ----------- | -------------- | +| `error_type` | Semantic error category (mellea-specific) | `rate_limit`, `timeout`, `auth`, `content_policy`, `invalid_request`, `transport_error`, `server_error`, `unknown` | +| `gen_ai.provider.name` | Backend provider name | `openai`, `ollama`, `watsonx`, `litellm`, `huggingface` | +| `gen_ai.request.model` | Model identifier | `gpt-4`, `llama3.2:7b`, `granite-3.1-8b-instruct` | +| `error.type` | Python exception class name (standard OTel) | `RateLimitError`, `TimeoutError`, `AuthenticationError` | + +### Error type categories + +The `error_type` attribute maps exceptions to human-friendly semantic labels: + +| Category | Description | Matched exceptions | +| -------- | ----------- | ------------------ | +| `rate_limit` | Request throttled by provider | `openai.RateLimitError`, class names containing `ratelimit` | +| `timeout` | Request or connection timed out | `TimeoutError`, `openai.APITimeoutError`, class names containing `timeout` | +| `auth` | Authentication or authorization failure | `openai.AuthenticationError`, `openai.PermissionDeniedError`, class names containing `auth` | +| `content_policy` | Request rejected by content moderation | `openai.BadRequestError` with `code="content_policy_violation"`, class names containing `content_policy` | +| `invalid_request` | Malformed or unsupported request | `openai.BadRequestError` (non-content-policy) | +| `transport_error` | Network or connection failure | `ConnectionError`, `openai.APIConnectionError`, class names containing `connection`/`transport` | +| `server_error` | Provider-side internal error | `openai.InternalServerError`, class names containing `server` | +| `unknown` | Unrecognized exception type | Any exception not matched above | + +### When errors are recorded + +Error metrics are recorded when a backend raises an exception during generation, +after the request has been dispatched to the provider. Construction-time errors +(e.g. missing API key) are not captured by the error counter. + ## Metrics export configuration Mellea supports multiple metrics exporters that can be used independently or @@ -369,8 +409,7 @@ streaming mode was used. ## Performance - **Zero overhead when disabled**: When `MELLEA_METRICS_ENABLED=false` (default), - neither `TokenMetricsPlugin` nor `LatencyMetricsPlugin` is registered and all - instrument calls are no-ops. + no auto-registered metrics plugins are active and all instrument calls are no-ops. - **Minimal overhead when enabled**: Counter increments and histogram recordings are extremely fast (~nanoseconds per operation). - **Async export**: Metrics are batched and exported asynchronously (default: diff --git a/docs/docs/evaluation-and-observability/telemetry.md b/docs/docs/evaluation-and-observability/telemetry.md index 12f669461..ba71fc1de 100644 --- a/docs/docs/evaluation-and-observability/telemetry.md +++ b/docs/docs/evaluation-and-observability/telemetry.md @@ -116,14 +116,15 @@ exporter configuration (Jaeger, Grafana Tempo, etc.), and debugging guidance. ## Metrics -Mellea automatically tracks token consumption and request latency across all -backends using OpenTelemetry metrics. No code changes are required — two -plugins hook into the generation pipeline and record metrics automatically: +Mellea automatically records the following metrics across all backends using +OpenTelemetry. No code changes are required: -- **`TokenMetricsPlugin`** — records `mellea.llm.tokens.input` and - `mellea.llm.tokens.output` counters after each LLM call. -- **`LatencyMetricsPlugin`** — records `mellea.llm.request.duration` (every - request) and `mellea.llm.ttfb` (streaming requests only) histograms. +- **Token counters** — `mellea.llm.tokens.input` and `mellea.llm.tokens.output` + after each LLM call. +- **Latency histograms** — `mellea.llm.request.duration` (every request) and + `mellea.llm.ttfb` (streaming requests only). +- **Error counter** — `mellea.llm.errors` on each failed backend call, + classified by semantic error type. The metrics API also exposes `create_counter`, `create_histogram`, and `create_up_down_counter` for instrumenting your own application code. @@ -134,9 +135,9 @@ Mellea supports three exporters that can run simultaneously: - **OTLP** — export to production observability platforms - **Prometheus** — register with `prometheus_client` for scraping -See [Metrics](../evaluation-and-observability/metrics) for token usage and -latency details, backend support matrix, exporter setup, custom instruments, -and troubleshooting. +See [Metrics](../evaluation-and-observability/metrics) for the full list of +metrics, backend support matrix, exporter setup, custom instruments, and +troubleshooting. ## Logging @@ -156,8 +157,8 @@ configuration, OTLP log export setup, and programmatic access via - [Tracing](../evaluation-and-observability/tracing) — distributed traces with Gen-AI semantic conventions. -- [Metrics](../evaluation-and-observability/metrics) — token usage metrics, - exporters, and custom instruments. +- [Metrics](../evaluation-and-observability/metrics) — metrics, exporters, + and custom instruments. - [Logging](../evaluation-and-observability/logging) — console logging and OTLP log export. - [Evaluate with LLM-as-a-Judge](../evaluation-and-observability/evaluate-with-llm-as-a-judge) — diff --git a/docs/docs/evaluation-and-observability/tracing.md b/docs/docs/evaluation-and-observability/tracing.md index 0595f7c1d..c8656fc11 100644 --- a/docs/docs/evaluation-and-observability/tracing.md +++ b/docs/docs/evaluation-and-observability/tracing.md @@ -242,8 +242,8 @@ import mellea # noqa: E402 - [Telemetry](../evaluation-and-observability/telemetry) — overview of all telemetry features and configuration. -- [Metrics](../evaluation-and-observability/metrics) — token usage metrics, - exporters, and custom instruments. +- [Metrics](../evaluation-and-observability/metrics) — metrics, exporters, + and custom instruments. - [Logging](../evaluation-and-observability/logging) — console logging and OTLP log export. - [Evaluate with LLM-as-a-Judge](../evaluation-and-observability/evaluate-with-llm-as-a-judge) — diff --git a/mellea/backends/huggingface.py b/mellea/backends/huggingface.py index 55f85be7b..78e578ce6 100644 --- a/mellea/backends/huggingface.py +++ b/mellea/backends/huggingface.py @@ -330,6 +330,16 @@ def __init__( self._generation_lock = threading.Lock() """Used to force generation requests to be non-concurrent. Necessary for preventing issues with adapters.""" + def _get_hf_model_id(self) -> str: + """Return the HuggingFace model name as a string. + + Returns the ``hf_model_name`` attribute when a ``ModelIdentifier`` is + provided, otherwise casts ``model_id`` to ``str``. + """ + if hasattr(self.model_id, "hf_model_name"): + return str(self.model_id.hf_model_name) # type: ignore + return str(self.model_id) + def _make_dc_cache(self, toks, **model_options): dc = DynamicCache() with torch.no_grad(): @@ -607,6 +617,10 @@ async def granite_formatters_processing( seed=seed, ) + # Set model/provider early so they are available in the error path + output.model = self._get_hf_model_id() + output.provider = "huggingface" + try: # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine. # We can also support synchronous calls by adding a flag and changing this ._generate function. @@ -855,6 +869,10 @@ async def _generate_from_context_with_kv_cache( seed=seed, ) + # Set model/provider early so they are available in the error path + output.model = self._get_hf_model_id() + output.provider = "huggingface" + try: # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine. # We can also support synchronous calls by adding a flag and changing this ._generate function. @@ -1000,6 +1018,10 @@ async def _generate_from_context_standard( seed=seed, ) + # Set model/provider early so they are available in the error path + output.model = self._get_hf_model_id() + output.provider = "huggingface" + try: # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine. # We can also support synchronous calls by adding a flag and changing this ._generate function. @@ -1153,10 +1175,7 @@ class used during generation, if any. } # Populate model and provider metadata - if hasattr(self.model_id, "hf_model_name"): - mot.model = str(self.model_id.hf_model_name) # type: ignore - else: - mot.model = str(self.model_id) + mot.model = self._get_hf_model_id() mot.provider = "huggingface" # Record tracing if span exists diff --git a/mellea/backends/litellm.py b/mellea/backends/litellm.py index bf3224ad0..dc074054c 100644 --- a/mellea/backends/litellm.py +++ b/mellea/backends/litellm.py @@ -372,6 +372,10 @@ async def _generate_from_chat_context_standard( _format=_format, ) + # Set model/provider early so they are available in the error path + output.model = str(self.model_id) + output.provider = "litellm" + try: # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine. # We can also support synchronous calls by adding a flag and changing this ._generate function. diff --git a/mellea/backends/ollama.py b/mellea/backends/ollama.py index f682a49d2..7278ddc2c 100644 --- a/mellea/backends/ollama.py +++ b/mellea/backends/ollama.py @@ -417,6 +417,10 @@ async def generate_from_chat_context( _format=_format, ) + # Set model/provider early so they are available in the error path + output.model = self._get_ollama_model_id() + output.provider = "ollama" + try: # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine. # We can also support synchronous calls by adding a flag and changing this ._generate function. @@ -705,11 +709,7 @@ async def post_processing( } # Populate model and provider metadata - mot.model = ( - self.model_id.ollama_name - if isinstance(self.model_id, ModelIdentifier) - else str(self.model_id) - ) + mot.model = str(self._get_ollama_model_id()) mot.provider = "ollama" # Record telemetry and close span now that response is available diff --git a/mellea/backends/openai.py b/mellea/backends/openai.py index 90d6c3646..60d859285 100644 --- a/mellea/backends/openai.py +++ b/mellea/backends/openai.py @@ -559,6 +559,10 @@ async def _generate_from_chat_context_standard( _format=_format, ) + # Set model/provider early so they are available in the error path + output.model = self._model_id + output.provider = "openai" + try: # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine. # We can also support synchronous calls by adding a flag and changing this ._generate function. diff --git a/mellea/backends/watsonx.py b/mellea/backends/watsonx.py index 5d414f52e..ccbcbd910 100644 --- a/mellea/backends/watsonx.py +++ b/mellea/backends/watsonx.py @@ -446,6 +446,10 @@ async def generate_from_chat_context( _format=_format, ) + # Set model/provider early so they are available in the error path + output.model = str(self._get_watsonx_model_id()) + output.provider = "watsonx" + try: # To support lazy computation, will need to remove this create_task and store just the unexecuted coroutine. # We can also support synchronous calls by adding a flag and changing this ._generate function. @@ -581,11 +585,7 @@ async def post_processing( mot.usage = usage # Populate model and provider metadata - mot.model = ( - self.model_id.watsonx_name - if isinstance(self.model_id, ModelIdentifier) - else str(self.model_id) - ) + mot.model = str(self._get_watsonx_model_id()) mot.provider = "watsonx" # Record tracing if span exists diff --git a/mellea/core/base.py b/mellea/core/base.py index c8ab7efc0..f18d092f5 100644 --- a/mellea/core/base.py +++ b/mellea/core/base.py @@ -510,6 +510,16 @@ async def astream(self) -> str: set_span_error(span, chunks[-1]) end_backend_span(span) del self._meta["_telemetry_span"] + + # Fire generation_error hook (FIRE_AND_FORGET — does not block the raise) + if has_plugins(HookType.GENERATION_ERROR): + from ..plugins.hooks.generation import GenerationErrorPayload + + err_payload = GenerationErrorPayload( + exception=chunks[-1], model_output=self + ) + await invoke_hook(HookType.GENERATION_ERROR, err_payload) + raise chunks[-1] for chunk in chunks: diff --git a/mellea/plugins/hooks/generation.py b/mellea/plugins/hooks/generation.py index 2b83c8c1f..18c0e057f 100644 --- a/mellea/plugins/hooks/generation.py +++ b/mellea/plugins/hooks/generation.py @@ -46,3 +46,20 @@ class GenerationPostCallPayload(MelleaBasePayload): prompt: str | list[dict[str, Any]] = "" model_output: Any = None latency_ms: float = 0.0 + + +class GenerationErrorPayload(MelleaBasePayload): + """Payload for ``generation_error`` — fires when the LLM backend raises an exception. + + This hook fires inside ``ModelOutputThunk.astream`` just before the exception + is re-raised, giving plugins a chance to observe (but not suppress) the error. + + Attributes: + exception: The exception raised by the backend. + model_output: The ``ModelOutputThunk`` at the time of the error. ``model`` + and ``provider`` are set when the backend set them early (before the + async task); otherwise they are ``None``. + """ + + exception: BaseException + model_output: Any = None diff --git a/mellea/plugins/types.py b/mellea/plugins/types.py index 715721177..312b21acc 100644 --- a/mellea/plugins/types.py +++ b/mellea/plugins/types.py @@ -48,6 +48,7 @@ class HookType(StrEnum): # Generation Pipeline GENERATION_PRE_CALL = "generation_pre_call" GENERATION_POST_CALL = "generation_post_call" + GENERATION_ERROR = "generation_error" # Validation VALIDATION_PRE_CHECK = "validation_pre_check" @@ -80,6 +81,7 @@ def _build_hook_registry() -> dict[str, tuple[type, type]]: ComponentPreExecutePayload, ) from mellea.plugins.hooks.generation import ( + GenerationErrorPayload, GenerationPostCallPayload, GenerationPreCallPayload, ) @@ -120,6 +122,7 @@ def _build_hook_registry() -> dict[str, tuple[type, type]]: # Generation Pipeline HookType.GENERATION_PRE_CALL.value: (GenerationPreCallPayload, PluginResult), HookType.GENERATION_POST_CALL.value: (GenerationPostCallPayload, PluginResult), + HookType.GENERATION_ERROR.value: (GenerationErrorPayload, PluginResult), # Validation HookType.VALIDATION_PRE_CHECK.value: (ValidationPreCheckPayload, PluginResult), HookType.VALIDATION_POST_CHECK.value: ( diff --git a/mellea/telemetry/metrics.py b/mellea/telemetry/metrics.py index f8b374715..eeb4f2ab0 100644 --- a/mellea/telemetry/metrics.py +++ b/mellea/telemetry/metrics.py @@ -53,6 +53,7 @@ Built-in metrics (auto-recorded via plugins when metrics are enabled): - Token counters: mellea.llm.tokens.input, mellea.llm.tokens.output (unit: tokens) - Latency histograms: mellea.llm.request.duration (unit: s), mellea.llm.ttfb (unit: s, streaming only) +- Error counter: mellea.llm.errors (unit: {error}), categorized by semantic error type Programmatic usage: from mellea.telemetry.metrics import create_counter, create_histogram @@ -72,6 +73,7 @@ latency_histogram.record(1.5, {"backend": "ollama"}) """ +import asyncio import os import warnings from importlib.metadata import version @@ -583,11 +585,147 @@ def record_ttfb(ttfb_s: float, model: str, provider: str) -> None: ) +# --------------------------------------------------------------------------- +# Error counters +# --------------------------------------------------------------------------- + +# Semantic error type constants (mellea-specific categories) +ERROR_TYPE_RATE_LIMIT = "rate_limit" +ERROR_TYPE_TIMEOUT = "timeout" +ERROR_TYPE_CONTENT_POLICY = "content_policy" +ERROR_TYPE_AUTH = "auth" +ERROR_TYPE_INVALID_REQUEST = "invalid_request" +ERROR_TYPE_TRANSPORT_ERROR = "transport_error" +ERROR_TYPE_SERVER_ERROR = "server_error" +ERROR_TYPE_UNKNOWN = "unknown" + + +def classify_error(exc: BaseException) -> str: + """Map an exception to a semantic error type string. + + Checks OpenAI SDK exception types first (when openai is installed), then + falls back to stdlib exceptions and name-based heuristics. + + Args: + exc: The exception to classify. + + Returns: + One of the ``ERROR_TYPE_*`` constants. + """ + # OpenAI SDK exceptions (optional dependency) + try: + import openai + + if isinstance(exc, openai.RateLimitError): + return ERROR_TYPE_RATE_LIMIT + if isinstance(exc, openai.APITimeoutError): + return ERROR_TYPE_TIMEOUT + if isinstance(exc, (openai.AuthenticationError, openai.PermissionDeniedError)): + return ERROR_TYPE_AUTH + if isinstance(exc, openai.BadRequestError): + # Content policy violations surface as BadRequestError with a specific code + if getattr(exc, "code", None) == "content_policy_violation": + return ERROR_TYPE_CONTENT_POLICY + return ERROR_TYPE_INVALID_REQUEST + if isinstance(exc, openai.APIConnectionError): + return ERROR_TYPE_TRANSPORT_ERROR + if isinstance(exc, openai.InternalServerError): + return ERROR_TYPE_SERVER_ERROR + except ImportError: + pass + + # Stdlib exceptions + if isinstance(exc, (TimeoutError, asyncio.TimeoutError)): + return ERROR_TYPE_TIMEOUT + if isinstance(exc, ConnectionError): + return ERROR_TYPE_TRANSPORT_ERROR + + # Name-based heuristics for provider-specific exceptions without explicit imports + name_lower = type(exc).__name__.lower() + if "ratelimit" in name_lower or "rate_limit" in name_lower: + return ERROR_TYPE_RATE_LIMIT + if "timeout" in name_lower: + return ERROR_TYPE_TIMEOUT + if "auth" in name_lower or "unauthorized" in name_lower: + return ERROR_TYPE_AUTH + if "content" in name_lower and "policy" in name_lower: + return ERROR_TYPE_CONTENT_POLICY + if ( + "connection" in name_lower + or "network" in name_lower + or "transport" in name_lower + ): + return ERROR_TYPE_TRANSPORT_ERROR + if "server" in name_lower: + return ERROR_TYPE_SERVER_ERROR + + return ERROR_TYPE_UNKNOWN + + +_error_counter: Any = None + + +def _get_error_counter() -> Any: + """Get or create the LLM error counter (internal use only). + + Returns: + Counter instrument for LLM errors. + """ + global _error_counter + + if _error_counter is None: + _error_counter = create_counter( + "mellea.llm.errors", + description="Total number of LLM errors categorized by semantic type", + unit="{error}", + ) + + return _error_counter + + +def record_error( + error_type: str, model: str, provider: str, exception_class: str +) -> None: + """Record an LLM error metric. + + This is a no-op when metrics are disabled, ensuring zero overhead. + + Args: + error_type: Semantic error category (use ``ERROR_TYPE_*`` constants). + model: Model identifier (e.g. "gpt-4", "llama2:7b"). + provider: Provider name (e.g. "openai", "ollama"). + exception_class: Python exception class name (e.g. "RateLimitError"). + + Example: + record_error( + error_type=ERROR_TYPE_RATE_LIMIT, + model="gpt-4", + provider="openai", + exception_class="RateLimitError", + ) + """ + if not _METRICS_ENABLED: + return + + counter = _get_error_counter() + counter.add( + 1, + { + "error_type": error_type, + "gen_ai.request.model": model, + "gen_ai.provider.name": provider, + "error.type": exception_class, + }, + ) + + __all__ = [ + "classify_error", "create_counter", "create_histogram", "create_up_down_counter", "is_metrics_enabled", + "record_error", "record_request_duration", "record_token_usage_metrics", "record_ttfb", diff --git a/mellea/telemetry/metrics_plugins.py b/mellea/telemetry/metrics_plugins.py index e9e671d0b..1c0561f58 100644 --- a/mellea/telemetry/metrics_plugins.py +++ b/mellea/telemetry/metrics_plugins.py @@ -5,6 +5,7 @@ - TokenMetricsPlugin: Records token usage statistics from ModelOutputThunk.usage - LatencyMetricsPlugin: Records request duration and TTFB latency histograms +- ErrorMetricsPlugin: Records LLM error counts categorized by semantic error type """ from __future__ import annotations @@ -16,7 +17,10 @@ from mellea.plugins.types import PluginMode if TYPE_CHECKING: - from mellea.plugins.hooks.generation import GenerationPostCallPayload + from mellea.plugins.hooks.generation import ( + GenerationErrorPayload, + GenerationPostCallPayload, + ) class TokenMetricsPlugin(Plugin, name="token_metrics", priority=50): @@ -92,5 +96,36 @@ async def record_latency_metrics( record_ttfb(ttfb_s=mot.ttfb_ms / 1000.0, model=model, provider=provider) +class ErrorMetricsPlugin(Plugin, name="error_metrics", priority=52): + """Records LLM error counts from generation errors. + + This plugin hooks into the generation_error event to classify exceptions + by semantic error type and increment the ``mellea.llm.errors`` counter. + """ + + @hook("generation_error", mode=PluginMode.FIRE_AND_FORGET) + async def record_error_metrics( + self, payload: GenerationErrorPayload, context: dict[str, Any] + ) -> None: + """Record error metrics when a generation error occurs. + + Args: + payload: Contains the exception and the ModelOutputThunk at the time of the error. + context: Plugin context (unused). + """ + from mellea.telemetry.metrics import classify_error, record_error + + mot = payload.model_output + error_type = classify_error(payload.exception) + record_error( + error_type=error_type, + model=mot.model if mot is not None and mot.model is not None else "unknown", + provider=mot.provider + if mot is not None and mot.provider is not None + else "unknown", + exception_class=type(payload.exception).__name__, + ) + + # All metrics plugins to auto-register when metrics are enabled -_METRICS_PLUGIN_CLASSES = (TokenMetricsPlugin, LatencyMetricsPlugin) +_METRICS_PLUGIN_CLASSES = (TokenMetricsPlugin, LatencyMetricsPlugin, ErrorMetricsPlugin) diff --git a/test/telemetry/test_metrics.py b/test/telemetry/test_metrics.py index d7c8688ac..d96d71687 100644 --- a/test/telemetry/test_metrics.py +++ b/test/telemetry/test_metrics.py @@ -611,6 +611,7 @@ def test_metric_instruments_lazy_initialization(enable_metrics): def test_record_metrics_noop_when_disabled(clean_metrics_env): """Test that all record functions are no-ops when metrics disabled.""" from mellea.telemetry.metrics import ( + record_error, record_request_duration, record_token_usage_metrics, ) @@ -619,10 +620,17 @@ def test_record_metrics_noop_when_disabled(clean_metrics_env): input_tokens=100, output_tokens=50, model="llama2:7b", provider="ollama" ) record_request_duration(duration_s=1.0, model="llama2:7b", provider="ollama") + record_error( + error_type="timeout", + model="llama2:7b", + provider="ollama", + exception_class="TimeoutError", + ) # No instruments should have been initialized from mellea.telemetry.metrics import ( _duration_histogram, + _error_counter, _input_token_counter, _output_token_counter, _ttfb_histogram, @@ -632,6 +640,7 @@ def test_record_metrics_noop_when_disabled(clean_metrics_env): assert _output_token_counter is None assert _duration_histogram is None assert _ttfb_histogram is None + assert _error_counter is None def test_record_functions_exported_in_public_api(): diff --git a/test/telemetry/test_metrics_backend.py b/test/telemetry/test_metrics_backend.py index 049291636..cd34ce194 100644 --- a/test/telemetry/test_metrics_backend.py +++ b/test/telemetry/test_metrics_backend.py @@ -88,6 +88,7 @@ def _setup_metrics_provider(metrics_module, metric_reader): metrics_module._output_token_counter = None metrics_module._duration_histogram = None metrics_module._ttfb_histogram = None + metrics_module._error_counter = None return provider @@ -427,3 +428,55 @@ async def test_huggingface_token_metrics_integration( assert output_tokens is not None, "Output tokens should be recorded" assert output_tokens > 0, f"Output tokens should be > 0, got {output_tokens}" + + +@pytest.mark.asyncio +@pytest.mark.openai +@pytest.mark.ollama +async def test_error_metrics_on_backend_failure(enable_metrics, metric_reader): + """Test that error metrics are recorded when a backend call fails. + + Uses OpenAI backend pointed at Ollama with a non-existent model so the + error fires during generation (through base.py:astream), which is where + GENERATION_ERROR is triggered. Also verifies that model/provider are + correctly populated in the error counter attributes (proving the early + output.model/provider set in generate_from_context works). + """ + from mellea.backends.openai import OpenAIBackend + from mellea.telemetry import metrics as metrics_module + + provider = _setup_metrics_provider(metrics_module, metric_reader) + + backend = OpenAIBackend( + model_id="nonexistent-model-xyz", + base_url="http://localhost:11434/v1", + api_key="dummy", + ) + ctx = SimpleContext() + ctx = ctx.add(Message(role="user", content="Say hello")) + + mot, _ = await backend.generate_from_context( + Message(role="assistant", content=""), ctx, model_options={} + ) + + # avalue() drives astream(), where the backend call fails, GENERATION_ERROR + # fires, and the exception is re-raised. pytest.raises catches that re-raise. + with pytest.raises(Exception): + await mot.avalue() + + # Yield to event loop so FIRE_AND_FORGET plugin task completes + await asyncio.sleep(0.05) + provider.force_flush() + metrics_data = metric_reader.get_metrics_data() + + error_count = get_metric_value( + metrics_data, + "mellea.llm.errors", + { + "gen_ai.provider.name": "openai", + "gen_ai.request.model": "nonexistent-model-xyz", + }, + ) + + assert error_count is not None, "Error counter should have been recorded" + assert error_count == 1, f"Expected 1 error, got {error_count}" diff --git a/test/telemetry/test_metrics_errors.py b/test/telemetry/test_metrics_errors.py new file mode 100644 index 000000000..f1e2ebcb4 --- /dev/null +++ b/test/telemetry/test_metrics_errors.py @@ -0,0 +1,125 @@ +"""Integration tests for error counter metrics recording. + +These tests verify that record_error() correctly records counter metrics with +proper attributes and values using OpenTelemetry. +""" + +import pytest + +# Check if OpenTelemetry is available +try: + from opentelemetry.sdk.metrics import MeterProvider + from opentelemetry.sdk.metrics.export import InMemoryMetricReader + + OTEL_AVAILABLE = True +except ImportError: + OTEL_AVAILABLE = False + +pytestmark = [ + pytest.mark.skipif(not OTEL_AVAILABLE, reason="OpenTelemetry not installed"), + pytest.mark.integration, +] + + +@pytest.fixture +def clean_metrics_env(monkeypatch): + """Enable metrics and reset module state for integration tests.""" + monkeypatch.setenv("MELLEA_METRICS_ENABLED", "true") + monkeypatch.delenv("MELLEA_METRICS_CONSOLE", raising=False) + + import importlib + + import mellea.telemetry.metrics + + importlib.reload(mellea.telemetry.metrics) + yield + importlib.reload(mellea.telemetry.metrics) + + +def _setup_in_memory_provider(metrics_module): + """Wire an InMemoryMetricReader into the metrics module globals.""" + reader = InMemoryMetricReader() + provider = MeterProvider(metric_readers=[reader]) + metrics_module._meter_provider = provider + metrics_module._meter = provider.get_meter("mellea") + metrics_module._error_counter = None + return reader, provider + + +def _find_error_data_points(metrics_data): + """Return all data points for mellea.llm.errors.""" + if metrics_data is None: + return [] + data_points = [] + for rm in metrics_data.resource_metrics: + for sm in rm.scope_metrics: + for metric in sm.metrics: + if metric.name == "mellea.llm.errors": + data_points.extend(metric.data.data_points) + return data_points + + +def test_record_error_basic(clean_metrics_env): + """Error counter is populated with correct value and attributes.""" + from mellea.telemetry import metrics as metrics_module + + reader, provider = _setup_in_memory_provider(metrics_module) + + from mellea.telemetry.metrics import record_error + + record_error( + error_type="rate_limit", + model="gpt-4", + provider="openai", + exception_class="RateLimitError", + ) + + provider.force_flush() + data_points = _find_error_data_points(reader.get_metrics_data()) + + assert len(data_points) == 1 + attrs = dict(data_points[0].attributes) + assert attrs["error_type"] == "rate_limit" + assert attrs["gen_ai.request.model"] == "gpt-4" + assert attrs["gen_ai.provider.name"] == "openai" + assert attrs["error.type"] == "RateLimitError" + assert data_points[0].value == 1 + + +def test_record_error_accumulation(clean_metrics_env): + """Multiple errors with the same attributes accumulate correctly.""" + from mellea.telemetry import metrics as metrics_module + + reader, provider = _setup_in_memory_provider(metrics_module) + + from mellea.telemetry.metrics import record_error + + record_error("timeout", "llama2:7b", "ollama", "TimeoutError") + record_error("timeout", "llama2:7b", "ollama", "TimeoutError") + record_error("timeout", "llama2:7b", "ollama", "TimeoutError") + + provider.force_flush() + data_points = _find_error_data_points(reader.get_metrics_data()) + + assert len(data_points) == 1 + assert data_points[0].value == 3 + + +def test_record_error_multiple_types(clean_metrics_env): + """Different error types are tracked as separate attribute sets.""" + from mellea.telemetry import metrics as metrics_module + + reader, provider = _setup_in_memory_provider(metrics_module) + + from mellea.telemetry.metrics import record_error + + record_error("rate_limit", "gpt-4", "openai", "RateLimitError") + record_error("timeout", "gpt-4", "openai", "APITimeoutError") + record_error("auth", "gpt-4", "openai", "AuthenticationError") + + provider.force_flush() + data_points = _find_error_data_points(reader.get_metrics_data()) + + assert len(data_points) == 3 + error_types = {dict(dp.attributes)["error_type"] for dp in data_points} + assert error_types == {"rate_limit", "timeout", "auth"} diff --git a/test/telemetry/test_metrics_helpers.py b/test/telemetry/test_metrics_helpers.py new file mode 100644 index 000000000..c196e6928 --- /dev/null +++ b/test/telemetry/test_metrics_helpers.py @@ -0,0 +1,188 @@ +"""Unit tests for classify_error().""" + +from unittest.mock import MagicMock + +import pytest + +from mellea.telemetry.metrics import ( + ERROR_TYPE_AUTH, + ERROR_TYPE_CONTENT_POLICY, + ERROR_TYPE_INVALID_REQUEST, + ERROR_TYPE_RATE_LIMIT, + ERROR_TYPE_SERVER_ERROR, + ERROR_TYPE_TIMEOUT, + ERROR_TYPE_TRANSPORT_ERROR, + ERROR_TYPE_UNKNOWN, + classify_error, +) + +# --------------------------------------------------------------------------- +# classify_error() — stdlib exceptions +# --------------------------------------------------------------------------- + + +def test_classify_timeout_error(): + assert classify_error(TimeoutError("timed out")) == ERROR_TYPE_TIMEOUT + + +def test_classify_connection_error(): + assert classify_error(ConnectionError("refused")) == ERROR_TYPE_TRANSPORT_ERROR + + +def test_classify_connection_refused_error(): + assert ( + classify_error(ConnectionRefusedError("refused")) == ERROR_TYPE_TRANSPORT_ERROR + ) + + +def test_classify_unknown_exception(): + assert classify_error(ValueError("something went wrong")) == ERROR_TYPE_UNKNOWN + + +def test_classify_runtime_error_is_unknown(): + assert classify_error(RuntimeError("unexpected")) == ERROR_TYPE_UNKNOWN + + +# --------------------------------------------------------------------------- +# classify_error() — name-based heuristics (no openai import needed) +# --------------------------------------------------------------------------- + + +def test_classify_name_rate_limit(): + class RateLimitError(Exception): + pass + + assert classify_error(RateLimitError()) == ERROR_TYPE_RATE_LIMIT + + +def test_classify_name_timeout_in_class_name(): + class RequestTimeoutError(Exception): + pass + + assert classify_error(RequestTimeoutError()) == ERROR_TYPE_TIMEOUT + + +def test_classify_name_auth_in_class_name(): + class AuthError(Exception): + pass + + assert classify_error(AuthError()) == ERROR_TYPE_AUTH + + +def test_classify_name_server_in_class_name(): + class ServerError(Exception): + pass + + assert classify_error(ServerError()) == ERROR_TYPE_SERVER_ERROR + + +def test_classify_name_transport_in_class_name(): + class TransportError(Exception): + pass + + assert classify_error(TransportError()) == ERROR_TYPE_TRANSPORT_ERROR + + +def test_classify_name_content_policy_in_class_name(): + class ContentPolicyError(Exception): + pass + + assert classify_error(ContentPolicyError()) == ERROR_TYPE_CONTENT_POLICY + + +# --------------------------------------------------------------------------- +# classify_error() — OpenAI SDK exceptions (mocked) +# --------------------------------------------------------------------------- + + +def _make_openai_mock(): + """Build a minimal openai module mock with the exception classes we test.""" + openai = MagicMock() + + class _Base(Exception): + pass + + class RateLimitError(_Base): + pass + + class APITimeoutError(_Base): + pass + + class AuthenticationError(_Base): + pass + + class PermissionDeniedError(_Base): + pass + + class BadRequestError(_Base): + def __init__(self, msg="", code=None): + super().__init__(msg) + self.code = code + + class APIConnectionError(_Base): + pass + + class InternalServerError(_Base): + pass + + openai.RateLimitError = RateLimitError + openai.APITimeoutError = APITimeoutError + openai.AuthenticationError = AuthenticationError + openai.PermissionDeniedError = PermissionDeniedError + openai.BadRequestError = BadRequestError + openai.APIConnectionError = APIConnectionError + openai.InternalServerError = InternalServerError + + return openai + + +@pytest.fixture +def openai_mock(monkeypatch): + mock = _make_openai_mock() + monkeypatch.setitem(__import__("sys").modules, "openai", mock) + return mock + + +def test_classify_openai_rate_limit(openai_mock): + assert classify_error(openai_mock.RateLimitError("429")) == ERROR_TYPE_RATE_LIMIT + + +def test_classify_openai_timeout(openai_mock): + assert classify_error(openai_mock.APITimeoutError()) == ERROR_TYPE_TIMEOUT + + +def test_classify_openai_auth(openai_mock): + assert classify_error(openai_mock.AuthenticationError("401")) == ERROR_TYPE_AUTH + + +def test_classify_openai_permission_denied(openai_mock): + assert classify_error(openai_mock.PermissionDeniedError("403")) == ERROR_TYPE_AUTH + + +def test_classify_openai_bad_request_invalid(openai_mock): + assert ( + classify_error(openai_mock.BadRequestError("400", code=None)) + == ERROR_TYPE_INVALID_REQUEST + ) + + +def test_classify_openai_bad_request_content_policy(openai_mock): + assert ( + classify_error( + openai_mock.BadRequestError("400", code="content_policy_violation") + ) + == ERROR_TYPE_CONTENT_POLICY + ) + + +def test_classify_openai_connection_error(openai_mock): + assert ( + classify_error(openai_mock.APIConnectionError()) == ERROR_TYPE_TRANSPORT_ERROR + ) + + +def test_classify_openai_internal_server_error(openai_mock): + assert ( + classify_error(openai_mock.InternalServerError("500")) + == ERROR_TYPE_SERVER_ERROR + ) diff --git a/test/telemetry/test_metrics_plugins.py b/test/telemetry/test_metrics_plugins.py index f99a8eb44..d78feea9d 100644 --- a/test/telemetry/test_metrics_plugins.py +++ b/test/telemetry/test_metrics_plugins.py @@ -1,4 +1,4 @@ -"""Unit tests for TokenMetricsPlugin and LatencyMetricsPlugin.""" +"""Unit tests for TokenMetricsPlugin, LatencyMetricsPlugin, and ErrorMetricsPlugin.""" from unittest.mock import patch @@ -7,8 +7,20 @@ pytest.importorskip("cpex", reason="cpex not installed — install mellea[hooks]") from mellea.core.base import ModelOutputThunk -from mellea.plugins.hooks.generation import GenerationPostCallPayload -from mellea.telemetry.metrics_plugins import LatencyMetricsPlugin, TokenMetricsPlugin +from mellea.plugins.hooks.generation import ( + GenerationErrorPayload, + GenerationPostCallPayload, +) +from mellea.telemetry.metrics import ( + ERROR_TYPE_TIMEOUT, + ERROR_TYPE_TRANSPORT_ERROR, + ERROR_TYPE_UNKNOWN, +) +from mellea.telemetry.metrics_plugins import ( + ErrorMetricsPlugin, + LatencyMetricsPlugin, + TokenMetricsPlugin, +) @pytest.fixture @@ -173,3 +185,85 @@ async def test_latency_missing_model_provider(latency_plugin): mock_dur.assert_called_once_with( duration_s=0.5, model="unknown", provider="unknown", streaming=False ) + + +# ErrorMetricsPlugin tests + + +@pytest.fixture +def error_plugin(): + return ErrorMetricsPlugin() + + +def _make_error_payload(exc, model="test-model", provider="test-provider"): + """Create a GenerationErrorPayload wrapping the given exception.""" + mot = ModelOutputThunk(value="") + mot.model = model + mot.provider = provider + return GenerationErrorPayload(exception=exc, model_output=mot) + + +@pytest.mark.asyncio +async def test_error_plugin_records_correct_type(error_plugin): + """Plugin classifies the exception and calls record_error with the right type.""" + payload = _make_error_payload(TimeoutError("timed out")) + + with patch("mellea.telemetry.metrics.record_error") as mock_record: + await error_plugin.record_error_metrics(payload, {}) + + mock_record.assert_called_once_with( + error_type=ERROR_TYPE_TIMEOUT, + model="test-model", + provider="test-provider", + exception_class="TimeoutError", + ) + + +@pytest.mark.asyncio +async def test_error_plugin_unknown_exception(error_plugin): + """Unrecognized exceptions are classified as unknown.""" + payload = _make_error_payload(ValueError("something broke")) + + with patch("mellea.telemetry.metrics.record_error") as mock_record: + await error_plugin.record_error_metrics(payload, {}) + + mock_record.assert_called_once_with( + error_type=ERROR_TYPE_UNKNOWN, + model="test-model", + provider="test-provider", + exception_class="ValueError", + ) + + +@pytest.mark.asyncio +async def test_error_plugin_falls_back_to_unknown_when_model_none(error_plugin): + """model/provider fall back to 'unknown' when None on the MOT.""" + payload = _make_error_payload(ConnectionError("refused"), model=None, provider=None) + + with patch("mellea.telemetry.metrics.record_error") as mock_record: + await error_plugin.record_error_metrics(payload, {}) + + mock_record.assert_called_once_with( + error_type=ERROR_TYPE_TRANSPORT_ERROR, + model="unknown", + provider="unknown", + exception_class="ConnectionError", + ) + + +@pytest.mark.asyncio +async def test_error_plugin_handles_none_model_output(error_plugin): + """Plugin handles a None model_output gracefully.""" + payload = GenerationErrorPayload( + exception=RuntimeError("queue error"), model_output=None + ) + + with patch("mellea.telemetry.metrics.record_error") as mock_record: + await error_plugin.record_error_metrics(payload, {}) + + mock_record.assert_called_once_with( + error_type=ERROR_TYPE_UNKNOWN, + model="unknown", + provider="unknown", + exception_class="RuntimeError", + )