Description:
Implement counters to track token usage across all LLM backends with model and backend labels.
Detailed Requirements:
- Create counters in
mellea/telemetry/metrics.py:
mellea.llm.tokens.input - Input/prompt tokens
mellea.llm.tokens.output - Output/completion tokens
mellea.llm.tokens.total - Total tokens (optional, can be computed)
- Add attributes/labels:
gen_ai.system - Backend system (openai, ollama, etc.)
gen_ai.request.model - Model identifier
mellea.backend - Backend class name
- Create helper function
record_token_usage(input_tokens, output_tokens, model, backend)
- Integrate with all backends that report token usage:
- OpenAI:
usage.prompt_tokens, usage.completion_tokens
- Ollama:
response.prompt_eval_count, response.eval_count
- WatsonX:
input_token_count, generated_token_count
- LiteLLM: Via litellm's usage tracking
- HuggingFace: Limited support (local models)
Files to Modify:
mellea/telemetry/metrics.py - Add token counters
mellea/telemetry/backend_instrumentation.py - Add metric recording alongside trace recording
mellea/backends/openai.py - Record token metrics
mellea/backends/ollama.py - Record token metrics
mellea/backends/watsonx.py - Record token metrics
mellea/backends/litellm.py - Record token metrics
Metric Definitions:
| Metric |
Type |
Unit |
Attributes |
mellea.llm.tokens.input |
Counter |
{token} |
model, backend, system |
mellea.llm.tokens.output |
Counter |
{token} |
model, backend, system |
Acceptance Criteria:
Description:
Implement counters to track token usage across all LLM backends with model and backend labels.
Detailed Requirements:
mellea/telemetry/metrics.py:mellea.llm.tokens.input- Input/prompt tokensmellea.llm.tokens.output- Output/completion tokensmellea.llm.tokens.total- Total tokens (optional, can be computed)gen_ai.system- Backend system (openai, ollama, etc.)gen_ai.request.model- Model identifiermellea.backend- Backend class namerecord_token_usage(input_tokens, output_tokens, model, backend)usage.prompt_tokens,usage.completion_tokensresponse.prompt_eval_count,response.eval_countinput_token_count,generated_token_countFiles to Modify:
mellea/telemetry/metrics.py- Add token countersmellea/telemetry/backend_instrumentation.py- Add metric recording alongside trace recordingmellea/backends/openai.py- Record token metricsmellea/backends/ollama.py- Record token metricsmellea/backends/watsonx.py- Record token metricsmellea/backends/litellm.py- Record token metricsMetric Definitions:
mellea.llm.tokens.input{token}mellea.llm.tokens.output{token}Acceptance Criteria: