Description:
Implement latency histograms to track request duration distribution and time-to-first-token (TTFB) for streaming requests.
Detailed Requirements:
- Create histograms in
mellea/telemetry/metrics.py:
mellea.llm.request.duration - Total request duration
mellea.llm.ttfb - Time to first byte/token (streaming only)
- Define appropriate bucket boundaries:
- Duration: [0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120] seconds
- TTFB: [0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10] seconds
- Add helper functions:
record_request_duration(duration_seconds, model, backend, streaming)
record_ttfb(ttfb_seconds, model, backend)
- Integrate with backend instrumentation:
- Measure total duration from request start to complete response
- For streaming: measure time from request to first chunk
- Add attributes:
gen_ai.system, gen_ai.request.model, mellea.backend
streaming - Boolean indicating streaming vs non-streaming
Files to Modify:
mellea/telemetry/metrics.py - Add latency histograms
mellea/telemetry/backend_instrumentation.py - Add timing helpers
mellea/backends/openai.py - Record latency metrics
- Other backends as applicable
Metric Definitions:
| Metric |
Type |
Unit |
Attributes |
mellea.llm.request.duration |
Histogram |
s |
model, backend, system, streaming |
mellea.llm.ttfb |
Histogram |
s |
model, backend, system |
Acceptance Criteria:
Description:
Implement latency histograms to track request duration distribution and time-to-first-token (TTFB) for streaming requests.
Detailed Requirements:
mellea/telemetry/metrics.py:mellea.llm.request.duration- Total request durationmellea.llm.ttfb- Time to first byte/token (streaming only)record_request_duration(duration_seconds, model, backend, streaming)record_ttfb(ttfb_seconds, model, backend)gen_ai.system,gen_ai.request.model,mellea.backendstreaming- Boolean indicating streaming vs non-streamingFiles to Modify:
mellea/telemetry/metrics.py- Add latency histogramsmellea/telemetry/backend_instrumentation.py- Add timing helpersmellea/backends/openai.py- Record latency metricsMetric Definitions:
mellea.llm.request.durationsmellea.llm.ttfbsAcceptance Criteria: