Skip to content

Implement latency histograms to track request duration distribution and time-to-first-token (TTFB) for streaming requests #463

Description

@psschwei

Description:
Implement latency histograms to track request duration distribution and time-to-first-token (TTFB) for streaming requests.

Detailed Requirements:

  1. Create histograms in mellea/telemetry/metrics.py:
    • mellea.llm.request.duration - Total request duration
    • mellea.llm.ttfb - Time to first byte/token (streaming only)
  2. Define appropriate bucket boundaries:
    • Duration: [0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120] seconds
    • TTFB: [0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10] seconds
  3. Add helper functions:
    • record_request_duration(duration_seconds, model, backend, streaming)
    • record_ttfb(ttfb_seconds, model, backend)
  4. Integrate with backend instrumentation:
    • Measure total duration from request start to complete response
    • For streaming: measure time from request to first chunk
  5. Add attributes:
    • gen_ai.system, gen_ai.request.model, mellea.backend
    • streaming - Boolean indicating streaming vs non-streaming

Files to Modify:

  • mellea/telemetry/metrics.py - Add latency histograms
  • mellea/telemetry/backend_instrumentation.py - Add timing helpers
  • mellea/backends/openai.py - Record latency metrics
  • Other backends as applicable

Metric Definitions:

Metric Type Unit Attributes
mellea.llm.request.duration Histogram s model, backend, system, streaming
mellea.llm.ttfb Histogram s model, backend, system

Acceptance Criteria:

  • Duration histogram captures all request latencies
  • TTFB histogram captures streaming first-token time
  • Bucket boundaries appropriate for LLM workloads
  • Metrics have correct labels

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions