Skip to content

feat: add token usage from previous turns to context length estimation #1319

Description

@jakelorocco

#1270 (review) adds support for token based context length limiting. It currently utilizes a few rough heuristics. For providers that return prompt tokens / new tokens, we should be able to more accurately estimate the cost of our inputs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/contextChatContext, message history, context windowing, serializationenhancementNew feature or requestp2Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions