Skip to content

feat(core): make system prompt immutable after session creation#33246

Open
0byte-coding wants to merge 3 commits into
anomalyco:devfrom
0byte-coding:fix/immutable-system-prompt
Open

feat(core): make system prompt immutable after session creation#33246
0byte-coding wants to merge 3 commits into
anomalyco:devfrom
0byte-coding:fix/immutable-system-prompt

Conversation

@0byte-coding

Copy link
Copy Markdown

Issue for this PR

Closes #29672

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Caches the system prompt in-memory per session ID so it becomes immutable after the session is created. This prevents unnecessary cache misses when the same session makes multiple requests.

The implementation stores system prompts in a HashMap keyed by session ID. This is very minimal and focused - just enough to lock the prompt after initialization.

How did you verify your code works?

I tested locally that:

  • Session initialization caches the system prompt correctly
  • Subsequent requests retrieve the cached prompt instead of regenerating it
  • Cache lookups return the exact same prompt object

Screenshots / recordings

N/A - This is not a UI change

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@github-actions

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

Based on the search results, I found several related PRs that involve system prompt and caching optimizations:

Potentially Related PRs:

  1. feat: improve deepseek prompt cache reuse #31867 - feat: improve deepseek prompt cache reuse

    • Related to prompt caching optimization for better cache hit rates
  2. fix(cache): improve Anthropic prompt cache hit rate with system split and tool stability #14743 - fix(cache): improve Anthropic prompt cache hit rate with system split and tool stability

    • Focuses on cache stability for system prompts with provider-specific optimization
  3. fix(session): move env block to tail of system prompt for cache stability #29949 - fix(session): move env block to tail of system prompt for cache stability

    • Another system prompt optimization for cache stability
  4. feat(provider): add provider-specific cache configuration system (significant token usage reduction) #5422 - feat(provider): add provider-specific cache configuration system (significant token usage reduction)

    • Related to caching systems at a broader level
  5. fix(session): fire system.transform before messages.transform #19961 - fix(session): fire system.transform before messages.transform

    • Related to system prompt handling in sessions

These PRs address similar domains of system prompt caching and session initialization, though none appear to be exact duplicates of PR #33246's specific implementation (in-memory HashMap caching per session ID). However, PRs #31867, #14743, and #29949 are closest in addressing cache efficiency and system prompt immutability concepts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Modify system prompt to improve cache hit rate

1 participant