Skip to content

fix: Update MultiTurnStrategy to include validation failure reasons in repair messages#633

Merged
jakelorocco merged 2 commits into
generative-computing:mainfrom
vabarbosa:fix/multiturnstrategy-repair
Mar 13, 2026
Merged

fix: Update MultiTurnStrategy to include validation failure reasons in repair messages#633
jakelorocco merged 2 commits into
generative-computing:mainfrom
vabarbosa:fix/multiturnstrategy-repair

Conversation

@vabarbosa

@vabarbosa vabarbosa commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

Description

This PR fixes MultiTurnStrategy to include detailed validation failure reasons from ValidationResult.reason in repair messages. The strategy now provides the model with actionable feedback about what went wrong (similar to RepairTemplateStrategy), significantly improving convergence rates. In addition, documentation updates were made and an example added.

Changes

  1. Core Fix (mellea/stdlib/sampling/base.py):

    • Added module-level docstring explaining all sampling strategies and their use cases
    • Updated MultiTurnStrategy.repair() to include ValidationResult.reason in user messages
  2. Documentation (docs/examples/instruct_validate_repair/README.md):

    • Added comprehensive "Sampling Strategies" section explaining when to use each strategy
    • Included code examples for all existing three strategies
  3. Example (docs/examples/instruct_validate_repair/multiturn_strategy_example.py):

    • New working example demonstrating MultiTurnStrategy with validation functions

Impact

  • MultiTurnStrategy now converges effectively with validation functions
  • ✅ Users get clear guidance on using the sampling strategy
  • ✅ Validation feedback is more actionable (and not ignored)

Testing

The new example can be run as a test:

uv run pytest docs/examples/instruct_validate_repair/multiturn_strategy_example.py

Related Issues

Fixes #631


Checklist

  • Code changes implement the fix
  • Module docstring added
  • Documentation updated
  • Working example provided
  • Backward compatible
  • Tests pass (uv run pytest -m "not qualitative")
  • Code formatted (uv run ruff format .)
  • Linting passes (uv run ruff check .)
  • Type checking passes (uv run mypy .)

append validation failure reasons to the repair message for MultiTurnStrategy

Fixes generative-computing#631

Signed-off-by: va <va@us.ibm.com>
@vabarbosa
vabarbosa requested a review from a team as a code owner March 12, 2026 17:59
@mergify

mergify Bot commented Mar 12, 2026

Copy link
Copy Markdown

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert|release)(?:\(.+\))?:

@jakelorocco jakelorocco left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vabarbosa, thank you for the PR. I pushed one change to correct the docstring for the sampling file since the rejection strat doesn't rely on validation_result.reason. Everything else looks good so I'll approve and it'll merge after the workflows pass.

Thank you!

@jakelorocco
jakelorocco enabled auto-merge March 13, 2026 12:56
@jakelorocco
jakelorocco added this pull request to the merge queue Mar 13, 2026
Merged via the queue into generative-computing:main with commit ebdd092 Mar 13, 2026
4 checks passed
@vabarbosa
vabarbosa deleted the fix/multiturnstrategy-repair branch March 13, 2026 15:50
akihikokuroda pushed a commit to akihikokuroda/mellea that referenced this pull request May 27, 2026
…n repair messages (generative-computing#633)

* fix: update multiturnstrategy repair message

append validation failure reasons to the repair message for MultiTurnStrategy

Fixes generative-computing#631

Signed-off-by: va <va@us.ibm.com>

* docs: update docstring to be correct

---------

Signed-off-by: va <va@us.ibm.com>
Co-authored-by: jakelorocco <59755218+jakelorocco@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MultiTurnStrategy doesn't include validation failure reasons in repair messages

2 participants