Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions docs/examples/instruct_validate_repair/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,15 @@ Shows how to use custom validation functions for complex requirements.
- Using `simple_validate()` helper
- Combining multiple validation strategies

### multiturn_strategy_example.py
Demonstrates MultiTurnStrategy for conversational repair with validation feedback.

**Key Features:**
- Using ChatContext for multi-turn conversations
- Validation functions with detailed failure reasons
- Iterative improvement through conversational feedback
- Understanding when to use different repair strategies

## Concepts Demonstrated

- **Instruct**: Generating outputs with natural language instructions
Expand Down Expand Up @@ -86,6 +95,66 @@ result = m.instruct(
)
```

## Sampling Strategies

Mellea provides three main sampling strategies for handling validation failures:

### RejectionSamplingStrategy
- **Use case**: Simple retry with the same prompt
- **Behavior**: Repeats the exact same instruction if validation fails
- **Best for**: Non-deterministic failures, simple requirements
- **Context**: Doesn't modify context between attempts

**Example:**
```python
from mellea.stdlib.sampling import RejectionSamplingStrategy

result = m.instruct(
"Write an email...",
requirements=["be formal", "under 50 words"],
strategy=RejectionSamplingStrategy(loop_budget=3)
)
```

### RepairTemplateStrategy
- **Use case**: Single-turn repair with feedback
- **Behavior**: Adds validation failure reasons to the instruction and retries
- **Best for**: Simple tasks where feedback can be added to the instruction
- **Context**: Doesn't modify context, only the instruction

**Example:**
```python
from mellea.stdlib.sampling import RepairTemplateStrategy

result = m.instruct(
"Write an email...",
requirements=["be formal", word_count_req],
strategy=RepairTemplateStrategy(loop_budget=3)
)
```

### MultiTurnStrategy
- **Use case**: Multi-turn conversational repair
- **Behavior**: Adds validation failure reasons as a new user message in the conversation
- **Best for**: Complex tasks, conversational contexts, agentic workflows
- **Context**: Builds conversation history with repair feedback
- **Requires**: ChatContext (conversational context)

**Example:**
```python
from mellea.stdlib.sampling import MultiTurnStrategy
from mellea.stdlib.context import ChatContext

m = start_session(ctx=ChatContext())
result = m.instruct(
"Write a detailed analysis...",
requirements=[...],
strategy=MultiTurnStrategy(loop_budget=3)
)
```

**Key Improvement**: All strategies now include detailed validation failure reasons (from `ValidationResult.reason`) when available, allowing the model to understand WHY requirements failed, not just WHICH requirements failed. This significantly improves convergence rates.

## Related Documentation

- See `mellea/stdlib/requirements/` for requirement types
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# pytest: ollama, llm, qualitative

"""MultiTurnStrategy Example with Validation Functions

Demonstrates how MultiTurnStrategy uses conversational repair with detailed
validation feedback to iteratively improve outputs.

This example shows the key difference between MultiTurnStrategy and other strategies:
it builds a conversation history where validation failures are communicated as user
messages, allowing the model to iteratively improve its response through dialogue.
"""

from mellea import start_session
from mellea.backends import ModelOption
from mellea.stdlib.context import ChatContext
from mellea.stdlib.requirements import req
from mellea.stdlib.requirements.requirement import simple_validate
from mellea.stdlib.sampling import MultiTurnStrategy

MIN_WORD_COUNT = 100


def validate_word_count(text: str) -> tuple[bool, str]:
"""A validation function that checks for minimum word count.

Returns detailed failure reasons to help the model understand what's wrong.
"""
word_count = len(text.split())
if word_count < MIN_WORD_COUNT:
return (
False,
f"Output has only {word_count} words. Need at least {MIN_WORD_COUNT} words.",
)
return True, ""


def demo_multiturn_repair():
"""Demonstrate MultiTurnStrategy with detailed validation feedback."""

# MultiTurnStrategy requires ChatContext for conversational repair
m = start_session(
ctx=ChatContext(), model_options={ModelOption.MAX_NEW_TOKENS: 300}
)

print("=== MultiTurnStrategy Demo ===\n")
print("Task: Write a detailed explanation of quantum computing\n")

result = m.instruct(
"Explain quantum computing like I am 5.",
requirements=[
req(
"Must be at least 100 words",
validation_fn=simple_validate(validate_word_count),
),
"Include at least one real-world application",
"Avoid technical jargon",
],
strategy=MultiTurnStrategy(loop_budget=5),
return_sampling_results=True,
)

# Show the repair process
print(f"Attempts made: {len(result.sample_generations)}")
print(f"Success: {result.success}\n")

for i, (gen, validations) in enumerate(
zip(result.sample_generations, result.sample_validations), 1
):
print(f"\n--- Attempt {i} ---")
output_text = str(gen.value) if gen.value else ""
print(f"Output length: {len(output_text.split())} words")

failed = [v for _, v in validations if not v.as_bool()]
if failed:
print("Failed validations:")
for val in failed:
if val.reason:
print(f" - {val.reason}")
else:
print("✓ All validations passed!")

print(f"\n{'=' * 60}")
print("Final output:")
print(f"{'=' * 60}")
print(result.value)
print(f"{'=' * 60}")

# Show the conversation history
print("\nConversation history:")
for i, msg in enumerate(m.ctx.as_list(), 1):
role = getattr(msg, "role", "unknown")
content = str(getattr(msg, "content", msg))[:100]
print(f"{i}. [{role}] {content}...")

return result


if __name__ == "__main__":
demo_multiturn_repair()
40 changes: 32 additions & 8 deletions mellea/stdlib/sampling/base.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,18 @@
"""Base Sampling Strategies."""
"""Base Sampling Strategies.

Sampling strategies control how Mellea handles validation failures during generation:

- **RejectionSamplingStrategy**: Simple retry with the same prompt. Best for non-deterministic
failures where the same instruction might succeed on retry.

- **RepairTemplateStrategy**: Single-turn repair by modifying the instruction with validation
feedback. Adds failure reasons to the instruction and retries. Best for simple tasks where
feedback can be incorporated into the instruction.

- **MultiTurnStrategy**: Multi-turn conversational repair (requires ChatContext). Adds validation
failure reasons as new user messages in the conversation, allowing iterative improvement through
dialogue. Best for complex tasks and agentic workflows.
"""

import abc
from copy import deepcopy
Expand Down Expand Up @@ -496,7 +510,7 @@ def repair(
past_results: list[ModelOutputThunk],
past_val: list[list[tuple[Requirement, ValidationResult]]],
) -> tuple[Component, Context]:
"""Returns a Message with a description of the failed requirements.
"""Returns a Message with a description (and validation reasons) of the failed requirements.

Args:
old_ctx: The context WITHOUT the last action + output.
Expand All @@ -512,15 +526,25 @@ def repair(
" Need chat context to run agentic sampling."
)

last_failed_reqs: list[Requirement] = [s[0] for s in past_val[-1] if not s[1]]
last_failed_reqs_str = "* " + "\n* ".join(
[str(r.description) for r in last_failed_reqs]
)
# TODO: what to do with checks ??
# Get failed requirements and their detailed validation reasons
failed_items = [(req, val) for req, val in past_val[-1] if not val.as_bool()]

# Build repair feedback using ValidationResult.reason when available
repair_lines = []
for req, validation in failed_items:
if validation.reason:
repair_lines.append(f"* {validation.reason}")
else:
# Fallback to requirement description if no reason
repair_lines.append(f"* {req.description}")

feedback = "\n".join(repair_lines)
next_action = Message(
role="user",
content=f"The following requirements have not been met: \n{last_failed_reqs_str}\n Please try again to fulfill the requirements.",
content=(
f"The following requirements have not been met:\n{feedback}\n"
f"Please try again to fulfill the requirements."
),
)

return next_action, new_ctx
Loading