Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/docs-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,11 @@ jobs:
- name: Generate API documentation
run: uv run python tooling/docs-autogen/build.py

# -- Run docs-autogen unit tests ------------------------------------------

- name: Run CLI reference tests
run: uv run pytest tooling/docs-autogen/test_cli_reference.py -v --tb=short

# -- Validate static docs ------------------------------------------------

- name: Lint static docs (markdownlint)
Expand Down
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -455,7 +455,8 @@ pyrightconfig.json
.claude/*
!.claude/settings.json

# Generated API documentation (built by tooling/docs-autogen/)
# Generated documentation (built by tooling/docs-autogen/)
docs/docs/api/
docs/docs/api-reference.mdx
docs/docs/reference/cli.md
.venv-docs-autogen/
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ mkdir -p .bob && ln -s ../.agents/skills .bob/skills
- Use `...` in `@generative` function bodies
- Prefer primitives over classes
- **Friendly Dependency Errors**: Wraps optional backend imports in `try/except ImportError` with a helpful message (e.g., "Please pip install mellea[hf]"). See `mellea/stdlib/session.py` for examples.
- **CLI command docstrings**: Typer command functions in `cli/` follow an enriched convention with `Prerequisites:` and `See Also:` sections — these feed the auto-generated CLI reference page. See [`docs/docs/guide/CONTRIBUTING.md`](docs/docs/guide/CONTRIBUTING.md) for the full pattern. Regenerate after changes: `uv run poe clidocs`. Test the generator: `uv run pytest tooling/docs-autogen/test_cli_reference.py -v`. Full pipeline docs: [`tooling/docs-autogen/README.md`](tooling/docs-autogen/README.md).
- **Backend telemetry fields**: All backends must populate `mot.usage` (dict with `prompt_tokens`, `completion_tokens`, `total_tokens`), `mot.model` (str), and `mot.provider` (str) in their `post_processing()` method. `mot.streaming` (bool) and `mot.ttfb_ms` (float | None) are set automatically in `astream()` — backends do not need to set them. Metrics are automatically recorded by `TokenMetricsPlugin` and `LatencyMetricsPlugin` — don't add manual `record_token_usage_metrics()` or `record_request_duration()` calls.

## 6. Commits & Hooks
Expand Down
56 changes: 54 additions & 2 deletions cli/alora/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,25 @@ def alora_train(
max_length: int = typer.Option(1024, help="Max sequence length"),
grad_accum: int = typer.Option(4, help="Gradient accumulation steps"),
):
"""Train an aLoRA or LoRA model on your dataset.
"""Train an aLoRA or LoRA adapter on a labelled dataset.

Fine-tunes a base causal language model using a JSONL dataset of item/label
pairs. Supports both aLoRA (asymmetric LoRA) and standard LoRA adapters.

Prerequisites:
Mellea installed with adapter extras (``uv add mellea[adapters]``).
A CUDA, MPS, or CPU device available for training.

Output:
Saves adapter weights to the path specified by ``--outfile``. The output
directory contains an ``adapter_config.json`` and the trained weight
files, ready for upload or local inference.

Examples:
m alora train data.jsonl --basemodel ibm-granite/granite-3.3-2b-instruct --outfile ./adapter

See Also:
guide: advanced/lora-and-alora-adapters

Args:
datafile: JSONL file with item/label pairs for training.
Expand Down Expand Up @@ -79,7 +97,23 @@ def alora_upload(
"processing if the model is invoked as an intrinsic.",
),
):
"""Upload trained adapter to remote model registry.
"""Upload a trained adapter to a remote model registry.

Pushes adapter weights to Hugging Face Hub, optionally packaging the adapter
as an intrinsic with an ``io.yaml`` configuration file.

Prerequisites:
Hugging Face CLI authenticated (``huggingface-cli login``).

Output:
Creates or updates a Hugging Face Hub repository at the name specified
by ``--name`` and uploads the adapter weight files.

Examples:
m alora upload ./adapter --name acme/my-alora

See Also:
guide: advanced/lora-and-alora-adapters

Args:
weight_path: Path to saved adapter weights directory.
Expand Down Expand Up @@ -140,6 +174,24 @@ def alora_add_readme(
):
"""Generate and upload an INTRINSIC_README.md for a trained adapter.

Uses an LLM to auto-generate documentation for a trained adapter based on
the training data and model configuration, then uploads it to the Hugging
Face Hub repository.

Prerequisites:
Hugging Face CLI authenticated (``huggingface-cli login``).
An LLM backend available for README generation.

Output:
Generates a README.md file, displays it for confirmation, and uploads
it to the Hugging Face Hub repository specified by ``--name``.

Examples:
m alora add-readme data.jsonl --basemodel ibm-granite/granite-3.3-2b-instruct --name acme/my-alora

See Also:
guide: advanced/lora-and-alora-adapters

Args:
datafile: JSONL file with item/label pairs used to train the adapter.
basemodel: Base model ID or path.
Expand Down
26 changes: 21 additions & 5 deletions cli/decompose/decompose.py
Original file line number Diff line number Diff line change
Expand Up @@ -236,12 +236,28 @@ def run(
),
] = False,
) -> None:
"""Runs the ``m decompose`` CLI workflow and writes generated outputs.
"""Break a complex task into ordered, executable subtasks.

Reads user queries from a file or interactive input, runs the decomposition
pipeline for each task job, and writes one JSON file, one rendered Python
program, and any generated validation modules under a per-job output
directory.
Reads user queries from a file or interactive input, runs the LLM-driven
decomposition pipeline for each task job, and writes one JSON file, one
rendered Python script, and any generated validation modules under a per-job
output directory.

Prerequisites:
Mellea installed (``uv add mellea``). An Ollama instance running locally,
or an OpenAI-compatible endpoint configured via ``--backend-endpoint``.

Output:
Creates a directory ``<out-dir>/<out-name>/`` containing a JSON
decomposition result file, a ready-to-run Python script, and any
generated validation modules. One directory per task job.

Examples:
m decompose run --out-dir ./output --input-file tasks.txt

See Also:
guide: guide/m-decompose
guide: how-to/refactor-prompts-with-cli

Args:
out_dir: Existing directory under which per-job output directories are
Expand Down
48 changes: 42 additions & 6 deletions cli/eval/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,38 @@

import typer

eval_app = typer.Typer(name="eval")
eval_app = typer.Typer(name="eval", help="LLM-as-a-judge evaluation pipelines.")


def eval_run(
test_files: list[str] = typer.Argument(
..., help="List of paths to json/jsonl files containing test cases"
),
backend: str = typer.Option("ollama", "--backend", "-b", help="Generation backend"),
model: str = typer.Option(None, "--model", help="Generation model name"),
backend: str = typer.Option(
"ollama",
"--backend",
"-b",
help="Inference backend for generating candidate responses (e.g. ollama, openai)",
),
model: str = typer.Option(
None,
"--model",
help="Model name/id for the generation backend; uses backend default if omitted",
),
max_gen_tokens: int = typer.Option(
256, "--max-gen-tokens", help="Max tokens to generate for responses"
),
judge_backend: str = typer.Option(
None, "--judge-backend", "-jb", help="Judge backend"
None,
"--judge-backend",
"-jb",
help="Inference backend for the judge model; reuses --backend if omitted",
),
judge_model: str = typer.Option(
None,
"--judge-model",
help="Model name/id for the judge; uses judge backend default if omitted",
),
judge_model: str = typer.Option(None, "--judge-model", help="Judge model name"),
max_judge_tokens: int = typer.Option(
256, "--max-judge-tokens", help="Max tokens for the judge model's judgement."
),
Expand All @@ -29,14 +45,34 @@ def eval_run(
output_format: str = typer.Option(
"json", "--output-format", help="Either json or jsonl format for results"
),
continue_on_error: bool = typer.Option(True, "--continue-on-error"),
continue_on_error: bool = typer.Option(
True,
"--continue-on-error",
help="Skip failed test cases instead of aborting the entire run",
),
):
"""Run LLM-as-a-judge evaluation on one or more test files.

Loads test cases from JSON/JSONL files, generates candidate responses using
the specified generation backend, scores them with a judge model, and writes
aggregated results to a file.

Prerequisites:
Mellea installed (``uv add mellea``). At least one inference backend
available (Ollama by default). A separate judge backend/model is
recommended but optional (defaults to the generation backend).

Output:
Writes evaluation results to ``<output-path>.<output-format>`` (default
``eval_results.json``). The file contains per-test-case scores, judge
verdicts, and aggregate statistics.

Examples:
m eval run tests.jsonl --backend ollama --model granite3.3:2b

See Also:
guide: evaluation-and-observability/evaluate-with-llm-as-a-judge

Args:
test_files: Paths to JSON/JSONL files containing test cases.
backend: Generation backend name.
Expand Down
31 changes: 29 additions & 2 deletions cli/fix/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,21 @@ def fix_async(
False, "--dry-run", help="Report locations without modifying files"
),
):
"""Fix async calls (aact, ainstruct, aquery) for the await_result default change.
"""Fix async calls for the await_result default change.

Scans Python source files for ``aact``, ``ainstruct``, and ``aquery`` calls
and applies an automated migration to restore blocking behaviour after the
``await_result`` default changed from ``True`` to ``False``.

Prerequisites:
Mellea installed (``uv add mellea``).

Output:
Modifies Python source files in place (unless ``--dry-run``). Prints a
summary of fixed call sites with file paths and line numbers.

Examples:
m fix async src/ --dry-run

Args:
path: File or directory to scan.
Expand Down Expand Up @@ -83,7 +97,20 @@ def fix_genslots(
False, "--dry-run", help="Report locations without modifying files"
),
):
"""Rewrite old genslot imports and class names to genstub equivalents.
"""Rewrite genslot imports and class names to genstub equivalents.

Scans Python source files and replaces deprecated ``GenerativeSlot`` imports
and class references with their ``GenerativeStub`` replacements.

Prerequisites:
Mellea installed (``uv add mellea``).

Output:
Modifies Python source files in place (unless ``--dry-run``). Prints a
summary of rewritten references with file paths and line numbers.

Examples:
m fix genslots src/ --dry-run

Args:
path: File or directory to scan.
Expand Down
11 changes: 9 additions & 2 deletions cli/m.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,15 @@ def callback() -> None:
"""Mellea command-line tool for LLM-powered workflows.

Provides sub-commands for serving models (``m serve``), training and uploading
adapters (``m alora``), decomposing tasks into subtasks (``m decompose``), and
running test-based evaluation pipelines (``m eval``).
adapters (``m alora``), decomposing tasks into subtasks (``m decompose``),
running test-based evaluation pipelines (``m eval``), and applying automated
code migrations (``m fix``).

Prerequisites:
Mellea installed (``uv add mellea``).

See Also:
guide: getting-started/quickstart
"""


Expand Down
22 changes: 21 additions & 1 deletion cli/serve/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,27 @@ def serve(
host: str = typer.Option("0.0.0.0", help="Host to bind to"),
port: int = typer.Option(8080, help="Port to bind to"),
):
"""Serve a FastAPI endpoint for a given script."""
"""Serve a Mellea program as an OpenAI-compatible HTTP endpoint.

Loads a Python script containing a Mellea generative function and exposes it
via a FastAPI server implementing the OpenAI chat completions API. The server
accepts ``POST /v1/chat/completions`` requests.

Prerequisites:
Mellea installed (``uv add mellea``). The target script must define at
least one generative function.

Output:
Starts a long-running HTTP server on the specified host and port.
The ``/v1/chat/completions`` endpoint accepts OpenAI-format chat
completion requests and returns ``ChatCompletion`` JSON responses.

Examples:
m serve my_app.py --port 9000

See Also:
guide: integrations/m-serve
"""
module = load_module_from_path(script_path)
route_path = "/v1/chat/completions"

Expand Down
3 changes: 2 additions & 1 deletion docs/docs/advanced/lora-and-alora-adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,4 +158,5 @@ affect other sessions.

**See also:** [Intrinsics](./intrinsics) |
[The Requirements System](../concepts/requirements-system) |
[Write Custom Verifiers](../how-to/write-custom-verifiers)
[Write Custom Verifiers](../how-to/write-custom-verifiers) |
[CLI Reference](../reference/cli)
Loading
Loading