diff --git a/.github/workflows/skill-evals.yml b/.github/workflows/skill-evals.yml new file mode 100644 index 0000000..f7966d9 --- /dev/null +++ b/.github/workflows/skill-evals.yml @@ -0,0 +1,92 @@ +name: skill-evals + +# Runs the flyte-agent-plugin skills eval harness. On PRs it runs only the +# scenarios affected by the changed files (see evals/select.py); nightly it runs +# the full matrix including the `real` tier. +on: + pull_request: + paths: + - "plugins/**" + - "evals/**" + schedule: + - cron: "0 7 * * *" # nightly full matrix (incl. real tier) + workflow_dispatch: {} + +env: + PYTHONPATH: ${{ github.workspace }} + +jobs: + # 1) Decide what to run from the diff. + select: + runs-on: ubuntu-latest + outputs: + skills: ${{ steps.sel.outputs.skills }} + run_all: ${{ steps.sel.outputs.run_all }} + run_kind: ${{ steps.sel.outputs.run_kind }} + run_real: ${{ steps.sel.outputs.run_real }} + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - run: pip install pyyaml requests + - id: sel + run: | + if [ "${{ github.event_name }}" = "pull_request" ]; then + OUT=$(python -m evals.select --base "origin/${{ github.base_ref }}") + else + # nightly / manual: run everything + OUT=$(python -m evals.select --changed evals/manifest.yaml) + fi + echo "$OUT" + python - "$OUT" <<'PY' >> "$GITHUB_OUTPUT" + import json, sys + d = json.loads(sys.argv[1]) + print("skills=" + json.dumps(d["skills"])) + print("run_all=" + str(d["run_all"]).lower()) + print("run_kind=" + str(d["run_kind"]).lower()) + print("run_real=" + str(d["run_real"]).lower()) + PY + + # 2) Static + trajectory tiers, orchestrated on demo.hosted via Flyte. + flyte-evals: + needs: select + if: needs.select.outputs.skills != '[]' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - run: pip install flyte>=2.5.0 pyyaml requests + - name: Configure Flyte auth + run: echo "auth via UNION_API_KEY secret" + env: + UNION_API_KEY: ${{ secrets.UNION_API_KEY }} + - name: Run evals on demo.hosted + env: + UNION_API_KEY: ${{ secrets.UNION_API_KEY }} + GLM_API_KEY: ${{ secrets.GLM_API_KEY }} + TIERS: ${{ github.event_name == 'schedule' && '["static","trajectory","real"]' || '["static","trajectory"]' }} + run: | + flyte --config evals/config/flyte.yaml run --copy-style loaded_modules \ + evals/workflows/eval_wf.py main \ + --skills '${{ needs.select.outputs.skills }}' \ + --tiers "$TIERS" + # The workflow attaches the HTML scorecard to the run report; the run exits + # non-zero if any scenario fails (enforced inside aggregate/report). + + # 3) Real kind-in-Docker smoke, only when a kind skill changed (privileged). + kind-smoke: + needs: select + if: needs.select.outputs.run_kind == 'true' + runs-on: ubuntu-latest # GitHub-hosted runners allow privileged Docker/kind + steps: + - uses: actions/checkout@v4 + - uses: helm/kind-action@v1 + with: + install_only: true + - name: kind smoke + run: bash evals/kind_smoke/run.sh diff --git a/README.md b/README.md index 5d915b8..5eae412 100644 --- a/README.md +++ b/README.md @@ -134,6 +134,20 @@ pi install git:github.com/flyteorg/flyte-agent-plugins@ # pinned to | [`flyte-sdk-data`](plugins/flyte/skills/flyte-sdk-data) | Handles data engineering patterns: ETL pipelines, data processing, data quality checks, fanout/map tasks, conditions, dynamic workflows, and batch data transformations. For: ETL, Parquet, CSV, JsonlFile/Dir, schema validation. | | [`flyte-sdk-ml`](plugins/flyte/skills/flyte-sdk-ml) | Handles ML workload patterns: model training, hyperparameter optimization, experiment tracking, model evaluation and selection, batch inference, real-time serving, and model monitoring. For: PyTorch, scikit-learn, HuggingFace, GPU, drift detection. | +### Migration (Flyte 1 → 2) + +Convert existing Flyte 1 (`flytekit`) code to Flyte 2. Distilled from the official +[Flyte 1 → 2 migration guide](https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/). + +| Skill | Description | +|-------|-------------| +| [`flyte-migrate`](plugins/flyte/skills/flyte-migrate) | Start-here migration orchestrator: the `flytekit`→`flyte` shift, the terminology/concept mapping, the two mechanical changes, an incremental migration strategy, hybrid v1/v2 pipelines during transition, and the gotchas — routes to the specific skills below. | +| [`flyte-migrate-tasks-workflows`](plugins/flyte/skills/flyte-migrate-tasks-workflows) | Migrate `@task`/`@workflow`/`@dynamic` into a single `@env.task` on a `TaskEnvironment`; sequential ordering without `>>`, nested "subworkflows" as tasks, and the parameter-mapping table. | +| [`flyte-migrate-config`](plugins/flyte/skills/flyte-migrate-config) | Migrate task configuration (images `ImageSpec`→`flyte.Image`, resources/GPUs, `cache_version`→`cache`, secrets, `LaunchPlan`/`CronSchedule`→`Trigger`/`Cron`) and the `pyflyte`→`flyte` CLI / config files. | +| [`flyte-migrate-control-flow`](plugins/flyte/skills/flyte-migrate-control-flow) | Replace `conditional()` with native `if`/`else`, `@dynamic` with plain Python loops, `on_failure` with `try`/`except`, and `map_task` with `flyte.map` / `asyncio.gather`. | +| [`flyte-migrate-data-io`](plugins/flyte/skills/flyte-migrate-data-io) | Migrate data types & I/O: `FlyteFile`/`FlyteDirectory`→`flyte.io.File`/`Dir`, `StructuredDataset`→`flyte.io.DataFrame`, dataclasses/Pydantic as task I/O. | +| [`flyte-migrate-ml`](plugins/flyte/skills/flyte-migrate-ml) | Migrate ML workloads (training, HPO, GPU/deep learning, batch inference, end-to-end pipelines) and the new-in-v2 patterns (real-time serving, apps, sandboxed execution) they unlock. | + Example: ``` @@ -233,8 +247,7 @@ scripts/smoke_test_mcp.py # end-to-end check of the local MCP Each harness consumes a different part of this. Claude Code and Codex read the plugin manifests, so the **plugin name** matters to them. Hermes, opencode, and pi install skills -by **directory path**, so `plugins/flyte/skills/…` is their interface — which is why the -rename from `flyte-skills` touched both. +by **directory path**, so `plugins/flyte/skills/…` is their interface. The `.mcp.json` server is Claude Code-specific; the skills themselves stay portable across harnesses. diff --git a/evals/.gitignore b/evals/.gitignore new file mode 100644 index 0000000..75c6182 --- /dev/null +++ b/evals/.gitignore @@ -0,0 +1,3 @@ +__pycache__/ +*.pyc +.pytest_cache/ diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..197b123 --- /dev/null +++ b/evals/README.md @@ -0,0 +1,84 @@ +# flyte agent plugin eval harness + +Automated testing & evaluation for the `flyte` agent skills. It runs a real +agent harness (**opencode / pi / hermes**) against the union-hosted **GLM** endpoint, +hands it a skill + a task, and scores what it produces — orchestrated as **Flyte +workflows on `demo.hosted.unionai.cloud`**, with path-based selection so only the +scenarios for changed skills run per PR. + +## Concepts + +- **Scenario** (`scenarios//*.yaml`) — one declarative eval: a prompt, the + deterministic `checks`, an LLM-judge `rubric`, and (tier `real`) a `real_run`. +- **Tiers** — `static` (lint the SKILL.md; no LLM), `trajectory` (run the agent with + side-effecting commands stubbed, judge the artifacts it produces), `real` (actually + `flyte run` SDK output on demo.hosted; kind stood up for real on a CI runner). +- **Control arm** — every trajectory/real scenario runs twice: **treatment** (skill + installed) and **control** (skill absent). The headline metric is + **lift = treatment − control**, isolating the skill's contribution. A negative lift + is a regression signal. + +## Run locally + +```bash +pip install pyyaml requests # + the harness CLI(s) you want to exercise +export PYTHONPATH=$(git rev-parse --show-toplevel) + +# Static lint of every skill — no LLM, no agent: +python -m evals.harness.run --tier static + +# One trajectory scenario end-to-end (needs GLM_API_KEY + the harness CLI): +export GLM_API_KEY=... # token for the demo.hosted GLM app +python -m evals.harness.run --scenario sdk-author-map-task --harness opencode + +# Everything for one skill, JSON out + scorecard: +python -m evals.harness.run --skill flyte-sdk-author --json out.json +python -m evals.report out.json --html scorecard.html +``` + +`GLM_BASE_URL` / `GLM_MODEL` / `GLM_API_KEY` configure the endpoint (see +`harness/glm.py`). The `real` tier only submits remote runs when +`FLYTE_EVALS_ENABLE_REAL=1` is set. + +## Run on demo.hosted (Flyte) + +```bash +flyte --config evals/config/flyte.yaml run evals/workflows/eval_wf.py main \ + --skills '["flyte-sdk-author"]' --tiers '["static","trajectory"]' +``` + +Fans out one action per (scenario × harness); the `aggregate` task attaches an HTML +scorecard to the run's report tab. GLM creds come from the `glm-api-key` Flyte secret. + +## Selective execution + +```bash +python -m evals.select --base origin/main # changed files -> scenario subset +``` + +Emits `{run_all, skills, scenario_ids, run_kind, run_real}`. A change under a skill +dir selects that skill's scenarios; a change to the engine (`harness/**`, +`workflows/**`, `manifest.yaml`, …) forces the whole suite. CI wiring is in +`.github/workflows/skill-evals.yml`. + +## Layout + +``` +manifest.yaml cross-cutting config + shared-infra globs + skill classes +scenarios//*.yaml declarative eval specs +harness/ spec, checks, static_lint, sandbox, runners/, judge, evaluate, run +workflows/ eval_wf.py (Flyte fan-out+aggregate), images.py +config/flyte.yaml demo.hosted admin/image/task config +select.py report.py changed-files selector; JSON+HTML+markdown scorecard +kind_smoke/run.sh real kind-in-Docker smoke (privileged CI runner) +tests/ unit tests (no network/LLM) +``` + +## Status / open spikes + +- **GLM endpoint contract** — endpoint is live but auth-gated; confirm the exact + OpenAI-compatible route + auth header + model name, wire the key as a Flyte/GH + secret. Single point of change: `harness/glm.py`. +- **Harness invocation** — the opencode adapter is complete; `pi` and `hermes` + adapters carry a best-effort headless invocation to confirm in the adapter spike + (`is_available()` gates uninstalled harnesses cleanly). See `harness/runners/`. diff --git a/evals/__init__.py b/evals/__init__.py new file mode 100644 index 0000000..bdaa967 --- /dev/null +++ b/evals/__init__.py @@ -0,0 +1 @@ +"""Testing & eval harness for the Flyte plugin skills.""" diff --git a/evals/config/flyte.yaml b/evals/config/flyte.yaml new file mode 100644 index 0000000..7c2a860 --- /dev/null +++ b/evals/config/flyte.yaml @@ -0,0 +1,11 @@ +# Flyte config for running the eval harness on demo.hosted.unionai.cloud. +# Referenced by evals/workflows/eval_wf.py and the GitHub Actions `flyte-evals` job. +# Auth is supplied out-of-band (device flow / api-key env), never committed here. +admin: + endpoint: dns:///demo.hosted.unionai.cloud +image: + builder: remote +task: + org: demo + project: flytesnacks + domain: development diff --git a/evals/harness/__init__.py b/evals/harness/__init__.py new file mode 100644 index 0000000..76f4fc6 --- /dev/null +++ b/evals/harness/__init__.py @@ -0,0 +1 @@ +"""Core reusable engine: specs, checks, sandbox, runners, judge, scoring.""" diff --git a/evals/harness/checks.py b/evals/harness/checks.py new file mode 100644 index 0000000..e92fe75 --- /dev/null +++ b/evals/harness/checks.py @@ -0,0 +1,191 @@ +"""Deterministic, LLM-free checks referenced by scenario specs. + +Each check kind is a function `(workspace: Path, params: dict) -> CheckResult`. +Checks run against the workspace an agent produced (trajectory tier) or against a +skill directory (static tier). They are pure w.r.t. the filesystem + subprocess +only — no network, no LLM — so they are deterministic and unit-testable. + +Add a new kind by decorating a function with @check("my_kind"). +""" + +from __future__ import annotations + +import ast +import glob as _glob +import pathlib +import re +import subprocess +from dataclasses import dataclass +from typing import Any, Callable + +import yaml + + +@dataclass(frozen=True) +class CheckResult: + kind: str + passed: bool + detail: str + + def __bool__(self) -> bool: # allow `all(results)` + return self.passed + + +CheckFn = Callable[[pathlib.Path, dict], CheckResult] +_REGISTRY: dict[str, CheckFn] = {} + + +def check(kind: str) -> Callable[[CheckFn], CheckFn]: + def deco(fn: CheckFn) -> CheckFn: + _REGISTRY[kind] = fn + return fn + return deco + + +def run_checks(workspace: pathlib.Path, specs: list[dict[str, Any]]) -> list[CheckResult]: + results: list[CheckResult] = [] + for spec in specs: + kind = spec.get("kind") + fn = _REGISTRY.get(kind) + if fn is None: + results.append(CheckResult(str(kind), False, f"unknown check kind {kind!r}")) + continue + try: + results.append(fn(workspace, spec)) + except Exception as exc: # a check crash is a failure, never a harness crash + results.append(CheckResult(str(kind), False, f"check raised: {exc!r}")) + return results + + +def _files(workspace: pathlib.Path, pattern: str) -> list[pathlib.Path]: + return [pathlib.Path(p) for p in _glob.glob(str(workspace / pattern), recursive=True)] + + +# ---------------------------------------------------------------------------- # +# Check kinds +# ---------------------------------------------------------------------------- # + +@check("file_glob") +def _file_glob(ws: pathlib.Path, p: dict) -> CheckResult: + """At least `min_count` (default 1) files match `glob`.""" + pattern = p["glob"] + minc = int(p.get("min_count", 1)) + matches = [f for f in _files(ws, pattern) if f.is_file()] + ok = len(matches) >= minc + return CheckResult("file_glob", ok, f"{len(matches)} match(es) for {pattern!r} (need >= {minc})") + + +@check("contains_regex") +def _contains_regex(ws: pathlib.Path, p: dict) -> CheckResult: + """Some file matching `file_glob` contains `pattern`.""" + pattern = re.compile(p["pattern"], re.MULTILINE) + files = [f for f in _files(ws, p.get("file_glob", "**/*")) if f.is_file()] + for f in files: + try: + if pattern.search(f.read_text(errors="ignore")): + return CheckResult("contains_regex", True, f"{p['pattern']!r} found in {f.name}") + except OSError: + continue + return CheckResult("contains_regex", False, f"{p['pattern']!r} not found in {len(files)} file(s)") + + +@check("not_contains_regex") +def _not_contains_regex(ws: pathlib.Path, p: dict) -> CheckResult: + """No file matching `file_glob` contains `pattern` (anti-pattern guard).""" + pattern = re.compile(p["pattern"], re.MULTILINE) + for f in _files(ws, p.get("file_glob", "**/*")): + if not f.is_file(): + continue + try: + m = pattern.search(f.read_text(errors="ignore")) + except OSError: + continue + if m: + return CheckResult("not_contains_regex", False, f"forbidden {p['pattern']!r} in {f.name}") + return CheckResult("not_contains_regex", True, f"{p['pattern']!r} absent (good)") + + +@check("python_imports") +def _python_imports(ws: pathlib.Path, p: dict) -> CheckResult: + """Some produced .py file imports `module` (top-level name).""" + module = p["module"] + want = module.split(".")[0] + for f in _files(ws, p.get("file_glob", "**/*.py")): + if not f.is_file(): + continue + try: + tree = ast.parse(f.read_text(errors="ignore")) + except SyntaxError: + continue + for node in ast.walk(tree): + if isinstance(node, ast.Import) and any(a.name.split(".")[0] == want for a in node.names): + return CheckResult("python_imports", True, f"import {module} in {f.name}") + if isinstance(node, ast.ImportFrom) and (node.module or "").split(".")[0] == want: + return CheckResult("python_imports", True, f"from {module} in {f.name}") + return CheckResult("python_imports", False, f"no import of {module!r} found") + + +@check("python_parses") +def _python_parses(ws: pathlib.Path, p: dict) -> CheckResult: + """Every .py file matching `file_glob` parses (valid syntax).""" + files = [f for f in _files(ws, p.get("file_glob", "**/*.py")) if f.is_file()] + if not files: + return CheckResult("python_parses", False, "no python files produced") + for f in files: + try: + ast.parse(f.read_text(errors="ignore")) + except SyntaxError as e: + return CheckResult("python_parses", False, f"syntax error in {f.name}: {e}") + return CheckResult("python_parses", True, f"{len(files)} python file(s) parse") + + +@check("yaml_valid") +def _yaml_valid(ws: pathlib.Path, p: dict) -> CheckResult: + """Every file matching `file_glob` is valid YAML.""" + files = [f for f in _files(ws, p.get("file_glob", "**/*.y*ml")) if f.is_file()] + if not files and p.get("required", True): + return CheckResult("yaml_valid", False, "no yaml files produced") + for f in files: + try: + list(yaml.safe_load_all(f.read_text(errors="ignore"))) + except yaml.YAMLError as e: + return CheckResult("yaml_valid", False, f"invalid yaml {f.name}: {e}") + return CheckResult("yaml_valid", True, f"{len(files)} yaml file(s) valid") + + +@check("cmd_succeeds") +def _cmd_succeeds(ws: pathlib.Path, p: dict) -> CheckResult: + """Run `cmd` in the workspace; pass iff exit code == expected (default 0).""" + cmd = p["cmd"] + expected = int(p.get("expect_code", 0)) + timeout = int(p.get("timeout", 120)) + try: + proc = subprocess.run( + cmd, shell=True, cwd=str(ws), capture_output=True, text=True, timeout=timeout + ) + except subprocess.TimeoutExpired: + return CheckResult("cmd_succeeds", False, f"timeout after {timeout}s: {cmd}") + ok = proc.returncode == expected + tail = (proc.stderr or proc.stdout or "").strip().splitlines()[-3:] + return CheckResult("cmd_succeeds", ok, f"exit {proc.returncode} (want {expected}) :: {' / '.join(tail)}") + + +@check("stub_called") +def _stub_called(ws: pathlib.Path, p: dict) -> CheckResult: + """A stubbed binary (e.g. kubectl) was invoked with an argument matching `pattern`. + + The stub-PATH bins append their argv to `.stublog` in the workspace (see + evals/harness/stubs/). This lets trajectory-tier deploy scenarios assert on + what the agent *would* have run, with zero real infra. + """ + log = ws / ".stublog" + if not log.exists(): + return CheckResult("stub_called", False, "no .stublog (no stubbed commands ran)") + text = log.read_text(errors="ignore") + pattern = re.compile(p["pattern"]) + ok = bool(pattern.search(text)) + return CheckResult("stub_called", ok, f"{p['pattern']!r} in stublog" if ok else f"{p['pattern']!r} not invoked") + + +def registered_kinds() -> list[str]: + return sorted(_REGISTRY) diff --git a/evals/harness/evaluate.py b/evals/harness/evaluate.py new file mode 100644 index 0000000..1b5d539 --- /dev/null +++ b/evals/harness/evaluate.py @@ -0,0 +1,184 @@ +"""Evaluate one scenario end-to-end and produce a scored verdict. + +Static tier -> lint the SKILL.md (no agent, no LLM). +Trajectory -> for each arm (treatment/control): sandbox -> run harness -> + deterministic checks + LLM judge. Report per-arm score and the + treatment-minus-control *lift*. +Real tier -> trajectory, plus (best-effort) execute the produced artifact + (`flyte run` on demo.hosted); the real-run outcome is recorded + as an extra check. Executed by the caller/workflow, gated on env. + +The whole module is defensive: a harness crash, missing CLI, or judge/network +error becomes a failed result with detail — never an exception that aborts a run. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field + +from . import checks as checks_mod +from .glm import GLMConfig +from .judge import JudgeResult, judge as run_judge +from .runners import get_runner +from .runners.base import Trajectory +from .sandbox import make_sandbox +from .spec import REPO_ROOT, Scenario +from .static_lint import lint_skill + + +@dataclass +class ArmResult: + arm: str + checks: list[checks_mod.CheckResult] = field(default_factory=list) + judge: JudgeResult | None = None + exit_code: int = 0 + error: str = "" + + @property + def checks_passed(self) -> bool: + return all(c.passed for c in self.checks) if self.checks else True + + @property + def score(self) -> float: + """Combined arm score in [0,1]: deterministic checks gate, judge grades.""" + if not self.checks_passed: + return 0.0 + if self.judge is not None: + return self.judge.score + return 1.0 if self.checks_passed else 0.0 + + @property + def passed(self) -> bool: + judge_ok = self.judge.passed if self.judge is not None else True + return self.checks_passed and judge_ok and not self.error + + +@dataclass +class ScenarioResult: + scenario_id: str + skill: str + tier: str + harness: str | None = None + arms: dict[str, ArmResult] = field(default_factory=dict) + + @property + def lift(self) -> float | None: + t, c = self.arms.get("treatment"), self.arms.get("control") + if t is None or c is None: + return None + return round(t.score - c.score, 4) + + @property + def passed(self) -> bool: + t = self.arms.get("treatment") + return bool(t and t.passed) + + def to_dict(self) -> dict: + return { + "scenario_id": self.scenario_id, + "skill": self.skill, + "tier": self.tier, + "harness": self.harness, + "passed": self.passed, + "lift": self.lift, + "arms": { + arm: { + "score": r.score, + "passed": r.passed, + "checks_passed": r.checks_passed, + "exit_code": r.exit_code, + "error": r.error, + "judge": None if r.judge is None else { + "score": r.judge.score, + "passed": r.judge.passed, + "dimensions": r.judge.dimensions, + "rationale": r.judge.rationale, + }, + "checks": [ + {"kind": c.kind, "passed": c.passed, "detail": c.detail} + for c in r.checks + ], + } + for arm, r in self.arms.items() + }, + } + + +def evaluate_static(scenario: Scenario) -> ScenarioResult: + skill_dir = REPO_ROOT / "plugins" / "flyte" / "skills" / scenario.skill + results = lint_skill(skill_dir) + arm = ArmResult(arm="treatment", checks=results) + return ScenarioResult(scenario.id, scenario.skill, "static", harness=None, + arms={"treatment": arm}) + + +def evaluate_scenario(scenario: Scenario, harness: str, glm: GLMConfig) -> ScenarioResult: + """Run a trajectory/real scenario for one harness across its arms.""" + if scenario.tier == "static": + return evaluate_static(scenario) + + res = ScenarioResult(scenario.id, scenario.skill, scenario.tier, harness=harness) + runner = get_runner(harness) + + for arm in scenario.arms(): + if not runner.is_available(): + res.arms[arm] = ArmResult(arm=arm, error=f"{harness} CLI not available") + continue + try: + with make_sandbox(tier=scenario.tier, glm=glm) as sb: + traj = runner.run(scenario, sb, arm, glm) + arm_res = _score_trajectory(scenario, traj, sb, glm) + # Real tier: actually execute the produced artifact (treatment only). + if scenario.tier == "real" and arm == "treatment" and scenario.real_run: + arm_res.checks.append(_maybe_real_run(scenario, sb)) + except Exception as exc: # never let one arm abort the suite + arm_res = ArmResult(arm=arm, error=f"harness crashed: {exc!r}") + res.arms[arm] = arm_res + + return res + + +def _maybe_real_run(scenario: Scenario, sandbox) -> "checks_mod.CheckResult": + """Execute the produced workflow via `flyte run` on demo.hosted (gated). + + Guarded by FLYTE_EVALS_ENABLE_REAL so trajectory/local testing never submits + a remote run by accident. The Flyte task/CI sets it explicitly. + """ + import os + import subprocess + + spec = scenario.real_run + if not (spec and spec.flyte_run): + return checks_mod.CheckResult("real_run", True, "no real_run requested") + if os.environ.get("FLYTE_EVALS_ENABLE_REAL", "").lower() not in ("1", "true", "yes"): + return checks_mod.CheckResult("real_run", True, "skipped (FLYTE_EVALS_ENABLE_REAL unset)") + + entry = (spec.entrypoint or "").split() + if not entry: + return checks_mod.CheckResult("real_run", False, "real_run.entrypoint not set") + cmd = ["flyte", "--config", str(REPO_ROOT / "evals" / "config" / "flyte.yaml"), "run", *entry] + for k, v in spec.inputs.items(): + cmd += [f"--{k.replace('_', '-')}", str(v)] + try: + proc = subprocess.run(cmd, cwd=str(sandbox.workspace), capture_output=True, + text=True, timeout=1800) + except (subprocess.TimeoutExpired, FileNotFoundError) as e: + return checks_mod.CheckResult("real_run", False, f"flyte run failed to start: {e}") + out = proc.stdout + proc.stderr + ok = proc.returncode == 0 and (spec.expect_status.upper() in out.upper() or proc.returncode == 0) + tail = " / ".join(out.strip().splitlines()[-3:]) + return checks_mod.CheckResult("real_run", ok, f"exit {proc.returncode} :: {tail}") + + +def _score_trajectory(scenario: Scenario, traj: Trajectory, sandbox, glm: GLMConfig) -> ArmResult: + check_results = checks_mod.run_checks(sandbox.workspace, list(scenario.checks)) + judge_result = None + if scenario.judge is not None: + judge_result = run_judge(scenario.judge, traj.as_judge_text(), glm) + return ArmResult( + arm=traj.arm, + checks=check_results, + judge=judge_result, + exit_code=traj.exit_code, + error=traj.error, + ) diff --git a/evals/harness/glm.py b/evals/harness/glm.py new file mode 100644 index 0000000..fa3e46e --- /dev/null +++ b/evals/harness/glm.py @@ -0,0 +1,71 @@ +"""Client config for the union-hosted GLM endpoint (the LLM under test + judge). + +The endpoint is OpenAI-compatible but auth-gated. Everything downstream (runner +adapters + judge) reads its connection details from here so there is exactly one +place to fix once the auth/schema spike is resolved. + +Environment variables (never hardcode creds): + GLM_BASE_URL default: the demo.hosted GLM app, with /v1 appended if missing + GLM_API_KEY bearer token / api key for the endpoint (required for live calls) + GLM_MODEL model name to request (default: glm-5.2) +""" + +from __future__ import annotations + +import os +from dataclasses import dataclass + +DEFAULT_BASE_URL = "https://glm-5-2-llm-service-development.apps.demo.hosted.unionai.cloud/v1" +DEFAULT_MODEL = "glm-5.2" + + +@dataclass(frozen=True) +class GLMConfig: + base_url: str + api_key: str + model: str + + @property + def has_key(self) -> bool: + return bool(self.api_key) + + @staticmethod + def from_env() -> "GLMConfig": + base = os.environ.get("GLM_BASE_URL", DEFAULT_BASE_URL).rstrip("/") + if not base.endswith("/v1"): + base = base + "/v1" + return GLMConfig( + base_url=base, + api_key=os.environ.get("GLM_API_KEY", ""), + model=os.environ.get("GLM_MODEL", DEFAULT_MODEL), + ) + + +def chat_completion(cfg: GLMConfig, messages: list[dict], *, temperature: float = 0.0, + max_tokens: int = 2048, timeout: int = 120) -> str: + """Minimal OpenAI-compatible chat call. Returns the assistant text. + + Kept dependency-light (requests) and isolated so the exact route/auth header + can be adjusted in one place after the endpoint spike. + """ + import requests + + if not cfg.has_key: + raise RuntimeError( + "GLM_API_KEY is not set — cannot call the GLM endpoint. " + "Export a valid token/api key for the demo.hosted GLM app." + ) + resp = requests.post( + f"{cfg.base_url}/chat/completions", + headers={"Authorization": f"Bearer {cfg.api_key}", "Content-Type": "application/json"}, + json={ + "model": cfg.model, + "messages": messages, + "temperature": temperature, + "max_tokens": max_tokens, + }, + timeout=timeout, + ) + resp.raise_for_status() + data = resp.json() + return data["choices"][0]["message"]["content"] diff --git a/evals/harness/judge.py b/evals/harness/judge.py new file mode 100644 index 0000000..10553b5 --- /dev/null +++ b/evals/harness/judge.py @@ -0,0 +1,94 @@ +"""LLM-as-judge over an agent trajectory, via the GLM endpoint. + +The judge is a *secondary* signal: deterministic `checks.py` gate pass/fail; the +judge produces a graded rubric score (0..1 per dimension) used for the lift +metric and for surfacing quality regressions. Judge output is forced to JSON and +parsed defensively so a malformed response degrades to score 0, never a crash. +""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass, field + +from .glm import GLMConfig, chat_completion +from .spec import JudgeSpec + +_JSON_RE = re.compile(r"\{.*\}", re.DOTALL) + +SYSTEM = ( + "You are a strict evaluator of an AI coding agent's work. You are given a " + "rubric and the agent's trajectory (what it said and the files it produced). " + "Score each named dimension from 0.0 to 1.0. Respond ONLY with a JSON object " + 'of the form {"scores": {"": , ...}, "rationale": ""}. ' + "Do not include any prose outside the JSON." +) + + +@dataclass(frozen=True) +class JudgeResult: + score: float # weighted aggregate in [0, 1] + passed: bool # score >= pass_threshold + dimensions: dict[str, float] = field(default_factory=dict) + rationale: str = "" + raw: str = "" + + @staticmethod + def error(msg: str) -> "JudgeResult": + return JudgeResult(score=0.0, passed=False, rationale=f"judge error: {msg}", raw=msg) + + +def build_prompt(rubric: str, dimensions: list[str], trajectory_text: str) -> list[dict]: + dims = ", ".join(dimensions) if dimensions else "correctness, skill_adherence, idiomatic" + user = ( + f"# Rubric\n{rubric}\n\n" + f"# Dimensions to score\n{dims}\n\n" + f"# Agent trajectory\n{trajectory_text}\n" + ) + return [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}] + + +def parse_response(text: str, weights: dict[str, float], threshold: float) -> JudgeResult: + m = _JSON_RE.search(text or "") + if not m: + return JudgeResult.error(f"no JSON in judge response: {text[:200]!r}") + try: + obj = json.loads(m.group(0)) + except json.JSONDecodeError as e: + return JudgeResult.error(f"bad JSON: {e}") + scores = {k: _clamp(v) for k, v in (obj.get("scores") or {}).items()} + agg = _weighted(scores, weights) + return JudgeResult( + score=agg, + passed=agg >= threshold, + dimensions=scores, + rationale=str(obj.get("rationale", "")), + raw=text, + ) + + +def judge(spec: JudgeSpec, trajectory_text: str, cfg: GLMConfig) -> JudgeResult: + dims = list(spec.weights) or ["correctness", "skill_adherence", "idiomatic"] + messages = build_prompt(spec.rubric, dims, trajectory_text) + try: + text = chat_completion(cfg, messages, temperature=0.0, max_tokens=1024) + except Exception as exc: + return JudgeResult.error(repr(exc)) + return parse_response(text, spec.weights, spec.pass_threshold) + + +def _clamp(v: object) -> float: + try: + return max(0.0, min(1.0, float(v))) + except (TypeError, ValueError): + return 0.0 + + +def _weighted(scores: dict[str, float], weights: dict[str, float]) -> float: + if not scores: + return 0.0 + if weights: + total_w = sum(weights.values()) or 1.0 + return sum(scores.get(k, 0.0) * w for k, w in weights.items()) / total_w + return sum(scores.values()) / len(scores) diff --git a/evals/harness/run.py b/evals/harness/run.py new file mode 100644 index 0000000..0fd0a6f --- /dev/null +++ b/evals/harness/run.py @@ -0,0 +1,89 @@ +"""Local CLI: run eval scenarios without Flyte. + +Examples +-------- + # static lint of every skill (no LLM, no agent): + python -m evals.harness.run --tier static + + # one scenario on one harness, end-to-end (needs GLM_API_KEY + the harness CLI): + python -m evals.harness.run --scenario sdk-author-map-task --harness opencode + + # every trajectory scenario for a skill, JSON out: + python -m evals.harness.run --skill flyte-sdk-author --json out.json +""" + +from __future__ import annotations + +import argparse +import json +import sys + +from .evaluate import ScenarioResult, evaluate_scenario, evaluate_static +from .glm import GLMConfig +from .spec import load_scenarios + + +def _select(args) -> list: + scenarios = load_scenarios() + out = [] + for sc in scenarios: + if args.scenario and sc.id != args.scenario: + continue + if args.skill and sc.skill != args.skill: + continue + if args.tier and sc.tier != args.tier: + continue + out.append(sc) + return out + + +def run(args) -> list[ScenarioResult]: + glm = GLMConfig.from_env() + results: list[ScenarioResult] = [] + for sc in _select(args): + if sc.tier == "static": + results.append(evaluate_static(sc)) + continue + harnesses = [args.harness] if args.harness else list(sc.harnesses) + for h in harnesses: + results.append(evaluate_scenario(sc, h, glm)) + return results + + +def _print_table(results: list[ScenarioResult]) -> int: + if not results: + print("no scenarios matched the filter", file=sys.stderr) + return 2 + print(f"{'SCENARIO':<32} {'HARNESS':<9} {'TIER':<11} {'PASS':<5} {'LIFT':>6}") + print("-" * 70) + failures = 0 + for r in results: + lift = "" if r.lift is None else f"{r.lift:+.2f}" + status = "ok" if r.passed else "FAIL" + failures += 0 if r.passed else 1 + print(f"{r.scenario_id:<32} {(r.harness or '-'):<9} {r.tier:<11} {status:<5} {lift:>6}") + print("-" * 70) + print(f"{len(results)} result(s), {failures} failing") + return 1 if failures else 0 + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(prog="flyte-evals", description="Run flyte-agent-plugin evals locally") + ap.add_argument("--scenario", help="scenario id") + ap.add_argument("--skill", help="filter by skill name") + ap.add_argument("--tier", choices=["static", "trajectory", "real"], help="filter by tier") + ap.add_argument("--harness", choices=["opencode", "pi", "hermes"], help="single harness") + ap.add_argument("--json", dest="json_out", help="write full results JSON to this path") + args = ap.parse_args(argv) + + results = run(args) + if args.json_out: + payload = [r.to_dict() for r in results] + with open(args.json_out, "w") as fh: + json.dump(payload, fh, indent=2) + print(f"wrote {args.json_out}", file=sys.stderr) + return _print_table(results) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/harness/runners/__init__.py b/evals/harness/runners/__init__.py new file mode 100644 index 0000000..9e7c1e7 --- /dev/null +++ b/evals/harness/runners/__init__.py @@ -0,0 +1,25 @@ +"""Per-harness runner adapters (opencode, pi, hermes) behind one interface.""" + +from __future__ import annotations + +from .base import Runner, Trajectory +from .opencode import OpenCodeRunner +from .pi import PiRunner +from .hermes import HermesRunner + +_RUNNERS: dict[str, type[Runner]] = { + "opencode": OpenCodeRunner, + "pi": PiRunner, + "hermes": HermesRunner, +} + + +def get_runner(name: str) -> Runner: + try: + return _RUNNERS[name]() + except KeyError: + raise ValueError(f"unknown harness {name!r}; known: {sorted(_RUNNERS)}") + + +def available_harnesses() -> list[str]: + return sorted(_RUNNERS) diff --git a/evals/harness/runners/base.py b/evals/harness/runners/base.py new file mode 100644 index 0000000..91b9c13 --- /dev/null +++ b/evals/harness/runners/base.py @@ -0,0 +1,108 @@ +"""Runner adapter interface + the Trajectory it returns. + +A Runner points a specific agent harness at the GLM endpoint, installs the skill +under test (treatment arm only), runs one non-interactive turn on the scenario +prompt inside the sandbox workspace, and returns a Trajectory: the transcript, +the files the agent produced, and the process outcome. +""" + +from __future__ import annotations + +import abc +import pathlib +import subprocess +from dataclasses import dataclass, field + +from ..glm import GLMConfig +from ..sandbox import Sandbox, install_skill +from ..spec import Scenario + +# Files we never treat as "agent-produced artifacts" when snapshotting a workspace. +_IGNORE = {".stublog", ".opencode", ".pi", ".git"} + + +@dataclass +class Trajectory: + harness: str + arm: str # "treatment" | "control" + transcript: str # what the agent said (stdout / json events) + files: dict[str, str] = field(default_factory=dict) # relpath -> content + exit_code: int = 0 + error: str = "" + + def as_judge_text(self, max_chars: int = 16000) -> str: + parts = [f"## Transcript\n{self.transcript.strip()}"] + for rel, content in sorted(self.files.items()): + parts.append(f"## File: {rel}\n```\n{content}\n```") + text = "\n\n".join(parts) + return text[:max_chars] + + +class Runner(abc.ABC): + name: str = "base" + + @abc.abstractmethod + def is_available(self) -> bool: + """True if the harness CLI is installed and usable.""" + + @abc.abstractmethod + def skills_dir(self, workspace: pathlib.Path) -> pathlib.Path: + """Where this harness discovers skills, relative to the workspace.""" + + @abc.abstractmethod + def _invoke(self, prompt: str, sandbox: Sandbox, glm: GLMConfig) -> tuple[str, int, str]: + """Run one non-interactive turn. Returns (transcript, exit_code, error).""" + + def run(self, scenario: Scenario, sandbox: Sandbox, arm: str, glm: GLMConfig) -> Trajectory: + if arm not in ("treatment", "control"): + raise ValueError(f"bad arm {arm!r}") + if arm == "treatment": + skill_dir = _repo_skill_dir(scenario.skill) + install_skill(skill_dir, self.skills_dir(sandbox.workspace)) + transcript, code, err = self._invoke(scenario.prompt, sandbox, glm) + return Trajectory( + harness=self.name, + arm=arm, + transcript=transcript, + files=snapshot_files(sandbox.workspace), + exit_code=code, + error=err, + ) + + +def snapshot_files(workspace: pathlib.Path, max_bytes: int = 200_000) -> dict[str, str]: + """Capture text files the agent produced in the workspace.""" + out: dict[str, str] = {} + for p in sorted(workspace.rglob("*")): + if not p.is_file(): + continue + rel = p.relative_to(workspace) + if any(part in _IGNORE for part in rel.parts): + continue + try: + if p.stat().st_size > max_bytes: + out[str(rel)] = f"<{p.stat().st_size} bytes, truncated>" + continue + out[str(rel)] = p.read_text(errors="ignore") + except OSError: + continue + return out + + +def _repo_skill_dir(skill: str) -> pathlib.Path: + from ..spec import REPO_ROOT + return REPO_ROOT / "plugins" / "flyte" / "skills" / skill + + +def sh(cmd: list[str], cwd: pathlib.Path, env: dict[str, str], timeout: int = 600 + ) -> tuple[str, int, str]: + """Helper: run a subprocess, capture combined output. Never raises.""" + try: + proc = subprocess.run( + cmd, cwd=str(cwd), env=env, capture_output=True, text=True, timeout=timeout + ) + return (proc.stdout + proc.stderr, proc.returncode, "") + except FileNotFoundError as e: + return ("", 127, f"binary not found: {e}") + except subprocess.TimeoutExpired: + return ("", 124, f"timeout after {timeout}s") diff --git a/evals/harness/runners/hermes.py b/evals/harness/runners/hermes.py new file mode 100644 index 0000000..af9518c --- /dev/null +++ b/evals/harness/runners/hermes.py @@ -0,0 +1,43 @@ +"""hermes adapter. + +hermes installs skills by repo path and runs headless. We install the skill into +a workspace-local dir and run one non-interactive turn pointed at the GLM +endpoint (OpenAI-compatible base URL via env). + +NOTE (spike): hermes was not installed in the dev environment; the exact +headless flag + custom-model config is confirmed in the adapter spike (see plan +Risks). `is_available()` gates it out cleanly until then. +""" + +from __future__ import annotations + +import pathlib +import shutil + +from ..glm import GLMConfig +from ..sandbox import Sandbox +from .base import Runner, sh + + +class HermesRunner(Runner): + name = "hermes" + + def is_available(self) -> bool: + return shutil.which("hermes") is not None + + def skills_dir(self, workspace: pathlib.Path) -> pathlib.Path: + return workspace / ".hermes" / "skills" + + def _invoke(self, prompt: str, sandbox: Sandbox, glm: GLMConfig): + env = dict(sandbox.env) + env["HERMES_SKILLS_DIR"] = str(self.skills_dir(sandbox.workspace)) + env["OPENAI_BASE_URL"] = glm.base_url + env["OPENAI_API_KEY"] = glm.api_key + cmd = [ + "hermes", "run", + "--model", glm.model, + "--non-interactive", + prompt, + ] + out, code, err = sh(cmd, cwd=sandbox.workspace, env=env) + return out, code, err diff --git a/evals/harness/runners/opencode.py b/evals/harness/runners/opencode.py new file mode 100644 index 0000000..5a7d91a --- /dev/null +++ b/evals/harness/runners/opencode.py @@ -0,0 +1,55 @@ +"""opencode adapter. + +Config strategy: + - skills discovered from `/.opencode/skills//` + - a project `opencode.json` registers the GLM endpoint as an OpenAI-compatible + custom provider `glm`, so `--model glm/` routes there + - headless run: `opencode run --model glm/ --format json --auto ""` +""" + +from __future__ import annotations + +import json +import pathlib +import shutil + +from ..glm import GLMConfig +from ..sandbox import Sandbox +from .base import Runner, sh + + +class OpenCodeRunner(Runner): + name = "opencode" + + def is_available(self) -> bool: + return shutil.which("opencode") is not None + + def skills_dir(self, workspace: pathlib.Path) -> pathlib.Path: + return workspace / ".opencode" / "skills" + + def _write_config(self, workspace: pathlib.Path, glm: GLMConfig) -> None: + cfg = { + "$schema": "https://opencode.ai/config.json", + "provider": { + "glm": { + "npm": "@ai-sdk/openai-compatible", + "name": "GLM (union-hosted)", + "options": {"baseURL": glm.base_url, "apiKey": glm.api_key}, + "models": {glm.model: {"name": glm.model}}, + } + }, + } + (workspace / "opencode.json").write_text(json.dumps(cfg, indent=2)) + + def _invoke(self, prompt: str, sandbox: Sandbox, glm: GLMConfig): + self._write_config(sandbox.workspace, glm) + cmd = [ + "opencode", "run", + "--model", f"glm/{glm.model}", + "--format", "json", + "--auto", + "--dir", str(sandbox.workspace), + prompt, + ] + out, code, err = sh(cmd, cwd=sandbox.workspace, env=sandbox.env) + return out, code, err diff --git a/evals/harness/runners/pi.py b/evals/harness/runners/pi.py new file mode 100644 index 0000000..fa4c55e --- /dev/null +++ b/evals/harness/runners/pi.py @@ -0,0 +1,46 @@ +"""pi adapter. + +pi discovers nested SKILL.md folders under its agent skills dir. We install the +skill into a workspace-local skills dir and run pi non-interactively pointed at +the GLM endpoint (OpenAI-compatible base URL via env). + +NOTE (spike): the exact non-interactive flag + custom-model config for pi is +confirmed in the adapter spike (see plan Risks). The invocation below is the +current best-effort; adjust `_invoke`/`skills_dir` once verified — everything +else in the harness is agnostic to it. +""" + +from __future__ import annotations + +import pathlib +import shutil + +from ..glm import GLMConfig +from ..sandbox import Sandbox +from .base import Runner, sh + + +class PiRunner(Runner): + name = "pi" + + def is_available(self) -> bool: + return shutil.which("pi") is not None + + def skills_dir(self, workspace: pathlib.Path) -> pathlib.Path: + # Workspace-local skills dir; PI_SKILLS_DIR points pi at it (set below). + return workspace / ".pi" / "skills" + + def _invoke(self, prompt: str, sandbox: Sandbox, glm: GLMConfig): + env = dict(sandbox.env) + # Point pi's skill discovery + OpenAI-compatible model at our config. + env["PI_SKILLS_DIR"] = str(self.skills_dir(sandbox.workspace)) + env["OPENAI_BASE_URL"] = glm.base_url + env["OPENAI_API_KEY"] = glm.api_key + cmd = [ + "pi", "run", + "--model", glm.model, + "--yes", + prompt, + ] + out, code, err = sh(cmd, cwd=sandbox.workspace, env=env) + return out, code, err diff --git a/evals/harness/sandbox.py b/evals/harness/sandbox.py new file mode 100644 index 0000000..04c7201 --- /dev/null +++ b/evals/harness/sandbox.py @@ -0,0 +1,100 @@ +"""Isolated workspace for one agent run: dir, skill install, stub-PATH, env. + +A `Sandbox` is a throwaway temp directory the agent runs inside. For the +trajectory tier it also lays down *stub* binaries (fake kubectl/helm/docker/...) +earlier on PATH than the real ones, so side-effecting commands the agent issues +log their argv to `/.stublog` and succeed without touching real infra. +The `stub_called` check then asserts on what the agent *would* have run. +""" + +from __future__ import annotations + +import os +import pathlib +import shutil +import stat +import tempfile +from dataclasses import dataclass, field + +from .glm import GLMConfig + +# Side-effecting tools the deploy/SDK skills may invoke. Stubbed in trajectory +# tier; real (unstubbed) in real tier. +STUBBED_TOOLS = [ + "kubectl", "helm", "docker", "kind", "k3d", "aws", "eksctl", + "gcloud", "doctl", "ssh", "scp", "terraform", +] + +_STUB_TEMPLATE = """#!/usr/bin/env bash +# Auto-generated trajectory-tier stub. Logs argv, succeeds, produces no effects. +echo "$(basename "$0") $*" >> "${STUBLOG:-$PWD/.stublog}" +exit 0 +""" + + +@dataclass +class Sandbox: + workspace: pathlib.Path + env: dict[str, str] + stub_bin: pathlib.Path | None = None + _tmp: tempfile.TemporaryDirectory | None = field(default=None, repr=False) + + @property + def stublog(self) -> pathlib.Path: + return self.workspace / ".stublog" + + def cleanup(self) -> None: + if self._tmp is not None: + self._tmp.cleanup() + + def __enter__(self) -> "Sandbox": + return self + + def __exit__(self, *exc) -> None: + self.cleanup() + + +def make_sandbox(*, tier: str, glm: GLMConfig | None = None, + fixtures: pathlib.Path | None = None) -> Sandbox: + tmp = tempfile.TemporaryDirectory(prefix="flyte-eval-") + ws = pathlib.Path(tmp.name) / "workspace" + ws.mkdir(parents=True) + + if fixtures and fixtures.exists(): + shutil.copytree(fixtures, ws, dirs_exist_ok=True) + + env = dict(os.environ) + env["STUBLOG"] = str(ws / ".stublog") + + stub_bin = None + if tier == "trajectory": + stub_bin = _write_stubs(pathlib.Path(tmp.name) / "stubbin") + env["PATH"] = f"{stub_bin}{os.pathsep}{env.get('PATH', '')}" + + if glm is not None: + # Surfaced to runner adapters so each harness can point at the GLM endpoint. + env.setdefault("GLM_BASE_URL", glm.base_url) + env.setdefault("GLM_MODEL", glm.model) + if glm.has_key: + env.setdefault("GLM_API_KEY", glm.api_key) + + return Sandbox(workspace=ws, env=env, stub_bin=stub_bin, _tmp=tmp) + + +def install_skill(skill_dir: pathlib.Path, dest_skills_dir: pathlib.Path) -> pathlib.Path: + """Copy a skill folder into a harness's skills directory. Returns the dest.""" + dest = dest_skills_dir / skill_dir.name + dest.parent.mkdir(parents=True, exist_ok=True) + if dest.exists(): + shutil.rmtree(dest) + shutil.copytree(skill_dir, dest) + return dest + + +def _write_stubs(bin_dir: pathlib.Path) -> pathlib.Path: + bin_dir.mkdir(parents=True, exist_ok=True) + for tool in STUBBED_TOOLS: + p = bin_dir / tool + p.write_text(_STUB_TEMPLATE) + p.chmod(p.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) + return bin_dir diff --git a/evals/harness/spec.py b/evals/harness/spec.py new file mode 100644 index 0000000..e1e7868 --- /dev/null +++ b/evals/harness/spec.py @@ -0,0 +1,174 @@ +"""Scenario and manifest schema + loaders. + +A *scenario* is a declarative YAML spec describing one eval: which skill, which +tier, the user prompt handed to the agent, deterministic checks, an LLM-judge +rubric, and (for tier: real) how to actually execute the produced artifact. + +Everything here is pure data + parsing — no LLM, no network, no subprocess — so +it is trivially unit-testable. +""" + +from __future__ import annotations + +import pathlib +from dataclasses import dataclass, field +from typing import Any + +import yaml + +REPO_ROOT = pathlib.Path(__file__).resolve().parents[2] +EVALS_ROOT = REPO_ROOT / "evals" + +VALID_TIERS = ("static", "trajectory", "real") +VALID_HARNESSES = ("opencode", "pi", "hermes") +VALID_ARMS = ("treatment", "control") + + +@dataclass(frozen=True) +class JudgeSpec: + rubric: str + weights: dict[str, float] = field(default_factory=dict) + pass_threshold: float = 0.7 + + @staticmethod + def from_dict(d: dict[str, Any] | None) -> "JudgeSpec | None": + if not d: + return None + return JudgeSpec( + rubric=d["rubric"], + weights=dict(d.get("weights") or {}), + pass_threshold=float(d.get("pass_threshold", 0.7)), + ) + + +@dataclass(frozen=True) +class RealRunSpec: + flyte_run: bool = False + entrypoint: str | None = None # e.g. "workflow.py main"; None -> autodetect + inputs: dict[str, Any] = field(default_factory=dict) + expect_status: str = "SUCCEEDED" + + @staticmethod + def from_dict(d: dict[str, Any] | None) -> "RealRunSpec | None": + if not d: + return None + return RealRunSpec( + flyte_run=bool(d.get("flyte_run", False)), + entrypoint=d.get("entrypoint"), + inputs=dict(d.get("inputs") or {}), + expect_status=str(d.get("expect_status", "SUCCEEDED")), + ) + + +@dataclass(frozen=True) +class Scenario: + id: str + skill: str + tier: str + prompt: str + harnesses: tuple[str, ...] + control: bool + setup: dict[str, Any] + checks: tuple[dict[str, Any], ...] + judge: JudgeSpec | None + real_run: RealRunSpec | None + source_path: pathlib.Path | None = None + + @staticmethod + def from_dict(d: dict[str, Any], source_path: pathlib.Path | None = None) -> "Scenario": + _require(d, "id", source_path) + _require(d, "skill", source_path) + + tier = d.get("tier", "trajectory") + if tier not in VALID_TIERS: + raise ValueError(f"{_loc(source_path)}: invalid tier {tier!r}, expected one of {VALID_TIERS}") + + # static tier lints the skill file directly — no agent prompt needed. + if tier != "static": + _require(d, "prompt", source_path) + + harnesses = tuple(d.get("harnesses") or VALID_HARNESSES) + bad = [h for h in harnesses if h not in VALID_HARNESSES] + if bad: + raise ValueError(f"{_loc(source_path)}: unknown harness(es) {bad}") + + return Scenario( + id=str(d["id"]), + skill=str(d["skill"]), + tier=tier, + prompt=str(d.get("prompt", "")), + harnesses=harnesses, + control=bool(d.get("control", True)), + setup=dict(d.get("setup") or {}), + checks=tuple(d.get("checks") or ()), + judge=JudgeSpec.from_dict(d.get("judge")), + real_run=RealRunSpec.from_dict(d.get("real_run")), + source_path=source_path, + ) + + def arms(self) -> tuple[str, ...]: + """Arms to run: static tier is skill-agnostic (treatment only).""" + if self.tier == "static" or not self.control: + return ("treatment",) + return ("treatment", "control") + + +@dataclass(frozen=True) +class Manifest: + harnesses: tuple[str, ...] + default_tiers: tuple[str, ...] + scenarios_dir: str + skill_dir_template: str + shared_infra_globs: tuple[str, ...] + kind_smoke_skills: tuple[str, ...] + sdk_real_skills: tuple[str, ...] + + @staticmethod + def load(path: pathlib.Path | None = None) -> "Manifest": + path = path or (EVALS_ROOT / "manifest.yaml") + d = yaml.safe_load(path.read_text()) + return Manifest( + harnesses=tuple(d.get("harnesses") or VALID_HARNESSES), + default_tiers=tuple(d.get("default_tiers") or ("static", "trajectory")), + scenarios_dir=d.get("scenarios_dir", "evals/scenarios"), + skill_dir_template=d.get("skill_dir_template", "plugins/flyte/skills/{skill}"), + shared_infra_globs=tuple(d.get("shared_infra_globs") or ()), + kind_smoke_skills=tuple(d.get("kind_smoke_skills") or ()), + sdk_real_skills=tuple(d.get("sdk_real_skills") or ()), + ) + + def skill_dir(self, skill: str, repo_root: pathlib.Path = REPO_ROOT) -> pathlib.Path: + return repo_root / self.skill_dir_template.format(skill=skill) + + +def load_scenarios(scenarios_dir: pathlib.Path | None = None) -> list[Scenario]: + """Load every scenario spec under scenarios_dir (recursively).""" + scenarios_dir = scenarios_dir or (EVALS_ROOT / "scenarios") + out: list[Scenario] = [] + seen: dict[str, pathlib.Path] = {} + for p in sorted(scenarios_dir.rglob("*.yaml")): + doc = yaml.safe_load(p.read_text()) + if not isinstance(doc, dict): + raise ValueError(f"{p}: scenario file must be a single YAML mapping") + sc = Scenario.from_dict(doc, source_path=p) + if sc.id in seen: + raise ValueError(f"duplicate scenario id {sc.id!r} in {p} and {seen[sc.id]}") + seen[sc.id] = p + out.append(sc) + return out + + +def scenarios_by_skill(scenarios: list[Scenario]) -> dict[str, list[Scenario]]: + by: dict[str, list[Scenario]] = {} + for sc in scenarios: + by.setdefault(sc.skill, []).append(sc) + return by + + +def _require(d: dict[str, Any], key: str, src: pathlib.Path | None) -> None: + if key not in d: + raise ValueError(f"{_loc(src)}: scenario missing required field {key!r}") + + +def _loc(src: pathlib.Path | None) -> str: + return str(src) if src else "" diff --git a/evals/harness/static_lint.py b/evals/harness/static_lint.py new file mode 100644 index 0000000..cbe0bee --- /dev/null +++ b/evals/harness/static_lint.py @@ -0,0 +1,102 @@ +"""Static tier: skill-agnostic lint of a SKILL.md. + +Runs with no LLM and no agent — it validates the skill file itself: + - YAML frontmatter present, with `name` + `description` + - `name` matches the containing directory + - description length is within agent-skill norms (not empty, not enormous) + - fenced code blocks parse (python -> ast, yaml -> safe_load) best-effort + - no obviously-hardcoded secrets or environment-specific hostnames/IDs + +Returns CheckResult list so it slots into the same reporting as other tiers. +""" + +from __future__ import annotations + +import ast +import pathlib +import re + +import yaml + +from .checks import CheckResult + +FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL) +FENCE_RE = re.compile(r"^```([\w-]*)\n(.*?)^```", re.DOTALL | re.MULTILINE) + +# Secret / environment-specific value patterns that must not be committed in a +# skill. Placeholders in are fine and explicitly allowed. +SECRET_PATTERNS = [ + (re.compile(r"AKIA[0-9A-Z]{16}"), "AWS access key id"), + (re.compile(r"aws_secret_access_key\s*=\s*[A-Za-z0-9/+]{40}"), "AWS secret key"), + (re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"), "private key"), + (re.compile(r"eyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{10,}"), "JWT"), + (re.compile(r"xox[baprs]-[0-9A-Za-z-]{10,}"), "Slack token"), + (re.compile(r"ghp_[0-9A-Za-z]{36}"), "GitHub PAT"), +] + + +def lint_skill(skill_dir: pathlib.Path) -> list[CheckResult]: + results: list[CheckResult] = [] + md = skill_dir / "SKILL.md" + if not md.exists(): + return [CheckResult("skill_exists", False, f"no SKILL.md in {skill_dir}")] + results.append(CheckResult("skill_exists", True, str(md))) + + text = md.read_text() + m = FRONTMATTER_RE.match(text) + if not m: + results.append(CheckResult("frontmatter", False, "missing/invalid YAML frontmatter")) + return results + + try: + fm = yaml.safe_load(m.group(1)) or {} + except yaml.YAMLError as e: + results.append(CheckResult("frontmatter", False, f"frontmatter not valid yaml: {e}")) + return results + results.append(CheckResult("frontmatter", True, "parsed")) + + name = fm.get("name") + results.append(CheckResult( + "frontmatter_name", bool(name) and name == skill_dir.name, + f"name={name!r} dir={skill_dir.name!r}", + )) + desc = (fm.get("description") or "").strip() + results.append(CheckResult( + "frontmatter_description", 20 <= len(desc) <= 1500, + f"description length {len(desc)} (want 20..1500)", + )) + + # Code fences parse. + py_bad, yaml_bad, n_py, n_yaml = [], [], 0, 0 + body = text[m.end():] + for lang, code in FENCE_RE.findall(body): + lang = lang.lower() + if lang in ("python", "py"): + n_py += 1 + try: + ast.parse(code) + except SyntaxError as e: + py_bad.append(f"L{e.lineno}: {e.msg}") + elif lang in ("yaml", "yml"): + n_yaml += 1 + # skip fences that are obviously templated (helm/${}) — best-effort + if "{{" in code or "${" in code: + continue + try: + list(yaml.safe_load_all(code)) + except yaml.YAMLError as e: + yaml_bad.append(str(e).splitlines()[0]) + results.append(CheckResult( + "python_fences_parse", not py_bad, f"{n_py} python fence(s); errors: {py_bad or 'none'}", + )) + results.append(CheckResult( + "yaml_fences_parse", not yaml_bad, f"{n_yaml} yaml fence(s); errors: {yaml_bad or 'none'}", + )) + + # No committed secrets. + found = [label for rx, label in SECRET_PATTERNS if rx.search(text)] + results.append(CheckResult( + "no_secrets", not found, f"secret-like values: {found or 'none'}", + )) + + return results diff --git a/evals/kind_smoke/run.sh b/evals/kind_smoke/run.sh new file mode 100755 index 0000000..443ee14 --- /dev/null +++ b/evals/kind_smoke/run.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# Real kind-in-Docker smoke test for the kind deploy skills (deploy-flyte-kind, +# start-dex-local). Runs on a PRIVILEGED GitHub Actions runner — NOT as a Flyte +# task (privileged DinD pods aren't assumed on the demo cluster). +# +# This is a coarse "does the skill's documented path actually stand up a cluster" +# check, independent of the LLM: it runs the canonical commands the skill teaches +# and asserts the flyte-binary pod becomes Ready. The trajectory tier separately +# judges whether the *agent* produces these steps. +# +# Requires: docker, kind, kubectl, helm on PATH. Env: +# PG_CONN external Postgres connection (defaults to an in-cluster throwaway) +# OBJ_* object-store creds (defaults to an in-cluster minio) +set -euo pipefail + +CLUSTER="${KIND_CLUSTER:-flyte-smoke}" +NS="${FLYTE_NS:-flyte}" + +cleanup() { kind delete cluster --name "$CLUSTER" >/dev/null 2>&1 || true; } +trap cleanup EXIT + +echo "==> preflight" +for t in docker kind kubectl helm; do command -v "$t" >/dev/null || { echo "MISSING: $t"; exit 1; }; done + +echo "==> create kind cluster with ingress port mappings (per deploy-flyte-kind Step 1)" +kind create cluster --name "$CLUSTER" --config - <<'EOF' +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: + - role: control-plane + extraPortMappings: + - containerPort: 30080 + hostPort: 80 + protocol: TCP + - containerPort: 30443 + hostPort: 443 + protocol: TCP +EOF + +echo "==> in-cluster minio + postgres (throwaway deps for the smoke test)" +kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f - +helm repo add bitnami https://charts.bitnami.com/bitnami >/dev/null +helm repo update >/dev/null +helm upgrade --install pg bitnami/postgresql -n "$NS" \ + --set auth.postgresPassword=flyte --set auth.database=flyte --wait --timeout 5m +helm upgrade --install minio bitnami/minio -n "$NS" \ + --set auth.rootUser=minio --set auth.rootPassword=miniostorage --wait --timeout 5m + +echo "==> install flyte-binary (per deploy-flyte-kind flyte-binary step)" +helm repo add flyteorg https://flyteorg.github.io/flyte >/dev/null +helm repo update >/dev/null +# NOTE: values wiring (db + storage endpoints) mirrors the skill; kept minimal here. +helm upgrade --install flyte-binary flyteorg/flyte-binary -n "$NS" \ + --set configuration.database.host="pg-postgresql.${NS}.svc.cluster.local" \ + --set configuration.database.password=flyte \ + --set configuration.storage.provider=s3 \ + --wait --timeout 10m || true + +echo "==> assert flyte-binary pod is present" +kubectl get pods -n "$NS" +kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=flyte-binary \ + -n "$NS" --timeout=5m + +echo "SMOKE OK: flyte-binary is Ready on kind cluster '$CLUSTER'" diff --git a/evals/manifest.yaml b/evals/manifest.yaml new file mode 100644 index 0000000..d54708b --- /dev/null +++ b/evals/manifest.yaml @@ -0,0 +1,39 @@ +# Top-level configuration for the flyte-agent-plugin eval harness. +# Scenario -> skill mapping is derived by scanning `scenarios_dir`; each scenario +# file declares its own `skill`. This manifest holds the cross-cutting config. + +# Full harness matrix. A scenario may narrow this via its own `harnesses:` list. +harnesses: [opencode, pi, hermes] + +# Tiers run by default in PR CI. The nightly cron adds `real`. +default_tiers: [static, trajectory] + +# Where scenario specs live (relative to repo root) and how to find a skill dir. +scenarios_dir: evals/scenarios +skill_dir_template: "plugins/flyte/skills/{skill}" + +# Changing any of these paths means the ENGINE changed -> run the whole suite, +# not just the scenarios for a changed skill (see evals/select.py). +shared_infra_globs: + - "evals/harness/**" + - "evals/workflows/**" + - "evals/select.py" + - "evals/report.py" + - "evals/manifest.yaml" + - "evals/pyproject.toml" + - "plugins/flyte/.claude-plugin/**" + - "plugins/flyte/.codex-plugin/**" + +# Deploy skills that get a REAL kind-in-Docker smoke test on a privileged +# GitHub Actions runner (see evals/kind_smoke/run.sh). All other deploy skills +# are trajectory-only (agent output judged, no infra stood up). +kind_smoke_skills: + - deploy-flyte-kind + - start-dex-local + +# SDK skills whose `tier: real` scenarios actually `flyte run` on demo.hosted. +sdk_real_skills: + - flyte-sdk-author + - flyte-sdk-run + - flyte-sdk-data + - flyte-sdk-eval diff --git a/evals/pyproject.toml b/evals/pyproject.toml new file mode 100644 index 0000000..01a09b5 --- /dev/null +++ b/evals/pyproject.toml @@ -0,0 +1,29 @@ +[project] +name = "flyte-agent-plugin-evals" +version = "0.0.1" +description = "Testing & eval harness for the Flyte agent plugin." +requires-python = ">=3.10" +dependencies = [ + "pyyaml>=6.0", + "requests>=2.31", +] + +[project.optional-dependencies] +# Installed on the Flyte task image / CI, not needed for the pure-static path. +flyte = ["flyte>=2.5.0"] +dev = ["pytest>=8.0"] + +[project.scripts] +flyte-evals = "evals.harness.run:main" + +[build-system] +requires = ["setuptools>=68"] +build-backend = "setuptools.build_meta" + +[tool.setuptools.packages.find] +# The package root is this directory (evals/), imported as `evals`. +where = [".."] +include = ["evals*"] + +[tool.pytest.ini_options] +testpaths = ["tests"] diff --git a/evals/report.py b/evals/report.py new file mode 100644 index 0000000..026c590 --- /dev/null +++ b/evals/report.py @@ -0,0 +1,113 @@ +"""Render a scorecard (JSON already exists; this adds a human-readable HTML + +a markdown summary suitable for a PR comment) from evaluate results. + +Input is the list-of-dicts produced by `ScenarioResult.to_dict()` (what +`evals.harness.run --json` writes and what the Flyte aggregate task collects). +""" + +from __future__ import annotations + +import argparse +import html +import json +import pathlib +import sys + + +def _cell(passed: bool) -> str: + return "✅" if passed else "❌" + + +def to_markdown(results: list[dict]) -> str: + total = len(results) + failed = [r for r in results if not r["passed"]] + lines = [ + f"### flyte-agent-plugin evals — {total - len(failed)}/{total} passing", + "", + "| scenario | skill | harness | tier | pass | treat | ctrl | lift |", + "|---|---|---|---|:--:|--:|--:|--:|", + ] + for r in results: + arms = r.get("arms", {}) + t = arms.get("treatment", {}) + c = arms.get("control", {}) + lift = "" if r.get("lift") is None else f"{r['lift']:+.2f}" + lines.append( + f"| {r['scenario_id']} | {r['skill']} | {r.get('harness') or '-'} | {r['tier']} | " + f"{_cell(r['passed'])} | {_fmt(t.get('score'))} | {_fmt(c.get('score'))} | {lift} |" + ) + if failed: + lines += ["", "
Failure detail", ""] + for r in failed: + lines.append(f"- **{r['scenario_id']}** ({r.get('harness') or '-'}):") + for arm, ar in r.get("arms", {}).items(): + for ch in ar.get("checks", []): + if not ch["passed"]: + lines.append(f" - [{arm}] check `{ch['kind']}`: {ch['detail']}") + if ar.get("error"): + lines.append(f" - [{arm}] error: {ar['error']}") + lines.append("
") + return "\n".join(lines) + + +def to_html(results: list[dict]) -> str: + rows = [] + for r in results: + arms = r.get("arms", {}) + t = arms.get("treatment", {}) + c = arms.get("control", {}) + lift = "" if r.get("lift") is None else f"{r['lift']:+.2f}" + rows.append( + "" + f"{html.escape(r['scenario_id'])}" + f"{html.escape(r['skill'])}" + f"{html.escape(r.get('harness') or '-')}" + f"{html.escape(r['tier'])}" + f"{_cell(r['passed'])}" + f"{_fmt(t.get('score'))}" + f"{_fmt(c.get('score'))}" + f"{lift}" + "" + ) + passed = sum(1 for r in results if r["passed"]) + return f""" +flyte-agent-plugin evals + +

flyte-agent-plugin evals — {passed}/{len(results)} passing

+ + + +{''.join(rows)}
scenarioskillharnesstierpasstreatmentcontrollift
+""" + + +def _fmt(v) -> str: + return "" if v is None else f"{v:.2f}" + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(prog="evals.report") + ap.add_argument("results_json", help="path to results JSON (list of ScenarioResult dicts)") + ap.add_argument("--html", help="write HTML scorecard here") + ap.add_argument("--markdown", help="write markdown summary here") + args = ap.parse_args(argv) + + results = json.loads(pathlib.Path(args.results_json).read_text()) + if args.html: + pathlib.Path(args.html).write_text(to_html(results)) + md = to_markdown(results) + if args.markdown: + pathlib.Path(args.markdown).write_text(md) + else: + print(md) + failed = sum(1 for r in results if not r["passed"]) + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/evals/scenarios/deploy-flyte-kind-vm/static.yaml b/evals/scenarios/deploy-flyte-kind-vm/static.yaml new file mode 100644 index 0000000..dbec329 --- /dev/null +++ b/evals/scenarios/deploy-flyte-kind-vm/static.yaml @@ -0,0 +1,5 @@ +id: deploy-flyte-kind-vm-static +skill: deploy-flyte-kind-vm +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/deploy-flyte-kind/kind-config.yaml b/evals/scenarios/deploy-flyte-kind/kind-config.yaml new file mode 100644 index 0000000..bb6ee4c --- /dev/null +++ b/evals/scenarios/deploy-flyte-kind/kind-config.yaml @@ -0,0 +1,38 @@ +id: deploy-kind-config +skill: deploy-flyte-kind +tier: trajectory +control: true +prompt: | + I want to deploy Flyte on a local kind cluster for evaluation. Produce the kind + cluster config and the exact commands to create the cluster with the required + host-port mappings, then show the helm values / commands to install flyte-binary + pointing at an external PostgreSQL and an S3-compatible object store. Write the + kind config to `kind-config.yaml`. You may run the create/apply commands. +setup: + workspace: empty +checks: + # The agent should produce a real kind cluster config. + - kind: file_glob + glob: "**/*.yaml" + - kind: yaml_valid + file_glob: "**/*.yaml" + - kind: contains_regex + file_glob: "**/*.yaml" + pattern: "kind:\\s*Cluster" + - kind: contains_regex + file_glob: "**/*.yaml" + pattern: "extraPortMappings" + # With stubbed kubectl/helm/kind on PATH, assert it actually drove the deploy. + - kind: stub_called + pattern: "kind create cluster" + - kind: stub_called + pattern: "helm .*(install|upgrade)" +judge: + rubric: | + correctness: is the kind config valid (control-plane node with extraPortMappings + for the ingress) and does the helm/flyte-binary install correctly reference an + external Postgres + object store? skill_adherence: did it follow the + deploy-flyte-kind skill's steps (cluster create -> deps -> flyte-binary) rather + than improvising? safety: it did NOT invent AWS EKS / GCP steps for a kind deploy. + weights: {correctness: 0.5, skill_adherence: 0.3, safety: 0.2} + pass_threshold: 0.7 diff --git a/evals/scenarios/deploy-flyte-kind/static.yaml b/evals/scenarios/deploy-flyte-kind/static.yaml new file mode 100644 index 0000000..5050afc --- /dev/null +++ b/evals/scenarios/deploy-flyte-kind/static.yaml @@ -0,0 +1,5 @@ +id: deploy-flyte-kind-static +skill: deploy-flyte-kind +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-deploy-aws/static.yaml b/evals/scenarios/flyte-deploy-aws/static.yaml new file mode 100644 index 0000000..415e407 --- /dev/null +++ b/evals/scenarios/flyte-deploy-aws/static.yaml @@ -0,0 +1,5 @@ +id: flyte-deploy-aws-static +skill: flyte-deploy-aws +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-migrate-config/resources.yaml b/evals/scenarios/flyte-migrate-config/resources.yaml new file mode 100644 index 0000000..18ebd18 --- /dev/null +++ b/evals/scenarios/flyte-migrate-config/resources.yaml @@ -0,0 +1,55 @@ +id: migrate-config-resources +skill: flyte-migrate-config +tier: trajectory +control: true +prompt: | + Migrate this Flyte 1 (flytekit) module to Flyte 2. Move the per-task image, + resources, and caching onto a shared TaskEnvironment. Write the result to + `migrated.py`. Do not run anything. + + ```python + import flytekit + from flytekit import task, workflow, Resources, ImageSpec + + image = ImageSpec(packages=["scikit-learn", "pandas"], python_version="3.11") + + @task(container_image=image, requests=Resources(cpu="2", mem="4Gi"), + cache=True, cache_version="1.0") + def train(n: int) -> float: + return float(n) + + @workflow + def main(n: int) -> float: + return train(n=n) + ``` +setup: + workspace: empty +checks: + - kind: file_glob + glob: "migrated.py" + - kind: python_parses + file_glob: "*.py" + - kind: python_imports + module: flyte + file_glob: "*.py" + - kind: contains_regex + file_glob: "*.py" + pattern: "TaskEnvironment" + - kind: contains_regex + file_glob: "*.py" + pattern: "Image|Resources" + # v1 image/caching constructs must be gone. + - kind: not_contains_regex + file_glob: "*.py" + pattern: "ImageSpec|cache_version" + - kind: not_contains_regex + file_glob: "*.py" + pattern: "import flytekit" +judge: + rubric: | + correctness: image/resources/cache moved onto a flyte.TaskEnvironment (image via + flyte.Image, cache="auto" or a CachePolicy replacing cache_version), train kept as + @env.task. skill_adherence: config declared once on the environment, not per-task. + completeness: no ImageSpec/cache_version/flytekit leftovers. + weights: {correctness: 0.5, skill_adherence: 0.3, completeness: 0.2} + pass_threshold: 0.7 diff --git a/evals/scenarios/flyte-migrate-config/static.yaml b/evals/scenarios/flyte-migrate-config/static.yaml new file mode 100644 index 0000000..85d5cce --- /dev/null +++ b/evals/scenarios/flyte-migrate-config/static.yaml @@ -0,0 +1,5 @@ +id: flyte-migrate-config-static +skill: flyte-migrate-config +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-migrate-control-flow/conditional.yaml b/evals/scenarios/flyte-migrate-control-flow/conditional.yaml new file mode 100644 index 0000000..960cd67 --- /dev/null +++ b/evals/scenarios/flyte-migrate-control-flow/conditional.yaml @@ -0,0 +1,63 @@ +id: migrate-control-flow-conditional +skill: flyte-migrate-control-flow +tier: trajectory +control: true +prompt: | + Migrate this Flyte 1 (flytekit) module that uses `conditional()` branching to + Flyte 2, replacing the DSL with native Python control flow. Write the result to + `migrated.py`. Do not run anything. + + ```python + import flytekit + from flytekit import conditional + + @flytekit.task + def is_positive(x: int) -> bool: + return x > 0 + + @flytekit.task + def double(x: int) -> int: + return x * 2 + + @flytekit.task + def negate(x: int) -> int: + return -x + + @flytekit.workflow + def main(x: int) -> int: + return ( + conditional("check") + .if_(is_positive(x=x).is_true()) + .then(double(x=x)) + .else_() + .then(negate(x=x)) + ) + ``` +setup: + workspace: empty +checks: + - kind: file_glob + glob: "migrated.py" + - kind: python_parses + file_glob: "*.py" + - kind: python_imports + module: flyte + file_glob: "*.py" + - kind: contains_regex + file_glob: "*.py" + pattern: "\\bif\\b" + # v1 branching DSL must be gone. + - kind: not_contains_regex + file_glob: "*.py" + pattern: "conditional\\(" + - kind: not_contains_regex + file_glob: "*.py" + pattern: "import flytekit" +judge: + rubric: | + correctness: does migrated.py express the branch as native Python if/else + (double when positive, negate otherwise) inside an orchestrating @env.task? + skill_adherence: no conditional()/if_()/then() DSL, uses TaskEnvironment + + @env.task. completeness: no leftover flytekit constructs. + weights: {correctness: 0.5, skill_adherence: 0.3, completeness: 0.2} + pass_threshold: 0.7 diff --git a/evals/scenarios/flyte-migrate-control-flow/static.yaml b/evals/scenarios/flyte-migrate-control-flow/static.yaml new file mode 100644 index 0000000..b86788d --- /dev/null +++ b/evals/scenarios/flyte-migrate-control-flow/static.yaml @@ -0,0 +1,5 @@ +id: flyte-migrate-control-flow-static +skill: flyte-migrate-control-flow +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-migrate-data-io/flytefile.yaml b/evals/scenarios/flyte-migrate-data-io/flytefile.yaml new file mode 100644 index 0000000..13d139a --- /dev/null +++ b/evals/scenarios/flyte-migrate-data-io/flytefile.yaml @@ -0,0 +1,57 @@ +id: migrate-data-io-flytefile +skill: flyte-migrate-data-io +tier: trajectory +control: true +prompt: | + Migrate this Flyte 1 (flytekit) module to Flyte 2, converting the FlyteFile and + StructuredDataset I/O to the Flyte 2 equivalents. Write the result to + `migrated.py`. Do not run anything. + + ```python + import flytekit + from flytekit import task, workflow + from flytekit.types.file import FlyteFile + from flytekit.types.structured import StructuredDataset + + @task + def load(path: str) -> FlyteFile: + return FlyteFile(path) + + @task + def to_table(f: FlyteFile) -> StructuredDataset: + import pandas as pd + return StructuredDataset(dataframe=pd.read_csv(f.path)) + + @workflow + def main(path: str) -> StructuredDataset: + return to_table(f=load(path=path)) + ``` +setup: + workspace: empty +checks: + - kind: file_glob + glob: "migrated.py" + - kind: python_parses + file_glob: "*.py" + - kind: python_imports + module: flyte + file_glob: "*.py" + # v2 offloaded types. + - kind: contains_regex + file_glob: "*.py" + pattern: "File|DataFrame" + # v1 types must be gone. + - kind: not_contains_regex + file_glob: "*.py" + pattern: "FlyteFile|StructuredDataset" + - kind: not_contains_regex + file_glob: "*.py" + pattern: "import flytekit" +judge: + rubric: | + correctness: FlyteFile -> flyte.io.File and StructuredDataset -> flyte.io.DataFrame, + with the load -> to_table -> main flow preserved as @env.task functions. + skill_adherence: uses TaskEnvironment + @env.task + flyte.io types. completeness: + no leftover flytekit / FlyteFile / StructuredDataset references. + weights: {correctness: 0.5, skill_adherence: 0.3, completeness: 0.2} + pass_threshold: 0.7 diff --git a/evals/scenarios/flyte-migrate-data-io/static.yaml b/evals/scenarios/flyte-migrate-data-io/static.yaml new file mode 100644 index 0000000..d0f1883 --- /dev/null +++ b/evals/scenarios/flyte-migrate-data-io/static.yaml @@ -0,0 +1,5 @@ +id: flyte-migrate-data-io-static +skill: flyte-migrate-data-io +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-migrate-ml/static.yaml b/evals/scenarios/flyte-migrate-ml/static.yaml new file mode 100644 index 0000000..f81b7a6 --- /dev/null +++ b/evals/scenarios/flyte-migrate-ml/static.yaml @@ -0,0 +1,5 @@ +id: flyte-migrate-ml-static +skill: flyte-migrate-ml +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-migrate-ml/train-gpu.yaml b/evals/scenarios/flyte-migrate-ml/train-gpu.yaml new file mode 100644 index 0000000..1186cca --- /dev/null +++ b/evals/scenarios/flyte-migrate-ml/train-gpu.yaml @@ -0,0 +1,50 @@ +id: migrate-ml-train-gpu +skill: flyte-migrate-ml +tier: trajectory +control: true +prompt: | + Migrate this Flyte 1 (flytekit) GPU training module to Flyte 2. Write the result + to `migrated.py`. Do not run anything. + + ```python + import flytekit + from flytekit import task, workflow, Resources, ImageSpec + + image = ImageSpec(packages=["torch", "torchvision"]) + + @task(container_image=image, requests=Resources(cpu="4", mem="16Gi", gpu="1")) + def train(epochs: int) -> str: + import torch + return f"trained {epochs} epochs on {torch.cuda.device_count()} gpus" + + @workflow + def main(epochs: int) -> str: + return train(epochs=epochs) + ``` +setup: + workspace: empty +checks: + - kind: file_glob + glob: "migrated.py" + - kind: python_parses + file_glob: "*.py" + - kind: python_imports + module: flyte + file_glob: "*.py" + - kind: contains_regex + file_glob: "*.py" + pattern: "TaskEnvironment" + - kind: contains_regex + file_glob: "*.py" + pattern: "gpu|GPU" + - kind: not_contains_regex + file_glob: "*.py" + pattern: "ImageSpec|import flytekit" +judge: + rubric: | + correctness: training task ported to @env.task with GPU resources expressed the + Flyte 2 way (e.g. a "T4:1"-style gpu string / flyte.Resources on the + TaskEnvironment), image via flyte.Image. skill_adherence: image/resources set once + on the environment. completeness: no ImageSpec/flytekit leftovers. + weights: {correctness: 0.5, skill_adherence: 0.3, completeness: 0.2} + pass_threshold: 0.7 diff --git a/evals/scenarios/flyte-migrate-tasks-workflows/hello-world.yaml b/evals/scenarios/flyte-migrate-tasks-workflows/hello-world.yaml new file mode 100644 index 0000000..5dc2236 --- /dev/null +++ b/evals/scenarios/flyte-migrate-tasks-workflows/hello-world.yaml @@ -0,0 +1,55 @@ +id: migrate-tasks-hello-world +skill: flyte-migrate-tasks-workflows +tier: trajectory +control: true +prompt: | + Migrate this Flyte 1 (flytekit) module to Flyte 2. Write the migrated code to + `migrated.py`. Do not run anything. + + ```python + import flytekit + + @flytekit.task + def say_hello(name: str) -> str: + return f"Hello, {name}!" + + @flytekit.task + def to_upper(greeting: str) -> str: + return greeting.upper() + + @flytekit.workflow + def main(name: str) -> str: + greeting = say_hello(name=name) + return to_upper(greeting=greeting) + ``` +setup: + workspace: empty +checks: + - kind: file_glob + glob: "migrated.py" + - kind: python_parses + file_glob: "*.py" + - kind: python_imports + module: flyte + file_glob: "*.py" + - kind: contains_regex + file_glob: "*.py" + pattern: "TaskEnvironment" + - kind: contains_regex + file_glob: "*.py" + pattern: "@\\w+\\.task" + # The whole point of the migration: v1 constructs must be gone. + - kind: not_contains_regex + file_glob: "*.py" + pattern: "import flytekit" + - kind: not_contains_regex + file_glob: "*.py" + pattern: "@\\w*\\.?workflow" +judge: + rubric: | + correctness: is migrated.py a faithful Flyte 2 port — say_hello + to_upper as + @env.task, and main as an orchestrating task that calls them (no @workflow)? + skill_adherence: TaskEnvironment created once, @env.task used, sequential calls + without the `>>` operator. completeness: no leftover flytekit imports/decorators. + weights: {correctness: 0.5, skill_adherence: 0.3, completeness: 0.2} + pass_threshold: 0.7 diff --git a/evals/scenarios/flyte-migrate-tasks-workflows/static.yaml b/evals/scenarios/flyte-migrate-tasks-workflows/static.yaml new file mode 100644 index 0000000..c3702e8 --- /dev/null +++ b/evals/scenarios/flyte-migrate-tasks-workflows/static.yaml @@ -0,0 +1,5 @@ +id: flyte-migrate-tasks-workflows-static +skill: flyte-migrate-tasks-workflows +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-migrate/end-to-end.yaml b/evals/scenarios/flyte-migrate/end-to-end.yaml new file mode 100644 index 0000000..80d6a90 --- /dev/null +++ b/evals/scenarios/flyte-migrate/end-to-end.yaml @@ -0,0 +1,60 @@ +id: migrate-overview-end-to-end +skill: flyte-migrate +tier: trajectory +control: true +prompt: | + Migrate this small Flyte 1 (flytekit) module to Flyte 2. Apply the standard + changes: rename imports, move config to a TaskEnvironment, and turn the workflow + into an orchestrating task. Write the result to `migrated.py`. Do not run anything. + + ```python + import flytekit + from flytekit import task, workflow, Resources, map_task + + @flytekit.task(requests=Resources(cpu="1", mem="2Gi")) + def square(x: int) -> int: + return x * x + + @flytekit.task + def total(xs: list[int]) -> int: + return sum(xs) + + @flytekit.workflow + def main(xs: list[int]) -> int: + squared = map_task(square)(x=xs) + return total(xs=squared) + ``` +setup: + workspace: empty +checks: + - kind: file_glob + glob: "migrated.py" + - kind: python_parses + file_glob: "*.py" + - kind: python_imports + module: flyte + file_glob: "*.py" + - kind: contains_regex + file_glob: "*.py" + pattern: "TaskEnvironment" + - kind: contains_regex + file_glob: "*.py" + pattern: "@\\w+\\.task" + # v1 constructs across several categories must all be gone. + - kind: not_contains_regex + file_glob: "*.py" + pattern: "import flytekit" + - kind: not_contains_regex + file_glob: "*.py" + pattern: "map_task" + - kind: not_contains_regex + file_glob: "*.py" + pattern: "@\\w*\\.?workflow" +judge: + rubric: | + correctness: full port — flytekit->flyte imports, Resources on a TaskEnvironment, + square/total as @env.task, main as an orchestrating task, and map_task replaced by + flyte.map or asyncio.gather. skill_adherence: applies the two mechanical changes + from the migration overview. completeness: no flytekit/map_task/@workflow leftovers. + weights: {correctness: 0.5, skill_adherence: 0.3, completeness: 0.2} + pass_threshold: 0.7 diff --git a/evals/scenarios/flyte-migrate/static.yaml b/evals/scenarios/flyte-migrate/static.yaml new file mode 100644 index 0000000..4acebd2 --- /dev/null +++ b/evals/scenarios/flyte-migrate/static.yaml @@ -0,0 +1,5 @@ +id: flyte-migrate-static +skill: flyte-migrate +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-sdk-agent/static.yaml b/evals/scenarios/flyte-sdk-agent/static.yaml new file mode 100644 index 0000000..d01d75b --- /dev/null +++ b/evals/scenarios/flyte-sdk-agent/static.yaml @@ -0,0 +1,5 @@ +id: flyte-sdk-agent-static +skill: flyte-sdk-agent +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-sdk-app/static.yaml b/evals/scenarios/flyte-sdk-app/static.yaml new file mode 100644 index 0000000..5644784 --- /dev/null +++ b/evals/scenarios/flyte-sdk-app/static.yaml @@ -0,0 +1,5 @@ +id: flyte-sdk-app-static +skill: flyte-sdk-app +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-sdk-author/map-task.yaml b/evals/scenarios/flyte-sdk-author/map-task.yaml new file mode 100644 index 0000000..4a7840e --- /dev/null +++ b/evals/scenarios/flyte-sdk-author/map-task.yaml @@ -0,0 +1,39 @@ +id: sdk-author-map-task +skill: flyte-sdk-author +tier: trajectory +control: true +prompt: | + Using the Flyte 2 SDK, scaffold a single Python file `workflow.py` containing a + workflow that fans out over a list of integers with a map/parallel task, squares + each one, and sums the results. Use flyte.TaskEnvironment and @env.task. Do not + use any Union-only APIs. Just write the file; do not run anything. +setup: + workspace: empty +checks: + - kind: file_glob + glob: "workflow.py" + - kind: python_parses + file_glob: "*.py" + - kind: python_imports + module: flyte + file_glob: "*.py" + - kind: contains_regex + file_glob: "*.py" + pattern: "@\\w+\\.task" + - kind: contains_regex + file_glob: "*.py" + pattern: "TaskEnvironment" + - kind: not_contains_regex + file_glob: "*.py" + pattern: "ReusePolicy" +judge: + rubric: | + Evaluate the agent's Flyte 2 workflow. Score: + - correctness: does it define tasks with TaskEnvironment/@env.task, implement a + real fan-out (flyte.map or async gather) that squares ints and sums them, and + is it runnable Python? + - skill_adherence: did it follow the flyte-sdk-author skill conventions (env + object, task decorators, a main entrypoint) rather than generic/guessed code? + - idiomatic: no Union-only APIs (ReusePolicy etc.), sensible types and I/O. + weights: {correctness: 0.5, skill_adherence: 0.3, idiomatic: 0.2} + pass_threshold: 0.7 diff --git a/evals/scenarios/flyte-sdk-author/real-run.yaml b/evals/scenarios/flyte-sdk-author/real-run.yaml new file mode 100644 index 0000000..a2da3be --- /dev/null +++ b/evals/scenarios/flyte-sdk-author/real-run.yaml @@ -0,0 +1,27 @@ +id: sdk-author-real-run +skill: flyte-sdk-author +tier: real +control: false +harnesses: [opencode] +prompt: | + Scaffold a minimal Flyte 2 workflow file `workflow.py` with a single @env.task + `greet(name: str) -> str` that returns "hello ", plus a `main(name: str)` + workflow that calls it. Use flyte.TaskEnvironment. Do not run it. +setup: + workspace: empty +checks: + - kind: file_glob + glob: "workflow.py" + - kind: python_parses + file_glob: "*.py" +real_run: + flyte_run: true + entrypoint: "workflow.py main" + inputs: {name: "world"} + expect_status: SUCCEEDED +judge: + rubric: | + correctness: is workflow.py a runnable Flyte 2 workflow with a greet task and a + main workflow wired correctly? skill_adherence: uses TaskEnvironment + @env.task. + weights: {correctness: 0.6, skill_adherence: 0.4} + pass_threshold: 0.7 diff --git a/evals/scenarios/flyte-sdk-author/static.yaml b/evals/scenarios/flyte-sdk-author/static.yaml new file mode 100644 index 0000000..3be1702 --- /dev/null +++ b/evals/scenarios/flyte-sdk-author/static.yaml @@ -0,0 +1,5 @@ +id: flyte-sdk-author-static +skill: flyte-sdk-author +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-sdk-data/etl-pipeline.yaml b/evals/scenarios/flyte-sdk-data/etl-pipeline.yaml new file mode 100644 index 0000000..e1bb131 --- /dev/null +++ b/evals/scenarios/flyte-sdk-data/etl-pipeline.yaml @@ -0,0 +1,29 @@ +id: sdk-data-etl-pipeline +skill: flyte-sdk-data +tier: trajectory +control: true +prompt: | + Write a Flyte 2 ETL pipeline in `etl.py`: a task that reads a CSV into a + DataFrame, a task that filters rows and adds a derived column, and a task that + writes Parquet. Wire them into a `main` workflow. Use flyte.TaskEnvironment and + proper Flyte types for the DataFrame/File I/O. Do not run it. +setup: + workspace: empty +checks: + - kind: file_glob + glob: "etl.py" + - kind: python_parses + file_glob: "*.py" + - kind: python_imports + module: flyte + file_glob: "*.py" + - kind: contains_regex + file_glob: "*.py" + pattern: "@\\w+\\.task" +judge: + rubric: | + correctness: three wired tasks (read CSV -> transform -> write Parquet) in a + runnable workflow. skill_adherence: uses Flyte types for DataFrame/File I/O per + the flyte-sdk-data skill, not ad-hoc file paths. idiomatic: sensible typing. + weights: {correctness: 0.5, skill_adherence: 0.3, idiomatic: 0.2} + pass_threshold: 0.7 diff --git a/evals/scenarios/flyte-sdk-data/static.yaml b/evals/scenarios/flyte-sdk-data/static.yaml new file mode 100644 index 0000000..72d3f02 --- /dev/null +++ b/evals/scenarios/flyte-sdk-data/static.yaml @@ -0,0 +1,5 @@ +id: flyte-sdk-data-static +skill: flyte-sdk-data +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-sdk-eval/static.yaml b/evals/scenarios/flyte-sdk-eval/static.yaml new file mode 100644 index 0000000..431746f --- /dev/null +++ b/evals/scenarios/flyte-sdk-eval/static.yaml @@ -0,0 +1,5 @@ +id: flyte-sdk-eval-static +skill: flyte-sdk-eval +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-sdk-ml/static.yaml b/evals/scenarios/flyte-sdk-ml/static.yaml new file mode 100644 index 0000000..3e9f87c --- /dev/null +++ b/evals/scenarios/flyte-sdk-ml/static.yaml @@ -0,0 +1,5 @@ +id: flyte-sdk-ml-static +skill: flyte-sdk-ml +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-sdk-optimize/static.yaml b/evals/scenarios/flyte-sdk-optimize/static.yaml new file mode 100644 index 0000000..b61b207 --- /dev/null +++ b/evals/scenarios/flyte-sdk-optimize/static.yaml @@ -0,0 +1,5 @@ +id: flyte-sdk-optimize-static +skill: flyte-sdk-optimize +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-sdk-run/static.yaml b/evals/scenarios/flyte-sdk-run/static.yaml new file mode 100644 index 0000000..a51ed23 --- /dev/null +++ b/evals/scenarios/flyte-sdk-run/static.yaml @@ -0,0 +1,5 @@ +id: flyte-sdk-run-static +skill: flyte-sdk-run +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-sdk-ship/static.yaml b/evals/scenarios/flyte-sdk-ship/static.yaml new file mode 100644 index 0000000..d41d0c2 --- /dev/null +++ b/evals/scenarios/flyte-sdk-ship/static.yaml @@ -0,0 +1,5 @@ +id: flyte-sdk-ship-static +skill: flyte-sdk-ship +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-sdk-types/static.yaml b/evals/scenarios/flyte-sdk-types/static.yaml new file mode 100644 index 0000000..48d8a62 --- /dev/null +++ b/evals/scenarios/flyte-sdk-types/static.yaml @@ -0,0 +1,5 @@ +id: flyte-sdk-types-static +skill: flyte-sdk-types +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/start-dex-local/static.yaml b/evals/scenarios/start-dex-local/static.yaml new file mode 100644 index 0000000..7c500d2 --- /dev/null +++ b/evals/scenarios/start-dex-local/static.yaml @@ -0,0 +1,5 @@ +id: start-dex-local-static +skill: start-dex-local +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/select.py b/evals/select.py new file mode 100644 index 0000000..0f08e75 --- /dev/null +++ b/evals/select.py @@ -0,0 +1,99 @@ +"""Map changed files -> the subset of scenarios to run. + +Used by CI to run only what a PR affects: + - a changed `plugins/flyte/skills//**` -> that skill's scenarios + - a change to engine/shared-infra paths (manifest.shared_infra_globs) -> run ALL + - flags whether the kind DinD smoke and the real tier are in scope + +Emits JSON on stdout: + {"run_all": bool, "skills": [...], "scenario_ids": [...], + "run_kind": bool, "run_real": bool} + +Usage: + python -m evals.select --base origin/main # diff vs a git ref + python -m evals.select --changed a.md b.py # explicit file list +""" + +from __future__ import annotations + +import argparse +import fnmatch +import json +import pathlib +import re +import subprocess +import sys + +from evals.harness.spec import Manifest, load_scenarios, scenarios_by_skill + +SKILL_PATH_RE = re.compile(r"plugins/flyte/skills/([^/]+)/") + + +def changed_from_git(base: str, repo_root: pathlib.Path) -> list[str]: + out = subprocess.run( + ["git", "diff", "--name-only", f"{base}...HEAD"], + cwd=str(repo_root), capture_output=True, text=True, + ) + if out.returncode != 0: + # fall back to a plain diff (e.g. no merge-base) so CI still selects something + out = subprocess.run( + ["git", "diff", "--name-only", base], + cwd=str(repo_root), capture_output=True, text=True, + ) + return [ln for ln in out.stdout.splitlines() if ln.strip()] + + +def is_shared_infra(path: str, manifest: Manifest) -> bool: + return any(fnmatch.fnmatch(path, g) for g in manifest.shared_infra_globs) + + +def select(changed: list[str], manifest: Manifest, scenarios) -> dict: + by_skill = scenarios_by_skill(scenarios) + run_all = any(is_shared_infra(p, manifest) for p in changed) + + if run_all: + chosen_skills = sorted(by_skill) + else: + chosen_skills = sorted({ + m.group(1) for p in changed if (m := SKILL_PATH_RE.search(p)) + }) + + chosen = [sc for sk in chosen_skills for sc in by_skill.get(sk, [])] + scenario_ids = sorted(sc.id for sc in chosen) + + run_kind = run_all or any(sk in manifest.kind_smoke_skills for sk in chosen_skills) + run_real = run_all or any(sk in manifest.sdk_real_skills for sk in chosen_skills) + + return { + "run_all": run_all, + "skills": chosen_skills, + "scenario_ids": scenario_ids, + "run_kind": run_kind, + "run_real": run_real, + } + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(prog="evals.select") + ap.add_argument("--base", help="git ref to diff against (e.g. origin/main)") + ap.add_argument("--changed", nargs="*", help="explicit changed file list") + args = ap.parse_args(argv) + + repo_root = pathlib.Path(__file__).resolve().parents[1] + manifest = Manifest.load() + scenarios = load_scenarios() + + if args.changed is not None: + changed = args.changed + elif args.base: + changed = changed_from_git(args.base, repo_root) + else: + print("provide --base or --changed ", file=sys.stderr) + return 2 + + print(json.dumps(select(changed, manifest, scenarios), indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/tests/__init__.py b/evals/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/evals/tests/test_checks.py b/evals/tests/test_checks.py new file mode 100644 index 0000000..400a375 --- /dev/null +++ b/evals/tests/test_checks.py @@ -0,0 +1,61 @@ +import pathlib + +from evals.harness import checks + + +def _ws(tmp_path, **files): + for name, content in files.items(): + p = tmp_path / name + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(content) + return tmp_path + + +def test_file_glob(tmp_path): + _ws(tmp_path, **{"workflow.py": "x=1"}) + r = checks.run_checks(tmp_path, [{"kind": "file_glob", "glob": "*.py"}])[0] + assert r.passed + r2 = checks.run_checks(tmp_path, [{"kind": "file_glob", "glob": "*.md"}])[0] + assert not r2.passed + + +def test_python_parses_and_imports(tmp_path): + _ws(tmp_path, **{"a.py": "import flyte\n@x.task\ndef f():\n return 1\n"}) + assert checks.run_checks(tmp_path, [{"kind": "python_parses"}])[0].passed + assert checks.run_checks(tmp_path, [{"kind": "python_imports", "module": "flyte"}])[0].passed + assert not checks.run_checks(tmp_path, [{"kind": "python_imports", "module": "torch"}])[0].passed + + +def test_python_parses_fails_on_syntax_error(tmp_path): + _ws(tmp_path, **{"bad.py": "def (:"}) + assert not checks.run_checks(tmp_path, [{"kind": "python_parses"}])[0].passed + + +def test_contains_and_not_contains(tmp_path): + _ws(tmp_path, **{"m.py": "TaskEnvironment()\n@env.task\ndef g(): ..."}) + assert checks.run_checks(tmp_path, [{"kind": "contains_regex", "pattern": "@\\w+\\.task"}])[0].passed + assert checks.run_checks(tmp_path, [{"kind": "not_contains_regex", "pattern": "ReusePolicy"}])[0].passed + assert not checks.run_checks(tmp_path, [{"kind": "not_contains_regex", "pattern": "TaskEnvironment"}])[0].passed + + +def test_yaml_valid(tmp_path): + _ws(tmp_path, **{"k.yaml": "kind: Cluster\nnodes: []\n"}) + assert checks.run_checks(tmp_path, [{"kind": "yaml_valid", "file_glob": "*.yaml"}])[0].passed + _ws(tmp_path, **{"bad.yaml": "a: b: c: :::"}) + assert not checks.run_checks(tmp_path, [{"kind": "yaml_valid", "file_glob": "bad.yaml"}])[0].passed + + +def test_cmd_succeeds(tmp_path): + assert checks.run_checks(tmp_path, [{"kind": "cmd_succeeds", "cmd": "true"}])[0].passed + assert not checks.run_checks(tmp_path, [{"kind": "cmd_succeeds", "cmd": "false"}])[0].passed + + +def test_stub_called(tmp_path): + (tmp_path / ".stublog").write_text("kind create cluster --name flyte\nhelm install foo\n") + assert checks.run_checks(tmp_path, [{"kind": "stub_called", "pattern": "kind create cluster"}])[0].passed + assert not checks.run_checks(tmp_path, [{"kind": "stub_called", "pattern": "terraform apply"}])[0].passed + + +def test_unknown_kind_is_failure_not_crash(tmp_path): + r = checks.run_checks(tmp_path, [{"kind": "nope"}])[0] + assert not r.passed and "unknown" in r.detail diff --git a/evals/tests/test_evaluate.py b/evals/tests/test_evaluate.py new file mode 100644 index 0000000..8b083ae --- /dev/null +++ b/evals/tests/test_evaluate.py @@ -0,0 +1,72 @@ +"""Integration test of the scoring/arm/lift plumbing with a mocked harness+judge +so it needs neither a real agent CLI nor the GLM endpoint.""" + +from evals.harness import evaluate as ev +from evals.harness.glm import GLMConfig +from evals.harness.judge import JudgeResult +from evals.harness.runners.base import Trajectory +from evals.harness.spec import Scenario + + +class FakeRunner: + """Treatment arm writes a valid workflow.py; control arm writes nothing.""" + + name = "fake" + + def is_available(self): + return True + + def run(self, scenario, sandbox, arm, glm): + if arm == "treatment": + (sandbox.workspace / "workflow.py").write_text( + "import flyte\n@env.task\ndef f():\n return 1\n" + ) + return Trajectory(harness="fake", arm=arm, transcript=f"ran {arm}") + + +def _scenario(): + return Scenario.from_dict({ + "id": "t", "skill": "flyte-sdk-author", "tier": "trajectory", "prompt": "p", + "checks": [ + {"kind": "file_glob", "glob": "*.py"}, + {"kind": "python_imports", "module": "flyte"}, + ], + "judge": {"rubric": "r", "weights": {"correctness": 1.0}, "pass_threshold": 0.7}, + }) + + +def test_treatment_beats_control_positive_lift(monkeypatch): + monkeypatch.setattr(ev, "get_runner", lambda name: FakeRunner()) + # Judge scores treatment high (checks pass) and is not even called for control + # (checks fail -> score 0 short-circuits in ArmResult.score). + monkeypatch.setattr(ev, "run_judge", + lambda spec, text, glm: JudgeResult(score=0.9, passed=True, + dimensions={"correctness": 0.9})) + res = ev.evaluate_scenario(_scenario(), "fake", GLMConfig("u", "k", "m")) + + assert set(res.arms) == {"treatment", "control"} + assert res.arms["treatment"].checks_passed is True + assert res.arms["treatment"].score == 0.9 + assert res.arms["control"].checks_passed is False # no file produced + assert res.arms["control"].score == 0.0 + assert res.lift == 0.9 + assert res.passed is True + + +def test_harness_unavailable_is_recorded_not_crash(monkeypatch): + class Unavailable(FakeRunner): + def is_available(self): + return False + monkeypatch.setattr(ev, "get_runner", lambda name: Unavailable()) + res = ev.evaluate_scenario(_scenario(), "fake", GLMConfig("u", "k", "m")) + assert res.arms["treatment"].error and not res.passed + + +def test_to_dict_serializable(monkeypatch): + monkeypatch.setattr(ev, "get_runner", lambda name: FakeRunner()) + monkeypatch.setattr(ev, "run_judge", + lambda spec, text, glm: JudgeResult(score=0.8, passed=True)) + d = ev.evaluate_scenario(_scenario(), "fake", GLMConfig("u", "k", "m")).to_dict() + import json + json.dumps(d) # must be JSON-serializable for Flyte I/O + report + assert d["lift"] == 0.8 and d["arms"]["treatment"]["passed"] is True diff --git a/evals/tests/test_judge.py b/evals/tests/test_judge.py new file mode 100644 index 0000000..c561d97 --- /dev/null +++ b/evals/tests/test_judge.py @@ -0,0 +1,32 @@ +from evals.harness.judge import parse_response +from evals.harness.spec import JudgeSpec + + +def test_parse_weighted_pass(): + text = '{"scores": {"correctness": 1.0, "idiomatic": 0.5}, "rationale": "good"}' + r = parse_response(text, {"correctness": 0.8, "idiomatic": 0.2}, 0.7) + assert r.passed + assert abs(r.score - (1.0 * 0.8 + 0.5 * 0.2)) < 1e-9 + assert r.rationale == "good" + + +def test_parse_below_threshold(): + text = 'noise {"scores": {"correctness": 0.2}} trailing' + r = parse_response(text, {"correctness": 1.0}, 0.7) + assert not r.passed and r.score == 0.2 + + +def test_parse_clamps_out_of_range(): + r = parse_response('{"scores": {"a": 5, "b": -3}}', {}, 0.7) + assert r.dimensions == {"a": 1.0, "b": 0.0} + + +def test_parse_garbage_is_error_not_crash(): + r = parse_response("the model refused", {}, 0.7) + assert not r.passed and r.score == 0.0 and "judge error" in r.rationale + + +def test_judgespec_from_dict(): + spec = JudgeSpec.from_dict({"rubric": "r", "weights": {"correctness": 1}, "pass_threshold": 0.9}) + assert spec.pass_threshold == 0.9 and spec.weights["correctness"] == 1 + assert JudgeSpec.from_dict(None) is None diff --git a/evals/tests/test_report.py b/evals/tests/test_report.py new file mode 100644 index 0000000..7b49594 --- /dev/null +++ b/evals/tests/test_report.py @@ -0,0 +1,23 @@ +from evals.report import to_html, to_markdown + +RESULTS = [ + {"scenario_id": "a", "skill": "flyte-sdk-author", "tier": "trajectory", + "harness": "opencode", "passed": True, "lift": 0.4, + "arms": {"treatment": {"score": 0.9, "checks": []}, "control": {"score": 0.5, "checks": []}}}, + {"scenario_id": "b", "skill": "deploy-flyte-kind", "tier": "static", + "harness": None, "passed": False, "lift": None, + "arms": {"treatment": {"score": 0.0, "error": "boom", + "checks": [{"kind": "frontmatter", "passed": False, "detail": "bad"}]}}}, +] + + +def test_markdown_has_summary_and_failure_detail(): + md = to_markdown(RESULTS) + assert "1/2 passing" in md + assert "flyte-sdk-author" in md and "+0.40" in md + assert "frontmatter" in md and "boom" in md + + +def test_html_renders(): + html = to_html(RESULTS) + assert "" in html and "flyte-sdk-author" in html and "1/2 passing" in html diff --git a/evals/tests/test_select.py b/evals/tests/test_select.py new file mode 100644 index 0000000..435fd12 --- /dev/null +++ b/evals/tests/test_select.py @@ -0,0 +1,47 @@ +from evals.harness.spec import Manifest, load_scenarios +from evals.select import select, is_shared_infra + + +def _manifest(): + return Manifest.load() + + +def test_single_skill_change_selects_only_that_skill(): + m = _manifest() + scs = load_scenarios() + out = select(["plugins/flyte/skills/flyte-sdk-author/SKILL.md"], m, scs) + assert out["run_all"] is False + assert out["skills"] == ["flyte-sdk-author"] + assert "flyte-sdk-author-static" in out["scenario_ids"] + assert "deploy-flyte-kind-static" not in out["scenario_ids"] + + +def test_engine_change_runs_all(): + m = _manifest() + scs = load_scenarios() + out = select(["evals/harness/checks.py"], m, scs) + assert out["run_all"] is True + assert len(out["skills"]) >= 14 + assert out["run_kind"] is True and out["run_real"] is True + + +def test_kind_skill_sets_run_kind(): + m = _manifest() + scs = load_scenarios() + out = select(["plugins/flyte/skills/deploy-flyte-kind/SKILL.md"], m, scs) + assert out["run_kind"] is True + out2 = select(["plugins/flyte/skills/flyte-sdk-types/SKILL.md"], m, scs) + assert out2["run_kind"] is False + + +def test_unrelated_change_selects_nothing(): + m = _manifest() + scs = load_scenarios() + out = select(["README.md"], m, scs) + assert out["skills"] == [] and out["scenario_ids"] == [] + + +def test_is_shared_infra(): + m = _manifest() + assert is_shared_infra("evals/workflows/eval_wf.py", m) + assert not is_shared_infra("plugins/flyte/skills/flyte-sdk-run/SKILL.md", m) diff --git a/evals/tests/test_spec_and_scenarios.py b/evals/tests/test_spec_and_scenarios.py new file mode 100644 index 0000000..90833cf --- /dev/null +++ b/evals/tests/test_spec_and_scenarios.py @@ -0,0 +1,50 @@ +"""Validate that every committed scenario spec loads and is well-formed, and that +every skill under plugins/ has at least a static scenario.""" + +import pathlib + +import pytest + +from evals.harness.spec import REPO_ROOT, Scenario, load_scenarios, scenarios_by_skill + +SKILLS_DIR = REPO_ROOT / "plugins" / "flyte" / "skills" + + +def test_all_scenarios_load(): + scenarios = load_scenarios() + assert scenarios, "no scenarios found" + + +def test_every_scenario_references_a_real_skill(): + valid = {p.name for p in SKILLS_DIR.iterdir() if (p / "SKILL.md").exists()} + for sc in load_scenarios(): + assert sc.skill in valid, f"{sc.id} references unknown skill {sc.skill}" + + +def test_every_skill_has_a_static_scenario(): + by_skill = scenarios_by_skill(load_scenarios()) + for p in SKILLS_DIR.iterdir(): + if (p / "SKILL.md").exists(): + tiers = {sc.tier for sc in by_skill.get(p.name, [])} + assert "static" in tiers, f"skill {p.name} has no static scenario" + + +def test_arms_logic(): + static = Scenario.from_dict({"id": "x", "skill": "flyte-sdk-run", "tier": "static"}) + assert static.arms() == ("treatment",) + traj = Scenario.from_dict({"id": "y", "skill": "flyte-sdk-run", "tier": "trajectory", + "prompt": "p"}) + assert set(traj.arms()) == {"treatment", "control"} + no_ctrl = Scenario.from_dict({"id": "z", "skill": "flyte-sdk-run", "tier": "trajectory", + "prompt": "p", "control": False}) + assert no_ctrl.arms() == ("treatment",) + + +def test_invalid_tier_rejected(): + with pytest.raises(ValueError): + Scenario.from_dict({"id": "bad", "skill": "flyte-sdk-run", "tier": "nope", "prompt": "p"}) + + +def test_trajectory_requires_prompt(): + with pytest.raises(ValueError): + Scenario.from_dict({"id": "bad", "skill": "flyte-sdk-run", "tier": "trajectory"}) diff --git a/evals/workflows/__init__.py b/evals/workflows/__init__.py new file mode 100644 index 0000000..0ac76c4 --- /dev/null +++ b/evals/workflows/__init__.py @@ -0,0 +1 @@ +"""Flyte orchestration of the eval harness (runs on demo.hosted.unionai.cloud).""" diff --git a/evals/workflows/eval_wf.py b/evals/workflows/eval_wf.py new file mode 100644 index 0000000..223097c --- /dev/null +++ b/evals/workflows/eval_wf.py @@ -0,0 +1,108 @@ +"""Flyte orchestration of the flyte-agent-plugin eval harness. + +Runs on demo.hosted.unionai.cloud (org demo / project flytesnacks / domain +development, remote image builder — see evals/config/flyte.yaml). The workflow +expands the scenario matrix, fans out one action per (scenario x harness) via +`flyte.map`, then aggregates verdicts into a scorecard. + +Run it: + flyte --config evals/config/flyte.yaml run evals/workflows/eval_wf.py main \\ + --skills '["flyte-sdk-author"]' --tiers '["static","trajectory"]' + +The GLM endpoint key is mounted from a Flyte secret `glm-api-key` as GLM_API_KEY. +""" + +from __future__ import annotations + +import json + +import flyte + +from evals.workflows.images import eval_image + +env = flyte.TaskEnvironment( + name="flyte-agent-plugin-evals", + image=eval_image, + resources=flyte.Resources(cpu="2", memory="4Gi"), + secrets=[flyte.Secret(key="glm-api-key", as_env_var="GLM_API_KEY")], + env_vars={ + "GLM_BASE_URL": "https://glm-5-2-llm-service-development.apps.demo.hosted.unionai.cloud/v1", + "GLM_MODEL": "glm-5.2", + "PYTHONPATH": "/root", + }, +) + + +@env.task +def eval_unit(unit: dict) -> dict: + """Evaluate one (scenario_id, harness) unit and return its verdict dict.""" + from evals.harness.evaluate import evaluate_scenario, evaluate_static + from evals.harness.glm import GLMConfig + from evals.harness.spec import load_scenarios + + scenarios = {s.id: s for s in load_scenarios()} + sc = scenarios[unit["scenario_id"]] + glm = GLMConfig.from_env() + if sc.tier == "static": + return evaluate_static(sc).to_dict() + return evaluate_scenario(sc, unit["harness"], glm).to_dict() + + +@env.task(report=True) +def aggregate(results: list[dict]) -> dict: + """Collect verdicts into a scorecard summary (also emits an HTML report).""" + import flyte.report + + from evals.report import to_html, to_markdown + + passed = sum(1 for r in results if r["passed"]) + summary = { + "total": len(results), + "passed": passed, + "failed": len(results) - passed, + "markdown": to_markdown(results), + "results": results, + } + # Attach the HTML scorecard to the Flyte run's report tab. + try: + flyte.report.replace(to_html(results), do_flush=True) + except Exception: + pass + return summary + + +@env.task +def main(skills: list[str] | None = None, + harnesses: list[str] | None = None, + tiers: list[str] | None = None) -> dict: + """Top-level workflow: build the matrix, fan out, aggregate.""" + from evals.harness.spec import load_scenarios + + tiers = tiers or ["static", "trajectory"] + scenarios = load_scenarios() + + units: list[dict] = [] + for sc in scenarios: + if sc.tier not in tiers: + continue + if skills and sc.skill not in skills: + continue + if sc.tier == "static": + units.append({"scenario_id": sc.id, "harness": None}) + continue + for h in (harnesses or list(sc.harnesses)): + units.append({"scenario_id": sc.id, "harness": h}) + + if not units: + return {"total": 0, "passed": 0, "failed": 0, "results": [], "markdown": "no units selected"} + + results = [r for r in flyte.map(eval_unit, units) if isinstance(r, dict)] + return aggregate(results) + + +if __name__ == "__main__": + # Local driver: `python -m evals.workflows.eval_wf` + flyte.init_from_config("evals/config/flyte.yaml") + run = flyte.run(main, skills=None, harnesses=None, tiers=["static"]) + print(run.url) + print(json.dumps(run.outputs(), indent=2, default=str)) diff --git a/evals/workflows/images.py b/evals/workflows/images.py new file mode 100644 index 0000000..0013236 --- /dev/null +++ b/evals/workflows/images.py @@ -0,0 +1,31 @@ +"""flyte.Image specs for the eval tasks. + +The eval task image needs: python + flyte + the harness CLIs (opencode, pi via +npm; hermes via its installer) + the harness package itself + judge deps. The +harness CLIs are node-based, so we install node and the npm globals in the image. + +Only the SDK/trajectory tiers run on Flyte. The kind DinD smoke runs on a +privileged GitHub Actions runner, not here (privileged pods aren't assumed on +the demo cluster) — so no Docker-in-Docker layer is baked in. +""" + +from __future__ import annotations + +import flyte + +# Base: flyte + eval harness python deps. +eval_image = ( + flyte.Image.from_debian_base() + .with_apt_packages("git", "curl", "ca-certificates") + # Node.js for the node-based agent harnesses (opencode, pi). + .with_commands([ + "curl -fsSL https://deb.nodesource.com/setup_22.x | bash -", + "apt-get install -y nodejs", + "npm install -g opencode-ai @mieszko/pi || true", + ]) + .with_pip_packages("pyyaml", "requests", "flyte>=2.5.0") + # Ship the eval package + the skills so tasks can install/lint them. + .with_source_folder("evals", "/root/evals") + .with_source_folder("plugins", "/root/plugins") + .with_env_vars({"PYTHONPATH": "/root"}) +) diff --git a/plugins/flyte/README.md b/plugins/flyte/README.md index ddab8a4..850a945 100644 --- a/plugins/flyte/README.md +++ b/plugins/flyte/README.md @@ -40,6 +40,24 @@ two bundled MCP servers. - **`flyte-sdk-ml`** — ML workload patterns (training, HPO, experiment tracking, evaluation, batch/real-time inference, monitoring). +### Migration (Flyte 1 → 2) + +Convert existing Flyte 1 (`flytekit`) code to Flyte 2, distilled from the official +[migration guide](https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/). + +- **`flyte-migrate`** — start-here orchestrator: the `flytekit`→`flyte` shift, concept + mapping, the two mechanical changes, incremental strategy, hybrid v1/v2 pipelines, gotchas. +- **`flyte-migrate-tasks-workflows`** — `@task`/`@workflow`/`@dynamic` → a single `@env.task` + on a `TaskEnvironment`; ordering without `>>`; subworkflows as tasks. +- **`flyte-migrate-config`** — images (`ImageSpec`→`flyte.Image`), resources/GPUs, caching, + secrets, scheduling (`LaunchPlan`/`CronSchedule`→`Trigger`/`Cron`), and `pyflyte`→`flyte`. +- **`flyte-migrate-control-flow`** — `conditional()`→`if`/`else`, `@dynamic`→plain loops, + `on_failure`→`try`/`except`, `map_task`→`flyte.map`/`asyncio.gather`. +- **`flyte-migrate-data-io`** — `FlyteFile`/`FlyteDirectory`→`flyte.io.File`/`Dir`, + `StructuredDataset`→`flyte.io.DataFrame`, dataclasses/Pydantic I/O. +- **`flyte-migrate-ml`** — training, HPO, GPU/deep learning, batch inference, and the + new-in-v2 patterns (serving, apps, sandboxed execution). + ## Bundled MCP servers **Claude Code only.** The servers live in `.mcp.json`, which Claude Code reads by diff --git a/plugins/flyte/skills/flyte-migrate-config/SKILL.md b/plugins/flyte/skills/flyte-migrate-config/SKILL.md new file mode 100644 index 0000000..61cf157 --- /dev/null +++ b/plugins/flyte/skills/flyte-migrate-config/SKILL.md @@ -0,0 +1,452 @@ +--- +name: flyte-migrate-config +description: Migrates Flyte 1 task configuration, container images, resources, caching, secrets, scheduling, and CLI/config-file usage to Flyte 2 equivalents. Use when migrating Flyte 1 task configuration, images, resources, secrets, scheduling, or CLI/config-file usage to Flyte 2. Trigger words include resources, ImageSpec, cache_version, secrets, LaunchPlan, CronSchedule, pyflyte, config, register, and deploy. +--- + +# Flyte 1 to Flyte 2 Migration: Task Configuration and CLI/Config + +In Flyte 1, image, resources, caching, secrets, and scheduling were configured per-task on the `@task` decorator or per-workflow on a `LaunchPlan`. In Flyte 2 most of this moves to the `flyte.TaskEnvironment`, so it is declared once and shared. The CLI is renamed from `pyflyte` to `flyte` and the config file is trimmed down. This skill covers migrating those settings and commands. + +## Grounding References + +| Resource | URL | +|---|---| +| Migration guide (Task configuration) | https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/configuration/ | +| Migration guide (CLI) | https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/cli-and-configuration/ | +| Official docs | https://www.union.ai/docs/v2/flyte | +| Docs index (LLMs) | https://www.union.ai/docs/v2/flyte/llms.txt | +| SDK API reference | https://www.union.ai/docs/v2/union/api-reference/flyte-sdk/ | +| CLI API reference | https://www.union.ai/docs/v2/union/api-reference/flyte-cli/ | +| Example code | https://github.com/unionai/unionai-examples | + +## Image, resources, and caching move to the TaskEnvironment + +Image, resources, and caching move from the `@task` decorator to the `TaskEnvironment`. Per-task settings like `retries` and `timeout` stay on `@env.task`. Note that `mem` is renamed to `memory`, and there are no separate `requests`/`limits` — a single `Resources` value serves as both. + +### Flyte 1 + +```python +from datetime import timedelta + +import flytekit +from flytekit import Resources + +image = flytekit.ImageSpec( + name="training-image", + packages=["scikit-learn", "pandas"], +) + +@flytekit.task( + container_image=image, + requests=Resources(cpu="2", mem="4Gi"), + limits=Resources(cpu="4", mem="8Gi"), + cache=True, + cache_version="1.0", + retries=3, + timeout=timedelta(minutes=30), +) +def train_epoch(step: int) -> float: + # A stand-in for a training step that returns the current loss. + return 1.0 / (step + 1) + +@flytekit.workflow +def main(step: int) -> float: + return train_epoch(step=step) +``` + +### Flyte 2 + +```python +from datetime import timedelta + +import flyte + +# Image, resources, and caching move to the TaskEnvironment, so they are declared +# once and shared by every task in the environment. +env = flyte.TaskEnvironment( + name="training", + image=flyte.Image.from_debian_base().with_pip_packages("scikit-learn", "pandas"), + resources=flyte.Resources(cpu="2", memory="4Gi"), # "memory", not "mem" + cache="auto", +) + +# retries and timeout stay on the task decorator. +@env.task(retries=3, timeout=timedelta(minutes=30)) +def train_epoch(step: int) -> float: + # A stand-in for a training step that returns the current loss. + return 1.0 / (step + 1) + +@env.task +def main(step: int) -> float: + return train_epoch(step) +``` + +## Container images: ImageSpec to flyte.Image + +Flyte 1's `ImageSpec` is replaced by Flyte 2's `flyte.Image` with a fluent builder API. Instead of one constructor with many arguments, you start from a base and chain builder methods. + +```python +from flyte import Image + +image = ( + Image.from_debian_base(name="my-image", registry="ghcr.io/myorg", python_version=(3, 11)) + .with_pip_packages("pandas", "numpy") + .with_apt_packages("curl", "git") + .with_env_vars({"MY_VAR": "value"}) +) +``` + +| Constructor | Use case | +|---|---| +| `Image.from_debian_base()` | Most common; includes the Flyte SDK | +| `Image.from_base(image_uri)` | Start from any existing image | +| `Image.from_dockerfile(path)` | Complex custom builds | +| `Image.from_uv_script(path)` | UV-based projects | + +Common chainable builder methods: `.with_pip_packages(...)`, `.with_requirements(path)`, `.with_uv_project(path)`, `.with_apt_packages(...)`, `.with_commands([...])`, `.with_source_file(path, dst=...)`, `.with_source_folder(path, dst=...)`, `.with_env_vars({...})`, and `.with_workdir(...)`. + +| Flyte 1 `ImageSpec` | Flyte 2 `Image` | Notes | +|---|---|---| +| `name` | `name` (constructor) | Same | +| `registry` | `registry` (constructor) | Same | +| `python_version` | `python_version` (tuple) | `"3.11"` becomes `(3, 11)` | +| `packages` | `.with_pip_packages()` | Method instead of param | +| `apt_packages` | `.with_apt_packages()` | Method instead of param | +| `requirements` | `.with_requirements()` | Supports txt, poetry.lock, uv.lock | +| `env` | `.with_env_vars()` | Method instead of param | +| `commands` | `.with_commands()` | Method instead of param | +| `copy` / `source_root` | `.with_source_file()` / `.with_source_folder()` | More explicit methods | +| `base_image` | `Image.from_base()` | Different constructor | +| `builder` | Config file or `flyte.init()` | Global setting | +| `platform` | `platform` (constructor) | Tuple: `("linux/amd64", "linux/arm64")` | + +For a private registry, create an image-pull secret and reference it: + +```bash +flyte create secret --type image_pull my-registry-secret --from-file ~/.docker/config.json +``` + +```python +image = Image.from_debian_base( + registry="private.registry.com", + name="my-image", + registry_secret="my-registry-secret", +) +``` + +## Resources and GPUs + +A single `flyte.Resources` value serves as both request and limit — there are no separate `requests`/`limits`. Several parameters were renamed. + +| Flyte 1 | Flyte 2 | Notes | +|---|---|---| +| `cpu="1"` | `cpu="1"` | Same | +| `mem="2Gi"` | `memory="2Gi"` | Renamed | +| `gpu="1"` | `gpu="A100:1"` | `Type:count` format | +| `ephemeral_storage="10Gi"` | `disk="10Gi"` | Renamed | +| N/A | `shm="auto"` | New: shared memory | + +GPU type and count are combined into one string, replacing the separate Flyte 1 `accelerator=` argument: + +```python +env = flyte.TaskEnvironment( + name="gpu_env", + resources=flyte.Resources( + cpu="4", + memory="32Gi", + gpu="A100:2", # Type:count + # gpu="A100 80G:1" # 80GB variant + # gpu=flyte.GPU("A100", count=1, partition="1g.5gb") # MIG partition + ), +) +``` + +Supported GPU types include A10, A10G, A100, A100 80G, B200, H100, H200, L4, L40s, T4, V100, RTX PRO 6000, and GB10. + +## Caching: cache_version to cache="auto" / CachePolicy + +Caching is enabled at the env level with `cache="auto"` (or per-task on `@env.task`). The explicit `cache_version` string moves into a `flyte.Cache` object. + +| Behavior | Description | +|---|---| +| `"auto"` | Cache results and reuse if available | +| `"override"` | Always execute and overwrite the cache | +| `"disable"` | No caching (default for a `TaskEnvironment`) | + +```python +# Flyte 1: @task(cache=True, cache_version="1.0") +# Flyte 2: +@env.task(cache="auto") +def cached_task(x: int) -> int: + return x * 2 + +# Advanced control (replaces cache_version, serialize, ignored_inputs, ...) +@env.task(cache=flyte.Cache( + behavior="auto", + version_override="v1.0", + serialize=True, + ignored_inputs=("debug",), +)) +def advanced(x: int, debug: bool = False) -> int: + return x * 2 +``` + +## Secrets: current_context().secrets to env vars + +Secrets move from `secret_requests` on the task to `secrets` on the `TaskEnvironment`, and you read them from environment variables instead of `current_context().secrets` — for example, an API key for a model registry or hosted LLM. + +### Flyte 1 + +```python +from flytekit import task, workflow, Secret, current_context + +@task(secret_requests=[Secret(group="openai", key="api_key")]) +def call_api() -> str: + token = current_context().secrets.get(group="openai", key="api_key") + return f"token has {len(token)} chars" + +@workflow +def main() -> str: + return call_api() +``` + +### Flyte 2 + +```python +import os + +import flyte + +# Secrets are declared on the TaskEnvironment and injected as environment +# variables (instead of read through current_context().secrets). +env = flyte.TaskEnvironment( + name="secrets", + secrets=[flyte.Secret(key="openai_api_key", as_env_var="OPENAI_API_KEY")], +) + +@env.task +def call_api() -> str: + token = os.getenv("OPENAI_API_KEY", "") + return f"token has {len(token)} chars" + +@env.task +def main() -> str: + return call_api() +``` + +A `flyte.Secret` can be mounted as an environment variable or as a file, and the access convention changes: + +```python +flyte.Secret(key="openai-key", as_env_var="OPENAI_API_KEY") # mount as env var +flyte.Secret(key="access-key", group="aws") # env var: AWS_ACCESS_KEY +flyte.Secret(key="ssl-cert", mount="/etc/flyte/secrets") # mount as a file +``` + +| Flyte 1 pattern | Flyte 2 pattern | +|---|---| +| `ctx.secrets.get(key="mykey", group="mygroup")` | `os.environ["MYGROUP_MYKEY"]` (auto-named) | +| `ctx.secrets.get(key="mykey")` | `os.environ["MY_SECRET"]` (with `as_env_var="MY_SECRET"`) | + +Create and manage secrets from the CLI: + +```bash +flyte create secret MY_SECRET_KEY --value my_secret_value +flyte create secret MY_SECRET_KEY --from-file /path/to/secret +flyte get secret +flyte delete secret MY_SECRET_KEY +``` + +## Scheduling: LaunchPlan + CronSchedule to flyte.Trigger + flyte.Cron + +A `LaunchPlan` with a `CronSchedule` (say, a nightly retraining job) becomes a `flyte.Trigger` attached directly to the task. Use `flyte.TriggerTime` to bind the scheduled fire time to an input, and deploy the trigger with `flyte deploy`. + +### Flyte 1 + +```python +from flytekit import task, workflow, LaunchPlan, CronSchedule + +@task +def retrain(kickoff_time: str) -> str: + return f"retrained model at {kickoff_time}" + +@workflow +def main(kickoff_time: str) -> str: + return retrain(kickoff_time=kickoff_time) + +# A LaunchPlan attaches a schedule (and default inputs) to a workflow. +nightly_retrain = LaunchPlan.get_or_create( + workflow=main, + name="nightly_retrain", + schedule=CronSchedule( + schedule="0 2 * * *", # 2 AM daily + kickoff_time_input_arg="kickoff_time", + ), +) +``` + +### Flyte 2 + +```python +from datetime import datetime + +import flyte + +env = flyte.TaskEnvironment(name="scheduling") + +# A Trigger replaces LaunchPlan + CronSchedule. It is attached directly to the +# task and deployed with it (flyte deploy). flyte.TriggerTime binds the +# scheduled fire time to a task input. +nightly_retrain = flyte.Trigger( + name="nightly_retrain", + automation=flyte.Cron("0 2 * * *"), # 2 AM daily + inputs={"trigger_time": flyte.TriggerTime}, + auto_activate=True, +) + +@env.task(triggers=nightly_retrain) +def main(trigger_time: datetime = datetime(2024, 1, 1, 2, 0)) -> str: + return f"retrained model at {trigger_time.isoformat()}" +``` + +Triggers support `flyte.Cron("0 9 * * *", timezone="America/New_York")` and `flyte.FixedRate(timedelta(hours=1))` as automations, plus convenience constructors like `flyte.Trigger.hourly()` and `flyte.Trigger.daily()`. + +## CLI command mapping: pyflyte to flyte + +The command-line tool is renamed from `pyflyte` to `flyte`, and remote is now the default. + +| Flyte 1 | Flyte 2 | Notes | +|---|---|---| +| `pyflyte run` | `flyte run` | Similar, different flags | +| `pyflyte run --remote` | `flyte run` | Remote is the default in Flyte 2 | +| `pyflyte run` (local) | `flyte run --local` | Local execution is now explicit | +| `pyflyte register` | `flyte deploy` | Different concept | +| `pyflyte package` | N/A | Not needed in Flyte 2 | +| `pyflyte serialize` | N/A | Not needed in Flyte 2 | + +### Running tasks — Flyte 1 + +```bash +# Local +pyflyte run my_module.py my_workflow --arg1 value1 + +# Remote +pyflyte --config config.yaml run --remote my_module.py my_workflow --arg1 value1 +``` + +### Running tasks — Flyte 2 + +```bash +# Remote (default) +flyte run my_module.py my_task --arg1 value1 + +# Local +flyte run --local my_module.py my_task --arg1 value1 + +# With an explicit config file +flyte --config config.yaml run my_module.py my_task --arg1 value1 +``` + +### Deploying (register to deploy) + +In Flyte 1 you registered a module; in Flyte 2 you deploy task environments. + +#### Flyte 1 + +```bash +pyflyte register my_module.py -p my-project -d development +``` + +#### Flyte 2 + +```bash +# Deploy a task environment +flyte deploy my_module.py my_env --project my-project --domain development + +# Deploy all environments in a file +flyte deploy --all my_module.py + +# Deploy with an explicit version, or recursively +flyte deploy --version v1.0.0 my_module.py my_env +flyte deploy --recursive --all ./src +``` + +### Key flag differences + +| Flyte 1 flag | Flyte 2 flag | Notes | +|---|---|---| +| `--remote` | (default) | Remote is the default | +| `--copy-all` | `--copy-style all` | File copying | +| N/A | `--copy-style loaded_modules` | Default: only imported modules | +| `-p, --project` | `--project` | Same | +| `-d, --domain` | `--domain` | Same | +| `-i, --image` | `--image` | Same format | +| N/A | `--follow, -f` | Follow execution logs | + +## Configuration files + +The config file lives in the same place (`~/.flyte/config.yaml`), but the environment variable changes from `FLYTECTL_CONFIG` to `FLYTE_CONFIG`, and the format is simpler. + +### Flyte 1 + +```yaml +admin: + endpoint: dns:///your-cluster.hosted.unionai.cloud + insecure: false + authType: Pkce +``` + +### Flyte 2 + +```yaml +admin: + endpoint: dns:///your-cluster.hosted.unionai.cloud + +image: + builder: remote # or "local" + +task: + domain: development + org: your-org + project: your-project +``` + +| Setting | Flyte 1 | Flyte 2 | +|---|---|---| +| Endpoint | `admin.endpoint` | `admin.endpoint` | +| Auth type | `admin.authType` | Auto-detected (PKCE default) | +| Project | CLI flag `-p` | `task.project` (default) | +| Domain | CLI flag `-d` | `task.domain` (default) | +| Organization | CLI flag `--org` | `task.org` (default) | +| Image builder | N/A | `image.builder` (`local` or `remote`) | + +### Configuring in code + +```python +import flyte + +# From a config file (auto-discovers, or pass a path) +flyte.init_from_config() +flyte.init_from_config("path/to/config.yaml") + +# Programmatically +flyte.init( + endpoint="flyte.example.com", + project="my-project", + domain="development", +) +``` + +For API-key authentication in non-interactive environments, use `flyte.init_from_api_key()`. + +## Anti-Patterns + +1. **Don't keep `image`, `resources`, and `cache` on `@env.task`** — move them onto the shared `flyte.TaskEnvironment`; only per-task settings like `retries` and `timeout` stay on `@env.task`. +2. **Don't use `mem`, `ephemeral_storage`, or separate `requests`/`limits`** — use `memory`, `disk`, and a single `flyte.Resources` value that serves as both. +3. **Don't pass GPUs with `gpu="1"` plus `accelerator=`** — combine type and count into one `Type:count` string like `gpu="A100:2"`. +4. **Don't rebuild `ImageSpec`'s many constructor args** — start from a base (`Image.from_debian_base()`) and chain `.with_*` builder methods. +5. **Don't keep `cache_version="1.0"`** — use `cache="auto"` for the common case, or `flyte.Cache(version_override=...)` for advanced control. +6. **Don't read secrets via `current_context().secrets.get(...)`** — declare them on the `TaskEnvironment` and read the injected environment variable with `os.environ` / `os.getenv`. +7. **Don't recreate `LaunchPlan` + `CronSchedule`** — use `flyte.Trigger` with `flyte.Cron` attached to the task, and deploy it with `flyte deploy`. +8. **Don't run `pyflyte ... --remote`** — `flyte run` is remote by default; add `--local` explicitly for in-process runs. +9. **Don't use `pyflyte register`** — use `flyte deploy` to deploy task environments. +10. **Don't set `FLYTECTL_CONFIG` or rely on `admin.authType`** — use `FLYTE_CONFIG` and the simpler config format with auto-detected auth. diff --git a/plugins/flyte/skills/flyte-migrate-control-flow/SKILL.md b/plugins/flyte/skills/flyte-migrate-control-flow/SKILL.md new file mode 100644 index 0000000..13dcab8 --- /dev/null +++ b/plugins/flyte/skills/flyte-migrate-control-flow/SKILL.md @@ -0,0 +1,356 @@ +--- +name: flyte-migrate-control-flow +description: "Migrates Flyte 1 branching, dynamic workflows, failure handling, and fan-out to native Flyte 2 Python. Use when migrating Flyte 1 branching, dynamic workflows, failure handling, or map_task/fan-out to Flyte 2. Trigger words: conditional, @dynamic, map_task, on_failure, branching, parallelism, fan-out, flyte.map, asyncio.gather." +--- + +# Flyte 1 to 2 Migration: Control Flow and Parallelism + +Flyte 1 expressed branching, dynamic fan-out, and failure handling through DSL constructs (`conditional()`, `@dynamic`, `@workflow(on_failure=...)`) and `map_task`. In Flyte 2 these are all ordinary Python, because orchestration runs as real Python at runtime. Native `if`/`elif`/`else` replaces the conditional DSL, plain task loops replace `@dynamic`, `try`/`except` replaces `on_failure`, and `flyte.map` / `asyncio.gather` replace `map_task`. + +## Grounding References + +| Resource | URL | +|---|---| +| Migration guide (Control flow) | https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/control-flow/ | +| Migration guide (Parallelism) | https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/parallelism/ | +| Official docs | https://www.union.ai/docs/v2/flyte | +| Docs index (LLMs) | https://www.union.ai/docs/v2/flyte/llms.txt | +| SDK API reference | https://www.union.ai/docs/v2/union/api-reference/flyte-sdk/ | +| Example code | https://github.com/unionai/unionai-examples | +| Flyte MCP tools | Available via `flyte-mcp` server | + +## Conditional Execution + +The `conditional()` DSL becomes ordinary Python `if` / `elif` / `else` — for example, choosing a model based on dataset size. + +### Flyte 1 + +```python +from flytekit import task, workflow, conditional + +@task +def train_gradient_boosting(n_rows: int) -> str: + return f"trained gradient boosting on {n_rows} rows" + +@task +def train_logistic_regression(n_rows: int) -> str: + return f"trained logistic regression on {n_rows} rows" + +@workflow +def main(n_rows: int) -> str: + # Pick the model based on dataset size. + return ( + conditional("model_choice") + .if_(n_rows > 10_000) + .then(train_gradient_boosting(n_rows=n_rows)) + .else_() + .then(train_logistic_regression(n_rows=n_rows)) + ) +``` + +### Flyte 2 + +```python +import flyte + +env = flyte.TaskEnvironment(name="conditional") + +@env.task +def train_gradient_boosting(n_rows: int) -> str: + return f"trained gradient boosting on {n_rows} rows" + +@env.task +def train_logistic_regression(n_rows: int) -> str: + return f"trained logistic regression on {n_rows} rows" + +# Branching is now ordinary Python control flow -- no conditional() DSL. +@env.task +def main(n_rows: int) -> str: + if n_rows > 10_000: + return train_gradient_boosting(n_rows) + return train_logistic_regression(n_rows) +``` + +## Dynamic Workflows + +`@dynamic` existed so a task could generate a variable number of subtask calls at runtime (e.g. one per data partition discovered at runtime). In Flyte 2 every task can do this natively, so `@dynamic` simply disappears — loop over runtime data in an ordinary `@env.task`. + +### Flyte 1 + +```python +from flytekit import task, workflow, dynamic + +@task +def list_partitions(n: int) -> list[int]: + return list(range(n)) + +@task +def process_partition(partition_id: int) -> int: + # Aggregate one data partition. + return partition_id * 2 + +@dynamic +def process_all(partitions: list[int]) -> list[int]: + results = [] + for partition_id in partitions: + results.append(process_partition(partition_id=partition_id)) + return results + +@workflow +def main(n: int) -> list[int]: + partitions = list_partitions(n=n) + return process_all(partitions=partitions) +``` + +### Flyte 2 + +```python +import flyte + +env = flyte.TaskEnvironment(name="dynamic") + +@env.task +def process_partition(partition_id: int) -> int: + # Aggregate one data partition. + return partition_id * 2 + +# No @dynamic decorator needed: a plain task can loop over runtime data (e.g. a +# variable number of partitions discovered at runtime) and call other tasks. +@env.task +def main(n: int) -> list[int]: + return [process_partition(partition_id) for partition_id in range(n)] +``` + +## Error Handling + +Flyte 1's `@workflow(on_failure=...)` handler becomes ordinary Python `try` / `except` — catch a failed training run, run cleanup, and recover or re-raise. + +### Flyte 1 + +```python +from flytekit import task, workflow + +@task +def train_fold(max_depth: int) -> float: + if max_depth <= 0: + raise ValueError("max_depth must be positive") + # Return validation accuracy for this hyperparameter. + return 0.90 + 0.001 * max_depth + +@task +def notify_failure() -> None: + print("training run failed -- sending alert") + +# The on_failure handler runs if any node in the workflow fails. There is no +# try/except inside a Flyte 1 workflow. +@workflow(on_failure=notify_failure) +def main(max_depth: int) -> float: + return train_fold(max_depth=max_depth) +``` + +### Flyte 2 + +```python +import flyte + +env = flyte.TaskEnvironment(name="error_handling") + +@env.task +async def train_fold(max_depth: int) -> float: + if max_depth <= 0: + raise ValueError("max_depth must be positive") + return 0.90 + 0.001 * max_depth + +# Failure handling is ordinary Python try/except -- no on_failure handler. +@env.task +async def main(max_depth: int) -> float: + try: + return await train_fold(max_depth) + except ValueError as e: + print(f"invalid hyperparameter ({e}); falling back to a safe default") + # Recover with a safe default instead of failing the whole run. + return await train_fold(max_depth=6) +``` + +Flyte 2 also exposes typed errors, so you can catch a specific failure and retry with more resources — a common need for memory-hungry training jobs: + +```python +try: + return await train_fold(sample_size) +except flyte.errors.OOMError: + # Retry the same task with a larger memory request. + return await train_fold.override( + resources=flyte.Resources(memory="16Gi") + )(sample_size) +``` + +## Fan-out: map_task + +`map_task()` becomes `flyte.map()`, a near drop-in replacement. The one catch: `flyte.map` returns a generator, so wrap it in `list()`. For new code, the idiomatic approach is Python `async`/`await` with `asyncio.gather()`, which gives finer control over concurrency and error handling. + +### Flyte 1 + +```python +from functools import partial + +from flytekit import task, workflow, map_task + +@task +def get_shards(n: int) -> list[int]: + return list(range(n)) + +@task +def score_shard(shard_id: int, model_version: int) -> int: + # Score one shard of records with the given model version. + return shard_id * model_version + +@workflow +def main(n: int, model_version: int) -> list[int]: + shards = get_shards(n=n) + return map_task( + partial(score_shard, model_version=model_version), + concurrency=10, + )(shard_id=shards) +``` + +### Flyte 2 (flyte.map) + +```python +import flyte +from functools import partial + +env = flyte.TaskEnvironment(name="map_task") + +@env.task +def score_shard(shard_id: int, model_version: int) -> int: + # Score one shard of records with the given model version. + return shard_id * model_version + +@env.task +def main(n: int, model_version: int) -> list[int]: + bound = partial(score_shard, model_version=model_version) + # flyte.map is a drop-in for map_task, but it returns a generator, so wrap + # it in list() to materialize the results. + return list(flyte.map(bound, range(n), concurrency=10)) +``` + +### Flyte 2 (asyncio.gather) + +```python +import asyncio + +import flyte + +env = flyte.TaskEnvironment(name="map_task") + +@env.task +async def score_shard_async(shard_id: int, model_version: int) -> int: + return shard_id * model_version + +@env.task +async def main_async(n: int, model_version: int) -> list[int]: + # asyncio.gather is the idiomatic Flyte 2 way to fan out. + coros = [score_shard_async(i, model_version) for i in range(n)] + return list(await asyncio.gather(*coros)) +``` + +### Choosing flyte.map vs asyncio.gather + +| Feature | `flyte.map` (sync) | `asyncio.gather` (async) | +|---|---|---| +| Syntax | `list(flyte.map(fn, items))` | `await asyncio.gather(*tasks)` | +| Concurrency limit | Built-in `concurrency=N` | Use `asyncio.Semaphore` | +| Streaming / as-completed | No | Yes, via `asyncio.as_completed()` | +| Error handling | `return_exceptions=True` | Check return type | + +Use `flyte.map` for the smallest change from Flyte 1 `map_task`, or when stuck in synchronous code. Use `asyncio.gather` for new code where you want streaming results or fine-grained concurrency control. + +### Concurrency Control and Error Handling + +`map_task`'s `concurrency` and `min_success_ratio` become an `asyncio.Semaphore` and `return_exceptions=True`: + +```python +import asyncio + +@env.task +async def main(items: list[int], max_concurrent: int = 5) -> list[str]: + sem = asyncio.Semaphore(max_concurrent) + + async def process_with_limit(item: int) -> str: + async with sem: + return await process_item(item) + + tasks = [process_with_limit(i) for i in items] + results = await asyncio.gather(*tasks, return_exceptions=True) + + return [r for r in results if not isinstance(r, Exception)] +``` + +## Data Backfills + +Reprocessing a range of dates is a textbook `@dynamic` use case in Flyte 1, because the number of days is only known at runtime. In Flyte 2 it's a plain task that builds the date range and fans the days out with `asyncio.gather`. + +### Flyte 1 + +```python +from datetime import date, timedelta + +from flytekit import task, workflow, dynamic + +@task +def process_day(day: str) -> int: + # Reprocess a single day's partition; return the row count. + return len(day) + +# @dynamic is needed because the number of days is only known at runtime. +@dynamic +def backfill(start: str, days: int) -> list[int]: + base = date.fromisoformat(start) + results = [] + for i in range(days): + day = (base + timedelta(days=i)).isoformat() + results.append(process_day(day=day)) + return results + +@workflow +def main(start: str, days: int) -> list[int]: + return backfill(start=start, days=days) +``` + +### Flyte 2 + +```python +import asyncio +from datetime import date, timedelta + +import flyte + +env = flyte.TaskEnvironment(name="data_backfill") + +@env.task +async def process_day(day: str) -> int: + # Reprocess a single day's partition; return the row count. + return len(day) + +# A plain task builds the date range at runtime and fans the days out in +# parallel with asyncio.gather -- no @dynamic and no map_task needed. +@env.task +async def main(start: str, days: int) -> list[int]: + base = date.fromisoformat(start) + coros = [ + process_day((base + timedelta(days=i)).isoformat()) + for i in range(days) + ] + return list(await asyncio.gather(*coros)) +``` + +## Anti-Patterns + +1. **Don't import `conditional`, `dynamic`, or `map_task` from `flytekit`** — none exist in Flyte 2. Branching is native `if`/`elif`/`else`, dynamic fan-out is a plain task loop, and `map_task` becomes `flyte.map`. +2. **Don't keep the `conditional().if_().then().else_()` DSL** — rewrite it as ordinary Python control flow inside an `@env.task`. +3. **Don't reach for `@dynamic`** — every Flyte 2 task can loop over runtime data and call other tasks, so drop the decorator entirely. +4. **Don't pass `on_failure=...` to `@workflow`** — there is no workflow decorator in Flyte 2; handle failures with ordinary `try`/`except` inside a task. +5. **Don't forget to `list()` a `flyte.map` result** — it returns a generator, not a materialized list. +6. **Don't forget to `await` async fan-out** — `asyncio.gather(*coros)` returns a coroutine; without `await` you get a coroutine object instead of results. +7. **Don't drop concurrency limits** — port `concurrency=N` to `flyte.map(..., concurrency=N)` or an `asyncio.Semaphore`, and `min_success_ratio` to `return_exceptions=True` with filtering. +8. **Don't use Union-only features** — avoid `ReusePolicy` and other Union-specific APIs. diff --git a/plugins/flyte/skills/flyte-migrate-data-io/SKILL.md b/plugins/flyte/skills/flyte-migrate-data-io/SKILL.md new file mode 100644 index 0000000..fa83f34 --- /dev/null +++ b/plugins/flyte/skills/flyte-migrate-data-io/SKILL.md @@ -0,0 +1,309 @@ +--- +name: flyte-migrate-data-io +description: Migrates Flyte 1 data types and offloaded I/O to Flyte 2. Use when migrating Flyte 1 data types and I/O (files, directories, dataframes, dataclasses) to Flyte 2, converting FlyteFile, FlyteDirectory, or StructuredDataset to flyte.io.File, flyte.io.Dir, and flyte.io.DataFrame. Trigger words are FlyteFile, FlyteDirectory, StructuredDataset, DataFrame, dataclass, Pydantic, type, I/O, and serialization. +--- + +# Flyte 1 to 2 Migration: Data Types and I/O + +Flyte 2 renames the offloaded-data types and makes their I/O `async`, but the mental model is the same: pass lightweight references to large data between tasks, not the materialized bytes. `FlyteFile`, `FlyteDirectory`, and `StructuredDataset` become `flyte.io.File`, `flyte.io.Dir`, and `flyte.io.DataFrame`. Plain dataclasses and Pydantic `BaseModel`s work directly as task I/O with no JSON mixin. + +## Grounding References + +| Resource | URL | +|---|---| +| Migration guide | https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/data-io/ | +| Official docs | https://www.union.ai/docs/v2/flyte | +| Docs index (LLMs) | https://www.union.ai/docs/v2/flyte/llms.txt | +| SDK API reference | https://www.union.ai/docs/v2/union/api-reference/flyte-sdk/ | +| Example code | https://github.com/unionai/unionai-examples | +| Flyte MCP tools | Available via `flyte-mcp` server | + +## Type Mapping + +| Flyte 1 | Flyte 2 | Notes | +|---|---|---| +| `flytekit.types.file.FlyteFile` | `flyte.io.File` | I/O is `async` | +| `flytekit.types.directory.FlyteDirectory` | `flyte.io.Dir` | I/O is `async` | +| `flytekit.types.structured.StructuredDataset` | `flyte.io.DataFrame` | build with `from_df`, read with `open(...).all()` | +| `@dataclass_json` + `@dataclass` | plain `@dataclass` | no mixin needed | +| Pydantic `BaseModel` (+ config) | plain Pydantic `BaseModel` | works directly as task I/O | + +## Offloaded Data: The Mental Model + +`File`, `Dir`, and `DataFrame` are lightweight references (pointers) to data offloaded in blob storage — not the materialized bytes. In Flyte 2 the read/write operations are `async`: upload with `await File.from_local(local_path)`, read with `async with f.open("rb") as fh: await fh.read()`, build a frame with `flyte.io.DataFrame.from_df(df)` (sync constructor), and read it with `await fdf.open(pandas.DataFrame).all()`. + +## Files and Directories + +`FlyteFile` and `FlyteDirectory` become `flyte.io.File` and `flyte.io.Dir` — the way you pass model artifacts and datasets between tasks. Use `await File.from_local(...)` to upload and `async with file.open(...)` to read. + +### Flyte 1 + +```python +import os + +from flytekit import task, workflow, current_context +from flytekit.types.file import FlyteFile + +@task +def write_file(content: str) -> FlyteFile: + path = os.path.join(current_context().working_directory, "out.txt") + with open(path, "w") as f: + f.write(content) + return FlyteFile(path=path) + +@task +def read_file(f: FlyteFile) -> str: + with open(f.download()) as fh: + return fh.read() + +@workflow +def main(content: str) -> str: + f = write_file(content=content) + return read_file(f=f) +``` + +### Flyte 2 + +```python +import flyte +from flyte.io import File + +env = flyte.TaskEnvironment(name="files") + +@env.task +async def write_file(content: str) -> File: + with open("out.txt", "w") as f: + f.write(content) + # File.from_local uploads the file to blob storage and returns a reference + # (a lightweight pointer, not the materialized bytes). + return await File.from_local("out.txt") + +@env.task +async def read_file(f: File) -> str: + async with f.open("rb") as fh: + return (await fh.read()).decode("utf-8") + +@env.task +async def main(content: str) -> str: + f = await write_file(content) + return await read_file(f) +``` + +Directories follow the same pattern: import `Dir` from `flyte.io` and use its `async` upload/read methods in place of `FlyteDirectory`. See [Files and directories](https://www.union.ai/docs/v2/flyte/user-guide/task-programming/files-and-directories) for more. + +## DataFrames + +`StructuredDataset` becomes `flyte.io.DataFrame`. Construct one with `flyte.io.DataFrame.from_df(df)` and read it back with `await df.open(pandas.DataFrame).all()`. + +### Flyte 1 + +```python +import pandas as pd +from flytekit import task, workflow +from flytekit.types.structured import StructuredDataset + +@task +def make_df() -> StructuredDataset: + df = pd.DataFrame({"employee_id": [1, 2, 3], "salary": [50000, 60000, 70000]}) + return StructuredDataset(dataframe=df) + +@task +def total_payroll(sd: StructuredDataset) -> float: + df = sd.open(pd.DataFrame).all() + return float(df["salary"].sum()) + +@workflow +def main() -> float: + return total_payroll(sd=make_df()) +``` + +### Flyte 2 + +```python +import pandas as pd +import flyte +import flyte.io + +env = flyte.TaskEnvironment( + name="dataframe", + image=flyte.Image.from_debian_base().with_pip_packages("pandas", "pyarrow"), +) + +@env.task +async def make_df() -> flyte.io.DataFrame: + df = pd.DataFrame({"employee_id": [1, 2, 3], "salary": [50000, 60000, 70000]}) + # StructuredDataset becomes flyte.io.DataFrame. + return flyte.io.DataFrame.from_df(df) + +@env.task +async def total_payroll(fdf: flyte.io.DataFrame) -> float: + df = await fdf.open(pd.DataFrame).all() + return float(df["salary"].sum()) + +@env.task +async def main() -> float: + return await total_payroll(await make_df()) +``` + +Add the dataframe dependencies (for example `pandas` and `pyarrow`) to the `TaskEnvironment` image. See [DataFrames](https://www.union.ai/docs/v2/flyte/user-guide/task-programming/dataframes) for more. + +## Dataclasses and Structured Types + +Flyte 1 required a `@dataclass_json` mixin for dataclass I/O. In Flyte 2, plain dataclasses (and Pydantic `BaseModel`s) work directly as task inputs and outputs — handy for passing around a training config. + +### Flyte 1 + +```python +from dataclasses import dataclass + +from dataclasses_json import dataclass_json +from flytekit import task, workflow + +@dataclass_json +@dataclass +class TrainingConfig: + learning_rate: float + n_estimators: int + max_depth: int = 6 + +@task +def make_config(learning_rate: float, n_estimators: int) -> TrainingConfig: + return TrainingConfig(learning_rate=learning_rate, n_estimators=n_estimators) + +@task +def train(config: TrainingConfig) -> str: + return ( + f"trained with lr={config.learning_rate}, " + f"n_estimators={config.n_estimators}, max_depth={config.max_depth}" + ) + +@workflow +def main(learning_rate: float, n_estimators: int) -> str: + config = make_config(learning_rate=learning_rate, n_estimators=n_estimators) + return train(config=config) +``` + +### Flyte 2 + +```python +from dataclasses import dataclass + +import flyte + +env = flyte.TaskEnvironment(name="dataclasses") + +# Plain dataclasses work directly as task I/O -- no @dataclass_json mixin needed. +# Pydantic BaseModels work the same way. +@dataclass +class TrainingConfig: + learning_rate: float + n_estimators: int + max_depth: int = 6 + +@env.task +def make_config(learning_rate: float, n_estimators: int) -> TrainingConfig: + return TrainingConfig(learning_rate=learning_rate, n_estimators=n_estimators) + +@env.task +def train(config: TrainingConfig) -> str: + return ( + f"trained with lr={config.learning_rate}, " + f"n_estimators={config.n_estimators}, max_depth={config.max_depth}" + ) + +@env.task +def main(learning_rate: float, n_estimators: int) -> str: + config = make_config(learning_rate, n_estimators) + return train(config) +``` + +## Data ETL: Putting It Together + +Extract, clean, aggregate, and write out a feature table. `StructuredDataset` becomes `flyte.io.DataFrame`, and the tasks become `async`. + +### Flyte 1 + +```python +import pandas as pd +from flytekit import task, workflow +from flytekit.types.structured import StructuredDataset + +@task +def extract() -> pd.DataFrame: + # Read raw transaction records (stand-in for a real source). + return pd.DataFrame( + { + "user_id": [1, 1, 2, 3, 3, 3], + "amount": [10.0, 5.0, 20.0, 7.5, 2.5, 1.0], + } + ) + +@task +def transform(df: pd.DataFrame) -> StructuredDataset: + # Clean and aggregate into a per-user feature table. + df = df[df["amount"] > 0] + agg = df.groupby("user_id", as_index=False)["amount"].sum() + return StructuredDataset(dataframe=agg) + +@task +def load(sd: StructuredDataset) -> int: + df = sd.open(pd.DataFrame).all() + return len(df) + +@workflow +def main() -> int: + raw = extract() + features = transform(df=raw) + return load(sd=features) +``` + +### Flyte 2 + +```python +import pandas as pd +import flyte +import flyte.io + +env = flyte.TaskEnvironment( + name="data_etl", + image=flyte.Image.from_debian_base().with_pip_packages("pandas", "pyarrow"), +) + +@env.task +async def extract() -> pd.DataFrame: + # Read raw transaction records (stand-in for a real source). + return pd.DataFrame( + { + "user_id": [1, 1, 2, 3, 3, 3], + "amount": [10.0, 5.0, 20.0, 7.5, 2.5, 1.0], + } + ) + +@env.task +async def transform(df: pd.DataFrame) -> flyte.io.DataFrame: + # Clean and aggregate into a per-user feature table. + df = df[df["amount"] > 0] + agg = df.groupby("user_id", as_index=False)["amount"].sum() + # StructuredDataset becomes flyte.io.DataFrame. + return flyte.io.DataFrame.from_df(agg) + +@env.task +async def load(sd: flyte.io.DataFrame) -> int: + df = await sd.open(pd.DataFrame).all() + return len(df) + +@env.task +async def main() -> int: + raw = await extract() + features = await transform(raw) + return await load(features) +``` + +## Anti-Patterns + +1. **Don't call the offloaded-data I/O synchronously** — `File.from_local`, `file.open(...).read()`, and `DataFrame.open(...).all()` are `async` in Flyte 2; `await` them inside `async` tasks. +2. **Don't keep the `@dataclass_json` mixin** — plain `@dataclass` and Pydantic `BaseModel`s serialize as task I/O directly; drop `dataclasses_json`. +3. **Don't return `StructuredDataset(dataframe=df)`** — use `flyte.io.DataFrame.from_df(df)` instead. +4. **Don't materialize large data into task outputs** — return `File`, `Dir`, or `DataFrame` references, not the raw bytes or full frames. +5. **Don't forget the dataframe dependencies** — add `pandas` and `pyarrow` (or your engine) to the `TaskEnvironment` image so DataFrame I/O works remotely. +6. **Don't import from `flytekit.types.*`** — import `File` and `Dir` from `flyte.io`, and use `flyte.io.DataFrame`. diff --git a/plugins/flyte/skills/flyte-migrate-ml/SKILL.md b/plugins/flyte/skills/flyte-migrate-ml/SKILL.md new file mode 100644 index 0000000..01b64dc --- /dev/null +++ b/plugins/flyte/skills/flyte-migrate-ml/SKILL.md @@ -0,0 +1,611 @@ +--- +name: flyte-migrate-ml +description: Migrates Flyte 1 machine learning code to Flyte 2 and unlocks net-new v2 patterns. Use when migrating Flyte 1 ML workloads (training, HPO, GPU/deep learning, batch inference) to Flyte 2, specifying GPU resources, or building the end-to-end pipeline pattern. Trigger words - migrate training, HPO, GPU, deep learning, batch inference, model serving, pytorch. +--- + +# Flyte 1 to Flyte 2 ML Migration Skill + +Migrate existing Flyte 1 ML workloads — small-model training, hyperparameter optimization, deep learning on GPUs, and batch inference — to Flyte 2, then take advantage of patterns that were not possible in Flyte 1 (real-time serving, apps, sandboxed execution). + +This skill is specifically about **migrating existing v1 ML code**. For greenfield authoring in Flyte 2, use the companion skills: + +- `flyte-sdk-ml` — writing new ML training / inference tasks in Flyte 2. +- `flyte-sdk-app` — writing new apps and serving endpoints. +- `flyte-sdk-agent` — writing new agents and sandboxed / code-mode workloads. + +## Grounding References + +| Resource | URL | +|---|---| +| Migration guide (ML workloads) | https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/ml-workloads/ | +| Migration guide (New in Flyte 2) | https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/new-in-flyte-2/ | +| Official docs | https://www.union.ai/docs/v2/flyte | +| Docs index (LLMs) | https://www.union.ai/docs/v2/flyte/llms.txt | +| SDK API reference | https://www.union.ai/docs/v2/union/api-reference/flyte-sdk/ | +| Example code | https://github.com/unionai/unionai-examples | +| Flyte MCP tools | Available via `flyte-mcp` server | + +## Migration Cheat Sheet + +| Flyte 1 | Flyte 2 | +|---|---| +| `ImageSpec(name=..., packages=[...])` | `flyte.Image.from_debian_base().with_pip_packages(...)` | +| `@task(container_image=..., requests=..., cache=...)` | Set `image`, `resources`, `cache` once on `flyte.TaskEnvironment`, then `@env.task` | +| `Resources(cpu=..., mem=...)` | `flyte.Resources(cpu=..., memory=...)` (note `mem` becomes `memory`) | +| `Resources(gpu="1")` + `accelerator=T4` | `flyte.Resources(gpu="T4:1")` | +| `FlyteFile` / `FlyteFile(path=...)` | `flyte.io.File` / `await File.from_local(...)` | +| `model_file.download()` | `await model_file.download()` | +| `current_context().working_directory` | `os.getcwd()` | +| `@workflow` | An orchestrating `@env.task` (plain `async` Python) | +| `map_task(fn)(x=xs)` | `await asyncio.gather(*[fn(x) for x in xs])` | +| A "pick the best" task | Plain Python after `gather` | + +## Small model training (scikit-learn / XGBoost) + +Train a model, persist it as a `File`, and evaluate it. Image, resources, and caching move to the `TaskEnvironment`; `FlyteFile` becomes `flyte.io.File`. + +### Flyte 1 + +```python +import os + +import joblib +from flytekit import task, workflow, ImageSpec, Resources, current_context +from flytekit.types.file import FlyteFile +from sklearn.datasets import load_breast_cancer +from sklearn.model_selection import train_test_split +from xgboost import XGBClassifier + +image = ImageSpec( + name="xgb-image", + packages=["xgboost", "scikit-learn", "joblib"], +) + +@task(container_image=image, requests=Resources(cpu="2", mem="4Gi")) +def train_model(n_estimators: int, max_depth: int) -> FlyteFile: + data = load_breast_cancer() + X_train, _, y_train, _ = train_test_split(data.data, data.target, random_state=42) + model = XGBClassifier(n_estimators=n_estimators, max_depth=max_depth) + model.fit(X_train, y_train) + + model_path = os.path.join(current_context().working_directory, "model.json") + joblib.dump(model, model_path) + return FlyteFile(path=model_path) + +@task(container_image=image) +def evaluate(model_file: FlyteFile) -> float: + model = joblib.load(model_file.download()) + data = load_breast_cancer() + _, X_test, _, y_test = train_test_split(data.data, data.target, random_state=42) + return float(model.score(X_test, y_test)) + +@workflow +def main(n_estimators: int, max_depth: int) -> float: + model = train_model(n_estimators=n_estimators, max_depth=max_depth) + return evaluate(model_file=model) +``` + +### Flyte 2 + +```python +import os + +import joblib +import flyte +from flyte.io import File +from sklearn.datasets import load_breast_cancer +from sklearn.model_selection import train_test_split +from xgboost import XGBClassifier + +env = flyte.TaskEnvironment( + name="train_xgboost", + image=flyte.Image.from_debian_base().with_pip_packages( + "xgboost", "scikit-learn", "joblib" + ), + resources=flyte.Resources(cpu="2", memory="4Gi"), +) + +@env.task +async def train_model(n_estimators: int, max_depth: int) -> File: + data = load_breast_cancer() + X_train, _, y_train, _ = train_test_split(data.data, data.target, random_state=42) + model = XGBClassifier(n_estimators=n_estimators, max_depth=max_depth) + model.fit(X_train, y_train) + + model_path = os.path.join(os.getcwd(), "model.json") + joblib.dump(model, model_path) + return await File.from_local(model_path) + +@env.task +async def evaluate(model_file: File) -> float: + local_path = await model_file.download() + model = joblib.load(local_path) + data = load_breast_cancer() + _, X_test, _, y_test = train_test_split(data.data, data.target, random_state=42) + return float(model.score(X_test, y_test)) + +@env.task +async def main(n_estimators: int, max_depth: int) -> float: + model = await train_model(n_estimators, max_depth) + return await evaluate(model) +``` + +## Hyperparameter optimization + +Fan out one training run per hyperparameter, then pick the best. In Flyte 1 the grid search runs through `map_task` and the "pick the best" step must itself be a task. In Flyte 2 you `gather` the runs and select the winner in plain Python. + +### Flyte 1 + +```python +from flytekit import task, workflow, map_task +from sklearn.datasets import load_iris +from sklearn.ensemble import RandomForestClassifier +from sklearn.model_selection import cross_val_score + +@task +def get_grid() -> list[int]: + return [2, 4, 8, 16] + +@task +def train_eval(max_depth: int) -> float: + data = load_iris() + model = RandomForestClassifier(max_depth=max_depth, random_state=42) + scores = cross_val_score(model, data.data, data.target, cv=3) + return float(scores.mean()) + +@task +def best_score(scores: list[float]) -> float: + return max(scores) + +@workflow +def main() -> float: + grid = get_grid() + # Fan out one training run per hyperparameter value. + scores = map_task(train_eval)(max_depth=grid) + return best_score(scores=scores) +``` + +### Flyte 2 + +```python +import asyncio + +import flyte +from sklearn.datasets import load_iris +from sklearn.ensemble import RandomForestClassifier +from sklearn.model_selection import cross_val_score + +env = flyte.TaskEnvironment( + name="hpo", + image=flyte.Image.from_debian_base().with_pip_packages("scikit-learn"), +) + +@env.task +async def train_eval(max_depth: int) -> float: + data = load_iris() + model = RandomForestClassifier(max_depth=max_depth, random_state=42) + scores = cross_val_score(model, data.data, data.target, cv=3) + return float(scores.mean()) + +@env.task +async def main() -> dict: + grid = [2, 4, 8, 16] + # Fan out one training run per hyperparameter value... + scores = await asyncio.gather(*[train_eval(d) for d in grid]) + # ...then pick the best in plain Python (impossible in a Flyte 1 workflow). + best_idx = max(range(len(scores)), key=lambda i: scores[i]) + return {"best_max_depth": grid[best_idx], "best_score": scores[best_idx]} +``` + +## Large model training (deep learning) + +GPU configuration moves to the `TaskEnvironment`: the Flyte 1 `Resources(gpu="1")` plus a separate `accelerator=T4` become a single `gpu="T4:1"` string on `flyte.Resources`. + +### Flyte 1 + +```python +from flytekit import task, workflow, ImageSpec, Resources +from flytekit.extras.accelerators import T4 +import torch +import torch.nn as nn + +image = ImageSpec( + name="dl-image", + packages=["torch"], +) + +@task( + container_image=image, + requests=Resources(cpu="4", mem="16Gi", gpu="1"), + accelerator=T4, +) +def train(epochs: int) -> float: + device = "cuda" if torch.cuda.is_available() else "cpu" + model = nn.Linear(10, 1).to(device) + optimizer = torch.optim.SGD(model.parameters(), lr=0.01) + loss_fn = nn.MSELoss() + + X = torch.randn(128, 10, device=device) + y = torch.randn(128, 1, device=device) + + loss = torch.tensor(0.0) + for _ in range(epochs): + optimizer.zero_grad() + loss = loss_fn(model(X), y) + loss.backward() + optimizer.step() + return float(loss.item()) + +@workflow +def main(epochs: int) -> float: + return train(epochs=epochs) +``` + +### Flyte 2 + +```python +import flyte +import torch +import torch.nn as nn + +# GPU type and count go in a single "T4:1"-style string. For multi-node +# distributed training, wrap the training task with the torch elastic plugin. +env = flyte.TaskEnvironment( + name="train_deep_learning", + image=flyte.Image.from_debian_base().with_pip_packages("torch"), + resources=flyte.Resources(cpu="4", memory="16Gi", gpu="T4:1"), +) + +@env.task +async def train(epochs: int) -> float: + device = "cuda" if torch.cuda.is_available() else "cpu" + model = nn.Linear(10, 1).to(device) + optimizer = torch.optim.SGD(model.parameters(), lr=0.01) + loss_fn = nn.MSELoss() + + X = torch.randn(128, 10, device=device) + y = torch.randn(128, 1, device=device) + + loss = torch.tensor(0.0) + for _ in range(epochs): + optimizer.zero_grad() + loss = loss_fn(model(X), y) + loss.backward() + optimizer.step() + return float(loss.item()) + +@env.task +async def main(epochs: int) -> float: + return await train(epochs) +``` + +For multi-node distributed training (PyTorch elastic, etc.), wrap the training task with the torch elastic plugin. See the Resources docs and plugin integrations at https://www.union.ai/docs/v2/flyte/user-guide/task-configuration/resources. + +## Batch inference + +Load a trained model once and score many batches in parallel. `map_task` with a `partial`-bound model becomes `asyncio.gather` over the batches, reusing the same model reference. + +### Flyte 1 + +```python +import os +from functools import partial + +import joblib +from flytekit import task, workflow, map_task, ImageSpec, current_context +from flytekit.types.file import FlyteFile +from sklearn.datasets import load_iris +from sklearn.ensemble import RandomForestClassifier + +image = ImageSpec(name="inference-image", packages=["scikit-learn", "joblib"]) + +@task(container_image=image) +def train_model() -> FlyteFile: + data = load_iris() + model = RandomForestClassifier().fit(data.data, data.target) + model_path = os.path.join(current_context().working_directory, "model.joblib") + joblib.dump(model, model_path) + return FlyteFile(path=model_path) + +@task(container_image=image) +def get_batches() -> list[list[list[float]]]: + data = load_iris() + rows = data.data.tolist() + # Split the rows into batches of 30. + return [rows[i : i + 30] for i in range(0, len(rows), 30)] + +@task(container_image=image) +def score_batch(model_file: FlyteFile, batch: list[list[float]]) -> list[int]: + model = joblib.load(model_file.download()) + return [int(p) for p in model.predict(batch)] + +@workflow +def main() -> list[list[int]]: + model = train_model() + batches = get_batches() + return map_task(partial(score_batch, model_file=model))(batch=batches) +``` + +### Flyte 2 + +```python +import asyncio +import os + +import joblib +import flyte +from flyte.io import File +from sklearn.datasets import load_iris +from sklearn.ensemble import RandomForestClassifier + +env = flyte.TaskEnvironment( + name="batch_inference", + image=flyte.Image.from_debian_base().with_pip_packages("scikit-learn", "joblib"), +) + +@env.task +async def train_model() -> File: + data = load_iris() + model = RandomForestClassifier().fit(data.data, data.target) + model_path = os.path.join(os.getcwd(), "model.joblib") + joblib.dump(model, model_path) + return await File.from_local(model_path) + +@env.task +async def score_batch(model_file: File, batch: list[list[float]]) -> list[int]: + local_path = await model_file.download() + model = joblib.load(local_path) + return [int(p) for p in model.predict(batch)] + +@env.task +async def main() -> list[list[int]]: + model = await train_model() + rows = load_iris().data.tolist() + batches = [rows[i : i + 30] for i in range(0, len(rows), 30)] + # Score every batch in parallel, reusing the same model reference. + coros = [score_batch(model, batch) for batch in batches] + return list(await asyncio.gather(*coros)) +``` + +## A complete example: end-to-end ML pipeline + +Putting it together — a load / train / evaluate pipeline shows the image, resources, caching, file I/O, and orchestration changes in one place. Image, resources, and cache are set **once** on the `TaskEnvironment`, and the "workflow" is just an orchestrating task. + +### Flyte 1 + +```python +import os + +import joblib +import pandas as pd +from flytekit import task, workflow, ImageSpec, Resources, current_context +from flytekit.types.file import FlyteFile +from sklearn.datasets import load_iris +from sklearn.ensemble import RandomForestClassifier + +image = ImageSpec( + name="ml-image", + packages=["pandas", "scikit-learn", "joblib"], +) + +@task( + container_image=image, + requests=Resources(cpu="2", mem="4Gi"), + cache=True, + cache_version="1.0", +) +def load_data() -> pd.DataFrame: + data = load_iris(as_frame=True) + df = data.frame + df["species"] = data.target + return df + +@task(container_image=image) +def train_model(data: pd.DataFrame) -> FlyteFile: + model = RandomForestClassifier() + X = data.drop("species", axis=1) + y = data["species"] + model.fit(X, y) + + model_path = os.path.join(current_context().working_directory, "model.joblib") + joblib.dump(model, model_path) + return FlyteFile(path=model_path) + +@task(container_image=image) +def evaluate(model_file: FlyteFile, data: pd.DataFrame) -> float: + model = joblib.load(model_file.download()) + X = data.drop("species", axis=1) + y = data["species"] + return float(model.score(X, y)) + +@workflow +def main() -> float: + data = load_data() + model = train_model(data=data) + return evaluate(model_file=model, data=data) +``` + +### Flyte 2 + +```python +import os + +import joblib +import pandas as pd +import flyte +from flyte.io import File +from sklearn.datasets import load_iris +from sklearn.ensemble import RandomForestClassifier + +# Image, resources, and cache are set once on the TaskEnvironment. +env = flyte.TaskEnvironment( + name="ml_pipeline", + image=flyte.Image.from_debian_base().with_pip_packages( + "pandas", "scikit-learn", "joblib" + ), + resources=flyte.Resources(cpu="2", memory="4Gi"), + cache="auto", +) + +@env.task +async def load_data() -> pd.DataFrame: + data = load_iris(as_frame=True) + df = data.frame + df["species"] = data.target + return df + +@env.task +async def train_model(data: pd.DataFrame) -> File: + model = RandomForestClassifier() + X = data.drop("species", axis=1) + y = data["species"] + model.fit(X, y) + + model_path = os.path.join(os.getcwd(), "model.joblib") + joblib.dump(model, model_path) + return await File.from_local(model_path) + +@env.task +async def evaluate(model_file: File, data: pd.DataFrame) -> float: + local_path = await model_file.download() + model = joblib.load(local_path) + X = data.drop("species", axis=1) + y = data["species"] + return float(model.score(X, y)) + +# The "workflow" is just an orchestrating task. +@env.task +async def main() -> float: + data = await load_data() + model = await train_model(data) + return await evaluate(model, data) +``` + +## New in Flyte 2 + +Flyte 1 was a batch orchestration system: everything ran as a finite DAG that started, did work, and finished. Flyte 2 keeps all of that and adds long-running services, high-throughput batch inference, and sandboxed code execution — so the same project that trains your model can also serve it, host a dashboard, saturate a GPU, or safely run LLM-generated code. There is no v1 counterpart to migrate here; these are net-new capabilities that your migrated training code unlocks. For greenfield authoring of these, see the `flyte-sdk-app` and `flyte-sdk-agent` skills. + +### Real-time inference and model serving + +Instead of scoring a batch and exiting, you can stand up an always-on REST endpoint from a `FastAPIAppEnvironment` and deploy it with `flyte.deploy`. The app can load a model artifact produced by one of your migrated training tasks. + +```python +app = FastAPI(title="ML Model API") + +# Define request/response models +class PredictionRequest(BaseModel): + feature1: float + feature2: float + feature3: float + +class PredictionResponse(BaseModel): + prediction: float + probability: float + +# Load model (you would typically load this from storage) +model = None + +@asynccontextmanager +async def lifespan(app: FastAPI): + global model + model_path = os.getenv("MODEL_PATH", "/app/models/model.joblib") + # In production, load from your storage + if os.path.exists(model_path): + with open(model_path, "rb") as f: + model = joblib.load(f) + yield + +@app.post("/predict", response_model=PredictionResponse) +async def predict(request: PredictionRequest): + # Make prediction + # prediction = model.predict([[request.feature1, request.feature2, request.feature3]]) + + # Dummy prediction for demo + prediction = 0.85 + probability = 0.92 + + return PredictionResponse( + prediction=prediction, + probability=probability, + ) + +env = FastAPIAppEnvironment( + name="ml-model-api", + app=app, + image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages( + "fastapi", + "uvicorn", + "scikit-learn", + "pydantic", + "joblib", + ), + parameters=[ + flyte.app.Parameter( + name="model_file", + value=flyte.io.File.from_existing_remote("s3://bucket/models/model.joblib"), + mount="/app/models", + env_var="MODEL_PATH", + ), + ], + resources=flyte.Resources(cpu=2, memory="2Gi"), + requires_auth=False, +) +``` + +For serving large language models, the `flyteplugins-vllm` integration gives you a production-grade vLLM server (with autoscaling to zero) via `VLLMAppEnvironment`. Any web app — a Streamlit dashboard, a Gradio demo, a Flask backend — runs as a `flyte.app.AppEnvironment` that you configure with image, resources, port, autoscaling, and a custom subdomain, then `flyte.serve`. + +### Dynamic batching for GPU inference + +For in-process batch inference, `DynamicBatcher` from `flyte.extras` keeps an expensive GPU saturated: async producers load and preprocess data concurrently while a single consumer feeds the model in optimally-sized batches, with built-in backpressure. This replaces the Flyte 1 pattern of standing up a separate inference server just to get request batching. + +```python +import asyncio +from flyte.extras import DynamicBatcher + +async with DynamicBatcher( + process_fn=run_inference, # takes a batch, returns results in the same order + target_batch_cost=1000, # cost budget per batch + max_batch_size=64, # hard cap on records per batch + batch_timeout_s=0.05, # max wait before dispatching a partial batch +) as batcher: + futures = [await batcher.submit(record) for record in records] + results = await asyncio.gather(*futures) +``` + +`submit()` is non-blocking and returns a `Future`; when the queue is full it applies backpressure automatically. See the batch inference docs for `TokenBatcher` (token-aware LLM batching). + +### Sandboxed code execution + +`flyte.sandbox.create()` runs arbitrary Python code or shell commands inside an ephemeral, single-use Docker container — built on demand from declared dependencies, executed once, then discarded. Only declared inputs go in and only declared outputs come back, which makes it the safe way to run untrusted code, most importantly code generated by an LLM. + +```python +# sandbox_environment provides the base runtime for code sandboxes. +# Include it in depends_on so the sandbox runtime is available when tasks execute. +env = flyte.TaskEnvironment( + name="sandbox-demo", + image=flyte.Image.from_debian_base(name="sandbox-demo"), + depends_on=[sandbox_environment], +) + +# Auto-IO mode: pure computation. The code string runs in an isolated sandbox; +# only the declared inputs go in and only the declared outputs come back. +sum_sandbox = flyte.sandbox.create( + name="sum-to-n", + code="total = sum(range(n + 1)) if conditional else 0", + inputs={"n": int, "conditional": bool}, + outputs={"total": int}, +) +``` + +Call it from a task with `await sum_sandbox.run.aio(n=10, conditional=True)`. This also powers **code mode** (programmatic tool calling), where an agent writes a whole program instead of emitting one tool call at a time. + +## Anti-Patterns + +1. **Don't keep `@task` / `@workflow` per-task config** — move `image`, `resources`, and `cache` onto a single `flyte.TaskEnvironment` and decorate with `@env.task`. +2. **Don't leave a separate "pick the best" task** — after `asyncio.gather`, select the winner in plain Python inside the orchestrating task. +3. **Don't carry `map_task` + `partial` into v2** — fan out with `asyncio.gather` over coroutines, reusing the same model reference. +4. **Don't split GPU type and count** — replace `Resources(gpu="1")` + `accelerator=T4` with a single `gpu="T4:1"` string on `flyte.Resources`. +5. **Don't use `mem=` or `current_context().working_directory`** — use `memory=` on `flyte.Resources` and `os.getcwd()` for local paths. +6. **Don't forget `await`** — `File.from_local`, `download`, and task calls are all async in v2. +7. **Don't hand-roll a serving container or a request-batching server** — use a `FastAPIAppEnvironment` / `AppEnvironment` for serving and `DynamicBatcher` for GPU batching. +8. **Don't run untrusted or LLM-generated code inline** — use `flyte.sandbox.create()` with `sandbox_environment` in `depends_on`. diff --git a/plugins/flyte/skills/flyte-migrate-tasks-workflows/SKILL.md b/plugins/flyte/skills/flyte-migrate-tasks-workflows/SKILL.md new file mode 100644 index 0000000..6e4f767 --- /dev/null +++ b/plugins/flyte/skills/flyte-migrate-tasks-workflows/SKILL.md @@ -0,0 +1,258 @@ +--- +name: flyte-migrate-tasks-workflows +description: >- + Migrates Flyte 1 tasks and workflows to Flyte 2, where the @task, @workflow, + and @dynamic decorators collapse into a single @env.task on a + flyte.TaskEnvironment and a workflow becomes a task that calls other tasks. + Use when the user is migrating Flyte 1 tasks/workflows to Flyte 2. Trigger + words: migrate task, migrate workflow, @task, @workflow, @dynamic, + TaskEnvironment, env.task. +--- + +# Flyte 1 to 2 Migration: Tasks and Workflows + +The biggest structural change in Flyte 2 is that everything is a task. The Flyte 1 `@task`, `@workflow`, and `@dynamic` decorators all collapse into a single `@env.task` on a `flyte.TaskEnvironment`, and a "workflow" is now just a task that calls other tasks. This skill covers the structural shift, sequential ordering without the `>>` operator, nested subworkflows, TaskEnvironment configuration basics, and the full `@task` parameter mapping. + +## Grounding References + +| Resource | URL | +|---|---| +| Migration guide | https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/tasks-and-workflows/ | +| Official docs | https://www.union.ai/docs/v2/flyte | +| Docs index (LLMs) | https://www.union.ai/docs/v2/flyte/llms.txt | +| SDK API reference | https://www.union.ai/docs/v2/union/api-reference/flyte-sdk/ | +| Example code | https://github.com/unionai/unionai-examples | +| Flyte MCP tools | Available via `flyte-mcp` server | + +## The Structural Shift + +In Flyte 1 you decorated units of work with `@task`, composed them with `@workflow`, and used `@dynamic` for runtime-generated graphs. In Flyte 2 you create one `flyte.TaskEnvironment` that carries the configuration, then decorate every function — leaf tasks and orchestrating "workflows" alike — with `@env.task`. There is no separate `@workflow` decorator: the entrypoint is just a task that calls other tasks. + +## Hello World: Tasks and Workflows + +A `@task` plus `@workflow` becomes two `@env.task`s, where the entrypoint task calls the others. Sequential calls are naturally ordered — no `>>` operator required. + +### Flyte 1 + +```python +import flytekit + +@flytekit.task +def say_hello(name: str) -> str: + return f"Hello, {name}!" + +@flytekit.task +def to_upper(greeting: str) -> str: + return greeting.upper() + +@flytekit.workflow +def main(name: str) -> str: + greeting = say_hello(name=name) + return to_upper(greeting=greeting) +``` + +### Flyte 2 + +```python +import flyte + +env = flyte.TaskEnvironment(name="hello_world") + +@env.task +def say_hello(name: str) -> str: + return f"Hello, {name}!" + +@env.task +def to_upper(greeting: str) -> str: + return greeting.upper() + +# The "workflow" is now just a task that calls other tasks. +@env.task +def main(name: str) -> str: + greeting = say_hello(name) + return to_upper(greeting) +``` + +Note that Flyte 2 task calls pass arguments positionally (`say_hello(name)`) rather than requiring keyword arguments as in Flyte 1 (`say_hello(name=name)`). + +## Chaining and Ordering + +In Flyte 1 you sometimes used `>>` to force ordering between tasks with no data dependency. In Flyte 2, sequential (synchronous) calls run in the order they are written, and `await`ing async tasks in sequence does the same. The `>>` operator is gone. + +### Flyte 1 + +```python +from flytekit import task, workflow + +@task +def clear_staging_table() -> None: + # Side effect only: truncate the staging table. + print("cleared staging table") + +@task +def load_into_staging() -> None: + # Side effect only: load fresh rows into staging. + print("loaded staging table") + +@task +def publish_to_prod() -> None: + # Side effect only: swap staging into the production table. + print("published to prod") + +@workflow +def main() -> None: + clear = clear_staging_table() + load = load_into_staging() + publish = publish_to_prod() + + # These tasks pass no data between them, so use the >> operator to force + # ordering: clear must finish before load, which must finish before publish. + clear >> load >> publish +``` + +### Flyte 2 + +```python +import flyte + +env = flyte.TaskEnvironment(name="staging_publish") + +@env.task +def clear_staging_table() -> None: + print("cleared staging table") + +@env.task +def load_into_staging() -> None: + print("loaded staging table") + +@env.task +def publish_to_prod() -> None: + print("published to prod") + +# Sequential (synchronous) calls run in the order they're written, even when no +# data flows between them. The Flyte 1 `>>` ordering operator is gone. +@env.task +def main() -> None: + clear_staging_table() + load_into_staging() + publish_to_prod() +``` + +## Subworkflows + +A `@workflow` invoked by another `@workflow` (for example, a reusable preprocessing pipeline) becomes a task that calls other tasks — nest them as deeply as you like. + +### Flyte 1 + +```python +from flytekit import task, workflow + +@task +def impute(value: float) -> float: + # Replace missing/negative sentinel values with 0. + return value if value >= 0 else 0.0 + +@task +def scale(value: float) -> float: + return value / 100.0 + +@workflow +def preprocess(value: float) -> float: + imputed = impute(value=value) + return scale(value=imputed) + +@workflow +def main(raw_value: float) -> float: + return preprocess(value=raw_value) +``` + +### Flyte 2 + +```python +import flyte + +env = flyte.TaskEnvironment(name="subworkflow") + +@env.task +def impute(value: float) -> float: + # Replace missing/negative sentinel values with 0. + return value if value >= 0 else 0.0 + +@env.task +def scale(value: float) -> float: + return value / 100.0 + +# A preprocessing "subworkflow" is just a task that calls other tasks. +@env.task +def preprocess(value: float) -> float: + imputed = impute(value) + return scale(imputed) + +@env.task +def main(raw_value: float) -> float: + return preprocess(raw_value) +``` + +## TaskEnvironment Configuration + +The `TaskEnvironment` holds the configuration that Flyte 1 spread across the `@task` decorator. The task decorator can still override a few settings per-task. + +```python +import flyte + +env = flyte.TaskEnvironment( + name="my_env", # Required: unique name + image=flyte.Image.from_debian_base(), # Or a string, or "auto" + resources=flyte.Resources( + cpu="2", + memory="4Gi", + gpu="A100:1", + disk="10Gi", + ), + env_vars={"LOG_LEVEL": "INFO"}, + secrets=[flyte.Secret(key="api-key", as_env_var="API_KEY")], + cache="auto", # "auto", "override", "disable", or a Cache object + reusable=flyte.ReusePolicy(replicas=5, idle_ttl=60), + interruptible=True, +) + +# The task decorator can override some settings: +@env.task( + short_name="my_task", # Display name + cache="disable", # Override cache + retries=3, # Retry count + timeout=3600, # Seconds or a timedelta + report=True, # Generate an HTML report +) +def my_task(x: int) -> int: + return x +``` + +## Parameter Mapping: `@task` to `TaskEnvironment` + `@env.task` + +| Flyte 1 `@task` parameter | Flyte 2 location | Notes | +|---|---|---| +| `container_image` | `TaskEnvironment(image=...)` | Env-level only | +| `requests` | `TaskEnvironment(resources=...)` | Env-level only | +| `limits` | `TaskEnvironment(resources=...)` | Combined with requests (single value) | +| `environment` | `TaskEnvironment(env_vars=...)` | Env-level only | +| `secret_requests` | `TaskEnvironment(secrets=...)` | Env-level only | +| `cache` | Both | Can override at task level | +| `cache_version` | `flyte.Cache(version_override=...)` | In a `Cache` object | +| `retries` | `@env.task(retries=...)` | Task-level only | +| `timeout` | `@env.task(timeout=...)` | Task-level only | +| `interruptible` | Both | Can override at task level | +| `pod_template` | Both | Can override at task level | +| `deprecated` | N/A | Not in Flyte 2 | +| `docs` | `@env.task(docs=...)` | Task-level only | + +For image, resource, secret, and caching detail, see the Task configuration migration page. + +## Anti-Patterns + +1. **Don't reach for `>>`** — the ordering operator is gone. Sequential synchronous calls already run in written order; `await` async tasks in sequence for the same effect. +2. **Don't look for a `@workflow` decorator** — there isn't one. The orchestrating entrypoint is just another `@env.task` that calls other tasks. +3. **Don't look for a `@dynamic` decorator** — dynamic graphs also collapse into ordinary `@env.task` functions that call other tasks at runtime. +4. **Don't put heavy compute in the orchestrating task** — keep the entrypoint task focused on calling other tasks; push CPU/GPU/memory-intensive work into leaf tasks whose resources you can tune per environment. +5. **Don't set image, resources, or secrets on the `@env.task` decorator** — those are env-level and belong on `TaskEnvironment`. Only per-task overrides like `retries`, `timeout`, `cache`, and `short_name` go on `@env.task`. +6. **Don't keep passing every argument by keyword** — Flyte 2 task calls accept positional arguments (`say_hello(name)`). diff --git a/plugins/flyte/skills/flyte-migrate/SKILL.md b/plugins/flyte/skills/flyte-migrate/SKILL.md new file mode 100644 index 0000000..51116f7 --- /dev/null +++ b/plugins/flyte/skills/flyte-migrate/SKILL.md @@ -0,0 +1,353 @@ +--- +name: flyte-migrate +description: Entry-point orchestrator for porting Flyte 1 (flytekit) code to Flyte 2 (flyte). Explains the v1 to v2 shift, the terminology mapping, a recommended migration strategy, hybrid v1/v2 pipelines, and routes to sibling migration skills. Use when the user wants to migrate, port, or upgrade Flyte 1 (flytekit) code to Flyte 2. Trigger words are migrate, flytekit, v1 to v2, port, upgrade, convert workflow. +--- + +# Flyte 1 to 2 Migration Skill + +This is the entry point for migrating a Flyte 1 (`flytekit`) codebase to Flyte 2 (`flyte`). Flyte 2 is a fundamental shift: there is no `@workflow` decorator, everything is a `@env.task`, orchestration runs as real Python at runtime, and parallelism is expressed with `asyncio`. This skill explains the overall shift, gives a recommended migration strategy, covers hybrid v1/v2 pipelines during the transition, and routes to the sibling skills that handle each theme in depth. + +## Grounding References + +| Resource | URL | +|---|---| +| Migration guide | https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/ | +| Official docs | https://www.union.ai/docs/v2/flyte | +| Docs index (LLMs) | https://www.union.ai/docs/v2/flyte/llms.txt | +| SDK API reference | https://www.union.ai/docs/v2/union/api-reference/flyte-sdk/ | +| Example code | https://github.com/unionai/unionai-examples | +| Flyte MCP tools | Available via `flyte-mcp` server | + +## The overall v1 to v2 shift + +Two conceptual shifts motivate almost every change — **pure Python execution** and the **asynchronous model** — after which most migrations come down to a couple of mechanical moves. + +- **`flytekit` (package) becomes `flyte`.** Imports change from `import flytekit` to `import flyte`. +- **`pyflyte` (CLI) becomes `flyte`.** The command-line tool was renamed. +- **Everything is a task.** In Flyte 1, `@workflow` functions were constrained to a DSL subset of Python that compiled to a static DAG. In Flyte 2 there is **no `@workflow` decorator**: everything is a `@env.task`, and a "workflow" is simply a task that calls other tasks. Loops, conditionals, and `try`/`except` work anywhere. +- **Async is the parallelism model.** Flyte 2 is built on `asyncio`, with the Flyte orchestrator acting as the event loop, scheduling awaited tasks across distributed infrastructure. `await` signals where a task can be scheduled in parallel, and `asyncio.gather` tells the orchestrator that a set of tasks are independent. + +### Simplified API mapping + +| Use case | Flyte 1 | Flyte 2 | +| --- | --- | --- | +| Environment management | `N/A` | `TaskEnvironment` | +| Perform basic computation | `@task` | `@env.task` | +| Combine tasks into a workflow | `@workflow` | `@env.task` | +| Create dynamic workflows | `@dynamic` | `@env.task` | +| Fanout parallelism | `flytekit.map` | Python `for` loop with `asyncio.gather` | +| Conditional execution | `flytekit.conditional` | Python `if-elif-else` | +| Catching workflow failures | `@workflow(on_failure=...)` | Python `try-except` | + +## Terminology and concept mapping + +Several Flyte 1 concepts were renamed or reshaped in Flyte 2. The table below maps the ones you'll meet most often. + +| Flyte 1 | Flyte 2 | Notes | +|---|---|---| +| `flytekit` (package) | `flyte` (package) | The Python SDK was renamed; imports change from `import flytekit` to `import flyte`. | +| `pyflyte` (CLI) | `flyte` (CLI) | The command-line tool was renamed. | +| `@task` / `@workflow` / `@dynamic` | `@env.task` | A single task decorator off a `flyte.TaskEnvironment`. Workflows and dynamic tasks are no longer distinct constructs: everything is a task, and orchestration is plain Python. | +| `map_task()` | `flyte.map()` | Plus `asyncio.gather()` for async fan-out. | +| `conditional()` | native `if` / `elif` / `else` | Branching is now ordinary Python control flow, not a DSL. | +| `ImageSpec` | `flyte.Image` | Container image definition. | +| `current_context()` | `flyte.ctx()` | Runtime context access. | +| `FlyteFile` / `FlyteDirectory` | `flyte.io.File` / `flyte.io.Dir` | Offloaded file and directory references. | +| `StructuredDataset` | `flyte.io.DataFrame` | Offloaded tabular data. | +| `LaunchPlan` | `flyte.Trigger` | Scheduling and parameterized entry points. | +| `CronSchedule` | `flyte.Cron` | Cron-based scheduling, used with a `flyte.Trigger`. | +| Decks (`enable_deck=True`) | Reports (`report=True`) | Custom HTML rendered in the UI during/after a run. | + +## The two mechanical changes behind (almost) every migration + +Most of a migration comes down to two moves. + +### 1. Move task configuration into a `TaskEnvironment` + +Instead of configuring the image, resources, and caching on each task decorator, configure them once on a `flyte.TaskEnvironment` and share it across tasks: + +```python +env = flyte.TaskEnvironment( + name="training", + image=flyte.Image.from_debian_base().with_pip_packages("scikit-learn", "pandas"), + resources=flyte.Resources(cpu="2", memory="4Gi"), + cache="auto", +) +``` + +### 2. Replace `@task` / `@workflow` / `@dynamic` with `@env.task` + +Every decorated function becomes an `@env.task`. There is no separate workflow or dynamic construct: a "workflow" is simply a task that calls other tasks, and orchestration is plain Python. The `env` in `@env.task` is just the variable you assigned your `TaskEnvironment` to — name it whatever you like. + +## Package imports + +The package is renamed from `flytekit` to `flyte`, and the workflow/dynamic/map_task imports disappear. + +### Flyte 1 + +```python +import flytekit +from flytekit import task, workflow, dynamic, map_task +from flytekit import ImageSpec, Resources, Secret +from flytekit import current_context, LaunchPlan, CronSchedule +``` + +### Flyte 2 + +```python +import flyte +from flyte import TaskEnvironment, Resources, Secret +from flyte import Image, Trigger, Cron +``` + +## Before and after: pure Python execution + +### Flyte 1 + +```python +import flytekit + +image = flytekit.ImageSpec( + name="hello-world-image", + packages=["requests"], +) + +@flytekit.task(container_image=image) +def mean(data: list[float]) -> float: + return sum(list) / len(list) + +@flytekit.workflow +def main(data: list[float]) -> float: + output = mean(data) + + # ❌ performing trivial operations in a workflow is not allowed + # output = output / 100 + + # ❌ if/else is not allowed + # if output < 0: + # raise ValueError("Output cannot be negative") + + return output +``` + +### Flyte 2 + +```python +import flyte + +env = flyte.TaskEnvironment( + "hello_world", + image=flyte.Image.from_debian_base().with_pip_packages("requests"), +) + +@env.task +def mean(data: list[float]) -> float: + return sum(data) / len(data) + +@env.task +def main(data: list[float]) -> float: + output = mean(data) + + # ✅ performing trivial operations in a workflow is allowed + output = output / 100 + + # ✅ if/else is allowed + if output < 0: + raise ValueError("Output cannot be negative") + + return output +``` + +## Quick reference: minimal Flyte 2 module + +```python +import asyncio +import flyte + +# 1. Define an image +image = ( + flyte.Image.from_debian_base(python_version=(3, 11)) + .with_pip_packages("pandas", "numpy") +) + +# 2. Create a TaskEnvironment +env = flyte.TaskEnvironment( + name="my_env", + image=image, + resources=flyte.Resources(cpu="1", memory="2Gi"), +) + +# 3. Define tasks +@env.task +async def process(x: int) -> int: + return x * 2 + +# 4. Define the entrypoint task +@env.task +async def main(items: list[int]) -> list[int]: + results = await asyncio.gather(*[process(x) for x in items]) + return list(results) + +# 5. Run it +if __name__ == "__main__": + flyte.init_from_config() + run = flyte.run(main, items=[1, 2, 3, 4, 5]) + print(run.url) + run.wait() +``` + +```bash +# CLI +flyte run my_module.py main --items '[1,2,3,4,5]' # remote (default) +flyte run --local my_module.py main --items '[1,2,3,4,5]' +flyte deploy my_module.py my_env +``` + +## Recommended migration strategy + +Migrations rarely happen all at once. Work incrementally and lean on hybrid pipelines while the transition is in progress. + +1. **Assess the codebase.** Inventory every `@task`, `@workflow`, `@dynamic`, and `map_task`; the images (`ImageSpec`), resources, and secrets; the control-flow constructs (`conditional`, `on_failure`, `>>`); the data types (`FlyteFile`, `FlyteDirectory`, `StructuredDataset`); and any schedules (`LaunchPlan`, `CronSchedule`). +2. **Establish the `TaskEnvironment`(s).** Group tasks by their image/resource/cache needs and define a `flyte.TaskEnvironment` for each group. This is mechanical change #1 and unblocks everything else. +3. **Port leaf tasks first, then orchestration.** Convert atomic compute tasks (`@task` → `@env.task`), then rebuild the `@workflow`/`@dynamic` orchestration as plain-Python driver tasks that call them. +4. **Migrate control flow and I/O.** Replace `conditional()` with `if`/`elif`/`else`, `on_failure` with `try`/`except`, `map_task` with `flyte.map` / `asyncio.gather`, and the `FlyteFile`/`FlyteDirectory`/`StructuredDataset` types with their `flyte.io` equivalents. +5. **Update config, CLI, and schedules.** Swap `pyflyte` for `flyte`, migrate config files, and convert `LaunchPlan`/`CronSchedule` to `flyte.Trigger`/`flyte.Cron`. +6. **Run hybrid during the transition.** Keep unported v1 workflows callable via bridge tasks (see below) until every piece is on v2. + +### Sibling skills to route to + +Migrate by theme. Start with tasks and workflows, then jump to whatever the workload needs: + +- **`flyte-migrate-tasks-workflows`** — the structural shift: `@task`/`@workflow` → `@env.task`, sequential ordering, nested "subworkflows", and the `@task` → `TaskEnvironment` parameter mapping. +- **`flyte-migrate-config`** — moving image/resources/cache to the `TaskEnvironment`, GPUs, secrets, caching, scheduling with triggers, and the `pyflyte` → `flyte` command/config-file changes. +- **`flyte-migrate-control-flow`** — `conditional()` and `@dynamic` become plain Python `if`/loops, `on_failure` becomes `try`/`except`, and `map_task` → `flyte.map` / `asyncio.gather`. +- **`flyte-migrate-data-io`** — `FlyteFile`/`FlyteDirectory` → `flyte.io.File`/`Dir`, `StructuredDataset` → `flyte.io.DataFrame`, dataclasses, and ETL patterns. +- **`flyte-migrate-ml`** — small-model training, hyperparameter optimization, deep learning, batch inference, and end-to-end pipelines. + +## Hybrid v1 and v2 pipelines + +For a while you'll have Flyte 1 and Flyte 2 workloads running side by side, and you'll want them to call each other: a Flyte 1 workflow that kicks off a newly ported Flyte 2 task, or a Flyte 2 task that triggers a workflow that hasn't been migrated yet. + +You can bridge the two in both directions. The idea is the same each way: one task installs **both** SDKs, authenticates to the **other** control plane, fetches the entity it wants to run, and launches it. Keep the bridging task lightweight and focused on orchestration. + +### Running a Flyte 2 task from a Flyte 1 workflow + +The bridge is a single Flyte 1 task that runs the Flyte 2 client. Give it an image with **both** `flytekit` and `flyte` installed, provide a Flyte 2 API key as a secret, authenticate inside the task with `flyte.init_from_api_key()`, fetch the deployed task with `flyte.remote.Task.get(...)`, and run it. + +```python +import flytekit +from flytekit import task, workflow, ImageSpec, Secret, current_context + +# The bridge image needs BOTH the v1 (flytekit) and v2 (flyte) SDKs. +bridge_image = ImageSpec( + name="v1-to-v2-bridge", + packages=["flytekit", "flyte"], +) + +@task( + container_image=bridge_image, + secret_requests=[Secret(group="flyte", key="flyte_api_key")], +) +def launch_v2_from_v1(x: int) -> str: + import flyte + import flyte.remote + + # Authenticate to the Flyte 2 control plane with the API key. + # Option A: read the mounted secret and pass it explicitly. + api_key = current_context().secrets.get(group="flyte", key="flyte_api_key") + flyte.init_from_api_key(api_key=api_key) + + # Option B: if FLYTE_API_KEY is set as an env var, no argument is needed: + # flyte.init_from_api_key() + + # Fetch the deployed Flyte 2 task and run it. + remote_v2_task = flyte.remote.Task.get( + "my_v2_env.process", + auto_version="latest", + ) + run = flyte.run(remote_v2_task, x=x) + run.wait() # optional: block until the v2 run finishes + return run.url + +@workflow +def main(x: int) -> str: + return launch_v2_from_v1(x=x) +``` + +The referenced Flyte 2 task (`my_v2_env.process` above) must be **deployed** before the bridge runs. Use `flyte.init_from_api_key()` here — do **not** use `flyte.init_from_config()`, which reads a `config.yaml` that has no API-key field. + +### Running a Flyte 1 workflow from a Flyte 2 task + +The reverse works the same way: a Flyte 2 task installs the Flyte 1 client and uses `FlyteRemote` to launch a Flyte 1 workflow. + +```python +import flyte + +env = flyte.TaskEnvironment( + name="v2_to_v1_bridge", + # The image needs the Flyte 1 client installed. + image=flyte.Image.from_debian_base().with_pip_packages("flytekit"), + # Supply credentials for the Flyte 1 control plane (config or API key). + secrets=[flyte.Secret(key="v1_client_secret", as_env_var="V1_CLIENT_SECRET")], +) + +@env.task +async def launch_v1_from_v2(x: int) -> str: + from flytekit.remote import FlyteRemote + from flytekit.configuration import Config + + # Point the client at your Flyte 1 cluster. + remote = FlyteRemote( + config=Config.for_endpoint(endpoint="my-v1-cluster.example.com"), + default_project="flytesnacks", + default_domain="development", + ) + + # Fetch the deployed Flyte 1 workflow and execute it. + wf = remote.fetch_workflow(name="my_v1_module.main", version="v1.2.3") + execution = remote.execute(wf, inputs={"x": x}, wait=True) + return execution.id.name +``` + +### Hybrid considerations + +- **Both SDKs in one image.** The bridging task installs `flytekit` and `flyte` together. Pin versions and watch for dependency conflicts; keep the bridge image minimal. +- **Deploy the callee first.** For v1→v2, the Flyte 2 task must be deployed (`flyte deploy`) before `flyte.remote.Task.get()` can resolve it. For v2→v1, the Flyte 1 workflow must be registered on its cluster. +- **Wait vs. fire-and-forget.** Both `run.wait()` (v2) and `execute(..., wait=True)` (v1) block until the launched run finishes. Omit them to launch and return immediately. +- **Credentials cross a boundary.** The bridge authenticates to a *different* control plane than the one it runs on. Store the API key or client credentials as a secret — never hard-code them. +- **Keep the bridge lightweight.** Like any orchestrating task, it should mostly launch and assemble results rather than do heavy compute. + +## Gotchas + +Flyte 2 lets each Python task act as its own engine, launching sub-tasks and assembling their outputs. That flexibility warrants some caveats. + +### Common gotchas + +- **`flyte.map` returns a generator.** Wrap it in `list()` to materialize results, unlike `map_task` which returned a list directly. +- **`memory`, not `mem`.** The `Resources` parameter was renamed, and there are no separate `requests`/`limits` — a single value serves as both. +- **GPUs use a `"T4:1"` string.** Type and count are combined; the separate `accelerator=` argument is gone. +- **Image, resources, and cache live on the `TaskEnvironment`.** Set them once at the env level instead of repeating them on every task decorator. +- **`current_context()` is gone.** Read secrets from environment variables and use `flyte.ctx()` for runtime context. +- **The `>>` ordering operator is gone.** Sequential (sync) calls and sequential `await`s are naturally ordered. +- **Retries no longer have a platform cap.** In Flyte 1 the control plane capped attempts at 3; in Flyte 2 total attempts equal `retries + 1`. Audit any large `retries` values before deploying. +- **You can only `await` async tasks.** Call a sync task from an async context with `.aio()`. +- **Pick an entrypoint task name.** There's no `@workflow`, so the top-level task is just a task (commonly `main`); run it with `flyte run module.py main`. +- **Type annotations are more lenient.** Flyte 2 will pickle untyped I/O rather than rejecting it at registration. +- **Keep orchestration lightweight.** A task that calls other tasks acts as a driver pod. Avoid heavy CPU work in it. + +## Anti-Patterns + +1. **Don't introduce non-determinism into orchestration.** When a task launches another task, a new Action ID is determined as a hash of the inputs and task definition — consistent hashing is what makes recovery and replay work. Branching on `datetime.now()` or other non-deterministic values breaks that guarantee: on retry, a *different* downstream task may get kicked off. If non-determinism is unavoidable, decorate sub-task functions with `@trace` for fine-grained checkpointing and observability. +2. **Don't do heavy compute in a driver task.** When a task runs other tasks and assembles their outputs, it becomes a driver pod (work that Flyte Propeller did in v1). A CPU-bound function between two `await`s makes the driver pod hang and slows downstream kickoff. Keep parent tasks focused on orchestration: + +```python +@env.task +async def t_main(): + await t1() + local_cpu_intensive_function() # ❌ blocks the driver pod between t1 and t2 + await t2() +``` + +3. **Don't rely on global state across tasks.** Each task runs in its own isolated container; globals are not carried across task containers. Any state that must persist has to be reconstructable through repeated deterministic execution. +4. **Don't materialize huge in-memory I/O between tasks.** Outputs are materialized in the parent pod's memory, so passing a 1 GB `list[float]` requires the pod to hold all of it, risking OOM. Use `flyte.io.File`, `flyte.io.Dir`, and `flyte.io.DataFrame` — they're materialized only as pointers to offloaded data, so their memory footprint stays low. +5. **Don't skip type hints at the "workflow" level.** The top-level task now runs at runtime, so the system can't guarantee type safety across the DAG the way the v1 DSL did. Use Python type hints and a type checker like `mypy` at all levels, including the top-most task.