add migration skills

.github/workflows/skill-evals.yml

@@ -0,0 +1,92 @@
+name: skill-evals
+
+# Runs the flyte-agent-plugin skills eval harness. On PRs it runs only the
+# scenarios affected by the changed files (see evals/select.py); nightly it runs
+# the full matrix including the `real` tier.
+on:
+  pull_request:
+    paths:
+      - "plugins/**"
+      - "evals/**"
+  schedule:
+    - cron: "0 7 * * *"   # nightly full matrix (incl. real tier)
+  workflow_dispatch: {}
+
+env:
+  PYTHONPATH: ${{ github.workspace }}
+
+jobs:
+  # 1) Decide what to run from the diff.
+  select:
+    runs-on: ubuntu-latest
+    outputs:
+      skills: ${{ steps.sel.outputs.skills }}
+      run_all: ${{ steps.sel.outputs.run_all }}
+      run_kind: ${{ steps.sel.outputs.run_kind }}
+      run_real: ${{ steps.sel.outputs.run_real }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install pyyaml requests
+      - id: sel
+        run: |
+          if [ "${{ github.event_name }}" = "pull_request" ]; then
+            OUT=$(python -m evals.select --base "origin/${{ github.base_ref }}")
+          else
+            # nightly / manual: run everything
+            OUT=$(python -m evals.select --changed evals/manifest.yaml)
+          fi
+          echo "$OUT"
+          python - "$OUT" <<'PY' >> "$GITHUB_OUTPUT"
+          import json, sys
+          d = json.loads(sys.argv[1])
+          print("skills=" + json.dumps(d["skills"]))
+          print("run_all=" + str(d["run_all"]).lower())
+          print("run_kind=" + str(d["run_kind"]).lower())
+          print("run_real=" + str(d["run_real"]).lower())
+          PY
+
+  # 2) Static + trajectory tiers, orchestrated on demo.hosted via Flyte.
+  flyte-evals:
+    needs: select
+    if: needs.select.outputs.skills != '[]'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install flyte>=2.5.0 pyyaml requests
+      - name: Configure Flyte auth
+        run: echo "auth via UNION_API_KEY secret"
+        env:
+          UNION_API_KEY: ${{ secrets.UNION_API_KEY }}
+      - name: Run evals on demo.hosted
+        env:
+          UNION_API_KEY: ${{ secrets.UNION_API_KEY }}
+          GLM_API_KEY: ${{ secrets.GLM_API_KEY }}
+          TIERS: ${{ github.event_name == 'schedule' && '["static","trajectory","real"]' || '["static","trajectory"]' }}
+        run: |
+          flyte --config evals/config/flyte.yaml run --copy-style loaded_modules \
+            evals/workflows/eval_wf.py main \
+            --skills '${{ needs.select.outputs.skills }}' \
+            --tiers "$TIERS"
+      # The workflow attaches the HTML scorecard to the run report; the run exits
+      # non-zero if any scenario fails (enforced inside aggregate/report).
+
+  # 3) Real kind-in-Docker smoke, only when a kind skill changed (privileged).
+  kind-smoke:
+    needs: select
+    if: needs.select.outputs.run_kind == 'true'
+    runs-on: ubuntu-latest   # GitHub-hosted runners allow privileged Docker/kind
+    steps:
+      - uses: actions/checkout@v4
+      - uses: helm/kind-action@v1
+        with:
+          install_only: true
+      - name: kind smoke
+        run: bash evals/kind_smoke/run.sh

README.md

@@ -134,6 +134,20 @@ pi install git:github.com/flyteorg/flyte-agent-plugins@<tag>         # pinned to
 | [`flyte-sdk-data`](plugins/flyte/skills/flyte-sdk-data) | Handles data engineering patterns: ETL pipelines, data processing, data quality checks, fanout/map tasks, conditions, dynamic workflows, and batch data transformations. For: ETL, Parquet, CSV, JsonlFile/Dir, schema validation. |
 | [`flyte-sdk-ml`](plugins/flyte/skills/flyte-sdk-ml) | Handles ML workload patterns: model training, hyperparameter optimization, experiment tracking, model evaluation and selection, batch inference, real-time serving, and model monitoring. For: PyTorch, scikit-learn, HuggingFace, GPU, drift detection. |
 
+### Migration (Flyte 1 → 2)
+
+Convert existing Flyte 1 (`flytekit`) code to Flyte 2. Distilled from the official
+[Flyte 1 → 2 migration guide](https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/).
+
+| Skill | Description |
+|-------|-------------|
+| [`flyte-migrate`](plugins/flyte/skills/flyte-migrate) | Start-here migration orchestrator: the `flytekit`→`flyte` shift, the terminology/concept mapping, the two mechanical changes, an incremental migration strategy, hybrid v1/v2 pipelines during transition, and the gotchas — routes to the specific skills below. |
+| [`flyte-migrate-tasks-workflows`](plugins/flyte/skills/flyte-migrate-tasks-workflows) | Migrate `@task`/`@workflow`/`@dynamic` into a single `@env.task` on a `TaskEnvironment`; sequential ordering without `>>`, nested "subworkflows" as tasks, and the parameter-mapping table. |
+| [`flyte-migrate-config`](plugins/flyte/skills/flyte-migrate-config) | Migrate task configuration (images `ImageSpec`→`flyte.Image`, resources/GPUs, `cache_version`→`cache`, secrets, `LaunchPlan`/`CronSchedule`→`Trigger`/`Cron`) and the `pyflyte`→`flyte` CLI / config files. |
+| [`flyte-migrate-control-flow`](plugins/flyte/skills/flyte-migrate-control-flow) | Replace `conditional()` with native `if`/`else`, `@dynamic` with plain Python loops, `on_failure` with `try`/`except`, and `map_task` with `flyte.map` / `asyncio.gather`. |
+| [`flyte-migrate-data-io`](plugins/flyte/skills/flyte-migrate-data-io) | Migrate data types & I/O: `FlyteFile`/`FlyteDirectory`→`flyte.io.File`/`Dir`, `StructuredDataset`→`flyte.io.DataFrame`, dataclasses/Pydantic as task I/O. |
+| [`flyte-migrate-ml`](plugins/flyte/skills/flyte-migrate-ml) | Migrate ML workloads (training, HPO, GPU/deep learning, batch inference, end-to-end pipelines) and the new-in-v2 patterns (real-time serving, apps, sandboxed execution) they unlock. |
+
 Example:
 
 ```
@@ -233,8 +247,7 @@ scripts/smoke_test_mcp.py                   # end-to-end check of the local MCP
 
 Each harness consumes a different part of this. Claude Code and Codex read the plugin
 manifests, so the **plugin name** matters to them. Hermes, opencode, and pi install skills
-by **directory path**, so `plugins/flyte/skills/…` is their interface — which is why the
-rename from `flyte-skills` touched both.
+by **directory path**, so `plugins/flyte/skills/…` is their interface.
 
 The `.mcp.json` server is Claude Code-specific; the skills themselves stay portable across
 harnesses.

evals/.gitignore

@@ -0,0 +1,3 @@
+__pycache__/
+*.pyc
+.pytest_cache/

evals/README.md

@@ -0,0 +1,84 @@
+# flyte agent plugin eval harness
+
+Automated testing & evaluation for the `flyte` agent skills. It runs a real
+agent harness (**opencode / pi / hermes**) against the union-hosted **GLM** endpoint,
+hands it a skill + a task, and scores what it produces — orchestrated as **Flyte
+workflows on `demo.hosted.unionai.cloud`**, with path-based selection so only the
+scenarios for changed skills run per PR.
+
+## Concepts
+
+- **Scenario** (`scenarios/<skill>/*.yaml`) — one declarative eval: a prompt, the
+  deterministic `checks`, an LLM-judge `rubric`, and (tier `real`) a `real_run`.
+- **Tiers** — `static` (lint the SKILL.md; no LLM), `trajectory` (run the agent with
+  side-effecting commands stubbed, judge the artifacts it produces), `real` (actually
+  `flyte run` SDK output on demo.hosted; kind stood up for real on a CI runner).
+- **Control arm** — every trajectory/real scenario runs twice: **treatment** (skill
+  installed) and **control** (skill absent). The headline metric is
+  **lift = treatment − control**, isolating the skill's contribution. A negative lift
+  is a regression signal.
+
+## Run locally
+
+```bash
+pip install pyyaml requests            # + the harness CLI(s) you want to exercise
+export PYTHONPATH=$(git rev-parse --show-toplevel)
+
+# Static lint of every skill — no LLM, no agent:
+python -m evals.harness.run --tier static
+
+# One trajectory scenario end-to-end (needs GLM_API_KEY + the harness CLI):
+export GLM_API_KEY=...     # token for the demo.hosted GLM app
+python -m evals.harness.run --scenario sdk-author-map-task --harness opencode
+
+# Everything for one skill, JSON out + scorecard:
+python -m evals.harness.run --skill flyte-sdk-author --json out.json
+python -m evals.report out.json --html scorecard.html
+```
+
+`GLM_BASE_URL` / `GLM_MODEL` / `GLM_API_KEY` configure the endpoint (see
+`harness/glm.py`). The `real` tier only submits remote runs when
+`FLYTE_EVALS_ENABLE_REAL=1` is set.
+
+## Run on demo.hosted (Flyte)
+
+```bash
+flyte --config evals/config/flyte.yaml run evals/workflows/eval_wf.py main \
+    --skills '["flyte-sdk-author"]' --tiers '["static","trajectory"]'
+```
+
+Fans out one action per (scenario × harness); the `aggregate` task attaches an HTML
+scorecard to the run's report tab. GLM creds come from the `glm-api-key` Flyte secret.
+
+## Selective execution
+
+```bash
+python -m evals.select --base origin/main        # changed files -> scenario subset
+```
+
+Emits `{run_all, skills, scenario_ids, run_kind, run_real}`. A change under a skill
+dir selects that skill's scenarios; a change to the engine (`harness/**`,
+`workflows/**`, `manifest.yaml`, …) forces the whole suite. CI wiring is in
+`.github/workflows/skill-evals.yml`.
+
+## Layout
+
+```
+manifest.yaml            cross-cutting config + shared-infra globs + skill classes
+scenarios/<skill>/*.yaml declarative eval specs
+harness/                 spec, checks, static_lint, sandbox, runners/, judge, evaluate, run
+workflows/               eval_wf.py (Flyte fan-out+aggregate), images.py
+config/flyte.yaml        demo.hosted admin/image/task config
+select.py  report.py     changed-files selector; JSON+HTML+markdown scorecard
+kind_smoke/run.sh        real kind-in-Docker smoke (privileged CI runner)
+tests/                   unit tests (no network/LLM)
+```
+
+## Status / open spikes
+
+- **GLM endpoint contract** — endpoint is live but auth-gated; confirm the exact
+  OpenAI-compatible route + auth header + model name, wire the key as a Flyte/GH
+  secret. Single point of change: `harness/glm.py`.
+- **Harness invocation** — the opencode adapter is complete; `pi` and `hermes`
+  adapters carry a best-effort headless invocation to confirm in the adapter spike
+  (`is_available()` gates uninstalled harnesses cleanly). See `harness/runners/`.

evals/__init__.py

@@ -0,0 +1 @@
+"""Testing & eval harness for the Flyte plugin skills."""

evals/config/flyte.yaml

@@ -0,0 +1,11 @@
+# Flyte config for running the eval harness on demo.hosted.unionai.cloud.
+# Referenced by evals/workflows/eval_wf.py and the GitHub Actions `flyte-evals` job.
+# Auth is supplied out-of-band (device flow / api-key env), never committed here.
+admin:
+  endpoint: dns:///demo.hosted.unionai.cloud
+image:
+  builder: remote
+task:
+  org: demo
+  project: flytesnacks
+  domain: development

evals/harness/__init__.py

@@ -0,0 +1 @@
+"""Core reusable engine: specs, checks, sandbox, runners, judge, scoring."""