Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions .github/aw/optimize-agentic-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,8 @@ Priority checks:
1. Is the task decomposable into smaller, faster sub-tasks?
2. Are there long-running tool calls that could be replaced with DataOps pre-steps?
3. Is the prompt asking the agent to do too much in one run?
4. Can `max-turns` or `timeout-minutes` be raised, or should the task be split?
4. For a large repetitive backlog, can each run process a manageable subset selected with a cache cursor or deterministic round-robin heuristic?
5. Can `max-turns` or `timeout-minutes` be raised, or should the task be split?

## Optimization Analysis Plan

Expand All @@ -100,11 +101,12 @@ After measuring token usage, produce a prioritized plan:
3. **Identify top cost drivers** — list the three most expensive phases/tool calls
4. **Apply quick wins first** — DataOps pre-steps, `gh-proxy`, `cli-proxy`, prompt trimming
5. **Sub-agent delegation** — identify repetitive per-item loops suitable for small-model workers
6. **Reuse execution experience** — preserve compact task features, configuration deltas, outcomes, costs, and diagnoses in `cache-memory` when cross-run reuse is useful; apply relevant recurring patterns to similar cases
7. **Prompt caching** — verify stable instructions and reusable experience appear before dynamic content
8. **Experiment correctness first** — add an `experiments:` entry, compare output quality first, and use `metric: "aic"` to choose among equivalent-quality variants
9. **Validate quality** — confirm the optimized run produces equivalent safe outputs
10. **Raise the per-run budget only if necessary** — consider increasing `max-ai-credits` only after all applicable optimizations have been exhausted and measured
6. **Bound repetitive work** — for very large backlogs, cap each run to a budget-safe subset and rotate through work with a persisted cache cursor or deterministic heuristic so items are not starved
7. **Reuse execution experience** — preserve compact task features, configuration deltas, outcomes, costs, and diagnoses in `cache-memory` when cross-run reuse is useful; apply relevant recurring patterns to similar cases
8. **Prompt caching** — verify stable instructions and reusable experience appear before dynamic content
9. **Experiment correctness first** — add an `experiments:` entry, compare output quality first, and use `metric: "aic"` to choose among equivalent-quality variants
10. **Validate quality** — confirm the optimized run produces equivalent safe outputs
11. **Raise the per-run budget only if necessary** — consider increasing `max-ai-credits` only after all applicable optimizations have been exhausted and measured

Present the plan clearly before making any edits. Confirm with the user before applying changes.

Expand Down
7 changes: 7 additions & 0 deletions .github/aw/token-optimization.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Apply these in order, measuring cost and quality after each change:
- [ ] **Correctness first**: compare quality before cost; use AIC or token count only to choose among equally successful variants
- [ ] **Cadence**: If the result is not time-sensitive, schedule less often (`hourly` → `daily`, `daily` → `weekly`)
- [ ] **Batching**: Prefer scheduled batch processing over reactive events when delayed processing is acceptable
- [ ] **Bounded subsets**: For large repetitive backlogs, process only a budget-safe subset per run and use a cache cursor or deterministic heuristic to rotate fairly through the remaining work
- [ ] **Telemetry**: Configure `observability.otlp` so token usage and run phases are measurable outside individual run logs
- [ ] **AgenticOps**: Add `copilot-token-audit` / `copilot-token-optimizer` workflows so the repository keeps finding waste automatically
- [ ] **Measure first**: Back every change with an `experiments:` field and `metric: "aic"` before promoting
Expand Down Expand Up @@ -305,6 +306,12 @@ The cheapest run is the one you don't execute. If a workflow doesn't need near-r

Reactive triggers (`issues:`, `pull_request:`, comment commands) suit immediate feedback. Otherwise prefer `schedule: daily on weekdays` and batch work. Typical batch-friendly tasks: triage summaries, stale backlog review, token audits, security digests. Combine with `cache-memory` or `repo-memory` to track processed items.

### Bound repetitive work to a manageable subset

Do not require one run to finish an unbounded backlog such as hundreds of lint violations. Set a per-run item, time, turn, or AI-credit budget and stop after a useful subset. Persist a compact cursor or processed-item set in `cache-memory` when stable state is available; otherwise use a deterministic heuristic such as file-path buckets, issue-number modulo, or oldest-first ordering. Rotate buckets round-robin across runs so every item eventually receives attention without repeatedly selecting the easiest items.

Keep each batch idempotent, skip items already fixed, and report the processed subset plus remaining work. Prefer smaller complete batches over a broad set of partial fixes that may exhaust the budget.

---

## Technique 7 — Measure Continuously with OpenTelemetry and AgenticOps
Expand Down