Skip to content

docs: turn automation creation into a phased interview, close reliability gaps - #455

Draft
VascoSch92 wants to merge 1 commit into
mainfrom
vasco/automation-skill-interview-and-reliability
Draft

docs: turn automation creation into a phased interview, close reliability gaps#455
VascoSch92 wants to merge 1 commit into
mainfrom
vasco/automation-skill-interview-and-reliability

Conversation

@VascoSch92

Copy link
Copy Markdown
Member

Summary

Based on an analysis of 72 real conversations that invoked this skill, cross-referenced with production run data (54k runs) from dev-oss-automation-vm. The skill's guidance was sound but agents routinely skipped the explicit user-facing checkpoints it mandates.

  • Interview-style creation flow: Automation Creation Process is now four explicit phases — Discovery (trigger, behavior, LLM-vs-deterministic call stated out loud, access checks, reachability, stakes) → Plan Presentation (shows the plan, ends the turn — never deploys in the same turn it's presented) → Deploy & Verify (dispatches one test run and polls to a terminal state before declaring success) → Report.
  • OPENHANDS_HOST moved to the very first thing the agent reads, ahead of everything else.
  • Sub-5-minute cron upgraded from a soft suggestion to a hard default to push back on, backed by real production numbers (every-minute automations failed 16–79% of the time vs. 1–31% for 5-minute-or-looser schedules).
  • Timeout-vs-cron-interval overlap note added (guards against the concurrent-run races seen in production).
  • Retry-with-backoff helper wired into the no-LLM script boilerplate (network fragility is the dominant real-world failure mode).
  • Tarball-integrity check added before upload (a corrupted tarball previously uploaded and created the automation fine, then failed silently on the first run).
  • Pre-flight health-check pattern added as a recommended custom-script pattern.
  • High-stakes-automation confirmation and secret-hygiene-in-prompts guidance folded into the existing Security Considerations section from docs: add Security Considerations section to openhands-automation skill #429 (money-moving/credential-spending/irreversible actions need a restated, specific confirmation — not a bare "yes" — and secrets must never be embedded verbatim in a persisted prompt or script).

Test plan

  • npm run build:skills — regenerated skills/index.js to match the updated SKILL.md content
  • uv run -p 3.12 --group test pytest tests/test_skills_catalog.py tests/test_sync_extensions.py tests/test_skills_have_readme.py tests/test_skill_plugin_loading.py tests/test_catalog_schema.py tests/test_catalogs.py — 236 passed
  • Manual read-through of both edited files for internal consistency (done locally; worth a second pass in review)

…lity gaps

Analysis of 72 real conversations that invoked this skill, cross-referenced
with production run data (54k runs) from dev-oss-automation-vm, found the
skill's guidance is sound but agents routinely skip the explicit
user-facing checkpoints it mandates: presenting a plan and deploying happen
in the same turn in the large majority of real creations, the cost-tradeoff
rule is reasoned about internally but rarely stated, and automations that
return a valid id often fail on every real run afterward with nothing
catching it.

Restructures Automation Creation Process into four explicit phases -
Discovery (trigger, behavior, LLM-vs-deterministic call, access checks,
stakes), Plan Presentation (ends the turn, no same-turn deploy), Deploy &
Verify (dispatch one test run and poll to a terminal state before
declaring success), Report. Also: moves OPENHANDS_HOST to the first thing
the agent reads, upgrades the sub-5-minute cron warning to a hard default
backed by the real 16-79% vs 1-31% failure-rate split, adds a
timeout-vs-cron-interval overlap note, wires a retry-with-backoff helper
into the no-LLM boilerplate, adds a tarball-integrity check before upload,
adds a pre-flight health-check pattern, and folds high-stakes-automation
confirmation and secret-hygiene-in-prompts guidance into the existing
Security Considerations section from #429.
@github-actions github-actions Bot added the type: docs Documentation only changes label Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: docs Documentation only changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant