docs: turn automation creation into a phased interview, close reliability gaps - #455
Draft
VascoSch92 wants to merge 1 commit into
Draft
docs: turn automation creation into a phased interview, close reliability gaps#455VascoSch92 wants to merge 1 commit into
VascoSch92 wants to merge 1 commit into
Conversation
…lity gaps Analysis of 72 real conversations that invoked this skill, cross-referenced with production run data (54k runs) from dev-oss-automation-vm, found the skill's guidance is sound but agents routinely skip the explicit user-facing checkpoints it mandates: presenting a plan and deploying happen in the same turn in the large majority of real creations, the cost-tradeoff rule is reasoned about internally but rarely stated, and automations that return a valid id often fail on every real run afterward with nothing catching it. Restructures Automation Creation Process into four explicit phases - Discovery (trigger, behavior, LLM-vs-deterministic call, access checks, stakes), Plan Presentation (ends the turn, no same-turn deploy), Deploy & Verify (dispatch one test run and poll to a terminal state before declaring success), Report. Also: moves OPENHANDS_HOST to the first thing the agent reads, upgrades the sub-5-minute cron warning to a hard default backed by the real 16-79% vs 1-31% failure-rate split, adds a timeout-vs-cron-interval overlap note, wires a retry-with-backoff helper into the no-LLM boilerplate, adds a tarball-integrity check before upload, adds a pre-flight health-check pattern, and folds high-stakes-automation confirmation and secret-hygiene-in-prompts guidance into the existing Security Considerations section from #429.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Based on an analysis of 72 real conversations that invoked this skill, cross-referenced with production run data (54k runs) from
dev-oss-automation-vm. The skill's guidance was sound but agents routinely skipped the explicit user-facing checkpoints it mandates.Automation Creation Processis now four explicit phases — Discovery (trigger, behavior, LLM-vs-deterministic call stated out loud, access checks, reachability, stakes) → Plan Presentation (shows the plan, ends the turn — never deploys in the same turn it's presented) → Deploy & Verify (dispatches one test run and polls to a terminal state before declaring success) → Report.OPENHANDS_HOSTmoved to the very first thing the agent reads, ahead of everything else.Security Considerationssection from docs: add Security Considerations section to openhands-automation skill #429 (money-moving/credential-spending/irreversible actions need a restated, specific confirmation — not a bare "yes" — and secrets must never be embedded verbatim in a persisted prompt or script).Test plan
npm run build:skills— regeneratedskills/index.jsto match the updatedSKILL.mdcontentuv run -p 3.12 --group test pytest tests/test_skills_catalog.py tests/test_sync_extensions.py tests/test_skills_have_readme.py tests/test_skill_plugin_loading.py tests/test_catalog_schema.py tests/test_catalogs.py— 236 passed