docs(release): add manual smoke checklist (#971)#1000
Merged
senamakel merged 4 commits intotinyhumansai:mainfrom Apr 29, 2026
Merged
docs(release): add manual smoke checklist (#971)#1000senamakel merged 4 commits intotinyhumansai:mainfrom
senamakel merged 4 commits intotinyhumansai:mainfrom
Conversation
Adds docs/RELEASE-MANUAL-SMOKE.md covering OS-level surfaces drivers cannot assert: macOS TCC prompts (Accessibility, Input Monitoring, Screen Recording, Microphone), Gatekeeper, codesign verify, DMG install, Windows SmartScreen, Linux .deb/.AppImage + libnotify toasts, and cross-platform first-launch / auto-update / sign-out flows. Active release line is 0.52.x (only stream in CHANGELOG). Sign-off block at the end pastes into release PRs. Closes tinyhumansai#971 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the two `(see tinyhumansai#971)` / `not yet created — tracked in tinyhumansai#971` placeholders in docs/TESTING-STRATEGY.md with markdown links to the new docs/RELEASE-MANUAL-SMOKE.md. Refs tinyhumansai#971 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…mansai#971) Adds step 5 to the Release checklist in docs/RELEASE_POLICY.md pointing release-cutters at docs/RELEASE-MANUAL-SMOKE.md and the sign-off block that pastes into the release PR description. Refs tinyhumansai#971 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
📝 WalkthroughWalkthroughIntroduces a new release-cut manual smoke checklist document and updates release policy and testing strategy to reference and require completion of that checklist for releases, replacing a prior pending-issue reference. The checklist enumerates OS-level verifications and includes a standardized sign-off block. (44 words) Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/TESTING-STRATEGY.md (1)
121-132:⚠️ Potential issue | 🟠 MajorAvoid ambiguity: “manual smoke list” should be exhaustive or explicitly delegated to the checklist doc.
This section says: “If a feature has no automated coverage AND is not on the manual smoke list, treat it as untested.” But the subsequent bullets appear to be a partial subset of what’s in
RELEASE-MANUAL-SMOKE.md(notably Windows and several cross-platform items).Suggestion: either (a) make the bullets exhaustive (mirror the checklist), or (b) change the section to clearly state that the authoritative/manual-smoke list is the full checklist in
RELEASE-MANUAL-SMOKE.md, and treat the bullets as examples only. This is important for process correctness.✅ Possible edit direction
- Keep the current intro sentence, but add something like:
- “Authoritative list is in
docs/RELEASE-MANUAL-SMOKE.md(see full checklist); bullets below are examples.”- Or remove/condense the bullets and point directly to the checklist for the complete list.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/TESTING-STRATEGY.md` around lines 121 - 132, The "Not driver-automatable — manual smoke required" section is ambiguous about whether the bullets are exhaustive; update that section to either (a) mirror the full checklist from docs/RELEASE-MANUAL-SMOKE.md so the bullets are complete, or (b) explicitly state that the authoritative manual-smoke list is docs/RELEASE-MANUAL-SMOKE.md and change the bullets to “examples only” (e.g., prepend “examples:” and add a sentence like “See docs/RELEASE-MANUAL-SMOKE.md for the full checklist; if a feature is not on that checklist and has no automated coverage, treat it as untested”).
🧹 Nitpick comments (2)
docs/RELEASE_POLICY.md (1)
29-35: Make the “paste sign-off block” requirement more explicit/standardized.Step 5 correctly requires completing the manual checklist + pasting the sign-off block into the release PR description. To reduce drift/mis-copying, consider specifying “paste the sign-off block verbatim from
RELEASE-MANUAL-SMOKE.md” (wording-only change).🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/RELEASE_POLICY.md` around lines 29 - 35, Clarify Step 5 by requiring the exact sign-off text be copied verbatim from RELEASE-MANUAL-SMOKE.md: update the bullet to read that after completing the manual smoke checklist you must "paste the sign-off block verbatim from RELEASE-MANUAL-SMOKE.md into the release PR description" (preserve the phrase RELEASE-MANUAL-SMOKE.md and the action "paste sign-off block verbatim") so reviewers can locate the source and verify it was copied exactly.docs/RELEASE-MANUAL-SMOKE.md (1)
41-42: Fix duplicated “only” in Linux toast expectation.In the Linux “OS-native notification toasts fire” bullet, “only” appears twice (“CI Linux only sees…” and “this surface only verifies…”). It’s minor, but it’s noisy and already flagged by your static analysis. Consider rewording one occurrence.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/RELEASE-MANUAL-SMOKE.md` around lines 41 - 42, In the "OS-native notification toasts fire" bullet (the line starting with "OS-native notification toasts fire"), remove the duplicated "only" by rephrasing either "CI Linux only sees Xvfb" or "this surface only verifies on a real desktop" (for example change the latter to "this surface verifies on a real desktop") so the sentence reads clearly without the repeated word; update the bullet text accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/RELEASE-MANUAL-SMOKE.md`:
- Around line 65-71: The fenced sign-off block in RELEASE-MANUAL-SMOKE.md lacks
a language tag (triggers MD040); update the opening fence for the block that
starts with ``` to include a language label such as text or md (e.g., change the
opening fence to ```text) so the block becomes a language-tagged code fence
around the "Release: vX.Y.Z / Tester: ..." lines.
---
Outside diff comments:
In `@docs/TESTING-STRATEGY.md`:
- Around line 121-132: The "Not driver-automatable — manual smoke required"
section is ambiguous about whether the bullets are exhaustive; update that
section to either (a) mirror the full checklist from
docs/RELEASE-MANUAL-SMOKE.md so the bullets are complete, or (b) explicitly
state that the authoritative manual-smoke list is docs/RELEASE-MANUAL-SMOKE.md
and change the bullets to “examples only” (e.g., prepend “examples:” and add a
sentence like “See docs/RELEASE-MANUAL-SMOKE.md for the full checklist; if a
feature is not on that checklist and has no automated coverage, treat it as
untested”).
---
Nitpick comments:
In `@docs/RELEASE_POLICY.md`:
- Around line 29-35: Clarify Step 5 by requiring the exact sign-off text be
copied verbatim from RELEASE-MANUAL-SMOKE.md: update the bullet to read that
after completing the manual smoke checklist you must "paste the sign-off block
verbatim from RELEASE-MANUAL-SMOKE.md into the release PR description" (preserve
the phrase RELEASE-MANUAL-SMOKE.md and the action "paste sign-off block
verbatim") so reviewers can locate the source and verify it was copied exactly.
In `@docs/RELEASE-MANUAL-SMOKE.md`:
- Around line 41-42: In the "OS-native notification toasts fire" bullet (the
line starting with "OS-native notification toasts fire"), remove the duplicated
"only" by rephrasing either "CI Linux only sees Xvfb" or "this surface only
verifies on a real desktop" (for example change the latter to "this surface
verifies on a real desktop") so the sentence reads clearly without the repeated
word; update the bullet text accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c987cf3c-59a4-4407-95f6-8284ab9aa35f
📒 Files selected for processing (3)
docs/RELEASE-MANUAL-SMOKE.mddocs/RELEASE_POLICY.mddocs/TESTING-STRATEGY.md
- RELEASE-MANUAL-SMOKE.md sign-off fence: add `text` language tag (markdownlint MD040). - RELEASE-MANUAL-SMOKE.md Linux toast: rephrase duplicate "only" — "CI Linux sees only Xvfb; this surface verifies on a real desktop". - TESTING-STRATEGY.md "Not driver-automatable" section: clarify that RELEASE-MANUAL-SMOKE.md is the source of truth and the bullets are examples, not the full list. - RELEASE_POLICY.md step 5: spell out that the sign-off block is pasted verbatim with checked items intact. Refs tinyhumansai#971 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/RELEASE-MANUAL-SMOKE.md`:
- Around line 23-60: Add an explicit owner field for each checklist section (the
headers "### macOS", "### Windows", "### Linux", "### Cross-platform", and the
"### 0.52.x — current" active line subsection) so reviewers know who must sign
off; for each header, insert a short "Owner: <team-or-username>" line (or
"Owners: ..." if multiple) directly under the heading and update any checklist
items if they map to sub-owners (e.g., Accessibility → `@mac-team`, Installer →
`@windows-installer`) to make responsibility unambiguous.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 08525a60-d8d6-4c7f-83ae-7653f5bb2034
📒 Files selected for processing (3)
docs/RELEASE-MANUAL-SMOKE.mddocs/RELEASE_POLICY.mddocs/TESTING-STRATEGY.md
✅ Files skipped from review due to trivial changes (1)
- docs/TESTING-STRATEGY.md
🚧 Files skipped from review as they are similar to previous changes (1)
- docs/RELEASE_POLICY.md
Comment on lines
+23
to
+60
| ### macOS | ||
|
|
||
| - [ ] **Gatekeeper accepts the signed `.app` on first launch** — Double-click the `.app` from a fresh download (Quarantine attribute set). Expected: app opens without `"OpenHuman" cannot be opened because the developer cannot be verified` dialog. If it appears, the build is unsigned or the notarization stapler is missing. | ||
| - [ ] **`codesign --verify --deep --strict <path-to-OpenHuman.app>` exits 0** — Run from terminal. Expected: no output, exit 0. Any `code object is not signed at all` or `invalid signature` output blocks the release. | ||
| - [ ] **DMG drag-to-Applications flow works** — Mount the `.dmg`, drag `OpenHuman.app` to the `Applications` alias. Expected: copy completes; eject succeeds; first launch from `/Applications` does not re-prompt Gatekeeper. | ||
| - [ ] **Accessibility permission prompt fires on first agent run** — Trigger an agent action that uses Accessibility (e.g. window-control skill). Expected: macOS prompts `OpenHuman would like to control this computer using accessibility features`. Granting it allows the action; denying it surfaces a clear in-app fallback. | ||
| - [ ] **Input Monitoring prompt fires on first hotkey use** — Press the registered global hotkey for the first time. Expected: `Input Monitoring` prompt; granting it makes the hotkey trigger; denying it does not crash the app. | ||
| - [ ] **Screen Recording prompt fires on first screen-share** — Use the screen-share skill or `getDisplayMedia` shim. Expected: `Screen Recording` prompt; granted → picker shows windows + screens; denied → in-app message explaining the requirement. | ||
| - [ ] **Microphone prompt fires on first voice capture** — Start a voice session. Expected: standard mic prompt; granted → capture begins; denied → fallback message, no panic. | ||
|
|
||
| ### Windows | ||
|
|
||
| - [ ] **SmartScreen does not block install** — Run the installer from a fresh download. Expected: SmartScreen passes (signed binary). If `Windows protected your PC` appears, the EV signature is missing or the reputation has not built up — escalate before shipping. | ||
| - [ ] **Installer creates Start Menu + Desktop shortcuts** — Defaults preserved. Expected: both shortcuts launch the app. | ||
| - [ ] **App registers `openhuman://` URL scheme** — From a browser, click an `openhuman://oauth/success?...` link. Expected: OS prompts to open in OpenHuman; clicking through delivers the deep link. | ||
|
|
||
| ### Linux | ||
|
|
||
| - [ ] **`.deb` and/or `.AppImage` install on a clean Ubuntu 22.04** — `sudo dpkg -i openhuman_*.deb` or `chmod +x openhuman-*.AppImage && ./openhuman-*.AppImage`. Expected: no missing-dependency errors; app launches. | ||
| - [ ] **OS-native notification toasts fire** — Trigger a notification from inside the app (e.g. memory captured, agent finished). Expected: a libnotify-style toast appears outside the app window. (CI Linux sees only Xvfb; this surface verifies on a real desktop.) | ||
|
|
||
| ### Cross-platform | ||
|
|
||
| - [ ] **First launch flow completes for a brand-new user** — Fresh OS user account, no `~/.openhuman` directory. Walk through onboarding to first agent reply. Expected: no crashes, no permission deadlocks, no stale-config errors. | ||
| - [ ] **Auto-update download + relaunch succeeds** — Install the previous release, point the updater feed at this release, trigger an update check. Expected: download completes, relaunch installs the new binary, version string in `Settings > About` matches the release tag. | ||
| - [ ] **Logging out + logging back in preserves nothing private** — Sign out, sign in as a different user. Expected: no leaked memory, threads, or skill state from the previous session (regression watch — see #900). | ||
|
|
||
| --- | ||
|
|
||
| ## Active release line | ||
|
|
||
| > If multiple stable release lines are in flight (security backports, LTS), add a sub-section per line and check the same boxes for each. As of writing, `0.52.x` is the only active line — older minor versions are end-of-life. Fold this section to suit when more release lines exist. | ||
|
|
||
| ### 0.52.x — current | ||
|
|
||
| - [ ] **OAuth gate respects `VITE_MINIMUM_SUPPORTED_APP_VERSION`** (per [`RELEASE_POLICY.md`](./RELEASE_POLICY.md)) — Set the variable to a value above this build's version, build, attempt OAuth from the older binary. Expected: gate blocks the deep link; opens `VITE_LATEST_APP_DOWNLOAD_URL`. | ||
| - [ ] **Gmail connect succeeds on a fresh install from `releases/latest`** — Per release-policy step 4. Expected: token exchange completes, inbox lists in-app. | ||
|
|
Contributor
There was a problem hiding this comment.
Add explicit owner fields per checklist section.
The checklist is comprehensive, but it still lacks named ownership per section (macOS / Windows / Linux / Cross-platform / active line), which was part of the linked scope. Without this, accountability for sign-off is ambiguous.
Suggested patch
### macOS
+Owner: @<owner-handle>
- [ ] **Gatekeeper accepts the signed `.app` on first launch** — Double-click the `.app` from a fresh download (Quarantine attribute set). Expected: app opens without `"OpenHuman" cannot be opened because the developer cannot be verified` dialog. If it appears, the build is unsigned or the notarization stapler is missing.
### Windows
+Owner: @<owner-handle>
- [ ] **SmartScreen does not block install** — Run the installer from a fresh download. Expected: SmartScreen passes (signed binary). If `Windows protected your PC` appears, the EV signature is missing or the reputation has not built up — escalate before shipping.
### Linux
+Owner: @<owner-handle>
- [ ] **`.deb` and/or `.AppImage` install on a clean Ubuntu 22.04** — `sudo dpkg -i openhuman_*.deb` or `chmod +x openhuman-*.AppImage && ./openhuman-*.AppImage`. Expected: no missing-dependency errors; app launches.
### Cross-platform
+Owner: @<owner-handle>
- [ ] **First launch flow completes for a brand-new user** — Fresh OS user account, no `~/.openhuman` directory. Walk through onboarding to first agent reply. Expected: no crashes, no permission deadlocks, no stale-config errors.
### 0.52.x — current
+Owner: @<owner-handle>
- [ ] **OAuth gate respects `VITE_MINIMUM_SUPPORTED_APP_VERSION`** (per [`RELEASE_POLICY.md`](./RELEASE_POLICY.md)) — Set the variable to a value above this build's version, build, attempt OAuth from the older binary. Expected: gate blocks the deep link; opens `VITE_LATEST_APP_DOWNLOAD_URL`.🧰 Tools
🪛 LanguageTool
[style] ~41-~41: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...4** — sudo dpkg -i openhuman_*.deb or chmod +x openhuman-*.AppImage && ./openhuman-*.AppImage. Expected: no missing-dependency error...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/RELEASE-MANUAL-SMOKE.md` around lines 23 - 60, Add an explicit owner
field for each checklist section (the headers "### macOS", "### Windows", "###
Linux", "### Cross-platform", and the "### 0.52.x — current" active line
subsection) so reviewers know who must sign off; for each header, insert a short
"Owner: <team-or-username>" line (or "Owners: ..." if multiple) directly under
the heading and update any checklist items if they map to sub-owners (e.g.,
Accessibility → `@mac-team`, Installer → `@windows-installer`) to make
responsibility unambiguous.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Problem
`TESTING-STRATEGY.md` (landed in #773 / PR #980) named `docs/RELEASE-MANUAL-SMOKE.md` as the manual-smoke source-of-truth but the file did not yet exist — two `(see #971)` / `not yet created — tracked in #971` placeholders pointed at this issue. Without the file, OS-level surfaces (TCC prompts, Gatekeeper, codesign, DMG install, auto-update) had nowhere documented to land sign-offs.
Solution
`docs/RELEASE-MANUAL-SMOKE.md` ships a fixed checklist with sections per platform:
Sign-off block at the end pastes into the release PR description.
`TESTING-STRATEGY.md` placeholders replaced with markdown links. `RELEASE_POLICY.md` Release checklist gets a step 5: "Complete the manual smoke checklist and paste the sign-off block into the release PR description before tagging."
Submission Checklist
Impact
Related
Summary by CodeRabbit