docs: annotate experiment 0021 with post-v2.1.119 corrections

[maruiz93]

Summary

• Add dated correction annotations to experiment 0021 (tool scoping)
• Strikethrough on stale "No effect" claims in summary table and section 1
• Add Note blocks after summary table and in Environment section explaining the version timing
• Cross-reference RCA #5182

The experiment ran on v2.1.118 (2026-04-23 02:05 UTC). Claude Code v2.1.119 was released 21 hours later, changing tools and disallowedTools enforcement in --agent/--print sessions. Results in section 1 are accurate for v2.1.118 but no longer reflect current behavior.

Related to RCA (Root Cause Analysis) #5182 and fullsend#5203.

🤖 Generated with Claude Code

[qodo-code-review]

PR Summary by Qodo

Annotate experiment 0021 results with post-v2.1.119 tool enforcement corrections

📝 Documentation 🕐 Less than 10 minutes

AI Description

• Add dated notes clarifying experiment 0021 ran on Claude Code v2.1.118
• Strike through stale “No effect” claims for frontmatter tool scoping in --agent
• Cross-reference RCA #5182 to explain current (post-v2.1.119) behavior and caveats

High-Level Assessment

The PR’s approach—adding dated correction notes plus striking through now-stale claims—is the clearest way to preserve historical experiment results while preventing readers from applying them to current versions. Alternatives like rewriting the experiment as-if-run-on-latest would lose important historical context.

Files changed (1) +19 / -5

Documentation (1) +19 / -5

README.mdAdd v2.1.119 timing note and mark stale tool-scoping conclusions +19/-5

Add v2.1.119 timing note and mark stale tool-scoping conclusions

• Updates the summary table and section 1 to strike through the prior “No effect” claim for 'tools'/'disallowedTools' in '--agent' sessions and point readers to an explanatory note. Adds dated note blocks near the summary and environment sections explaining that v2.1.119 changed enforcement after the experiment and links to RCA #5182 for known unreliability details.

0021-tool-scoping/README.md

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 10:44 AM UTC · Completed 10:48 AM UTC
Commit: a726f0d · View workflow run →

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 19 rules

✅ Skills: writing-how-to

1. Contradictory scoping claim 🐞 Bug ≡ Correctness

Description

Section 1 now strikes the "no effect in --agent" claim but still states frontmatter
tools/disallowedTools "only work" for subagents, which conflicts with the new note stating
v2.1.119 applies them in --agent/--print sessions. This leaves the README giving contradictory
guidance about current behavior.

Code

0021-tool-scoping/README.md[R36-38]

+`tools` and `disallowedTools` in agent frontmatter ~~**have no effect in
+`--agent` sessions**~~ see note. They only work when the agent runs as
+a subagent spawned via the Agent tool.

Relevance

⭐⭐ Medium

No prior review suggestions found about resolving contradictory README guidance; only general docs
consistency fixes (PR #36).

PR-#36

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new note explicitly states v2.1.119 applies tools/disallowedTools in --agent/--print
sessions, while the updated section 1 text still claims they only work when run as a subagent,
creating a direct contradiction in the same document.

0021-tool-scoping/README.md[26-32]
0021-tool-scoping/README.md[36-38]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The README now says v2.1.119 applies `tools`/`disallowedTools` in `--agent`/`--print` sessions, but section 1 still contains an unconditional statement that these mechanisms "only work" for subagents. This is internally inconsistent.

### Issue Context
The PR introduces a dated note explaining post-v2.1.119 behavior, but the surrounding section 1 prose still reads like a universal statement rather than "v2.1.118-only" guidance.

### Fix Focus Areas
- 0021-tool-scoping/README.md[26-38]

### Suggested change
Rewrite the section-1 opening to explicitly scope the old behavior to v2.1.118 (or rewrite it to describe both behaviors), e.g.:
- "On v2.1.118, `tools`/`disallowedTools` have no effect in `--agent` sessions and only work for subagents." 
- "As of v2.1.119, they are applied in `--agent`/`--print` sessions (see note), but are unreliable; subagent behavior differs (see RCA)."

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

1. Contradictory scoping claim 🐞 Bug ≡ Correctness

Section 1 now strikes the "no effect in --agent" claim but still states frontmatter
tools/disallowedTools "only work" for subagents, which conflicts with the new note stating
v2.1.119 applies them in --agent/--print sessions. This leaves the README giving contradictory
guidance about current behavior.

Agent Prompt

### Issue description
The README now says v2.1.119 applies `tools`/`disallowedTools` in `--agent`/`--print` sessions, but section 1 still contains an unconditional statement that these mechanisms "only work" for subagents. This is internally inconsistent.

### Issue Context
The PR introduces a dated note explaining post-v2.1.119 behavior, but the surrounding section 1 prose still reads like a universal statement rather than "v2.1.118-only" guidance.

### Fix Focus Areas
- 0021-tool-scoping/README.md[26-38]

### Suggested change
Rewrite the section-1 opening to explicitly scope the old behavior to v2.1.118 (or rewrite it to describe both behaviors), e.g.:
- "On v2.1.118, `tools`/`disallowedTools` have no effect in `--agent` sessions and only work for subagents." 
- "As of v2.1.119, they are applied in `--agent`/`--print` sessions (see note), but are unreliable; subagent behavior differs (see RCA)."

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[fullsend-ai-review]

Review — approve

Summary

Clean documentation-only change that annotates experiment 0021 (Tool scoping) with correction notes for the v2.1.119 behavior changes. The approach of strikethrough + dated blockquote notes is exactly right — it preserves the experiment's historical accuracy for v2.1.118 while clearly flagging that results no longer reflect current behavior.

What changed

• Summary table (rows 20–21): Stale "No effect" claims for tools and disallowedTools in --agent sessions are struck through with "see note"
• Post-table note: Blockquote explains v2.1.119 timing, known bugs (scoped patterns stripping entire tools, subagent inheritance gap), and links RCA #5182
• Section 1 prose (line 28): Corresponding stale claim struck through
• Environment section: Blockquote reiterates version timing for readers who jump to the bottom

Checks

Check	Result
Scope	✅ Single file, docs-only, surgical edits
Frontmatter	✅ Unchanged — status: Concluded remains correct
README index	✅ No update needed (no title/status change)
Commit message	✅ Conventional Commits format (docs: prefix)
DCO sign-off	✅ Present
Markdown syntax	✅ Valid GFM strikethrough and blockquotes
Cross-references	✅ RCA #5182 link is appropriate
Injection defense	✅ No instruction-like patterns in PR body or diff

---

Labels: PR is a documentation-only annotation of experiment results

[fullsend-ai-retro]

Retro: PR #44 — Annotate experiment 0021 with post-v2.1.119 corrections

Originating PR: fullsend-ai/experiments#44

What happened

A contributor (maruiz93) opened a documentation-only PR adding correction annotations to experiment 0021 (tool scoping). The experiment originally ran on Claude Code v2.1.118 and found that tools/disallowedTools in agent frontmatter had no effect in --agent sessions. Version v2.1.119, released 21 hours later, changed this enforcement. The PR adds dated Note (2026-07) blockquotes and applies markdown strikethrough to the now-stale claims, preserving historical accuracy while flagging that results no longer reflect current behavior.

Timeline:

• 10:43 — PR opened (1 commit, 1 file, +19/−5)
• 10:44 — Qodo bot posts PR summary; fullsend review agent starts (run 29574451249)
• 10:46 — Qodo posts an inline finding ("Contradictory scoping claim", labeled as Bug/Medium) noting that section 1 still says tools "only work" for subagents while the new note says v2.1.119 applies them in --agent/--print sessions
• 10:48 — Fullsend review agent posts APPROVE review with a thorough checklist (scope, frontmatter, index, commit format, DCO, markdown syntax, cross-references, injection defense). Post-script applies ready-for-merge label
• 10:53 — PR merged by author

Assessment

Workflow quality: Good. The PR went from open to merged in 10 minutes with zero rework iterations. The review agent correctly identified it as a low-risk, docs-only, single-file change and approved with a thorough review. The agent ran on Opus via the fullsend-ai/agents harness (commit 937a16b) with docs-review and pr-review skills loaded.

Qodo finding assessment: The Qodo finding is understandable but overstated. It labels a readability concern as a "Bug." The document uses a deliberate annotative pattern — strikethrough marks historical claims as superseded, and dated blockquote notes explain what changed. The text after the PR is internally consistent when read as annotated experimental results rather than current-state guidance. The claim "They only work when the agent runs as a subagent" is the experiment's observed finding on v2.1.118, not a statement about current behavior. The fullsend agent's approval was correct.

Review agent strengths: Thorough structural verification (frontmatter, index, DCO, commit format, injection defense). Correctly assessed the PR's intent and approach. Completed in ~4 minutes.

Existing issue coverage: Multiple existing issues in fullsend-ai/fullsend already track the improvement areas touched by this workflow: #1480 (shallow docs review), #3901 (contradiction detection in docs), #1772 (defer ready-for-merge on docs PRs), #4107 (overlapping findings with Qodo), #1406 (weigh bot warnings). No novel proposals filed — see below.

Proposals

None. The workflow produced a correct outcome efficiently. All potentially relevant improvement areas are already tracked in the existing issue backlog.

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 10:55 AM UTC · Completed 11:04 AM UTC
Commit: a726f0d · View workflow run →