Skip to content

fix(addie): audit and remove always-available tool duplicates from set arrays#3005

Merged
bokelley merged 1 commit into
mainfrom
claude/issue-2998-always-available-audit
Apr 24, 2026
Merged

fix(addie): audit and remove always-available tool duplicates from set arrays#3005
bokelley merged 1 commit into
mainfrom
claude/issue-2998-always-available-audit

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Closes #2998

Summary

PR #2979 fixed the specific Addie hallucination where get_github_issue/draft_github_issue/create_github_issue were duplicated between ALWAYS_AVAILABLE_TOOLS and a set's tools array. The same pattern existed in 9 other places. This PR completes the systematic audit.

Root cause: When a tool appears in both ALWAYS_AVAILABLE_TOOLS and a set's tools array, the set's description (by construction) describes that capability. When the set lands in the "Capabilities Not Available" hint, Sonnet reads the description and hallucinates the capability is off — even though the tool is loaded via ALWAYS_AVAILABLE_TOOLS.

Changes

server/src/addie/tool-sets.ts

  • member.tools: remove propose_content, get_my_content, set_outreach_preference
  • content.tools: remove list_pending_content, approve_content, reject_content
  • knowledge.tools: remove get_github_issue
  • admin.tools: remove list_escalations, resolve_escalation
  • Reword member and content descriptions to not claim ownership of always-available capabilities
  • Expand ALWAYS_AVAILABLE_BLURBS with 7 new entries: get_escalation_status, propose_content, get_my_content, list_pending_content, approve_content, reject_content, set_outreach_preference

server/tests/unit/addie/tool-sets.test.ts

  • Add tools-array invariant test: asserts no tool in ALWAYS_AVAILABLE_TOOLSALWAYS_AVAILABLE_ADMIN_TOOLS appears in any set's tools array. Fails loudly with the exact set and tool name. This replaces the hand-auditing the issue requested as a "spreadsheet" — it's zero-maintenance and catches future regressions automatically.

Non-breaking justification

All removed tools remain reachable in every conversation via ALWAYS_AVAILABLE_TOOLS / ALWAYS_AVAILABLE_ADMIN_TOOLS. Removing from set arrays is strictly a deduplication with no functional change to what tools Sonnet can call. getToolsForSets builds the final tool list as a Set, so ALWAYS_AVAILABLE already guarantees these tools are loaded regardless of routing.

Expert consensus

prompt-engineer and adtech-product-expert reviewed; both approved. Key finding: propose_content/get_my_content is the highest-priority fix (content submission is the primary member workflow and the hallucination destroys trust at the moment a member is most invested).

Session: https://claude.ai/code/session_01Qyi4Qs9RU6WkxmMxQ3yjrv


Generated by Claude Code

…t arrays

Closes #2998

Removes 9 tools from set arrays where they duplicated entries in
ALWAYS_AVAILABLE_TOOLS / ALWAYS_AVAILABLE_ADMIN_TOOLS. The duplicates caused
Sonnet to hallucinate capability unavailability (e.g. "I can't submit content")
when the router didn't select the set whose description mentioned the capability.

Changes:
- member.tools: remove propose_content, get_my_content, set_outreach_preference
- content.tools: remove list_pending_content, approve_content, reject_content
- knowledge.tools: remove get_github_issue
- admin.tools: remove list_escalations, resolve_escalation
- Reword member and content descriptions to not claim ownership of always-available capabilities
- Add 7 entries to ALWAYS_AVAILABLE_BLURBS so the correction section covers reclaimed tools
- Add tools-array invariant test that unions both always-available arrays

Non-breaking: tools remain reachable via ALWAYS_AVAILABLE_TOOLS /
ALWAYS_AVAILABLE_ADMIN_TOOLS; removing from set arrays has no functional impact
on tool availability for end users.

ad-tech-protocol-expert and prompt-engineer reviewed; both approved.

Session: https://claude.ai/code/session_01Qyi4Qs9RU6WkxmMxQ3yjrv
@github-actions

Copy link
Copy Markdown
Contributor

IPR Policy Agreement Required

Thank you for your contribution! Before we can accept your pull request, you must agree to our Intellectual Property Rights Policy.

By making a Contribution, you agree that:

  • You grant the Foundation a perpetual, irrevocable, worldwide, non-exclusive, royalty-free copyright license to your Contribution
  • You grant a patent license under any Necessary Claims
  • You represent that you own or have sufficient rights to grant these licenses

To agree, please comment below with the exact phrase:

I have read the IPR Policy

You can read the full IPR Policy here.


I have read the IPR Policy


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

@bokelley

Copy link
Copy Markdown
Contributor Author

I have read the IPR Policy

@bokelley bokelley marked this pull request as ready for review April 24, 2026 11:20
@bokelley bokelley merged commit d830562 into main Apr 24, 2026
11 of 12 checks passed
@bokelley bokelley deleted the claude/issue-2998-always-available-audit branch April 24, 2026 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Audit tool-set descriptions for always-available overlap to prevent Addie hallucinations

2 participants