Skip to content

REQ-002 "Satisfied" overstates model-autonomous invocation — criterion verifies only the plumbing #105

Description

@AlonMalach

Motivation

The spec should not read as if a capability works when it doesn't. REQ-002 (a MUST / MVP
requirement) is marked Satisfied, implying the model can autonomously decide to switch
adapters. In reality only the mechanism is verified — a model-emitted control token takes
effect. Whether the model has learned when to invoke an adapter is unsolved. ADR-022 (Proposed)
now formally targets that gap, so REQ-002's status needs reconciling to keep the spec honest.

Gaps

  • REQ-002's satisfied_when verifies only that a model-emitted control token changes the active
    adapter (plumbing) — not that the model decides appropriately when to emit it.
  • No requirement or shipped mechanism covers a learned invocation policy bounded by precision /
    low false-invocation (not over-invoking the base).
  • INSIGHT-0001 records the trap directly: "freely generatable ≠ learned-to-emit." A reader taking
    REQ-002 at face value would conclude autonomous invocation is delivered.

Task overview

Decide how to reconcile REQ-002's status/criterion with the fact that model-autonomous invocation
(the learned when-to-invoke policy) is not yet delivered, now that ADR-022 records that goal.

What we solve

The spec accurately reflects reality: the plumbing is done; the learned invocation policy is open
work. Downstream planning (and INSIGHT-0001's warning) stop being contradicted by a Satisfied MUST.

Related requirements / ADRs

  • REQ-002 — Model-driven adapter switching (Satisfied; status in question)
  • REQ-003 — Function-call-style dynamic switching (Not satisfied; adjacent call/return goal)
  • ADR-022 — Learned when-to-invoke policy for model-autonomous adapter invocation (Proposed)
  • INSIGHT-0001 — Distinguish externally-driven adapter selection from model-autonomous invocation
  • BG-01 — Capabilities compose like software (goal REQ-002 supports)

Proposed approach (high level)

Reconcile REQ-002 via one of these (to be decided, not pre-judged):

  • Tighten REQ-002's satisfied_when to add a learned-policy criterion (precision / false-
    invocation bound, shown by an eval), flipping it Satisfied → Not satisfied until ADR-022's
    follow-up delivers a mechanism.
  • Split concerns: keep REQ-002 as the (Satisfied) plumbing requirement and add a new
    requirement for the learned invocation policy that ADR-022's mechanism would serve.
  • Leave as-is and rely on ADR-022's consequences note to record the gap (lowest effort; keeps
    the overstatement risk).

Open questions & alternatives

  • Open question: Is "model-driven switching" meant to mean the plumbing (current criterion) or the
    learned policy? The answer picks between tightening vs splitting.
  • Open question: What precision / false-invocation bound and eval would define "satisfied" for the
    policy?
  • Alternative trade-off: tightening reopens a MUST/MVP requirement (cleaner spec, but a MUST goes
    red); splitting preserves REQ-002's history at the cost of a new REQ id; leaving as-is is cheapest
    but perpetuates the overstatement.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions