feat(crawler): register publisher_properties agents into periodic probe loop

[bokelley]

Closes #4846

Problem

Agents discovered via authorization_type: 'publisher_properties' in adagents.json were never included in the periodic sales-agent probe loop. The crawler fan-out (#4840) correctly persisted per-child-publisher rows, but the declaring manager agent itself was never re-probed via list_authorized_properties in subsequent crawl cycles. That meant new child publishers added to a manager's delegation list were only discovered on a full re-crawl of the manager's adagents.json — not via the periodic probe.

Solution

1. sales_candidate registration (fanOutPublisherPropertiesAuthorizations)

After writing the per-child rows, the fan-out helper now calls upsertSalesCandidate(agentUrl, sourceDomain). The upsert uses a CASE guard so confirmed types (sales, creative, etc.) are never downgraded — only NULL or unknown rows get written as sales_candidate. This path runs from all three entry points: full cycle, manager-revalidation queue, and single-domain refresh.

2. Probe inclusion (crawlAllAgents)

Before calling crawler.crawlAgents(), the crawler fetches sales_candidate rows via getSalesCandidatesForProbe() (filtered by next_probe_after <= NOW() or NULL) and appends them to the AgentInfo[] list. The SDK then probes list_authorized_properties on those agents alongside registered agents in the same call.

3. Positive-signal promotion

Promotion requires index.getAgentAuthorizations(agentUrl) returning non-empty publisher_domains from the in-memory PropertyIndex — absence of an error is not sufficient evidence. Candidates that return no authorizations receive a recordSalesCandidateProbeFailure call applying exponential backoff: 1 day → 7 days → 30 days.

4. Migration 487

ALTER TABLE discovered_agents
  ADD COLUMN probe_failure_count INT NOT NULL DEFAULT 0,
  ADD COLUMN next_probe_after TIMESTAMPTZ;

CREATE INDEX idx_discovered_agents_sales_candidate_probe
  ON discovered_agents (next_probe_after)
  WHERE agent_type = 'sales_candidate';

5. getAllDiscoveredAgents extended

The general-purpose SELECT now includes probe_failure_count and next_probe_after so callers get the full DiscoveredAgent shape.

Files changed

File	Change
server/src/db/migrations/487_discovered_agents_sales_candidate_probe.sql	New migration
server/src/db/federated-index-db.ts	4 new methods + extended SELECT
server/src/federated-index.ts	Proxy methods for new DB methods
server/src/crawler.ts	Candidate fetch/probe/promote/backoff logic
server/tests/integration/crawler-publisher-properties-fanout.test.ts	6 new integration tests covering the full lifecycle
.changeset/crawler-sales-candidate-probe.md	Empty changeset (server-only)

Test coverage

The new integration test block (FederatedIndexDatabase — sales_candidate probe lifecycle) covers:

• upsertSalesCandidate creates a row with sales_candidate type
• Idempotent across repeated fan-out calls
• Does not downgrade confirmed sales rows
• promoteSalesCandidateToSales upgrades the row and clears backoff state
• recordSalesCandidateProbeFailure applies correct backoff intervals
• getSalesCandidatesForProbe respects next_probe_after filtering

---

Generated by Claude Code

[aao-release-bot]

LGTM. Follow-ups noted below. Right shape: gates promotion on positive publisher_domains evidence rather than absence-of-error, and the candidate state lives in the same row that eventually becomes sales — no separate table to keep in sync.

Things I checked

• Migration 492 adds two columns to discovered_agents and a partial index gated on agent_type = 'sales_candidate'. discovered_agents is a global directory (no account_id / tenant_id), so tenant-scoping doesn't apply. ADD COLUMN ... NOT NULL DEFAULT 0 is metadata-only on PG 11+. Migration number 491 is the only gap in the 482–492 sequence — not a collision.
• upsertSalesCandidate (federated-index-db.ts:751-765) — CASE guard is correct: only NULL or 'unknown' get overwritten; sales/creative/signals/buyer are preserved. Test at crawler-publisher-properties-fanout.test.ts:478-494 exercises the no-downgrade path.
• TOCTOU between getSalesCandidatesForProbe (crawler.ts:163) and promoteSalesCandidateToSales / recordSalesCandidateProbeFailure (crawler.ts:227/230) is bounded by WHERE agent_type = 'sales_candidate' in both UPDATEs (federated-index-db.ts:793,812). A concurrent promote turns the second writer's UPDATE into a no-op. Safe, but load-bearing — don't remove that gate.
• Candidate publisher_domains from the probe flow through populate. The listDiscoveredAgents() walk at crawler.ts:660 has no agent_type filter, so sales_candidate rows go through index.getAgentAuthorizations on the same pass. Comment update at L654-660 is accurate.
• Changeset present (.changeset/crawler-sales-candidate-probe.md). Empty front matter is the server-only convention used elsewhere in .changeset/. No wire-touching files, so no major/minor/patch bump required.

Follow-ups (non-blocking — file as issues)

• Backoff schedule prose drift. The docstring at federated-index-db.ts:800 says "first failure → 1 day, 1–2 failures → 7 days, 3+ → 30 days," which the SQL implements correctly via pre-increment CASE values. But the PR body, changeset, and migration header at 492_*.sql:5 all collapse it to "1 day → 7 days → 30 days," implying a three-step ladder. The actual ladder is 1d → 7d → 7d → 30d → 30d (two 7-day steps because < 3 matches pre-increment values 1 and 2). Either tighten the changeset/migration prose to match, or change < 3 to < 2. Lean toward fixing the prose — two 7-day retries before escalating to 30 is the more humane curve.
• Empty-inventory probes accrue backoff. crawler.ts:226 treats a reachable agent that returns publisher_domains: [] (e.g., a freshly-onboarded agent with no publishers yet) as a probe failure and applies backoff. After three such "failures" the agent is parked for 30 days. Worth distinguishing reachable-but-empty from unreachable in a follow-up so newly-onboarded agents don't get exiled.
• Crawler-level promotion test missing. The new test block at crawler-publisher-properties-fanout.test.ts:450-573 covers the DB state machine cleanly (insert, no-downgrade, due-filter, promote-resets, failure-increments, idempotency) but never exercises crawler.ts:221-238 — the wiring that reads PropertyIndex and decides promote-vs-backoff. A test that seeds PropertyIndex with a publisher_domains entry for the candidate URL and asserts promotion happens would close that gap.

Minor nits (non-blocking)

1. Protocol cast pattern. (candidate.protocol as 'mcp' | 'a2a') || 'mcp' at crawler.ts:170 mirrors the existing pattern at L147 — fine to leave, but a coerceProtocol(s) helper would centralize the assumption. Notable that the column has no CHECK constraint, so anything could land there.

LGTM. Follow-ups noted below.

[aao-release-bot]

⚠️ Argus review could not complete

The automated review encountered an issue (possibly reached max turns, timed out, or failed to post the final gh pr review). A human reviewer should take this PR.

View workflow run

_{This is an automated message from the Argus AI review workflow.}