feat(registry): one-shot re-probe of unknown agents with extended timeout#3558
Merged
Merged
Conversation
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…eout Closes #3551 (Problem 2 quick-win for #3538). 77% of agents in the public registry currently render type='unknown'. Before designing the full retry-with-backoff system, ship a one-shot script that re-probes every currently-unknown agent with an extended 30s timeout. The output tells us whether unknown is mostly transient or mostly dead — informs the full Problem 2 PR's scope. ## Script server/scripts/reprobe-unknown-agents.ts: - Selects every agent with snapshot.inferred_type='unknown' or missing snapshot - Re-probes with 30s timeout (vs current 10s in crawler.ts:548) - Reuses the crawler's probe + write helpers — no bypass - Calls resolveAgentTypes from member-profiles.ts (#3541) so member_profiles.agents[] picks up new types - Reports {still_unknown, newly_classified by type, probe_failed, dns_failed} - Idempotent. --dry-run mode for staging validation before real runs. ## Test server/tests/unit/reprobe-unknown-agents.test.ts pins the report-shape contract across success/failure/dns-fail cases. ## Operator note Same protocol as the #3541 backfill: dry-run on staging, capture stdout, eyeball the diff, then real run. Repeat on prod. Operator paste-back of the final report into a PR comment so the full Problem 2 PR can scope from real data. ## Out of scope - Full retry-with-backoff loop with last_probe_attempt_at tracking - Mark-as-dead semantics for permanently unreachable agents Both depend on this script's output to scope correctly.
…r-agent timing #3541 has merged, so resolveAgentTypes is statically imported from member-profiles.ts (the dynamic-import workaround is gone). Adds a load-bearing silent-corruption guard: a transient probe failure on an agent that already had a snapshot row never overwrites that row back to NULL — decideWrite() routes probe_failed/dns_failed to "preserve". Adds per-agent elapsed_ms + slowest-N report so operators see the slow tail. Fails loud on missing DATABASE_URL. Test coverage: success / probe-failed (timeout) / DNS-failed / discovery_error routing / silent-corruption guard / dry-run / per-agent timing — 18/18 pass.
Previous shim was a no-op due to ESM import hoisting — imports run before module-body statements, so member-profiles.ts (transitively imported via the script's static import of resolveAgentTypes) constructed WorkOS before WORKOS_API_KEY got set. Passed locally where the env was already set; failed in CI where it wasn't. Mock member-profiles directly so WorkOS init never triggers. The script's behavior around resolveAgentTypes is exercised through the mock — unit tests pin the behavior, not the implementation. 18/18 reprobe tests pass.
EmmaLouise2018
force-pushed
the
EmmaLouise2018/registry-reprobe-unknown
branch
from
9ce641d to
a65e235
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3551.
77% of agents in the public registry currently render
type: 'unknown'. Before designing the full retry/backoff system (Problem 2 in #3538), ship a one-shot script that re-probes every currently-unknown agent with an extended timeout. The output tells us whether unknown is mostly transient (timeouts/network) or mostly dead — informs the full Problem 2 PR's scope.Script
server/scripts/reprobe-unknown-agents.ts:agent_capabilities_snapshot.inferred_type IS NULLor the snapshot row is missing.crawler.ts:548).CapabilityDiscovery.discoverCapabilities,inferTypeFromProfile,AgentSnapshotDatabase.upsertCapabilities) — no parallel probe path.{still_unknown, newly_classified: {sales, creative, signals, buying}, probe_failed, dns_failed}.--dry-runmode probes everything but writes nothing.Stack ordering / known gap
PR #3541 will export
resolveAgentTypes()fromserver/src/routes/member-profiles.tsso this script can also refreshmember_profiles.agents[]after each re-probe. As of this branch,resolveAgentTypesis not yet exported onmain. The script tries a dynamic import, logs[warn] resolveAgentTypes not exported … skipping member_profiles.agents[] refresh. Land #3541 to enable it., and continues. Once #3541 lands, the import resolves automatically — no code change here. Land #3541 first; merge order: 3540 → 3542 → 3543 → 3541 → this PR.Test
server/tests/unit/reprobe-unknown-agents.test.tspins the report-shape contract across success / probe-failed / DNS-failed cases. 5/5 pass, mirrors thecrawler-type-update-policy.test.tsstyle.Operator runbook
Same protocol as #3541's backfill:
Test plan
npx vitest run server/tests/unit/reprobe-unknown-agents.test.ts— 5/5 pass.npm run typecheck— clean.--dry-runon staging.Out of scope
last_probe_attempt_attracking — depends on this script's output to scope correctly.