Skip to content

feat: on-demand property/brand refresh + discovery improvements#2078

Merged
bokelley merged 3 commits into
mainfrom
bokelley/property-discovery-bug
Apr 11, 2026
Merged

feat: on-demand property/brand refresh + discovery improvements#2078
bokelley merged 3 commits into
mainfrom
bokelley/property-discovery-bug

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Summary

  • Refresh buttons on property-viewer and brand-viewer pages so members can trigger immediate re-crawl of adagents.json and brand.json (resolves escalation Add Deals Protocol for inventory package discovery and activation (formerly known as Curation Protocol) #218 — gatavo.com not visible despite valid adagents.json)
  • New endpoint POST /api/registry/brand-crawl-request for on-demand brand.json re-crawl, matching existing crawl-request pattern
  • Auto-crawl when members add publisher domains to their profile (with SSRF validation + rate cap)
  • Enqueue verified WorkOS domains into catalog_crawl_queue for the 30-minute discovery cycle
  • Shared SSRF utility validateCrawlDomain() in url-security.ts — uses existing validateHostResolution (IPv4+IPv6 checks) instead of inline IPv4-only DNS check
  • crawlSingleDomain now also scans brand.json via extracted scanBrandForDomain() method

Test plan

  • TypeScript compiles cleanly
  • All 565 unit tests pass
  • Playwright browser tests: refresh buttons hidden when logged out, visible when logged in, click behavior correct (text changes, button disables, handles rate limit response)
  • Local server testing: both endpoints return 202, rate limiting works (shared domain key), SSRF rejects private IPs and invalid domains, auth required
  • Code review: addressed SSRF bypass in member-profiles, duplicate code extraction, domain normalization consistency
  • Security review: addressed all Must Fix (SSRF on auto-crawl, unverified domain enqueue) and Should Fix (rate limit key namespace, domain format validation)

🤖 Generated with Claude Code

bokelley and others added 3 commits April 11, 2026 10:18
…provements

Add refresh buttons to property-viewer and brand-viewer pages so members
can trigger immediate re-crawl of adagents.json and brand.json instead of
waiting for the 6-hour crawl cycle. Auto-crawl when members add publisher
domains to their profile, and enqueue verified WorkOS domains for discovery.

- Extract scanBrandForDomain() from batch method; crawlSingleDomain now scans both files
- New POST /api/registry/brand-crawl-request endpoint (rate-limited, SSRF-safe)
- Extract validateCrawlDomain() to shared url-security utility (IPv4+IPv6)
- Auto-crawl on member profile publisher domain save (with SSRF validation)
- Enqueue verified WorkOS domains into catalog_crawl_queue

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant