fix: Dedupe alerts by URL to prevent spam from existing duplicates#808
Merged
Conversation
The previous fix (1760aed) prevented new duplicate perspectives from being created, but existing duplicates still matched the alert query. This caused the same article to be posted multiple times. Now the alert query checks if ANY perspective with the same external_url has been alerted to a channel, not just the specific perspective_id. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ring 1. Alert deduplication: Check external_url across ALL perspectives, not just the specific perspective_id. Prevents spam from cross-feed duplicates. 2. Content relevance: Tighten mentions_agentic detection to require BOTH agentic AI terms AND advertising context. Prevents general AI news (ChatGPT updates, etc.) from being flagged as relevant. 3. Performance: Add composite index on industry_alerts(channel_id, perspective_id) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
external_urlacross all perspectivesmentions_agenticdetection to require advertising contextindustry_alerts(channel_id, perspective_id)for query performanceProblem
Spam from duplicates: The same Adweek article was being posted 5+ times because it appeared in multiple RSS feeds. Each feed created a duplicate perspective before the cross-feed deduplication fix.
Irrelevant content: Articles about general AI (ChatGPT updates, OpenAI agents) were being flagged as relevant because they mentioned "AI agent" even without any advertising context.
Solution
Alert deduplication
Changed the NOT EXISTS subquery from checking
perspective_idto checking if ANY perspective with the sameexternal_urlhas been alerted to that channel.Content relevance
Tightened
checkMentionsAgentic()to require BOTH:agentic,ai agent,autonomous agent, etc.advertising,programmatic,media buy,dsp,campaign, etc.Before: Article about "OpenAI's new AI agent" →
mentions_agentic = trueAfter: Same article →
mentions_agentic = false(no advertising context)Test plan
🤖 Generated with Claude Code