Skip to content

Add integration test for message deduplication #2872

@hubcio

Description

@hubcio

Message deduplication has unit tests in MessageDeduplicator but no integration test verifying the full pipeline: client sends duplicate messages -> server drops them at partition level.

What to focus on:

  • Enable deduplication via server config (system.message_deduplication.enabled = true)
  • Send messages with explicit IDs, then resend the same IDs
  • Poll and verify only unique messages were persisted
  • Test TTL expiry: after expiry window, previously seen IDs should be accepted again

Files to reference:

  • core/common/src/deduplication/message_deduplicator.rs - core dedup logic + existing unit tests
  • core/common/src/types/message/messages_batch_mut.rs - prepare_for_persistence() dedup integration
  • core/server/src/streaming/partitions/helpers.rs - create_message_deduplicator() from config
  • core/integration/tests/server/scenarios/ - existing server scenario tests for pattern reference

Related (closed):

Done when an integration test proves that duplicate messages are dropped end-to-end with deduplication enabled, and unique messages pass through.

Metadata

Metadata

Assignees

Labels

good first issueGood for newcomersserveriggy-server related changetestNew testcase, or improvement of existing

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions