Skip to content

Add poison message handling for queue deserialization failures#455

Merged
niemyjski merged 1 commit into
mainfrom
feature/poison-message-handling
Feb 15, 2026
Merged

Add poison message handling for queue deserialization failures#455
niemyjski merged 1 commit into
mainfrom
feature/poison-message-handling

Conversation

@niemyjski
Copy link
Copy Markdown
Member

Summary

  • When a message fails to deserialize during dequeue, it is now abandoned through the normal retry flow instead of crashing the consumer. This gives operators a window to fix transient serializer issues (e.g., missing JsonConverters) before messages are permanently dead-lettered.
  • Adds FaultInjectingSerializer test infrastructure (implements ISerializer) for simulating serialization/deserialization failures in tests.
  • Adds two new virtual test methods to QueueTestBase: DequeueAsync_WithPoisonMessage_MovesToDeadletterAsync and EnqueueAsync_WithSerializationError_ThrowsAndLeavesQueueEmptyAsync.
  • Makes QueueEntry null-safe for poison message entries where deserialization yields null values.
  • Documents the poison message handling flow in the queues guide.

Poison Message Flow

Dequeue -> Deserialize fails -> Abandon (attempt incremented) -> Still has retries? -> Re-queued with backoff delay -> Retries exhausted? -> Moved to dead letter queue

Changes

File Change
docs/guide/queues.md New section documenting poison message handling
src/Foundatio.TestHarness/Serializer/FaultInjectingSerializer.cs New test serializer for injecting failures
src/Foundatio.TestHarness/Queue/QueueTestBase.cs New virtual tests + ISerializer param on GetQueue
src/Foundatio/Queues/QueueEntry.cs Null-safe Value handling
tests/Foundatio.Tests/Queue/InMemoryQueueTests.cs Skip new tests (no serialization), remove #region

Provider implementations (separate repos)

Each provider repo has corresponding changes to wire up the poison message handling in their DequeueImplAsync:

  • Foundatio.Redis - Catches deserialization errors, creates QueueEntry with null value, calls AbandonAsync
  • Foundatio.AzureStorage - Catches deserialization errors, abandons with attempt tracking
  • Foundatio.AzureServiceBus - Catches deserialization errors, abandons for retry
  • Foundatio.AWS - Catches deserialization errors, abandons for retry

Test plan

  • All 1799 tests pass in Foundatio.slnx (1786 succeeded, 13 skipped, 0 failed)
  • All provider repos build successfully with ReferenceFoundatioSource=true
  • InMemoryQueue tests skip serialization tests (InMemoryQueue does not use serialization)
  • Run provider-specific tests against live services (Redis, Azure Storage, Azure Service Bus, SQS)

Made with Cursor

When a message fails to deserialize during dequeue, it is now abandoned through the normal retry flow instead of crashing the consumer. This gives operators a window to fix transient serializer issues before messages are permanently dead-lettered.

- Add FaultInjectingSerializer test infrastructure for simulating failures
- Add DequeueAsync_WithPoisonMessage_MovesToDeadletterAsync base test
- Add EnqueueAsync_WithSerializationError_ThrowsAndLeavesQueueEmptyAsync base test
- Make QueueEntry null-safe for poison message entries with null values
- Add ISerializer parameter to GetQueue for test serializer injection
- Document poison message handling flow in queues guide
- Remove #region directives from InMemoryQueueTests

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant