feat(connectors): add generic HTTP sink connector

[mlevkov]

Summary

Adds a generic HTTP sink connector that delivers consumed Iggy stream messages to any HTTP endpoint — webhooks, Lambda functions, REST APIs, or SaaS integrations.

• 4 batch modes: individual (one request per message), nd_json (newline-delimited), json_array (single array), raw (bytes)
• Automatic retry via reqwest-middleware: RetryTransientMiddleware with ExponentialBackoff and custom HttpSinkRetryStrategy that respects user-configured success codes
• Flexible metadata: optional Iggy envelope (stream/topic/offset/timestamp), checksum, origin timestamp, message headers
• Strongly typed serialization: MetadataEnvelope, IggyMetadata, EncodedPayload, EncodedHeader structs (no json! macros)
• Simple hex message IDs: 32-character lowercase hex string (no dashes)
• Connection pooling: ClientWithMiddleware wrapping reqwest with TCP keep-alive (30s), pool idle timeout (90s), configurable max connections
• Health checks: opt-in startup probe (HEAD/GET/OPTIONS) with graceful degradation
• TLS: optional danger_accept_invalid_certs for dev environments
• Custom headers: arbitrary HTTP headers for auth tokens, API keys, routing
• strum Display: BatchMode variants have human-readable Display for logs

Files Changed

File	Purpose
sinks/http_sink/src/lib.rs	Core implementation — types, config, Sink trait, retry, batch modes, 57 unit tests
sinks/http_sink/README.md	Usage guide, config reference, runtime model, deployment patterns, message flow
sinks/http_sink/Cargo.toml	Crate manifest
sinks/http_sink/config.toml	Example connector configuration
fixtures/http/container.rs	Docker test container with WireMock
fixtures/http/sink.rs	Test fixtures (single-topic, multi-topic, batch mode variants)
http/http_sink.rs	7 integration tests (delivery, metadata, batch modes, multi-topic)

Architecture

Iggy Stream → [Runtime polls topic] → consume(messages) → [batch mode] → HTTP endpoint
                                            │
                                    ┌───────┴────────┐
                                    │  individual    │ → 1 request per message
                                    │  nd_json       │ → all messages, newline-delimited
                                    │  json_array    │ → all messages, JSON array
                                    │  raw           │ → 1 request per message, raw bytes
                                    └────────────────┘

Code Review History

Round	Agents	Findings	Fixed
Round 1	code-reviewer, silent-failure-hunter, comment-analyzer, code-simplifier	12	12
Round 2	Same 4 agents (follow-up)	7	7
Round 3	Same 4 agents (post-feature additions)	17	15 (2 deferred)
Round 4	Same 4 agents (double-review follow-up)	6	6
Round 5	Maintainer review (hubcio)	14	14
Round 6	6 agents (code-reviewer ×2, silent-failure-hunter, code-architect, comment-analyzer, type-design-analyzer)	8	8
Round 7	Same 6 agents (follow-up on remediation)	0	—
Round 8	5 agents (code-reviewer, silent-failure-hunter, comment-analyzer, code-architect, type-design-analyzer) — post spetz review refactoring	6	6
Round 9	Same 5 agents (follow-up on Round 8 remediation)	0	—
Total		70	68 (2 deferred)

Round 5 (maintainer review — hubcio) key changes:

• Consuming iterator — eliminate .clone() on payloads in all send methods
• DRY refactor — extracted send_per_message() shared by individual/raw modes
• Pre-built HeaderMap in open() — avoid per-request header parsing
• HashSet for success_status_codes — O(1) hot-path lookup
• Forward iggy message headers as iggy_headers in metadata envelope
• Success/transient status code overlap warning in open()
• Worst-case latency documentation on consume()

Round 8 (post spetz review — 12 items addressed in 10 commits + 1 remediation):

• BatchMode::Ndjson → NdJson, serializes as "nd_json"
• Field name constants (FIELD_*, ENCODING_*) replacing magic strings
• strum::Display on BatchMode, removed mode_name/batch_mode string params
• content_type() moved from HttpSink to BatchMode impl
• json! macros → strongly typed MetadataEnvelope/IggyMetadata/EncodedPayload/EncodedHeader
• Comment separators removed, test helpers reordered below tests
• client parameter threading removed — fn client() helper
• UUID v8 → 32-char hex (uuid crate removed)
• Hand-rolled retry → reqwest-middleware (RetryTransientMiddleware + HttpSinkRetryStrategy)
• retry_backoff_multiplier: f64 → u32 (matches ExponentialBackoff::base())
• retries_count field removed (retries transparent inside middleware)

Round 8 remediation (2 CRITICAL, 4 HIGH):

• build_envelope() returns Result — no more unwrap_or(Null) silent data loss
• retry_delay > max_retry_delay auto-swaps to prevent ExponentialBackoff panic
• Retry-After warn uses actual status code (was hardcoded "429")
• Middleware errors logged with {:#} for full retry context
• requests_sent → send_attempts (middleware retries make counter ambiguous)
• Doc comment default: 2.0 → default: 2 for u32 backoff multiplier

Deferred (tracked in issues)

• L4: Structured error type enum replacing string-based Error::HttpRequestFailed — #2927
• D1: Expose internal metrics (errors_count) via runtime health API — #2928

Test Plan

• 57 unit tests covering config parsing, validation, serialization, batch modes, header forwarding, edge cases
• 7 integration tests with WireMock in Docker:
  • individual_json_messages_delivered_as_separate_posts — per-message delivery + envelope
  • ndjson_messages_delivered_as_single_request — NDJSON batch mode
  • json_array_messages_delivered_as_single_request — JSON array batch mode
  • raw_binary_messages_delivered_without_envelope — raw byte passthrough
  • metadata_disabled_sends_bare_payload — include_metadata=false
  • individual_messages_have_sequential_offsets — offset ordering integrity
  • multi_topic_messages_delivered_with_correct_topic_metadata — 2 topics, metadata accuracy
• cargo clippy -p iggy_connector_http_sink -- -D warnings — 0 warnings
• cargo clippy -p integration -- -D warnings — 0 warnings

Related

• Discussion: Proposal: Generic HTTP Sink Connector #2919
• Issue (structured errors): bug(connectors): consume() return value discarded in sink runtime #2927
• Issue (runtime metrics): bug(connectors): PollingMessages auto-commit commits offsets before sink processing #2928

[codecov]

Codecov Report

❌ Patch coverage is 58.68114% with 495 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.37%. Comparing base (411a697) to head (fd7d6d4).

Files with missing lines	Patch %	Lines
core/connectors/sinks/http_sink/src/lib.rs	57.97%	481 Missing and 14 partials ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #2925       +/-   ##
=============================================
- Coverage     71.74%   59.37%   -12.38%     
  Complexity      943      943               
=============================================
  Files          1121     1123        +2     
  Lines         93800    94978     +1178     
  Branches      71124    72314     +1190     
=============================================
- Hits          67301    56390    -10911     
- Misses        23863    36068    +12205     
+ Partials       2636     2520      -116     

Components	Coverage Δ
Rust Core	55.87% <58.68%> (-16.49%)	⬇️
Java SDK	62.30% <ø> (ø)
C# SDK	67.43% <ø> (-0.21%)	⬇️
Python SDK	81.43% <ø> (ø)
Node SDK	91.53% <ø> (+0.21%)	⬆️
Go SDK	38.68% <ø> (ø)

Files with missing lines	Coverage Δ
core/connectors/sdk/src/convert.rs	100.00% <100.00%> (ø)
core/connectors/sdk/src/lib.rs	56.00% <ø> (ø)
...ore/connectors/sinks/elasticsearch_sink/src/lib.rs	66.17% <ø> (-4.34%)	⬇️
core/connectors/sinks/http_sink/src/lib.rs	57.97% <57.97%> (ø)

... and 314 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hubcio]

overall good direction, please fix the CI (markdown lint) and rest of the comments. looks pretty solid to me.

[mlevkov]

Thank you @spetz — excellent feedback across the board. Here's how I've grouped the 12 comments for implementation:

Grouped by theme

A. Constants & naming (straightforward)

#	Line	Change
1	77	Ndjson → NdJson (nd_json in snake_case)
3	318	pub const for "data", "iggy_payload_encoding" field names
5	349	Constants for all metadata field names (iggy_checksum, iggy_origin_timestamp, etc.)
11	1184	Remove // ── ... ── comment sections; reorder helpers below callers
12	1669	Use constants for field names in test assertions

B. Strongly typed structs + enum methods

#	Line	Change
2	295	Move content_type() to BatchMode impl
4/6	340/378	Replace serde_json::json! with serializable MetadataEnvelope struct
8	584	strum Display on BatchMode for mode_name instead of string parameter

C. Method signature cleanup

#	Line	Change
9	444	Use self.client directly instead of passing client parameter

D. Design decisions (want to discuss)

#	Line	Question
7	443	reqwest-middleware — would replace hand-rolled retry with the workspace's middleware + retry policy. Leaning toward yes — aligns with the rest of the connectors runtime and gets retry improvements for free on workspace upgrades.
10	942	UUID format — currently v8 (per @hubcio 's earlier request for proper RFC 4122 bits). Options: (a) v4 random UUID (loses correlation to iggy message ID), (b) v8 custom (current — preserves most of the ID but overwrites 6 bits), (c) simple hex without dashes (lossless but not a valid UUID), (d) user-configurable. What's your preference?

Will work through A → B → C → D in order. Addressing #10 and #7 last since they need alignment.

[mlevkov]

@spetz and @hubcio

Please note the limitations introduced by the shift to reqwest-middleware:

1. Retry-After header no longer used for backoff timing

The hand-rolled retry loop previously parsed integer Retry-After headers and used them as the delay between retries (capped to max_retry_delay). The reqwest-retry middleware does not support Retry-After natively — it always uses computed exponential backoff. The custom HttpSinkRetryStrategy now logs a warning when a response carries Retry-After, so operators know the middleware's backoff may be insufficient:

WARN Server returned 429 with Retry-After: 120 — middleware uses computed backoff which may be insufficient

Impact: If a downstream endpoint returns Retry-After: 120 (wait 2 minutes), the middleware will retry sooner (e.g., after 1s, 2s, 4s). This could result in additional 429 responses during the wait period, but the middleware will keep retrying up to max_retries with backoff, so delivery still succeeds in most cases.

2. Retry count no longer tracked as a connector-level metric

The retries_count field was removed because retries are now transparent inside the middleware. The close() stats log reports send_attempts (logical calls to send_with_retry) rather than physical HTTP requests. With max_retries=3, a single send attempt may produce up to 4 actual HTTP requests, but this is invisible to the connector's counters. The reqwest-tracing TracingMiddleware emits spans for each physical request, so retry visibility is available via tracing/logging infrastructure.

3. retry_backoff_multiplier changed from f64 to u32

The ExponentialBackoff::base() method in retry-policies accepts u32. The old f64 type allowed fractional multipliers like 1.5, but these were never actually supported by the backoff library (internally it uses u32::checked_pow). The type change makes the API honest — 2 means "double the delay each retry". This is a breaking config change for any existing configs using fractional values, but since the connector is brand new (never deployed), there are no existing configs to break.

These are acceptable trade-offs for the simplification (120 → 40 lines in send_with_retry, removal of compute_retry_delay/parse_retry_after/is_transient_status), and align with the retry pattern used in the Iggy runtime's HttpProvider.