fix: source redis lag/requeue, lane-growth bound, and honest reserved-seam docs#148
Merged
Conversation
…loor Lag now returns the group's lag from XINFO GROUPS as-is, including zero when the group is caught up; the prior positive-only guard fell through to XLEN and reported the full stream length for a fully-consumed group. ActionNak now records a per-entry redelivery floor when Result.Requeue exceeds the configured minimum idle, and NakRedeliver holds an entry back until its floor passes, matching the documented contract. Message headers are built lazily so a message whose headers are never read pays no per-message header allocation. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
…rt docs Add WithMaxLanes so the Hopper's per-key lane set (and its goroutines and map entries) is bounded under unbounded key cardinality; keys beyond the bound fold onto lanes by hash without reordering a key's messages. laneKey hashes the partition-key string directly, dropping a per-message []byte allocation. dispatchBatch now records ErrBatchResultCount on a batch result over-count so the discarded extra results are visible in traces rather than swallowed. Correct three inert-surface docs: Deduper is a reserved seam adapted by the idempotency middleware, not consulted by the Hopper; Batched.SettleBatch is a one-call settle seam the Hopper does not use (it settles per message); and remove retry's dead AttemptHeader and WithClock/cfg.now, and make WithJitterSource's scope honest. Add tests for ReceiveBatch, batch over-count, WithMaxLanes folding, InProgress/Manual dispositions, and the cdc DecodeEvent registry-error and RawJSON.As unmarshal-error paths. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
…ignal DriveFunc's godoc dropped the broken [Deduper] link for concrete guidance (use Drive or the idempotency middleware). doc.go now documents the transactional DriveTx mode alongside the durable and stateless modes. Drive and DriveTx mark the span with statemachine.exactly_once=false when a message yields no event id, so the silent loss of dedup is visible in traces rather than passing unnoticed; EventID's godoc explains it. Add a DriveTx Save-failure test covering the in-transaction persist-error abort. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
A transactional inlet is backed by one GroupTransactSession that fences a single consumer; a second Subscribe could not share it and would silently return a subscription with no transact session (Begin would then report not-transactional). Subscribe now rejects the second call with a sentinel error. Add coverage for the transactional option assembly and the guard. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
…s): add README and doc.go jetstream newMessage no longer eagerly builds the header slice; Headers materializes and caches it on first read, so a message whose headers are never read pays no per-message header allocation. Add a README and a doc.go for source/cloudevents, the only source adapter that lacked both; the package comment moves to doc.go to match the suite's documentation layout. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this change does
Remediates the source review findings (correctness + perf + coverage; no new feature behavior):
Lag()no longer hides a caught-up group (dropped thelag>0guard);ActionNaknow honorsResult.Requeuevia a per-entry requeue floor (with an injectable clock + tests).ErrBatchResultCountinstead of silently dropped; kafka guards a secondSubscribeon a transactional Inlet (errTransactionalSingleSubscribe).EventIDnow emits astatemachine.exactly_once=falsetelemetry signal and is documented;DriveTxSave-failure-aborts-transaction path gets the missing test;doc.godocuments theDriveTxmode; thedrivefuncgodoc no longer links a nonexistent type.Deduperis actually consumed bysource/idempotency— kept, and the misleading "the Hopper uses it" doc corrected to "reserved seam adapted by the idempotency middleware."Batched.SettleBatchdocumented as a reserved one-call seam (per-message Results make a single-result batch settle wrong). DeadretryAttemptHeader/WithClockremoved (writing the header would need backend cooperation = feature work, explicitly skipped).WithMaxLanes(default 4096) bounds the per-key lane map/goroutine growth (the unbounded-growth leak) by folding keys onto a bounded lane set without breaking per-key ordering;laneKeyhashes the partition key directly (no[]bytealloc); jetstream/redisnewMessageheaders are now lazy.source/cloudeventsgainsdoc.go+ README; addedReceiveBatch, kafka transactional-Subscribe, and cdc error-path coverage.Note:
WithMaxLanesis a new option (additive) rather than silent idle-eviction; flagging since it is the one piece of new public surface here.Checklist
-race) clean;examples/sourcedrive+e2ebuild