Draft
Conversation
haileyok
force-pushed
the
hailey/tap-kafka
branch
from
bf81e16 to
15255d7
Compare
haileyok
force-pushed
the
hailey/tap-kafka
branch
from
15255d7 to
90263fa
Compare
haileyok
commented
Dec 27, 2025
Comment on lines
+135
to
+136
| atkafka.WithEnsureTopic(true), // ensures that the topic has been created | ||
| atkafka.WithTopicPartitions(24), // TODO: partition count env var |
Member
Author
There was a problem hiding this comment.
these should actually be configurable, probably
haileyok
commented
Dec 27, 2025
Comment on lines
+151
to
+154
| if backoffUntil, ok := r.pdsBackoff.Load(pdsURL); ok { | ||
| if time.Now().Before(backoffUntil) { | ||
| return false, nil | ||
| } |
Member
Author
There was a problem hiding this comment.
since we are almost exclusively doing reads out of this map, swapping it out with sync.Map instead of that mutex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With this set of changes, I was able to get from ~6k evt/s over the websocket to ~ 9k evt/s when using SQLite, and roughly 17k evt/s when using Postgres + tuning. this might not be a thing for merging, but showing an example of how we can increase the backfill speed significantly under the right conditions...