Closed
Conversation
6d36012 to
a649a49
Compare
Contributor
|
@ableegoldman regarding the options we should make it a priority that we can increase/decrease number of partitions without having to make a massive storage change. we may even want to consider less-efficient range scans and using the partitioner to filter client-side based on key. |
Contributor
Author
|
closing in favor of the alternative approach via client-side filtering: #251 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation of
rangeandallqueries for the MongoKVTable.In order to follow Streams semantics, we need to be able to differentiate data that falls within a kafka partition. There are two options:
kafkaPartitionfield to KVDoc and filter range queries with thisThis PR implements option 1, although we may want to go with option 2 for two reasons: (a) increased read/write/storage due to added field (though it's just an int) and (b) in the current implementation for Mongo window stores, we need separate collections for each kafka partition anyway (to track stream-time on a partition basis), so it probably makes sene to be consistent.
I'll follow up with a PR for option 2 after finishing the window store range queries