Skip to content

Mongo KV range queries (option 1)#223

Closed
ableegoldman wants to merge 4 commits intomainfrom
MONGO-range-queries
Closed

Mongo KV range queries (option 1)#223
ableegoldman wants to merge 4 commits intomainfrom
MONGO-range-queries

Conversation

@ableegoldman
Copy link
Copy Markdown
Contributor

Implementation of range and all queries for the MongoKVTable.

In order to follow Streams semantics, we need to be able to differentiate data that falls within a kafka partition. There are two options:

  1. Add a kafkaPartition field to KVDoc and filter range queries with this
  2. Separate data into different collections for each kafka partition

This PR implements option 1, although we may want to go with option 2 for two reasons: (a) increased read/write/storage due to added field (though it's just an int) and (b) in the current implementation for Mongo window stores, we need separate collections for each kafka partition anyway (to track stream-time on a partition basis), so it probably makes sene to be consistent.

I'll follow up with a PR for option 2 after finishing the window store range queries

@agavra
Copy link
Copy Markdown
Contributor

agavra commented Feb 15, 2024

@ableegoldman regarding the options we should make it a priority that we can increase/decrease number of partitions without having to make a massive storage change. we may even want to consider less-efficient range scans and using the partitioner to filter client-side based on key.

@ableegoldman
Copy link
Copy Markdown
Contributor Author

closing in favor of the alternative approach via client-side filtering: #251

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants