Conversation
25ad40f to
fc310cc
Compare
|
Quickly drafting to check whether there are some remnants from the first four iterations that have made it into the final thing and that could just be removed/simplified now, which is very possible because it's all a big blur at this point 🫠 |
Nope, I mostly just ran into differently shaped walls. |
f417f9d to
2fc33cc
Compare
|
🙏 |
zehiko
left a comment
There was a problem hiding this comment.
I learned a lot reviewing this PR and I really appreciate nice documentation! I believe it's clear to me how this addresses use cases like streaming and why the protection is in place, but folks with more experience with the code base I'm sure can provide more valuable feedback.
|
|
||
| // NOTE: None of these fields should ever be publicly exposed, either directly or through a method, | ||
| // as it is always possible to go back to an actual `RwLock` via `RwLockWriteGuard::rwlock`. | ||
| // Doing so would defeat the deadlock protection that the `StorageEngine` offers. |
There was a problem hiding this comment.
perhaps it's worth documenting an example use case of doing the wrong thing without the protection that StorageEngine provides?
This implements
ChunkStoreHandle&QueryCacheHandlewhich, among other things, allow for streaming dataframes across FFI and network barriers.Those handles on their own are extremely problematic though: letting them loose all across the codebase effectively wraps the entire codebase in a semantically-unsafe{} block where every row interacting with these handles can lead to very nasty race conditions and even deadlocks.
That's why this PR also introduces the
StorageEnginetype, which makes using these handles actually safe in practice:Balancing these safety guarantees with the flexibility we need (today... and tomorrow!) for all our corner use cases has proven to be a subtle art...
ChunkStoreHandleandQueryCacheHandle#7486Checklist
mainbuild: rerun.io/viewernightlybuild: rerun.io/viewerCHANGELOG.mdand the migration guideTo run all checks from
main, comment on the PR with@rerun-bot full-check.