make sure that only concat preallocates buffers#382
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #382 +/- ##
==========================================
- Coverage 82.61% 82.58% -0.03%
==========================================
Files 162 162
Lines 44228 44329 +101
==========================================
+ Hits 36538 36609 +71
- Misses 7690 7720 +30 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Thanks @ritchie46 . My small concerns with this PR:
I left some ideas as a comment on the issue |
|
@jorgecarleitao I implemented your proposal from #347. As we now need to define an For now, only the |
|
@jorgecarleitao what is the status of this one? |
jorgecarleitao
left a comment
There was a problem hiding this comment.
Good to go: this is a solid improvement to me.
* MutableArrayData::with_capacities * better pattern matching * add binary capacities * add list child data * add struct capacities * add panic for dictionary type * change dictionary capacity enum variant
|
Included in #411 cherry pick |
…se (#411) * Reduce memory usage of concat (large)utf8 (#348) * reduce memory needed for concat * reuse code for str allocation buffer * make sure that only concat preallocates buffers (#382) * MutableArrayData::with_capacities * better pattern matching * add binary capacities * add list child data * add struct capacities * add panic for dictionary type * change dictionary capacity enum variant Co-authored-by: Ritchie Vink <ritchie46@gmail.com>
Which issue does this PR close?
Partial fix for #347. In #348 @jorgecarleitao pointed out that the memory savings don't work for the
filterand thezipkernel.This PR restores the implementation for those two kernels, and keeps the new
preallocationfor theconcatkernel.This is achieved by create a builder pattern for
MutableArrayData. This way we don't break the API, and we may choose topreallocatebuffers.Could you take a look at this @jorgecarleitao ?
Are there any user-facing changes?
There is an builder pattern struct for
MutableArrayData.If there are any breaking changes to public APIs, please add the
breaking changelabel.No