When converting python objects to arrow array currently we always collect the input to a sequence, but this may be memory consuming in certain cases.
For unknown sized iterators we could consume and temporarily store the seen items during inference potentially improving both the conversion time and peak memory usage.
Reporter: Krisztian Szucs / @kszucs
Related issues:
Note: This issue was originally created as ARROW-10006. Please see the migration documentation for further details.
When converting python objects to arrow array currently we always collect the input to a sequence, but this may be memory consuming in certain cases.
For unknown sized iterators we could consume and temporarily store the seen items during inference potentially improving both the conversion time and peak memory usage.
Reporter: Krisztian Szucs / @kszucs
Related issues:
Note: This issue was originally created as ARROW-10006. Please see the migration documentation for further details.