Add blosc getitem#1
Conversation
| buf = ensure_contiguous_ndarray(buf, self.max_buffer_size) | ||
| return decompress_partial(buf, start, nitems, | ||
| typesize, encoding_size, dest=out) | ||
|
|
There was a problem hiding this comment.
Here we could try to add an option to "guess" the encoding size, if we know the typesize then we know blosc will return at least typesize bytes if we ask to decode the first element.
IIRC when blosc_getitem succeed it tells you how much it wrote to the buffer you gave it, so if you might be able to figure out whether the encoded typesize is correct, or 1 (if stored as bytes).
Not sure it is worth it though and there is a performance impact to chec that as you need to decompress twice.
There was a problem hiding this comment.
We could also just peek at buf[3]...
https://github.com/Blosc/c-blosc/blob/403d50ced2cd35d3e72a007d29d73110ac45e62b/blosc/blosc.c#L1613
There was a problem hiding this comment.
I added some code to infer the typesize / encoding size as you suggested here, and its passing the tests. I changed typesize and encoding size to kwargs that default to 0, and if they are zero, they use the inferences to do the partial decompression
|
That looks good to me, I'm thinking that when this is used in a higher level API we might be able to guess the typesize of the underlying blosc serialization once and reuse it – this would then let the users be unaware of this detail. |
The advantage of using warnings instead of globals is performance and the ability to also still get the warning printed by default.
Example of unstable-allowing context manager.
… is the right size
This PR demonstrates a way to do partial decompression of a blosc compressed array buffer. The biggest limitation of the
decompress_partialmethod is the inability to properly decompress parts of a multidimensional array. The user must also must account for typesize and the encoding size of the encoded array to function properly.TODO:
tox -e py38passes locallytox -e docspasses locally