Skip to content

Add blosc getitem#1

Open
andrewfulton9 wants to merge 22 commits into
masterfrom
add_blosc_getitem
Open

Add blosc getitem#1
andrewfulton9 wants to merge 22 commits into
masterfrom
add_blosc_getitem

Conversation

@andrewfulton9

Copy link
Copy Markdown
Owner

This PR demonstrates a way to do partial decompression of a blosc compressed array buffer. The biggest limitation of the decompress_partial method is the inability to properly decompress parts of a multidimensional array. The user must also must account for typesize and the encoding size of the encoded array to function properly.

TODO:

  • Unit tests and/or doctests in docstrings
  • tox -e py38 passes locally
  • Docstrings and API docs for any new/modified user-facing classes and functions
  • Changes documented in docs/release.rst
  • tox -e docs passes locally
  • AppVeyor and Travis CI passes
  • Test coverage to 100% (Coveralls passes)

Comment thread numcodecs/blosc.pyx Outdated
Comment thread numcodecs/blosc.pyx Outdated
Comment thread numcodecs/blosc.pyx Outdated
Comment thread numcodecs/blosc.pyx
buf = ensure_contiguous_ndarray(buf, self.max_buffer_size)
return decompress_partial(buf, start, nitems,
typesize, encoding_size, dest=out)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we could try to add an option to "guess" the encoding size, if we know the typesize then we know blosc will return at least typesize bytes if we ask to decode the first element.

IIRC when blosc_getitem succeed it tells you how much it wrote to the buffer you gave it, so if you might be able to figure out whether the encoded typesize is correct, or 1 (if stored as bytes).

Not sure it is worth it though and there is a performance impact to chec that as you need to decompress twice.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added some code to infer the typesize / encoding size as you suggested here, and its passing the tests. I changed typesize and encoding size to kwargs that default to 0, and if they are zero, they use the inferences to do the partial decompression

Comment thread numcodecs/tests/common.py Outdated
Comment thread numcodecs/tests/test_blosc.py
@Carreau

Carreau commented May 21, 2020

Copy link
Copy Markdown

That looks good to me, I'm thinking that when this is used in a higher level API we might be able to guess the typesize of the underlying blosc serialization once and reuse it – this would then let the users be unaware of this detail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants