Skip to content

Fix deserialization of bytes chunks larger than 64MB#2637

Merged
mrocklin merged 3 commits into
dask:masterfrom
pentschev:fix-large-chunks-deserializer
Apr 29, 2019
Merged

Fix deserialization of bytes chunks larger than 64MB#2637
mrocklin merged 3 commits into
dask:masterfrom
pentschev:fix-large-chunks-deserializer

Conversation

@pentschev

@pentschev pentschev commented Apr 29, 2019

Copy link
Copy Markdown
Member

When chunks of bytes are larger than 64MB they get split into multiple 64MB frames within the serializer. When _deserialize_bytes() returns only the first frame, it truncates the object and it's unable to deserialize it. The solution is simply to join all the frames.

@pentschev

Copy link
Copy Markdown
Member Author

I'm still writing a test for this, please hold on to merging until then.

@pentschev pentschev changed the title Fix deserialization of chunks larger than 64MB Fix deserialization of bytes chunks larger than 64MB Apr 29, 2019
Comment thread distributed/protocol/tests/test_serialize.py Outdated
@mrocklin

Copy link
Copy Markdown
Member

Thanks for the fix @pentschev . I have a small suggestion, but am +1 regardless. You should be able to press the "commit suggestion" button.

Co-Authored-By: pentschev <peter@entschev.com>
@pentschev

Copy link
Copy Markdown
Member Author

@mrocklin the failure doesn't seem to be related to this PR.

@mrocklin

Copy link
Copy Markdown
Member

I actually don't see a failure at all.

Regardless, merging.

@mrocklin mrocklin merged commit e0cf7e7 into dask:master Apr 29, 2019
@pentschev

Copy link
Copy Markdown
Member Author

Thanks @mrocklin !

@mrocklin

mrocklin commented Apr 29, 2019 via email

Copy link
Copy Markdown
Member

muammar added a commit to muammar/distributed that referenced this pull request May 8, 2019
* upstream/master:
  Fix deserialization of bytes chunks larger than 64MB (dask#2637)
  bump version to 1.27.1
  Updated logging module doc links from docs.python.org/2 to docs.python.org/3. (dask#2635)
  Adaptive: recommend close workers when any are idle (dask#2330)
@pentschev pentschev deleted the fix-large-chunks-deserializer branch November 11, 2020 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants