[Storage] Download stream refactor by annatisch · Pull Request #7848 · Azure/azure-sdk-for-python

annatisch · 2019-10-12T01:21:34Z

Refactored the StorageStreamDownloader objects to:

[Breaking] Moved iterable behaviour to separate chunks() function ([StorageStreamDownloader] [Breaking Change] Replace iterable behavior with the chunk #7731).
Added readall() and readinto(stream) functions to start implementing stream interface (Storage Stream Downloader #7504).
Deprecated content_as_bytes(), content_as_text(), download_to_stream().
[Breaking] Made most of the public attributes private, and documented those remaining public ones.
Reduced some duplicate code (still lots of work to be done here).
Added async module level upload/download functions.

adxsdk6 · 2019-10-12T01:21:36Z

Can one of the admins verify this patch?

rakshith91 · 2019-10-12T22:52:46Z

Would be great if you can update History.md :)
Thanks.

…e-sdk-for-python into storage-downloads

annatisch · 2019-10-14T03:57:13Z

@rakshith91 - Done.

rakshith91

LGTM..thanks

xiafu-msft · 2019-10-16T23:19:48Z

-                self.download_size = min(self.file_size, self.length - self.offset + 1)
-            elif self.offset is not None:
-                self.download_size = self.file_size - self.offset
+            self._file_size = parse_length_from_content_range(response.properties.content_range)


I have a question about file_size. If we uploaded a 30 bytes file with client side encryption, then the encrypted file is 32 bytes. When we want to download the file, the parsed self._file_size is 32 I assume, are we intended to keep 32?

The file size here is purely with regards to the bytes being downloaded, and the values used in the range header. So if the encrypted file is 32, that will be the value we use. If this does not match the decrypted value size, we don't update it... I think that's valid, as the size attribute is to determine the size of the Stream (i.e. what's being downloaded).

… storage-downloads

annatisch · 2019-10-18T13:12:30Z

/azp run python - storage - ci

azure-pipelines · 2019-10-18T13:12:44Z

Azure Pipelines successfully started running 1 pipeline(s).

xiafu-msft · 2019-10-18T20:35:40Z

It looks great 😀

annatisch · 2019-10-18T23:51:57Z

/azp run python - storage - ci

azure-pipelines · 2019-10-18T23:52:09Z

Azure Pipelines successfully started running 1 pipeline(s).

* Fix SubStream to respect IOBase protocol (#7843) * removes NoRetry policy (#7845) * [storage] Makes signed_identifiers a required param (#7844) * makes signed_identifiers a required param * fix tests and update history * removes unecessary check * fix files and queues tests * [storage] Changes `file_permission_key` param to `permission_key` (#7841) * changes file_permission_key param to permission_key * update history * Change the param directive to keyword in docstrings (#7855) * Change the directive to keyword in docstrings * pylint :( * [storage] Adds from_string to models (#7870) * adds from_string to models * fixes to naming and history * remove _str in docstrings * pylint * [storage-file] await async poller (#7872) * await files async poller * fix docstrings * fix type hints * [File and Queue] Client constructors (#7853) * from_queue_url * file client constructors * update CHANGELOG * fix test * lint fix * some doc changes * queue docs * blob docs * [storage-queue] rename queue messages (#7895) * rename queue messages * update history * [Storage-Queue][Storage-File] Kwargify positional params (#7877) * Queue kwargification * kwargify Files * small fix * minor fix * comments + lint * some changes * pylint * [storage-file, queue] moves param protocol to kwargs in gen_shared_access_signature() (#7897) * moves protocol in gen_sas to kwargs * delete whitespace * [storage] Unexposes models from aio (#7881) * unexposes models from aio * unexpose all except clients and pylint fixes * update history * unexpose sync paged models * update history phrasing * [Storage-queue] Allow None message encode policy (#7898) * Don't expose NoEncode policies * Support None value for encode/decode policies * Updated tests * Updated sync encryption tests * Updated async encryption tests * Updated async encoding tests * Fixed test * Fix test * Updated release notes * Rename enqueue_message to send_messgae (#7928) * [storage-blob, queue] Renames Logging to <ServiceName>AnalyticsLogging (#7921) * renames logging to ServiceAnalyticsLogging * changes param logging to analytics_logging * fixes tests due to param name change * [Blob][File][Encryption]Fix Download Encrypted Blob/File Bug (#7965) #7957 * Consolidate offset - length behavior in files (#7942) * Initital Commit * length changes * some test changes * some more changes * history.md * Update a couple of recrodings * comments address * ops * Stop modifying internal_response body (#7958) * Stop modifying internal body * Replicate for queues and files too * [Storage] Relocated SAS generation (#7955) * Expose account name on clients * Updated blob sas gen * Updated file sas gen * Update queue sas gen * Updated docstrings * Some pylint fixes * More pylint * More pylint * Reverted auth change * Added release notes * Fixed kwarg docstring * [Storage] Updated close handles (#7940) * Updated close handles * Pylint fixes * Fix tests * Review feedback * Missing recursive parameter * Added release notes * [Storage] Conditional etag parameters (#8047) * Updated clients * Process conditional headers * Updated tests * Fixed test * Updated error scenarios * Updated release notes * Revert skipped tests * [Storage] Download stream refactor (#7848) * Refactored download stream API * Unskip tests * Test warnings * Missing await * Fixed append tests * Fixed page tests * Pylint * Py2.7 iter support * Refactor downloaders * Added missing async module functions * Updated release notes * Keyword params * Documented more keyword options * Added module functions to exports * Fixed merge * Fix for Files download stream * Pylint fix * Updated File stream downloads * Fix tests * Updated error message * Added Files release notes * Pylint fix * Download stream error handling * Fixed some tests * Design Pipeline ownership (#7981) * Initital Commit * Pipeline Ownership * slight modifications * pipeline ownership for queues and files * fixes tests, adds docstrings to wrapper classes * pylint * Batching APIs Raise on Any Failure (#7963) * Raise on Single Failure * some changes * lint * some changes * oops * comments + lint * some optimization * history * comments * lint * recording * Plug HttpLoggingPolicy to Storage (#8081) * Plug HttpLoggingPolicy to Storage * Update dependencies * Skip depends job for Storage * Fix shared req * [storage] make storage files _internal (#7949) * blobs internal * files internal * queues internal * some fixes to docstrings * update history.md * fix async test import * some file mypy fixes * some queue mypy fixes * some blob mypy fixes * fix mistake in type annot * history edits * Merge fix * Revert "Merge fix" This reverts commit 748fbd6. * Better merge fix * fix some tests (#8100) * Fix tests (#8103) * minor fix special char * fix auth * comments * [Storage] Bumped version + Pipeline fix (#8089) * Bumped version * Docs tweak * Re-bumped version * Added 3.8 classifier * Added proxy policy to pipeline * Synced base clients * Added batch exception * Comment the echo check to workaround batch headers issue (#8118) * Fix 8091: expose generated enum to customers (#8117) * Fix #8091 * Add one in file

annatisch added 4 commits October 11, 2019 17:52

Refactored download stream API

48671cb

Unskip tests

c741675

Test warnings

40914d9

Missing await

ec2c05c

annatisch added 3 commits October 11, 2019 20:16

Fixed append tests

4ba1b20

Fixed page tests

32db209

Pylint

5bf0d7f

annatisch added the Storage Storage Service (Queues, Blobs, Files) label Oct 12, 2019

annatisch added 2 commits October 11, 2019 21:09

Py2.7 iter support

3e063e4

Refactor downloaders

cb3144c

annatisch marked this pull request as ready for review October 12, 2019 15:39

annatisch requested review from rakshith91 and zezha-msft as code owners October 12, 2019 15:39

annatisch requested review from johanste and lmazuel October 12, 2019 15:40

annatisch added 2 commits October 12, 2019 11:14

Added missing async module functions

f1fadfe

Merge branch 'feature/storage-preview5' into storage-downloads

48ae6f5

rakshith91 added the P0 label Oct 12, 2019

annatisch added 2 commits October 13, 2019 20:52

Updated release notes

4b369a3

Merge branch 'storage-downloads' of https://github.com/annatisch/azur…

8d90594

…e-sdk-for-python into storage-downloads

rakshith91 reviewed Oct 14, 2019

View reviewed changes

Comment thread sdk/storage/azure-storage-blob/azure/storage/blob/__init__.py Outdated

rakshith91 reviewed Oct 14, 2019

View reviewed changes

Comment thread sdk/storage/azure-storage-blob/azure/storage/blob/aio/blob_client_async.py Outdated

rakshith91 reviewed Oct 14, 2019

View reviewed changes

Comment thread sdk/storage/azure-storage-blob/azure/storage/blob/aio/download_async.py

rakshith91 reviewed Oct 14, 2019

View reviewed changes

Comment thread sdk/storage/azure-storage-blob/azure/storage/blob/aio/download_async.py

annatisch added 2 commits October 14, 2019 12:22

Keyword params

3b1d6b5

Merge branch 'feature/storage-preview5' into storage-downloads

366847f

rakshith91 approved these changes Oct 14, 2019

View reviewed changes

annatisch requested a review from xiafu-msft October 14, 2019 23:53

mayurid added the blocking-release Blocks release label Oct 15, 2019

annatisch added 3 commits October 15, 2019 16:41

Added module functions to exports

d895acb

Merge branch 'feature/storage-preview5' into storage-downloads

d46f074

Fixed merge

51777cc

xiafu-msft reviewed Oct 16, 2019

View reviewed changes

Comment thread sdk/storage/azure-storage-blob/azure/storage/blob/aio/download_async.py

xiafu-msft approved these changes Oct 16, 2019

View reviewed changes

xiafu-msft reviewed Oct 16, 2019

View reviewed changes

annatisch added 9 commits October 17, 2019 08:15

Merge remote-tracking branch 'upstream/feature/storage-preview5' into…

d5911c3

… storage-downloads

Fix for Files download stream

cb17af9

Pylint fix

233eb40

Updated File stream downloads

75e4b33

Merge remote-tracking branch 'upstream/feature/storage-preview5' into…

8cbfb4e

… storage-downloads

Fix tests

7788730

Updated error message

e8878a1

Added Files release notes

145d6c1

Pylint fix

e80b075

annatisch added 2 commits October 18, 2019 10:45

Download stream error handling

5d08ca4

Fixed some tests

c8e8f7a

Merge branch 'feature/storage-preview5' into storage-downloads

be38f71

annatisch added 2 commits October 18, 2019 17:27

Merge branch 'feature/storage-preview5' into storage-downloads

10a08d8

Merge branch 'feature/storage-preview5' into storage-downloads

a61d220

annatisch merged commit 6e31ddf into Azure:feature/storage-preview5 Oct 19, 2019

Uh oh!

Conversation

annatisch commented Oct 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adxsdk6 commented Oct 12, 2019

Uh oh!

rakshith91 commented Oct 12, 2019

Uh oh!

annatisch commented Oct 14, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rakshith91 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xiafu-msft Oct 16, 2019

Choose a reason for hiding this comment

Uh oh!

annatisch Oct 17, 2019

Choose a reason for hiding this comment

Uh oh!

annatisch commented Oct 18, 2019

Uh oh!

azure-pipelines Bot commented Oct 18, 2019

Uh oh!

xiafu-msft commented Oct 18, 2019

Uh oh!

annatisch commented Oct 18, 2019

Uh oh!

azure-pipelines Bot commented Oct 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

annatisch commented Oct 12, 2019 •

edited

Loading