Skip to content

[Storage-queue] Allow None message encode policy #7898

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
annatisch merged 11 commits into from
Oct 16, 2019
Merged

[Storage-queue] Allow None message encode policy #7898

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
1586078
Don't expose NoEncode policies
annatisch Oct 15, 2019
1595ff4
Support None value for encode/decode policies
annatisch Oct 15, 2019
77332eb
Updated tests
annatisch Oct 15, 2019
2ec7777
Updated sync encryption tests
annatisch Oct 15, 2019
9c00ea1
Updated async encryption tests
annatisch Oct 15, 2019
b4bd8fb
Updated async encoding tests
annatisch Oct 15, 2019
50dbdfe
Fixed test
annatisch Oct 15, 2019
a4b50a4
Fix test
annatisch Oct 15, 2019
00a36d0
Updated release notes
annatisch Oct 15, 2019
febe858
Merge branch 'feature/storage-preview5' into annatisch-patch-1
annatisch Oct 15, 2019
9ff8b34
Merge branch 'feature/storage-preview5' into annatisch-patch-1
annatisch Oct 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions sdk/storage/azure-storage-queue/HISTORY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@
- `QueueClient` now accepts only `account_url` with mandatory a string param `queue_name`.
To use a queue_url, the method `from_queue_url` must be used.
- `set_queue_access_policy` has required parameter `signed_identifiers`.
- NoRetry policy has been removed. Use keyword argument `retry_total=0` for no retries.
- `NoRetry` policy has been removed. Use keyword argument `retry_total=0` for no retries.
- `NoEncodePolicy` and `NoDecodePolicy` have been removed. Use `message_encode_policy=None` and `message_decode_policy=None`.
- Removed types that were accidentally exposed from two modules. Only `QueueServiceClient` and `QueueClient`
should be imported from azure.storage.queue.aio
- NoRetry policy has been removed. Use keyword argument `retry_total=0` for no retries.
- Some parameters have become keyword only, rather than positional. Some examples include:
- `loop`
- `max_concurrency`
Expand Down Expand Up @@ -135,4 +135,4 @@ https://aka.ms/azure-sdk-preview1-python.

## Version 1.0.0:

- The package has switched from Apache 2.0 to the MIT license.
- The package has switched from Apache 2.0 to the MIT license.
6 changes: 1 addition & 5 deletions sdk/storage/azure-storage-queue/azure/storage/queue/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,7 @@
BinaryBase64EncodePolicy,
BinaryBase64DecodePolicy,
TextXMLEncodePolicy,
TextXMLDecodePolicy,
NoEncodePolicy,
NoDecodePolicy
TextXMLDecodePolicy
)
from .models import (
QueueMessage,
Expand Down Expand Up @@ -56,8 +54,6 @@
'BinaryBase64DecodePolicy',
'TextXMLEncodePolicy',
'TextXMLDecodePolicy',
'NoEncodePolicy',
'NoDecodePolicy',
'Logging',
'Metrics',
'CorsRule',
Expand Down
12 changes: 9 additions & 3 deletions sdk/storage/azure-storage-queue/azure/storage/queue/queue_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
process_storage_error,
return_response_headers,
return_headers_and_deserialized)
from ._message_encoding import TextXMLEncodePolicy, TextXMLDecodePolicy
from ._message_encoding import TextXMLEncodePolicy, TextXMLDecodePolicy, NoEncodePolicy, NoDecodePolicy
from ._deserialize import deserialize_queue_properties, deserialize_queue_creation
from ._generated import AzureQueueStorage
from ._generated.models import StorageErrorException, SignedIdentifier
Expand Down Expand Up @@ -105,8 +105,14 @@ def __init__(
self._query_str, credential = self._format_query_string(sas_token, credential)
super(QueueClient, self).__init__(parsed_url, service='queue', credential=credential, **kwargs)

self._config.message_encode_policy = kwargs.get('message_encode_policy') or TextXMLEncodePolicy()
self._config.message_decode_policy = kwargs.get('message_decode_policy') or TextXMLDecodePolicy()
if 'message_encode_policy' in kwargs:
self._config.message_encode_policy = kwargs['message_encode_policy'] or NoEncodePolicy()
else:
self._config.message_encode_policy = TextXMLEncodePolicy()
if 'message_decode_policy' in kwargs:
self._config.message_decode_policy = kwargs['message_decode_policy'] or NoDecodePolicy()

@kristapratico kristapratico Oct 15, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we document these kwargs in the client docstring or should they be part of the configuration kwargs in #6165 ?

@annatisch annatisch Oct 15, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They should be documented in the Client I think - as they are not part of the pipeline and not common between the storage library. I will add this :)

@annatisch annatisch Oct 15, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, correction - after the meeting today it seems that the other languages do not currently support this feature, therefore we will not be documenting it right now.

else:
self._config.message_decode_policy = TextXMLDecodePolicy()
self._client = AzureQueueStorage(self.url, pipeline=self._pipeline)

@classmethod
Expand Down
20 changes: 17 additions & 3 deletions sdk/storage/azure-storage-queue/tests/test_queue_encodings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,8 @@
BinaryBase64EncodePolicy,
BinaryBase64DecodePolicy,
TextXMLEncodePolicy,
TextXMLDecodePolicy,
NoEncodePolicy,
NoDecodePolicy)
TextXMLDecodePolicy)
from azure.storage.queue._message_encoding import NoEncodePolicy, NoDecodePolicy

from queuetestcase import (
QueueTestCase
Expand Down Expand Up @@ -71,6 +70,8 @@ def test_message_text_xml(self, resource_group, location, storage_account, stora
queue = qsc.get_queue_client(self.get_resource_name(TEST_QUEUE_PREFIX))

# Asserts
assert isinstance(queue._config.message_encode_policy, TextXMLEncodePolicy)
assert isinstance(queue._config.message_decode_policy, TextXMLDecodePolicy)
self._validate_encoding(queue, message)

@ResourceGroupPreparer()
Expand Down Expand Up @@ -189,6 +190,19 @@ def test_message_base64_decode_fails(self, resource_group, location, storage_acc

# Asserts
self.assertNotEqual(-1, str(e.exception).find('Message content is not valid base 64'))

def test_message_no_encoding(self):
# Arrange
queue = QueueClient(
account_url="https://account.queue.core.windows.net",
queue_name="queue",
credential="account_key",
message_encode_policy=None,
message_decode_policy=None)

# Asserts
assert isinstance(queue._config.message_encode_policy, NoEncodePolicy)
assert isinstance(queue._config.message_decode_policy, NoDecodePolicy)


# ------------------------------------------------------------------------------
Expand Down
2 changes: 0 additions & 2 deletions sdk/storage/azure-storage-queue/tests/test_queue_encodings_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,6 @@
BinaryBase64DecodePolicy,
TextXMLEncodePolicy,
TextXMLDecodePolicy,
NoEncodePolicy,
NoDecodePolicy
)

from azure.storage.queue.aio import (
Expand Down
22 changes: 7 additions & 15 deletions sdk/storage/azure-storage-queue/tests/test_queue_encryption.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,6 @@
QueueClient,
BinaryBase64EncodePolicy,
BinaryBase64DecodePolicy,
NoEncodePolicy,
NoDecodePolicy
)
from encryption_test_helper import (
KeyWrapper,
Expand All @@ -59,13 +57,13 @@ def _decode_base64_to_bytes(data):

class StorageQueueEncryptionTest(QueueTestCase):
# --Helpers-----------------------------------------------------------------
def _get_queue_reference(self, qsc, prefix=TEST_QUEUE_PREFIX):
def _get_queue_reference(self, qsc, prefix=TEST_QUEUE_PREFIX, **kwargs):
queue_name = self.get_resource_name(prefix)
queue = qsc.get_queue_client(queue_name)
queue = qsc.get_queue_client(queue_name, **kwargs)
return queue

def _create_queue(self, qsc, prefix=TEST_QUEUE_PREFIX):
queue = self._get_queue_reference(qsc, prefix)
def _create_queue(self, qsc, prefix=TEST_QUEUE_PREFIX, **kwargs):
queue = self._get_queue_reference(qsc, prefix, **kwargs)
try:
created = queue.create_queue()
except ResourceExistsError:
Expand Down Expand Up @@ -191,10 +189,8 @@ def test_update_encrypted_message(self, resource_group, location, storage_accoun
def test_update_encrypted_binary_message(self, resource_group, location, storage_account, storage_account_key):
# Arrange
qsc = QueueServiceClient(self._account_url(storage_account.name), storage_account_key)
queue = self._create_queue(qsc)
queue = self._create_queue(qsc, message_encode_policy=BinaryBase64EncodePolicy(), message_decode_policy=BinaryBase64DecodePolicy())
queue.key_encryption_key = KeyWrapper('key1')
queue._config.message_encode_policy = BinaryBase64EncodePolicy()
queue._config.message_decode_policy = BinaryBase64DecodePolicy()

binary_message = self.get_random_bytes(100)
queue.enqueue_message(binary_message)
Expand Down Expand Up @@ -224,10 +220,8 @@ def test_update_encrypted_raw_text_message(self, resource_group, location, stora
return
# Arrange
qsc = QueueServiceClient(self._account_url(storage_account.name), storage_account_key)
queue = self._create_queue(qsc)
queue = self._create_queue(qsc, message_encode_policy=None, message_decode_policy=None)
queue.key_encryption_key = KeyWrapper('key1')
queue._config.message_encode_policy = NoEncodePolicy()
queue._config.message_decode_policy = NoDecodePolicy()

raw_text = u'Update Me'
queue.enqueue_message(raw_text)
Expand All @@ -252,10 +246,8 @@ def test_update_encrypted_json_message(self, resource_group, location, storage_a
return
# Arrange
qsc = QueueServiceClient(self._account_url(storage_account.name), storage_account_key)
queue = self._create_queue(qsc)
queue = self._create_queue(qsc, message_encode_policy=None, message_decode_policy=None)
queue.key_encryption_key = KeyWrapper('key1')
queue._config.message_encode_policy = NoEncodePolicy()
queue._config.message_decode_policy = NoDecodePolicy()

message_dict = {'val1': 1, 'val2': '2'}
json_text = dumps(message_dict)
Expand Down
22 changes: 7 additions & 15 deletions sdk/storage/azure-storage-queue/tests/test_queue_encryption_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,6 @@
VERSION,
BinaryBase64EncodePolicy,
BinaryBase64DecodePolicy,
NoEncodePolicy,
NoDecodePolicy
)

from azure.storage.queue.aio import (
Expand Down Expand Up @@ -76,13 +74,13 @@ async def send(self, request, **config):

class StorageQueueEncryptionTestAsync(AsyncQueueTestCase):
# --Helpers-----------------------------------------------------------------
def _get_queue_reference(self, qsc, prefix=TEST_QUEUE_PREFIX):
def _get_queue_reference(self, qsc, prefix=TEST_QUEUE_PREFIX, **kwargs):
queue_name = self.get_resource_name(prefix)
queue = qsc.get_queue_client(queue_name)
queue = qsc.get_queue_client(queue_name, **kwargs)
return queue

async def _create_queue(self, qsc, prefix=TEST_QUEUE_PREFIX):
queue = self._get_queue_reference(qsc, prefix)
async def _create_queue(self, qsc, prefix=TEST_QUEUE_PREFIX, **kwargs):
queue = self._get_queue_reference(qsc, prefix, **kwargs)
try:
created = await queue.create_queue()
except ResourceExistsError:
Expand Down Expand Up @@ -220,10 +218,8 @@ async def test_update_encrypted_message(self, resource_group, location, storage_
async def test_update_encrypted_binary_message(self, resource_group, location, storage_account, storage_account_key):
qsc = QueueServiceClient(self._account_url(storage_account.name), storage_account_key, transport=AiohttpTestTransport())
# Arrange
queue = await self._create_queue(qsc)
queue = await self._create_queue(qsc, message_encode_policy=BinaryBase64EncodePolicy(), message_decode_policy=BinaryBase64DecodePolicy())
queue.key_encryption_key = KeyWrapper('key1')
queue._config.message_encode_policy = BinaryBase64EncodePolicy()
queue._config.message_decode_policy = BinaryBase64DecodePolicy()

binary_message = self.get_random_bytes(100)
await queue.enqueue_message(binary_message)
Expand Down Expand Up @@ -253,10 +249,8 @@ async def test_update_encrypted_raw_text_message(self, resource_group, location,
if not self.is_live:
return
# Arrange
queue = await self._create_queue(qsc)
queue = await self._create_queue(qsc, message_encode_policy=None, message_decode_policy=None)
queue.key_encryption_key = KeyWrapper('key1')
queue._config.message_encode_policy = NoEncodePolicy()
queue._config.message_decode_policy = NoDecodePolicy()

raw_text = u'Update Me'
await queue.enqueue_message(raw_text)
Expand Down Expand Up @@ -284,10 +278,8 @@ async def test_update_encrypted_json_message(self, resource_group, location, sto
if not self.is_live:
return
# Arrange
queue = await self._create_queue(qsc)
queue = await self._create_queue(qsc, message_encode_policy=None, message_decode_policy=None)
queue.key_encryption_key = KeyWrapper('key1')
queue._config.message_encode_policy = NoEncodePolicy()
queue._config.message_decode_policy = NoDecodePolicy()

message_dict = {'val1': 1, 'val2': '2'}
json_text = dumps(message_dict)
Expand Down