QueueClient.receive_messages() receives more than one message

[markkuleinio]

• Package Name: azure-storage-queue
• Package Version: 12.1.1
• Operating System: Debian Linux 10
• Python Version: 3.7.3

Describe the bug
QueueClient.receive_messages(messages_per_page=1) returns more than one message from the queue.

To reproduce

help(QueueClient.receive_messages) says:

Help on function receive_messages in module azure.storage.queue._queue_client:

receive_messages(self, **kwargs)
    Removes one or more messages from the front of the queue.
...
    :keyword int messages_per_page:
        A nonzero integer value that specifies the number of
        messages to retrieve from the queue, up to a maximum of 32. If
        fewer are visible, the visible messages are returned. By default,
        a single message is retrieved from the queue with this operation.

Testing code:

>>> from azure.storage.queue import QueueClient
>>> q = QueueClient(account_url="https://the_account_that_is_used.queue.core.windows.net", credential="xxx", queue_name="receive-test-2020-06-02", connection_timeout=10)
>>> q.get_queue_properties()
{'name': 'receive-test-2020-06-02', 'metadata': {}, 'approximate_message_count': 0}
>>> q.send_message("test1")
{'id': '28dd2ab0-ccdc-4f8e-a8a1-be6a8f980b21', 'inserted_on': datetime.datetime(2020, 6, 2, 6, 43, 9, tzinfo=datetime.timezone.utc), 'expires_on': datetime.datetime(2020, 6, 9, 6, 43, 9, tzinfo=datetime.timezone.utc), 'dequeue_count': None, 'content': 'test1', 'pop_receipt': 'AgAAAAMAAAAAAAAAkutnFKk41gE=', 'next_visible_on': datetime.datetime(2020, 6, 2, 6, 43, 9, tzinfo=datetime.timezone.utc)}
>>> q.send_message("test2")
{'id': '5944194a-1643-4614-9a5b-102cc2466544', 'inserted_on': datetime.datetime(2020, 6, 2, 6, 43, 17, tzinfo=datetime.timezone.utc), 'expires_on': datetime.datetime(2020, 6, 9, 6, 43, 17, tzinfo=datetime.timezone.utc), 'dequeue_count': None, 'content': 'test2', 'pop_receipt': 'AgAAAAMAAAAAAAAA2TX4GKk41gE=', 'next_visible_on': datetime.datetime(2020, 6, 2, 6, 43, 17, tzinfo=datetime.timezone.utc)}
>>> q.send_message("test3")
{'id': 'd61739d4-39f8-463c-b7f2-d71d5e8b1c2a', 'inserted_on': datetime.datetime(2020, 6, 2, 6, 43, 20, tzinfo=datetime.timezone.utc), 'expires_on': datetime.datetime(2020, 6, 9, 6, 43, 20, tzinfo=datetime.timezone.utc), 'dequeue_count': None, 'content': 'test3', 'pop_receipt': 'AgAAAAMAAAAAAAAAGeKTGqk41gE=', 'next_visible_on': datetime.datetime(2020, 6, 2, 6, 43, 20, tzinfo=datetime.timezone.utc)}
>>>
>>> msgs = q.receive_messages(visibility_timeout=120, messages_per_page=1)
>>> for msg_batch in msgs.by_page():
...     for msg in msg_batch:
...             print(msg.content)
...
test1
test2
test3
>>>

As you can see, all three messages were returned with the single call, even though messages_per_page is 1.

How do I get only one message from the queue?

One problem in this current state is the visibility_timeout: When there are lots of messages in the queue, receive_messages() gets them all, and the handling of each message takes a long time, it is hard to tune the visibility_timeout to a useful value.

Expected behavior
It is expected to be possible to fetch only one message from the queue.

Screenshots
See above

Additional context
The Python Azure storage packages are confusing: various GitHub projects (some abandoned?) and various documentation sites. Hopefully this is the current project.

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[xiafu-msft]

hi @markkuleinio
you need to do like this:

pages = QueueClient.receive_messages(messages_per_page=1).by_page()
next_page = next(pages)
items_in_page = list(a_page)

[markkuleinio]

Thanks @xiafu-msft, with that I tried this (using the same q as in my first message):

>>> def get_one_message(queue_client, visibility_timeout=60):
...     pages = queue_client.receive_messages(visibility_timeout=visibility_timeout, messages_per_page=1).by_page()
...     try:
...         page = next(pages)
...         msg = next(page)
...     except StopIteration:
...         return None
...     return msg
...
>>> msg = get_one_message(q, visibility_timeout=30)
>>> msg
>>> type(msg)
<class 'NoneType'>
>>> q.get_queue_properties()["approximate_message_count"]
0
>>>
>>> q.send_message("test1")
{'id': '3772a944-7ebc-4a08-8821-b6ec1e153317', 'inserted_on': datetime.datetime(2020, 6, 9, 5, 50, 5, tzinfo=datetime.timezone.utc), 'expires_on': datetime.datetime(2020, 6, 16, 5, 50, 5, tzinfo=datetime.timezone.utc), 'dequeue_count': None, 'content': 'test1', 'pop_receipt': 'AgAAAAMAAAAAAAAAtp9u0yE+1gE=', 'next_visible_on': datetime.datetime(2020, 6, 9, 5, 50, 5, tzinfo=datetime.timezone.utc)}
>>> q.send_message("test2")
{'id': 'd5abd2b8-3afc-4d78-be1d-1d457c4a6c52', 'inserted_on': datetime.datetime(2020, 6, 9, 5, 50, 8, tzinfo=datetime.timezone.utc), 'expires_on': datetime.datetime(2020, 6, 16, 5, 50, 8, tzinfo=datetime.timezone.utc), 'dequeue_count': None, 'content': 'test2', 'pop_receipt': 'AgAAAAMAAAAAAAAAGgFu1SE+1gE=', 'next_visible_on': datetime.datetime(2020, 6, 9, 5, 50, 8, tzinfo=datetime.timezone.utc)}
>>> q.send_message("test3")
{'id': 'f7b2bfcb-23ac-4dd2-826b-2478d300ccef', 'inserted_on': datetime.datetime(2020, 6, 9, 5, 50, 11, tzinfo=datetime.timezone.utc), 'expires_on': datetime.datetime(2020, 6, 16, 5, 50, 11, tzinfo=datetime.timezone.utc), 'dequeue_count': None, 'content': 'test3', 'pop_receipt': 'AgAAAAMAAAAAAAAARFTR1iE+1gE=', 'next_visible_on': datetime.datetime(2020, 6, 9, 5, 50, 11, tzinfo=datetime.timezone.utc)}
>>> q.get_queue_properties()["approximate_message_count"]
3
>>>
>>> msg = get_one_message(q, visibility_timeout=30)
>>> type(msg)
<class 'azure.storage.queue._models.QueueMessage'>
>>> msg.content
'test1'
>>> q.delete_message(msg)
>>> q.get_queue_properties()["approximate_message_count"]
2
>>> msg = get_one_message(q, visibility_timeout=30)
>>> type(msg)
<class 'azure.storage.queue._models.QueueMessage'>
>>> msg.content
'test2'
>>> q.delete_message(msg)
>>> q.get_queue_properties()["approximate_message_count"]
1
>>> msg = get_one_message(q, visibility_timeout=30)
>>> type(msg)
<class 'azure.storage.queue._models.QueueMessage'>
>>> msg.content
'test3'
>>> q.delete_message(msg)
>>> q.get_queue_properties()["approximate_message_count"]
0
>>> msg = get_one_message(q, visibility_timeout=30)
>>> type(msg)
<class 'NoneType'>
>>>

Seems to be working.

Please make sure that the receive_messages() docstring gets corrected.

Thanks!

[xiafu-msft]

Hi @markkuleinio

Thanks for the feedback, we will try to make the docstring more clear