Flaky Emulator Tests

[bryevdv]

There are a handful of flaky Cosmos Emulator tests that are currently failing unrelated PRs (even after a recent fix for emulator/live discrepancies was recently merged, AFAIK).

Specifically: #8607 and #8548 at least are not passing due to Cosmos emulator tests.

The tests often seem to be these query and retry tests:

self = <query_tests.QueryTest testMethod=test_max_item_count_honored_in_order_by_query>

    def test_max_item_count_honored_in_order_by_query(self):
        created_collection = self.config.create_multi_partition_collection_with_custom_pk_if_not_exist(self.client)
        docs = []
        for i in range(10):
            document_definition = {'pk': 'pk', 'id': 'myId' + str(uuid.uuid4())}
            docs.append(created_collection.create_item(body=document_definition))
    
        query = 'SELECT * from c ORDER BY c._ts'
        query_iterable = created_collection.query_items(
            query=query,
            max_item_count=1,
            enable_cross_partition_query=True
        )
>       self.validate_query_requests_count(query_iterable, 15 * 2 + 1)

test\query_tests.py:193: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test\query_tests.py:210: in validate_query_requests_count
    self.assertEqual(self.count, expected_count)
E   AssertionError: 33 != 31
_________ Test_retry_policy_tests.test_default_retry_policy_for_query _________

self = <retry_policy_tests.Test_retry_policy_tests testMethod=test_default_retry_policy_for_query>

    def test_default_retry_policy_for_query(self):
        document_definition_1 = { 'id': 'doc1',
                                  'name': 'sample document',
                                  'key': 'value'}
        document_definition_2 = { 'id': 'doc2',
                                  'name': 'sample document',
                                  'key': 'value'}
    
        self.created_collection.create_item(body=document_definition_1)
        self.created_collection.create_item(body=document_definition_2)
    
        try:
            original_execute_function = _retry_utility.ExecuteFunction
            mf = self.MockExecuteFunctionConnectionReset(original_execute_function)
            _retry_utility.ExecuteFunction = mf
    
            docs = self.created_collection.query_items(query="Select * from c", max_item_count=1, enable_cross_partition_query=True)
    
            result_docs = list(docs)
            self.assertEqual(result_docs[0]['id'], 'doc1')
            self.assertEqual(result_docs[1]['id'], 'doc2')
    
            # TODO: Differing result between live and emulator
            if 'localhost' in self.host or '127.0.0.1' in self.host:
>               self.assertEqual(mf.counter, 12)
E               AssertionError: 18 != 12

test\retry_policy_tests.py:198: AssertionError

@annatisch suggested

I wonder if we can mark these as xfailures so as not to break CI for everyone

Which I am inclined to agree is worth considering. But this is also probably worth a wider discussion.

cc @mayurid @srinathnarayanan thoughts? Can we just mark these xfail for now?

[srinathnarayanan]

yes, we can mark it as xfail. i was going to rework these tests in a different PR.

[bryevdv]

OK will do it as part of #8607 to get things passing there.