Skip to content

[Cosmos] Add documentation around client retries #28433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
simorenoh merged 5 commits into from
Mar 16, 2023
Merged

[Cosmos] Add documentation around client retries #28433

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f1444a4
Create ErrorCodesAndRetries.md
simorenoh Jan 20, 2023
c82013c
Update ErrorCodesAndRetries.md
simorenoh Feb 14, 2023
6d5fc43
small changes
simorenoh Mar 16, 2023
ed7021e
Update _retry_utility.py
simorenoh Mar 16, 2023
48ad09b
Merge branch 'main' into retry-docs
simorenoh Mar 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions sdk/cosmos/azure-cosmos/azure/cosmos/_retry_utility.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,8 @@ def Execute(client, global_endpoint_manager, function, *args, **kwargs):

return result
except exceptions.CosmosHttpResponseError as e:
retry_policy = None
retry_policy = defaultRetry_policy
# Re-assign retry policy based on error code
if e.status_code == StatusCodes.FORBIDDEN and e.sub_status == SubStatusCodes.WRITE_FORBIDDEN:
retry_policy = endpointDiscovery_retry_policy
elif e.status_code == StatusCodes.TOO_MANY_REQUESTS:
Expand All @@ -111,8 +112,6 @@ def Execute(client, global_endpoint_manager, function, *args, **kwargs):
retry_policy = partition_key_range_gone_retry_policy
elif e.status_code == StatusCodes.REQUEST_TIMEOUT or e.status_code == StatusCodes.SERVICE_UNAVAILABLE:
retry_policy = timeout_failover_retry_policy
else:
retry_policy = defaultRetry_policy

# If none of the retry policies applies or there is no retry needed, set the
# throttle related response headers and re-throw the exception back arg[0]
Expand Down
18 changes: 18 additions & 0 deletions sdk/cosmos/azure-cosmos/docs/ErrorCodesAndRetries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
## Cosmos DB Python SDK – Detailing Exceptions and Retries

The Cosmos DB Python SDK has several default policies that will deal with retrying certain errors and exceptions. More information on these can be found below.

| Status code | Cause of exception and retry behavior |
| :--- |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 400 | For all operations: </br><ul><li> This exception is encountered when the request is invalid, which could be for any of the following reasons: </br><ul><li>Syntax error in query text</li><li>Malformed JSON document for a write request</li><li>Incorrectly formatted REST API request body etc.</li></ul></li><li>The client does NOT retry the request when a Bad Request (400) exception is thrown by the server.</li></ul> |
| 401 | For all operations: </br><ul><li> This is an unauthorized exception due to invalid auth tokens being used for the request. The client does NOT retry requests when this exception is encountered.</li></ul> |
| 403 | <ul><li>For Substatus 3 (Write Forbidden): </br><ul><li>This exception occurs when a geo-replicated database account runs into writable/readable location changes (say, after a failover).</li><li>This exception can occur regardless of the Consistency level set for the account. </li><li>The client refreshes it's location endpoints and retries requests when the user has enabled endpoint discovery in their client (default behavior).</li></ul></li><li>For all other cases: </br><ul><li> The client does NOT retry requests when this exception is encountered. </li> |
| 404/1002 | <ul><li>For write operations: </br><ul><li>If multiple write locations are enabled for the account, the SDK will fetch the write endpoints and retry once per each of these. </li><li>The client refreshes it's location endpoints and retries requests when the user has enabled endpoint discovery in their client (default behavior).</li><li>If the account does not have multiple write locations enabled, the SDK will retry only once. </li></ul></li><li>For read operations: </br><ul><li>If multiple write locations are enabled for the account, the SDK will fetch the read endpoints and retry once per each of these. </li><li>The client refreshes it's location endpoints and retries requests when the user has enabled endpoint discovery in their client (default behavior).</li><li>If the account does not have multiple write locations enabled, the SDK will retry only once. </li> |
| 408 | <ul><li>For Write Operations: <br><ul><li>Timeout exceptions can be encountered by both the client as well as the server. </li><li>Server-side timeout exceptions are not retried for write operations as it is not possible to determine if the write was in fact successfully committed on the server. To avoid overwriting a previously committed write, the client does NOT retry this operation and bubbles up the exception to the application as a Request Timeout Exception (408). </li><li>For a client-generated timeout exception, there are two scenarios: </br><ul><li>The request was sent over the wire to the server by the client, but the network request timeout exceeded, while waiting for a response. In this case, it is unclear if the write was received and committed by the server and thus, the operation is not retried. A timeout exception is bubbled up to the application by the client. </li><li>The request was not sent over the wire to the server which resulted in a client-generated timeout. The client does NOT retry.</li></ul></li><li>For Query and Point Read Operations:</br><ul><li>The request is retried locally for up to 120 attempts with exponential backoff. </li><li>The SDK will mark the current endpoint as unavailable, refresh its location based routing directives, and set the new endpoint before retrying.</li></ul> </li></ul> |

@xinlian12 xinlian12 Mar 16, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think in Java and .net, we are marking region as unavailable based on http timeout, as it is not a strong signal to say the region is down

| 409 | <ul><li>For Write Operations: </br><ul><li>This exception occurs when an attempt is made by the application to Create/Insert an Item that already exists.</li><li>This exception can occur regardless of the Consistency level set for the account. </li><li>This exception can occur for write operations when an attempt is made to create an existing item or when a unique key constraint violation occurs. </li><li>The client does NOT retry on Conflict exceptions </li></ul></li><li>For Query and Point Read Operations: </br><ul><li>N/A as this exception is only encountered for Create/Insert operations. </li></ul></li> |
| 410/1002 | <ul><li>For all operations: </br><ul><li>This exception occurs when a partition is split (or merged in the future) and no longer exists, and can occur regardless of the Consistency level set for the account.</li><li>The SDK will refresh its partition key range cache and trigger a single retry, fetching the new ranges from the gateway once it finds an empty cache. </li> |
| 412 | <ul><li>For Write Operations: </br><ul><li>This exception is encountered when the etag that is sent to the server for validation prior to updating an Item, does not match the etag of the Item on the server. </li><li>The client does NOT retry this operation locally or against any of the remote regions for the account as retries would not help alleviate the etag mismatch. </li><li>The application would need to trigger a retry by first reading the Item, fetching the latest etag and issuing the Upsert/Replace operation. </br><ul><li>This operation can continue to fail with the same exception when multiple updates are executed concurrently for the same Item. </li><li>An upper bound on the number of retries before handing off the Item to a dead letter queue should be implemented by the application. </li></ul></li></ul></li><li>For Query and point read Operations: </br><ul><li>N/A as this exception is only encountered for Create/Insert/Replace/Upsert operations. </li></ul></li></ul> |
| 429 | For all Operations: </br><ul><li>By default, the client retries the request for a maximum of 9 times (or for a maximum of 30 seconds, whichever limit is reached first). </li><li>The client can also be initialized with a custom retry policy, which overrides the two limits mentioned above. </li><li>After all the retries are exhausted, the client bubbles up the exception to the application. </li><li>**For a multi-region account**, the client does NOT retry the request against a remote region for the account. </li><li>When the application receives a Request Rate too large exception (429), the application would need to instrument its own retry logic and dead letter queues. </li></ul> |
| 449 | <ul><li>For Write Operations: </br><ul><li>This exception is encountered when a resource is concurrently updated on the server, which can happen due to concurrent writes, user triggered while conflicts are concurrently being resolved etc. </li><li>Only one update can be executed at a time per item. The other concurrent requests will fail with a Concurrent Execution Exception (449). </li><li>The client does NOT retry requests that failed with a 449. </li></ul></li><li>For Query and point read Operations: </br><ul><li>N/A as this exception is only encountered for Create/Insert/Replace/Upsert operations. </li></ul></li></ul> |
Comment thread
simorenoh marked this conversation as resolved.
| 500 | For all Operations: </br><ul><li>The occurrence of an Invalid Exception (500) is extremely rare, and the client does NOT retry a request that encounters this exception. </li></ul> |
| 503 | When a Service Unavailable exception is encountered: </br><ul><li>For data plane Write Operations: </br><ul><li>The request will NOT be retried by the SDK, and will get bubbled up to the user as a ServiceRequestError. </li></ul><li>For Read/Query operations: </br><ul><li> The SDK will mark the current endpoint as unavailable, refresh its location based routing directives, and set the new endpoint before retrying **once**, and after this retry is exhausted the exception is bubbled up to the application as a ServiceRequestError. </li></ul><li>For a metadata operation to retrieve the address of replicas and query plans: </br><ul><li> The SDK will mark the current endpoint as unavailable, refresh its location based routing directives, and set the new endpoint before retrying once, and after this retry is exhausted the exception is bubbled up to the application as a ServiceRequestError.</li></ul></li></ul></li></li></li></ul></li></li></ul></li></ul></li></ul> |