allow configuring max timeout for get_secret function in azure-keyvault-secrets

[AlmogBentz]

related problem:
on key vault outage on 19.3.21 we experienced long running requests to key vault using the azure keyvault get_secret function.
seems like there is no timeout configured.

requested solution:
Please set a default timeout for your requests, and more importantly let user set a timeout for get_secret.

we really do not want to start wrapping functions with timeout decorators using subprocesses.

[chlowell]

Thank you for opening this issue! Key Vault clients--really, Azure SDK clients in general--do have a default timeout you can override with a keyword argument:

# all requests time out after 42 seconds
client = SecretClient(VAULT_URL, credential, timeout=42)

# this one request times out after 43 seconds
secret = client.get_secret(secret_name, timeout=43)

The default timeout is extremely long today due to #17481. We'll fix that soon. In the meantime you can work around it with the timeout argument.

[AlmogBentz]

Awesome! Thank you for the quick reply!

Did I miss it in the documentation? Is there someplace where all the posible keyword arguments are documented for functions like get secret?

[chlowell]

We maintain a table of them in the azure-core readme, which is part of that package's documentation (for example). We ought to connect that to client documentation. I'm uncertain of the best place for a link though. Did you consult any particular documentation while you were trying to sort this out?

[AlmogBentz]

Yes, I used this pypi link https://pypi.org/project/azure-keyvault-secrets/
https://azuresdkdocs.blob.core.windows.net/$web/python/azure-keyvault-secrets/latest/azure.keyvault.secrets.html#azure.keyvault.secrets.SecretClient

And also looked in github (same doc)

And couldnt find it.

[rez10191]

Hello,

I've been experimenting with this timeout value as well. It appears that there is a minimum of 120 seconds applied regardless of the value entered in timeout=. If I set timeout=0 it times out immediately without making the request, so I'm reasonably certain the parameter is being applied at some level. My test setup is running on an AWS server with no credentials and so I know it can never connect.

demo_keyvault_timeout.py.zip
Note all the timeout / retry parameters are set pretty aggressively trying to better demonstrate the issue.

I ran under python3.7 on a fresh venv on an ubuntu 18.04 with pip install azure.identity azure-keyvault-secrets

$ pip freeze

azure-common==1.1.27
azure-core==1.13.0
azure-identity==1.5.0
azure-keyvault-secrets==4.2.0
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
cryptography==3.4.7
idna==2.10
isodate==0.6.0
msal==1.11.0
msal-extensions==0.3.0
msrest==0.6.21
oauthlib==3.1.0
pkg-resources==0.0.0
portalocker==1.7.1
pycparser==2.20
PyJWT==2.1.0
requests==2.25.1
requests-oauthlib==1.3.0
six==1.15.0
urllib3==1.26.4

Example output is as follows:

2021-05-04 15:16:22,388 - __main__ - INFO - about to list_properties_of_secrets
2021-05-04 15:16:22,395 - __main__ - INFO - Request URL: 'https://foobar.vault.azure.net/secrets?api-version=REDACTED'
2021-05-04 15:16:22,395 - __main__ - INFO - Request method: 'GET'
2021-05-04 15:16:22,395 - __main__ - INFO - Request headers:
2021-05-04 15:16:22,395 - __main__ - INFO -     'Accept': 'application/json'
2021-05-04 15:16:22,395 - __main__ - INFO -     'x-ms-client-request-id': 'af570582-aceb-11eb-a667-0ac604366a34'
2021-05-04 15:16:22,395 - __main__ - INFO -     'User-Agent': 'azsdk-python-keyvault-secrets/4.2.0 Python/3.7.5 (Linux-5.4.0-1045-aws-x86_64-with-Ubuntu-18.04-bionic)'
2021-05-04 15:16:22,395 - __main__ - INFO -     'Content-Length': '0'
2021-05-04 15:16:22,395 - __main__ - INFO - No body was attached to the request
2021-05-04 15:16:22,607 - __main__ - INFO - Response status: 401
2021-05-04 15:16:22,608 - __main__ - INFO - Response headers:
2021-05-04 15:16:22,608 - __main__ - INFO -     'Cache-Control': 'no-cache'
2021-05-04 15:16:22,609 - __main__ - INFO -     'Pragma': 'no-cache'
2021-05-04 15:16:22,609 - __main__ - INFO -     'Content-Length': '87'
2021-05-04 15:16:22,609 - __main__ - INFO -     'Content-Type': 'application/json; charset=utf-8'
2021-05-04 15:16:22,609 - __main__ - INFO -     'Expires': '-1'
2021-05-04 15:16:22,609 - __main__ - INFO -     'WWW-Authenticate': 'REDACTED'
2021-05-04 15:16:22,609 - __main__ - INFO -     'x-ms-keyvault-region': 'eastus'
2021-05-04 15:16:22,609 - __main__ - INFO -     'x-ms-client-request-id': 'af570582-aceb-11eb-a667-0ac604366a34'
2021-05-04 15:16:22,609 - __main__ - INFO -     'x-ms-request-id': '3b12702f-e3ee-419f-af8f-d29b99664769'
2021-05-04 15:16:22,609 - __main__ - INFO -     'x-ms-keyvault-service-version': '1.2.265.0'
2021-05-04 15:16:22,609 - __main__ - INFO -     'x-ms-keyvault-network-info': 'conn_type=Ipv4;addr=18.220.224.127;act_addr_fam=InterNetwork;'
2021-05-04 15:16:22,609 - __main__ - INFO -     'X-Powered-By': 'REDACTED'
2021-05-04 15:16:22,610 - __main__ - INFO -     'Strict-Transport-Security': 'REDACTED'
2021-05-04 15:16:22,611 - __main__ - INFO -     'X-Content-Type-Options': 'REDACTED'
2021-05-04 15:16:22,612 - __main__ - INFO -     'Date': 'Tue, 04 May 2021 15:16:22 GMT'
ManagedIdentityCredential.get_token failed: Unexpected response '<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
		 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
'
ChainedTokenCredential.get_token failed: ManagedIdentityCredential raised unexpected error "Unexpected response '<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
		 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
'"
ChainedTokenCredential failed to retrieve a token from the included credentials.
Attempted credentials:
	ManagedIdentityCredential: Unexpected response '<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
		 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
'
2021-05-04 15:18:22,733 - __main__ - ERROR - ChainedTokenCredential failed to retrieve a token from the included credentials.
Attempted credentials:
	ManagedIdentityCredential: Unexpected response '<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
		 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
'
Traceback (most recent call last):
  File "demo_keyvault_timeout.py", line 81, in load_secret_from_azure
    retry_backoff_factor=0.1
  File "demo_keyvault_timeout.py", line 69, in <listcomp>
    secret.name for secret in client.list_properties_of_secrets(
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/core/paging.py", line 129, in __next__
    return next(self._page_iterator)
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/core/paging.py", line 76, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/keyvault/secrets/_generated/v7_1/operations/_key_vault_client_operations.py", line 359, in get_next
    pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/core/pipeline/_base.py", line 211, in run
    return first_node.send(pipeline_request)  # type: ignore
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/core/pipeline/_base.py", line 71, in send
    response = self.next.send(request)
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/core/pipeline/_base.py", line 71, in send
    response = self.next.send(request)
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/core/pipeline/_base.py", line 71, in send
    response = self.next.send(request)
  [Previous line repeated 2 more times]
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/core/pipeline/policies/_redirect.py", line 158, in send
    response = self.next.send(request)
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/core/pipeline/policies/_retry.py", line 445, in send
    response = self.next.send(request)
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/keyvault/secrets/_shared/challenge_auth_policy.py", line 111, in send
    self._handle_challenge(request, challenge)
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/keyvault/secrets/_shared/challenge_auth_policy.py", line 137, in _handle_challenge
    self._token = self._credential.get_token(scope)
  File "/home/ubuntu/squashfs-root/usr/lib/python3.7/site-packages/hlazureutils/venv/lib/python3.7/site-packages/azure/identity/_credentials/chained.py", line 90, in get_token
    raise ClientAuthenticationError(message=message)
azure.core.exceptions.ClientAuthenticationError: ChainedTokenCredential failed to retrieve a token from the included credentials.
Attempted credentials:
	ManagedIdentityCredential: Unexpected response '<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
		 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
'
2021-05-04 15:18:22,735 - __main__ - WARNING - Failed to lookup key from Azure key vault! - ChainedTokenCredential failed to retrieve a token from the included credentials.
Attempted credentials:
	ManagedIdentityCredential: Unexpected response '<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
		 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
'

Note the two minute delay between the initial logs and the exception getting thrown to the client. I am aware that ManagedIdentityCredential won't work outside of Azure. The intention is that this is part of a larger wrapper that is agnostic to the hosting platform, and it is desirable to avoid a long timeout like this if possible.

Thanks for any insight!

[chlowell]

These HTTP configuration kwargs are documented in the azure-core README. Client and credential instances have independent configuration, so your settings for SecretClient don't automatically apply to ManagedIdentityCredential. You can manually apply the same configuration to both. For example, tweaking your code:

pipeline_configuration = {
    # Note "logging_enable" controls DEBUG level logging that contains authentication secrets.
    # Ensure your logs are secure before enabling it
    "logging_enable": True,  
    "logger": logger,
    "timeout": 2,
    "connection_timeout": 1.0,
    "read_timeout": 1.0,
    "retry_total": 1,
}

credential = ManagedIdentityCredential(**pipeline_configuration)

client = SecretClient(
    vault_url="https://{}.vault.azure.net".format(key_vault_name),
    credential=credential,
    **pipeline_configuration
)

In general the same arguments allow you to override a client's configuration for a single request, as in your code:

client.list_properties_of_secrets(
    logging_enable=True,
    logger=logger,
    timeout=0.5,
    ...

However, configuration set this way doesn't apply to an authentication request. There's no way to override a credential's HTTP configuration for a single request today (#7989).

[rez10191]

I updated the demo code as you suggested and saw a similar 120s delay in my test setup (not running on an azure server).

demo_keyvault_timeout.py.zip

Just to make sure I'm following: this is expected / would be resolved by #7989, or is it an unrelated issue?

As an interesting aside, when I run this code on my development PC, it times out swiftly with a more reasonable exception.
ManagedIdentityCredential.get_token failed: ManagedIdentityCredential authentication unavailable, no managed identity endpoint found. The only difference that springs to mind is that I have the azure-cli package installed, but the test code doesn't use AzureCliCredential in the chain. pip freeze matches the output as my AWS test setup.

If this is unrelated, I can open a separate issue as I didn't mean to hijack this one if it is unrelated to the configuration @AlmogBentz had originally asked about.

[chlowell]

Your 120s delay is caused by a couple overlapping issues unrelated to Key Vault:

1. ManagedIdentityCredential ignores your retry_total setting when it thinks it's on an Azure VM
   • fixed on our master branch, which will release as azure-identity 1.6.0 next week
   • but note the credential's default retry configuration is aligned with the platform's guidance; overriding it could cause a client to raise an error instead of retrying when deployed on an Azure VM
2. Our HTTP retry policy will sleep after an operation has taken longer than your timeout setting
   • fixed by Retry policies don't sleep after operations time out #18548

Given your settings, with both those fixes it will take ~8 seconds for ManagedIdentityCredential to give up on getting a token from AWS (#18552 and #18554 would further reduce the delay).

#7989 tracks applying per-request configuration to authentication requests. That is to say, having client.list_properties_of_secrets(timeout=1) apply timeout=1 to an authentication request. Resolving it wouldn't resolve your problem, I mention it to explain why those settings don't affect ManagedIdentityCredential in your code.