az keyvault secret fails with error Session.request() got an unexpected keyword argument 'enable_cae'

[stephan-uhlmann]

• Package Name: azure-keyvault-secrets
• Package Version: 4.9.0
• Operating System: Linux (openSUSE Tumbleweed)
• Python Version: 3.11.10

Describe the bug
Since the last update of azure-keyvault-secrets the az keyvault secret commands fail with the error

Session.request() got an unexpected keyword argument 'enable_cae'

To Reproduce
Steps to reproduce the behavior:

1. Example command which fails: az keyvault secret list --vault-name myvault
2. Also: az keyvault secret show --vault-name myvault --name mysecret --query value -o tsv

Expected behavior
Command runs successfully.

Screenshots

I'll post a debug log as an additional comment, to keep this descripton readable.

Additional context

Commands az keyvault list run successfully.
Command az keyvault certificate list --vault-name myvault also fails with the same error (azure-keyvault-certificates==4.9.0).
Maybe there are more, I can test others as well if needed.

[stephan-uhlmann]

Debug log (tenantId zeroed out)

DEBUG: cli.knack.cli: Command arguments: ['keyvault', 'secret', 'list', '--vault-name', 'mysecrets', '--debug']
DEBUG: cli.knack.cli: __init__ debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f90bece68e0>, <function OutputProducer.on_global_arguments at 0x7f90beaa1f80>, <function CLIQuery.on_global_arguments at 0x7f90beadfa60>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'keyvault': ['azure.cli.command_modules.keyvault']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands
DEBUG: cli.azure.cli.core: keyvault                  0.004        20       113
DEBUG: cli.azure.cli.core: Total (1)                 0.004        20       113
DEBUG: cli.azure.cli.core: Loaded 20 groups, 113 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command  : keyvault secret list
DEBUG: cli.azure.cli.core: Command table: keyvault secret list
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f90bdd20900>]
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/su/.azure/commands/2024-10-24.18-37-04.keyvault_secret_list.14289.log'.
INFO: az_command_data_logger: command args: keyvault secret list --vault-name {} --debug
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f90bdaf6d40>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f90bdb1cea0>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f90bdb1cfe0>, <function register_upcoming_breaking_change_info.<locals>.update_breaking_change_info at 0x7f90bdb1d080>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f90beaa2020>, <function CLIQuery.handle_query_parameter at 0x7f90beadfb00>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f90bdb1cf40>]
DEBUG: cli.azure.cli.core.auth.persistence: build_persistence: location='/home/su/.azure/msal_token_cache.json', encrypt=False
DEBUG: cli.azure.cli.core.auth.binary_cache: load: /home/su/.azure/msal_http_cache.bin
DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG: msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000
DEBUG: msal.authority: openid_config("https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
DEBUG: msal.application: Broker enabled? None
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): mysecrets.vault.azure.net:443
DEBUG: urllib3.connectionpool: https://mysecrets.vault.azure.net:443 "GET /secrets?api-version=7.4 HTTP/11" 401 97
DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://vault.azure.net/.default',), kwargs={'claims': None, 'tenant_id': '00000000-0000-0000-0000-000000000000', 'enable_cae': True}
DEBUG: cli.azure.cli.core.auth.msal_credentials: UserCredential.get_token: scopes=('https://vault.azure.net/.default',), claims=None, kwargs={'enable_cae': True}
DEBUG: msal.application: Found 2 RTs matching {'environment': 'login.microsoftonline.com', 'home_account_id': '********.00000000-0000-0000-0000-000000000000', 'family_id': '1'}
DEBUG: msal.telemetry: Generate or reuse correlation_id: f77eb449-2615-47dc-b228-d9f4cec4769a
DEBUG: msal.application: Cache attempts an RT
DEBUG: cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/azure/cli/command_modules/keyvault/_command_type.py", line 113, in keyvault_command_handler
    return _encode_hex(transform_result(result, **{**command_args, 'kv_transform': True}))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/cli/command_modules/keyvault/_transformers.py", line 12, in _multi_transformers
    output = t(output, **command_args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/cli/command_modules/keyvault/_transformers.py", line 29, in filter_out_managed_resources
    return [_ for _ in output if not getattr(_, 'managed')] if output else output
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/cli/command_modules/keyvault/_transformers.py", line 29, in <listcomp>
    return [_ for _ in output if not getattr(_, 'managed')] if output else output
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/core/paging.py", line 123, in __next__
    return next(self._page_iterator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/core/paging.py", line 75, in __next__
    self._response = self._get_next(self.continuation_token)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/keyvault/secrets/_generated/_operations/_operations.py", line 850, in get_next
    pipeline_response: PipelineResponse = self._client._pipeline.run(  # pylint: disable=protected-access
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/core/pipeline/_base.py", line 229, in run
    return first_node.send(pipeline_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
               ^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 2 more times]
  File "/usr/lib/python3.11/site-packages/azure/core/pipeline/policies/_redirect.py", line 197, in send
    response = self.next.send(request)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/core/pipeline/policies/_retry.py", line 532, in send
    response = self.next.send(request)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/keyvault/secrets/_shared/challenge_auth_policy.py", line 120, in send
    return self.handle_challenge_flow(request, response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/keyvault/secrets/_shared/challenge_auth_policy.py", line 149, in handle_challenge_flow
    request_authorized = self.on_challenge(request, response)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/keyvault/secrets/_shared/challenge_auth_policy.py", line 239, in on_challenge
    self.authorize_request(request, scope, claims=challenge.claims, tenant_id=challenge.tenant_id)
  File "/usr/lib/python3.11/site-packages/azure/core/pipeline/policies/_authentication.py", line 133, in authorize_request
    self._request_token(*scopes, **kwargs)
  File "/usr/lib/python3.11/site-packages/azure/core/pipeline/policies/_authentication.py", line 96, in _request_token
    self._token = cast(TokenCredential, self._credential).get_token(*scopes, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/cli/core/auth/credential_adaptor.py", line 65, in get_token
    token, _ = self._get_token(scopes, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/cli/core/auth/credential_adaptor.py", line 38, in _get_token
    token = self._credential.get_token(*scopes, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/cli/core/auth/msal_credentials.py", line 60, in get_token
    result = self._msal_app.acquire_token_silent_with_error(list(scopes), self._account, claims_challenge=claims,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/application.py", line 1416, in acquire_token_silent_with_error
    return _clean_up(self._acquire_token_silent_with_error(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/application.py", line 1439, in _acquire_token_silent_with_error
    result = self._acquire_token_silent_from_cache_and_possibly_refresh_it(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/application.py", line 1574, in _acquire_token_silent_from_cache_and_possibly_refresh_it
    result = self._acquire_token_silent_by_finding_rt_belongs_to_me_or_my_family(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/application.py", line 1635, in _acquire_token_silent_by_finding_rt_belongs_to_me_or_my_family
    last_resp = at = self._acquire_token_silent_by_finding_specific_refresh_token(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/application.py", line 1683, in _acquire_token_silent_by_finding_specific_refresh_token
    response = client.obtain_token_by_refresh_token(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/oauth2cli/oauth2.py", line 835, in obtain_token_by_refresh_token
    resp = super(Client, self).obtain_token_by_refresh_token(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/oauth2cli/oauth2.py", line 265, in obtain_token_by_refresh_token
    return self._obtain_token("refresh_token", data=data, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/oauth2cli/oidc.py", line 170, in _obtain_token
    ret = super(Client, self)._obtain_token(grant_type, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/oauth2cli/oauth2.py", line 776, in _obtain_token
    resp = super(Client, self)._obtain_token(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/oauth2cli/oauth2.py", line 237, in _obtain_token
    resp = (post or self._http_client.post)(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/individual_cache.py", line 269, in wrapper
    value = function(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/msal/individual_cache.py", line 269, in wrapper
    value = function(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Session.request() got an unexpected keyword argument 'enable_cae'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 666, in execute
    raise ex
  File "/usr/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 733, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 703, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 336, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/cli/command_modules/keyvault/_command_type.py", line 135, in keyvault_command_handler
    return keyvault_exception_handler(ex)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/azure/cli/command_modules/keyvault/_command_type.py", line 49, in keyvault_exception_handler
    raise CLIError(ex)
knack.util.CLIError: Session.request() got an unexpected keyword argument 'enable_cae'

ERROR: cli.azure.cli.core.azclierror: Session.request() got an unexpected keyword argument 'enable_cae'
ERROR: az_command_data_logger: Session.request() got an unexpected keyword argument 'enable_cae'
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f90bdd20b80>]
INFO: az_command_data_logger: exit code: 1
INFO: cli.__main__: Command ran in 0.626 seconds (init: 0.086, invoke: 0.541)
INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
INFO: telemetry.client: Accumulated 0 events. Flush the clients.
INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
INFO: telemetry.save: Save telemetry record of length 3891 in cache
INFO: telemetry.main: Begin creating telemetry upload process.
INFO: telemetry.process: Creating upload process: "/usr/bin/python3.11 /usr/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /home/su/.azure"
INFO: telemetry.process: Return from creating process
INFO: telemetry.main: Finish creating telemetry upload process.

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[laiapat]

Hi @stephan-uhlmann, thank you for opening this issue. Rolling back your versions of each azure-keyvault-* to the previous versions (e.g. 4.8.0 for azure-keyvault-secrets) should resolve the issue. More details are below.

enable_cae is passed to all token requests in the latest version of each azure-keyvault-* library. This is to enable Continuous Access Evaluation, which adds an extra layer of security to authentication.

The error you're seeing is caused by enable_cae being incorrectly handled by the CLI's underlying credential. The Azure CLI has pinned dependencies for Key Vault SDKs, so I'm surprised to see that the latest versions are being used and exposing this bug. cc @evelyn-ys @jiasli

[stephan-uhlmann]

Thanks for the quick reply! My Azure CLI version is 2.65.0. I install it as an RPM from my distributions (openSUSE Tumbleweed) repository, together with the Python modules. So it might be a packaging error by the distribution, that they don't use the pinned versions but the newest ones?

[stephan-uhlmann]

Looks like Azure/azure-cli#30062 will fix the issue?

[laiapat]

Sorry for the delay; I was out of office.

Looks like Azure/azure-cli#30062 will fix the issue?

Yes, that change should fortunately fix the keyword argument issue 🙂 I'll mark this as resolved since this was ultimately a CLI bug and they've merged a fix.

It looks like openSUSE has a bug tracker -- I didn't see anything that looked relevant when searching for "pinned", but it may be worth opening an issue if you can't find one. The CLI resolved a similar bug in the past by pinning their KV dependencies, and this is the first I've heard of this versioning issue since then. That makes me wonder if the distribution is fine and the conflict came from elsewhere.

If it is confirmed to be an error from the distribution though, please feel free to update this issue so that we can take that into account in the future. Thanks again for opening this issue!