test(integ): absorb iam:SimulatePrincipalPolicy throttling across integ suites - #6081
Merged
Merged
Conversation
Integ tests run under pytest -n auto (dozens of xdist workers). Many resolve/ validate an IAM execution role via resolve_and_validate_role, which calls the low-TPS iam:SimulatePrincipalPolicy API. Under concurrent load IAM throttles it (ClientError: Throttling / Rate exceeded), failing tests during setup or build. Add an identical test-harness mitigation to the serve, train, and mlops integ conftests: - autouse session fixture sets adaptive retries via AWS_RETRY_MODE / AWS_MAX_ATTEMPTS env vars, so EVERY boto client in the worker inherits them (including IAM clients the resolver builds from an explicitly-passed Session, which the previous serve-only DEFAULT_SESSION approach missed). - pytest_runtest_makereport converts residual SimulatePrincipalPolicy throttling into a skip (setup + call phases). Throttling on any other op still fails loud. train gets a new parent tests/integ/conftest.py covering train/, ai_registry/, and jumpstart/. sagemaker-core integ is unaffected (no resolver call path).
mujtaba1747
reviewed
Jul 23, 2026
|
|
||
|
|
||
| @pytest.fixture(autouse=True, scope="session") | ||
| def _configure_boto_adaptive_retries(): |
Collaborator
There was a problem hiding this comment.
I think retries are a good idea.
But instead of skipping can we do the following:
Can we go to IAM team to increase rate limit
Or add feature to skip iam validations
lucasjia-aws
approved these changes
Jul 23, 2026
Collaborator
|
with batch-call we should not actually go into skip branch right? But we should keep monitoring if tests skips happen a lot with this change |
4 tasks
jam-jee
added a commit
to jam-jee/sagemaker-python-sdk
that referenced
this pull request
Jul 23, 2026
…merge artifacts The master merge into this branch reintroduced the pytest_runtest_makereport skip hook in the train and mlops integ conftests (both landed by aws#6081), so a SimulatePrincipalPolicy throttle surviving the retries would still be silently skipped there. Remove the hook and its now-unused helper/constants from both, so exhausted throttling fails loudly in every suite. Also fix two artifacts the merge left in the serve conftest: - restore the fixture teardown that resets AWS_RETRY_MODE / AWS_MAX_ATTEMPTS (the 'previous' dict was captured but never restored -> unused-variable lint and env leak across the session); - update the stale docstring that still described the removed _configure_default_boto_retries / DEFAULT_SESSION approach.
lucasjia-aws
pushed a commit
that referenced
this pull request
Jul 23, 2026
…6094) * test(integ): let exhausted IAM throttling fail instead of skipping The prior mitigation converted a SimulatePrincipalPolicy throttle that survived the adaptive retries into a skipped test (pytest_runtest_makereport). That hid a persistent rate-limit regression: a genuinely throttled run would silently drop out of the results instead of showing up as a failure. Keep the autouse adaptive-retry fixture (it still absorbs transient bursts), but remove the skip conversion and its now-unused helper/constants. Throttling that exhausts the retry budget now fails the test loudly so the regression is visible. * test(integ): remove throttle skip hook from train & mlops; fix serve merge artifacts The master merge into this branch reintroduced the pytest_runtest_makereport skip hook in the train and mlops integ conftests (both landed by #6081), so a SimulatePrincipalPolicy throttle surviving the retries would still be silently skipped there. Remove the hook and its now-unused helper/constants from both, so exhausted throttling fails loudly in every suite. Also fix two artifacts the merge left in the serve conftest: - restore the fixture teardown that resets AWS_RETRY_MODE / AWS_MAX_ATTEMPTS (the 'previous' dict was captured but never restored -> unused-variable lint and env leak across the session); - update the stale docstring that still described the removed _configure_default_boto_retries / DEFAULT_SESSION approach.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Integration tests run under
pytest -n auto(dozens of xdist workers). Many of them resolve/validate an IAM execution role viaresolve_and_validate_role— invoked fromTrainDefaults.get_role,ModelBuilder.build()/.deploy(),Pipelinecreate/upsert, and the feature-processor scheduler. That function calls the low-TPSiam:SimulatePrincipalPolicyAPI (plus aget_rolefor the trust check). With many workers hitting it at once, IAM throttles the request:_evaluate_permissionsonly degradesAccessDenied/NoSuchEntity;Throttlingre-raises and fails the test — often during fixture setup, not just the call phase.Fix (test layer only — no SDK source change)
Add an identical, self-contained mitigation to the serve, train, and mlops integ conftests:
_configure_boto_adaptive_retries(autouse, session-scoped) — sets adaptive retries via theAWS_RETRY_MODE/AWS_MAX_ATTEMPTSenvironment variables. Env vars apply to every boto3 client created in the worker, so the IAM calls ride out transient throttling whether the resolver falls back to the default session or builds its client from an explicitly-passedSession. Pre-existing env values are restored on teardown.pytest_runtest_makereport— belt-and-suspenders: converts a residualSimulatePrincipalPolicythrottle that survives retries into a skip (checked in bothsetupandcallphases). Throttling on any other operation still fails loudly.Why not the previous serve-only approach?
The prior serve conftest set a retry
Configonboto3.DEFAULT_SESSION. That only helps when the resolver falls back to the default session — several serve tests (and the train/mlops fixtures) pass their ownSession(boto_session=...), whose IAM client carried botocore's default 4-attempt policy and was unaffected. Env vars close that gap and let all three suites share one implementation.Scope
sagemaker-train/tests/integ/conftest.pycoverstrain/,ai_registry/, andjumpstart/.sagemaker-coreinteg is intentionally not modified —resolve_and_validate_rolehas no call path there, so it never hitsSimulatePrincipalPolicy.Testing
py_compile) and lint clean.Throttling/ThrottlingException/RequestLimitExceededonly onSimulatePrincipalPolicy; ignores other ops, other error codes, and non-ClientErrorexceptions.Type of change
Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.