Skip to content

test(integ): absorb iam:SimulatePrincipalPolicy throttling across integ suites - #6081

Merged
jam-jee merged 1 commit into
aws:masterfrom
jam-jee:fix/integ-iam-simulate-throttle
Jul 23, 2026
Merged

test(integ): absorb iam:SimulatePrincipalPolicy throttling across integ suites#6081
jam-jee merged 1 commit into
aws:masterfrom
jam-jee:fix/integ-iam-simulate-throttle

Conversation

@jam-jee

@jam-jee jam-jee commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Issue

Integration tests run under pytest -n auto (dozens of xdist workers). Many of them resolve/validate an IAM execution role via resolve_and_validate_role — invoked from TrainDefaults.get_role, ModelBuilder.build()/.deploy(), Pipeline create/upsert, and the feature-processor scheduler. That function calls the low-TPS iam:SimulatePrincipalPolicy API (plus a get_role for the trust check). With many workers hitting it at once, IAM throttles the request:

botocore.exceptions.ClientError: An error occurred (Throttling) when calling the
SimulatePrincipalPolicy operation (reached max retries: 4): Rate exceeded

_evaluate_permissions only degrades AccessDenied/NoSuchEntity; Throttling re-raises and fails the test — often during fixture setup, not just the call phase.

Fix (test layer only — no SDK source change)

Add an identical, self-contained mitigation to the serve, train, and mlops integ conftests:

  1. _configure_boto_adaptive_retries (autouse, session-scoped) — sets adaptive retries via the AWS_RETRY_MODE / AWS_MAX_ATTEMPTS environment variables. Env vars apply to every boto3 client created in the worker, so the IAM calls ride out transient throttling whether the resolver falls back to the default session or builds its client from an explicitly-passed Session. Pre-existing env values are restored on teardown.
  2. pytest_runtest_makereport — belt-and-suspenders: converts a residual SimulatePrincipalPolicy throttle that survives retries into a skip (checked in both setup and call phases). Throttling on any other operation still fails loudly.

Why not the previous serve-only approach?

The prior serve conftest set a retry Config on boto3.DEFAULT_SESSION. That only helps when the resolver falls back to the default session — several serve tests (and the train/mlops fixtures) pass their own Session(boto_session=...), whose IAM client carried botocore's default 4-attempt policy and was unaffected. Env vars close that gap and let all three suites share one implementation.

Scope

  • New parent sagemaker-train/tests/integ/conftest.py covers train/, ai_registry/, and jumpstart/.
  • sagemaker-core integ is intentionally not modified — resolve_and_validate_role has no call path there, so it never hits SimulatePrincipalPolicy.

Testing

  • All three conftests compile (py_compile) and lint clean.
  • Unit-checked the throttle-detection helper: matches Throttling/ThrottlingException/RequestLimitExceeded only on SimulatePrincipalPolicy; ignores other ops, other error codes, and non-ClientError exceptions.

Type of change

  • Bug fix / test-infrastructure fix (non-breaking; test-layer only)

Checklist

  • I have read the CONTRIBUTING doc
  • Changes are limited to test conftests; no SDK source or public API is touched
  • No new dependencies introduced

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Integ tests run under pytest -n auto (dozens of xdist workers). Many resolve/
validate an IAM execution role via resolve_and_validate_role, which calls the
low-TPS iam:SimulatePrincipalPolicy API. Under concurrent load IAM throttles it
(ClientError: Throttling / Rate exceeded), failing tests during setup or build.

Add an identical test-harness mitigation to the serve, train, and mlops integ
conftests:
- autouse session fixture sets adaptive retries via AWS_RETRY_MODE /
  AWS_MAX_ATTEMPTS env vars, so EVERY boto client in the worker inherits them
  (including IAM clients the resolver builds from an explicitly-passed Session,
  which the previous serve-only DEFAULT_SESSION approach missed).
- pytest_runtest_makereport converts residual SimulatePrincipalPolicy throttling
  into a skip (setup + call phases). Throttling on any other op still fails loud.

train gets a new parent tests/integ/conftest.py covering train/, ai_registry/,
and jumpstart/. sagemaker-core integ is unaffected (no resolver call path).


@pytest.fixture(autouse=True, scope="session")
def _configure_boto_adaptive_retries():

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think retries are a good idea.

But instead of skipping can we do the following:
Can we go to IAM team to increase rate limit
Or add feature to skip iam validations

@jam-jee
jam-jee merged commit a76fb88 into aws:master Jul 23, 2026
16 of 25 checks passed
@lucasjia-aws

Copy link
Copy Markdown
Collaborator

with batch-call we should not actually go into skip branch right? But we should keep monitoring if tests skips happen a lot with this change

jam-jee added a commit to jam-jee/sagemaker-python-sdk that referenced this pull request Jul 23, 2026
…merge artifacts

The master merge into this branch reintroduced the pytest_runtest_makereport
skip hook in the train and mlops integ conftests (both landed by aws#6081), so a
SimulatePrincipalPolicy throttle surviving the retries would still be silently
skipped there. Remove the hook and its now-unused helper/constants from both, so
exhausted throttling fails loudly in every suite.

Also fix two artifacts the merge left in the serve conftest:
- restore the fixture teardown that resets AWS_RETRY_MODE / AWS_MAX_ATTEMPTS
  (the 'previous' dict was captured but never restored -> unused-variable lint
  and env leak across the session);
- update the stale docstring that still described the removed
  _configure_default_boto_retries / DEFAULT_SESSION approach.
lucasjia-aws pushed a commit that referenced this pull request Jul 23, 2026
…6094)

* test(integ): let exhausted IAM throttling fail instead of skipping

The prior mitigation converted a SimulatePrincipalPolicy throttle that survived
the adaptive retries into a skipped test (pytest_runtest_makereport). That hid a
persistent rate-limit regression: a genuinely throttled run would silently drop
out of the results instead of showing up as a failure.

Keep the autouse adaptive-retry fixture (it still absorbs transient bursts), but
remove the skip conversion and its now-unused helper/constants. Throttling that
exhausts the retry budget now fails the test loudly so the regression is visible.

* test(integ): remove throttle skip hook from train & mlops; fix serve merge artifacts

The master merge into this branch reintroduced the pytest_runtest_makereport
skip hook in the train and mlops integ conftests (both landed by #6081), so a
SimulatePrincipalPolicy throttle surviving the retries would still be silently
skipped there. Remove the hook and its now-unused helper/constants from both, so
exhausted throttling fails loudly in every suite.

Also fix two artifacts the merge left in the serve conftest:
- restore the fixture teardown that resets AWS_RETRY_MODE / AWS_MAX_ATTEMPTS
  (the 'previous' dict was captured but never restored -> unused-variable lint
  and env leak across the session);
- update the stale docstring that still described the removed
  _configure_default_boto_retries / DEFAULT_SESSION approach.
@github-actions github-actions Bot mentioned this pull request Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants