Skip to content

Fix evaluation loop behavior when max_eval_steps is set - #42980

Closed
PRATHAMESHG1234 wants to merge 6 commits into
huggingface:mainfrom
PRATHAMESHG1234:eval-step-limit-clean
Closed

Fix evaluation loop behavior when max_eval_steps is set#42980
PRATHAMESHG1234 wants to merge 6 commits into
huggingface:mainfrom
PRATHAMESHG1234:eval-step-limit-clean

Conversation

@PRATHAMESHG1234

Copy link
Copy Markdown

This PR follows up on #41299 and fixes remaining issues around max_eval_steps.

  • Adds max_eval_steps to TrainingArguments
  • Limits evaluation by number of batches
  • Fixes num_samples handling for iterable datasets
  • Preserves callback event ordering

All trainer tests pass locally.

@github-actions

Copy link
Copy Markdown
Contributor

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42980&sha=ed59aa

@PRATHAMESHG1234

Copy link
Copy Markdown
Author

The tests_torch failure is due to a known ResNet initialization bug (see #31841), not this PR.

Failing test: tests/models/resnet/test_modeling_resnet.py::ResNetModelTest::test_can_load_ignoring_mismatched_shapes
Error: AssertionError: 0.11898990720510483 not less than or equal to 0.1 for classifier.1.bias

This is a pre-existing issue where ResNet's ignore_mismatched_sizes=True produces out-of-bound initialization values. My changes only modify evaluation loop behavior (max_eval_steps) and don't touch model initialization code.

All other tests passed. Requesting maintainer review.

@PRATHAMESHG1234

Copy link
Copy Markdown
Author

@sgugger @ArthurZucker @muellerzr Could you review this PR? The ResNet test failure appears unrelated to the changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants