Skip to content

Fix unhandled exception noise from background safetensors conversion thread - #45752

Merged
ArthurZucker merged 3 commits into
huggingface:mainfrom
dhruv7477:fix/background-thread-conversion-error
Jun 1, 2026
Merged

Fix unhandled exception noise from background safetensors conversion thread#45752
ArthurZucker merged 3 commits into
huggingface:mainfrom
dhruv7477:fix/background-thread-conversion-error

Conversation

@dhruv7477

Copy link
Copy Markdown
Contributor

The background Thread-auto_conversion in modeling_utils.py was spawned with ignore_errors_during_conversion=False. When get_repo_discussions() raises HfHubHTTPError 403 (discussions disabled on the repo), the exception propagated uncaught inside the thread, and Python printed the full traceback to stderr — the noise reported in #44403.

Since this thread is explicitly fire-and-forget (the comment on line 720 reads "try to launch safetensors conversion for next time"), errors from it should never surface to the user. Changing the flag to True causes auto_conversion to catch and suppress the exception cleanly.

A prior attempt (#44440) was closed because it was opened two days after the issue while discussion was still ongoing, and made additional changes to safetensors_conversion.py. This PR is a single-line fix to the actual root cause.

Fixes #44403


Tests run: python -m pytest tests/utils/test_modeling_utils.py -x -v
Result: 94 passed, 38 skipped, 1 xfailed

I used an AI assistant to help trace the root cause and identify the relevant code path, but I reviewed the change, ran the tests, and verified the fix against the stack trace in the issue.

…thread

Signed-off-by: Dhruv Sharma <dhruv7477@gmail.com>
@dhruv7477

Copy link
Copy Markdown
Contributor Author

Re: Failing CI checks

The two failing tests (test_tp_generation for exaone4 and test_ep_forward for gpt_oss) are pre-existing failures unrelated to this PR.

GptOssModelTest::test_ep_forward — tracked by open issue #45161 ("Only TP not working with GPT-OSS MoE model", filed April 1, 2026). The ProcessRaisedException in EP forward is a known infrastructure issue.

Exaone4ModelTest::test_tp_generation — SIGABRT (process crash) in the distributed subprocess. This is a CUDA/NCCL-level crash that cannot be caused by changing the ignore_errors_during_conversion flag on a background Python thread in modeling_utils.py.

This PR touches a single line in the background safetensors conversion thread. It has no interaction with tensor-parallel or expert-parallel computation paths. Could a maintainer re-run the required run_tests check?

@LysandreJik

Copy link
Copy Markdown
Member

Thanks @dhruv7477 this looks good to me; will update branch and merge

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ty

@ArthurZucker
ArthurZucker merged commit 855aad5 into huggingface:main Jun 1, 2026
5 of 8 checks passed
@dhruv7477
dhruv7477 deleted the fix/background-thread-conversion-error branch June 1, 2026 11:24
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

kashif pushed a commit to kashif/transformers that referenced this pull request Jun 1, 2026
…thread (huggingface#45752)

Signed-off-by: Dhruv Sharma <dhruv7477@gmail.com>
Co-authored-by: Lysandre Debut <hi@lysand.re>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
khushali9 pushed a commit to khushali9/transformers that referenced this pull request Jun 8, 2026
…thread (huggingface#45752)

Signed-off-by: Dhruv Sharma <dhruv7477@gmail.com>
Co-authored-by: Lysandre Debut <hi@lysand.re>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
stevhliu pushed a commit to stevhliu/transformers that referenced this pull request Jul 30, 2026
…thread (huggingface#45752)

Signed-off-by: Dhruv Sharma <dhruv7477@gmail.com>
Co-authored-by: Lysandre Debut <hi@lysand.re>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unnecessary noise when loading a transformer

5 participants