[CI] Allow single file tests to run as fast tests

[DN6]

What does this PR do?

We've had trouble running single file testing with fast tests for a while because creating a dummy checkpoint for models in their original format is non-trivial. In this PR, we leverage the fact that the hub allows downloading safetensors metadata via
HfApi().parse_safetensors_file_metadata. This includes information about the checkpoint keys, which we can use to build a dummy state dict with empty tensors on meta device. The primary thing we want to test with single file loading is the loading and conversion step, and we can use these dummy checkpoints for that purpose.

To run a check on whether converted parameters match 1:1 would still require downloading a real checkpoints, so we move just that test to nightly.

Also added coverage for models that support single file loading that were missed during the first test refactor pass.

Fixes # (issue)

Before submitting

• Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
  • Did you read the Coding with AI agents guide?
  • Did you run the self-review skill on the diff?
  • Did you share the final self-review notes in the PR description or a comment?
• Did you read the contributor guideline?
• Did you read our philosophy doc? (important for complex PRs)
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?
• Are you the author (or part of the team) of the model/pipeline (only applicable for model/pipeline related PRs)?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

Really nice!

[sayakpaul]

Maybe we don't need init_empty_weights() here and we could just use with torch.device("meta")?

[sayakpaul]

Should it resemble

diffusers/tests/models/testing_utils/common.py

Lines 519 to 523 in 3c46892

	for name, param in model.named_parameters():
	if any(module_to_keep_in_fp32 in name.split(".") for module_to_keep_in_fp32 in fp32_modules):
	assert param.dtype == torch.float32, f"Parameter {name} should be float32 but got {param.dtype}"
	else:
	assert param.dtype == torch.float16, f"Parameter {name} should be float16 but got {param.dtype}"

?