Add D-FINE Model into Transformers

[VladOS95-cyber]

What does this PR do?

D-FINE, a powerful real-time object detector that achieves outstanding localization precision by redefining the bounding box regression task in DETR models. D-FINE comprises two key components: Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD).

As backbone, it is used HGNet-V2 (High Performance GPU Net) image classification model.

This PR add D-FINE and HGNet-V2 into the Transformers library. There is a new thing in transformers called modular, which adds new models by creating a modeling_modelname.py file. Since D-FINE updates several RT-DETR arch parts while keeping the rest of the model unchanged, it serves as an ideal use case for this modular approach.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case. Request to add D-FINE #35283
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@qubvel @Rocketknight1 @ArthurZucker @NielsRogge

[VladOS95-cyber]

Hi @qubvel! I decided to create a new PR as previous one (#35400) was broken, sorry for the mess. In this PR all the latest changes.
To sum up, I cleaned up modulars according to your comments. On this step we have two unresolved problems: 1. Wrong namespace import for modeling_d_fine_resnet.py where conversion script provides from .configuration_d_fine_resnet_resnet. 2. DFine config conversion using modular does not work properly because it provides a call super().init(is_encoder_decoder=is_encoder_decoder, **kwargs) too early in the very beginning of init and before all self.args assignments, which is wrong as well.

[Cyrilvallez]

Hey very sorry, I had very low bandwith last week as I was presenting at a conference! Are you still experiencing issue with how modular works, or is it resolved?

[VladOS95-cyber]

Hey very sorry, I had very low bandwith last week as I was presenting at a conference! Are you still experiencing issue with how modular works, or is it resolved?

Hey @Cyrilvallez! Thank you for your answer, yes, there are still some issues: 1. Wrong namespace import for modeling_d_fine_resnet.py where conversion script provides from .configuration_d_fine_resnet_resnet (too much resnet). 2. DFine config conversion using modular does not work properly because it provides a call super().init(is_encoder_decoder=is_encoder_decoder, **kwargs) too early in the very beginning of init and before all self.args assignments, which is wrong as well. It must be placed after all self.args assignments.

[Cyrilvallez]

Ok @VladOS95-cyber, I looked into it, thanks for the feedback 🤗

1. is indeed an issue, as you have 2 modular files in the same folder, this was unprecedented and we were not parsing correctly: [modular] allow multiple modular files in the same model folder #36287 will solve it
2. This is actually by design, as usually (but not always, true) super().__init__() calls are placed at the beginning. In this instance it should not be an issue, is it? Meaning that I don't think the order has any importance in this case

[Cyrilvallez]

Oh, or is 2. an issue because of the attribute_map? I may have seen this issue when reviewing RtDetrV2. But in this case, I would argue that using the attribute_map itself is usually very confusing, and it would definitely be best not to if possible (e.g. by renaming the arg directly)

[VladOS95-cyber]

Ok @VladOS95-cyber, I looked into it, thanks for the feedback 🤗

1. is indeed an issue, as you have 2 modular files in the same folder, this was unprecedented and we were not parsing correctly: [modular] allow multiple modular files in the same model folder #36287 will solve it
2. This is actually by design, as usually (but not always, true) super().__init__() calls are placed at the beginning. In this instance it should not be an issue, is it? Meaning that I don't think the order has any importance in this case

Hi @Cyrilvallez! Well, I suppose it is an issue in case of config and attribute map usage, yes. Because if we call super().init(kwargs**) in the beginning and do args.self assignment later, we override some attributes values that were passed before. In RTDetrV2 this problem was resolved by implementing config from the scratch, but in our case, we reuse RTDetrConfig, so this problem came up unfortunately.

[VladOS95-cyber]

@Cyrilvallez So, what do you suggest in this case?

[qubvel]

Thanks, @VladOS95-cyber for addressing the comments, refactoring, and adding type hints! I left a few minor comments, but it's in good shape. As the next step, let's also ensure:

• Modular issues are resolved (cc @Cyrilvallez)
• torch.compile (fullgraph) and torch.export are supported (see comments)
• All checkpoints can be converted (please provide links for converted checkpoints on the Hub) and logits/boxes match the original implementation
• Model can be fine-tuned (here is an example for RT-DETRv2, it would work with the change of checkpoint path only)
• CI is happy: consistency, docs, style issues are resolved

Thanks for the great work!

[VladOS95-cyber]

Hi @qubvel! There is a short summary of what we've achieved so far by this moment:

1. Modular issues are resolved (cc @Cyrilvallez) - still in progress. Moreover, I would like to discuss what we are going to do with config and attribute map problem.
2. torch.compile (fullgraph) works and compiles the model, but I cannot execute this model later on inference as it is failed with exception I provided in comments. But I checked the behavior of other models in this case, it is the same. So, I assume either I do something wrong or it is expected.
3. torch.export is supported. I added it into tests and it was executed successfully.
4. All main checkpoints were converted: https://huggingface.co/vladislavbro/dfine_x_coco, https://huggingface.co/vladislavbro/dfine_l_coco, https://huggingface.co/vladislavbro/dfine_m_coco, https://huggingface.co/vladislavbro/dfine_s_coco. Model cards I'll add later. All logits/boxes are matched with original implementation
5. Model is able to be fine tuned, I checked on the script you provided and changed checkpoint on d-fine one, and it worked.
6. CI is not happy yet because it depends on issues from modular conversion. Once it will be resolved, CI will be completed.

[qubvel]

Hi @VladOS95-cyber, thanks for iterating and addressing comments! I left a few more comments, but in general looks great!

I am having difficulty running fine-tuning for the "m" checkpoint, while the "l" checkpoint is fine. Can you please double-check the modeling code, losses and initialization strategy? Here is the error I got:

[/usr/local/lib/python3.11/dist-packages/transformers/loss/loss_for_object_detection.py](https://localhost:8080/#) in generalized_box_iou(boxes1, boxes2)
    416     # so do an early check
    417     if not (boxes1[:, 2:] >= boxes1[:, :2]).all():
--> 418         raise ValueError(f"boxes1 must be in [x0, y0, x1, y1] (corner) format, but got {boxes1}")
    419     if not (boxes2[:, 2:] >= boxes2[:, :2]).all():
    420         raise ValueError(f"boxes2 must be in [x0, y0, x1, y1] (corner) format, but got {boxes2}")

ValueError: boxes1 must be in [x0, y0, x1, y1] (corner) format, but got tensor([[4.0848e-02, 6.0155e-02, 9.7476e-01, 9.4098e-01],
        [4.3698e-02, 1.0271e-01, 9.6435e-01, 8.6763e-01],
        [4.0578e-06, 6.4026e-06, 1.4135e-05, 2.2186e-05],
        ...,
        [9.9940e-01, 3.7292e-02, 1.0063e+00, 4.3024e-02],
        [9.8414e-01, 4.3545e-02, 1.0219e+00, 4.8508e-02],
        [7.8416e-01, 6.7056e-01, 8.3837e-01, 7.0462e-01]], device='cuda:0')

P.S. torch.compile(model, fullgraph=True) works fine with my env, thanks!

[VladOS95-cyber]

@qubvel do you know if anything was changed for transformers recently? I cannot download model for fine-tuning anymore.
Just one thing, I am using my remote branch for transformer package like: !uv run pip install --upgrade -q -U git+https://github.com/VladOS95-cyber/transformers.git@add-dfine-model . But it worked yesterday

model = AutoModelForObjectDetection.from_pretrained(
    checkpoint,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True)

It is failed with: 

RuntimeError: Error(s) in loading state_dict for DFineForObjectDetection:
	size mismatch for model.denoising_class_embed.weight: copying a param with shape torch.Size([81, 256]) from checkpoint, the shape in current model is torch.Size([6, 256]).
	size mismatch for model.enc_score_head.weight: copying a param with shape torch.Size([80, 256]) from checkpoint, the shape in current model is torch.Size([5, 256]).
	size mismatch for model.enc_score_head.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([5]).
	size mismatch for model.decoder.class_embed.0.weight: copying a param with shape torch.Size([80, 256]) from checkpoint, the shape in current model is torch.Size([5, 256]).
	size mismatch for model.decoder.class_embed.0.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([5]).
	size mismatch for model.decoder.class_embed.1.weight: copying a param with shape torch.Size([80, 256]) from checkpoint, the shape in current model is torch.Size([5, 256]).
	size mismatch for model.decoder.class_embed.1.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([5]).
	size mismatch for model.decoder.class_embed.2.weight: copying a param with shape torch.Size([80, 256]) from checkpoint, the shape in current model is torch.Size([5, 256]).
	size mismatch for model.decoder.class_embed.2.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([5]).
	size mismatch for model.decoder.class_embed.3.weight: copying a param with shape torch.Size([80, 256]) from checkpoint, the shape in current model is torch.Size([5, 256]).
	size mismatch for model.decoder.class_embed.3.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([5]).

[qubvel]

@VladOS95-cyber hmm, yes, there ware some PRs regarding from_pretrained refactoring.. it might be broken, let me check

[qubvel]

PR with fix merged 👍 thanks for reporting

[VladOS95-cyber]

Hi @qubvel! Latest state: I added tests for resnet. I don't know how many times I checked modeling code but I could not find any problems that might be a root cause of this issue with fine-tuning. I made small fix but it did not help and it seems weird as loss calculation is always failed on different steps and sometimes is not failed at all. Returned scores and boxes are the same as in original implementation. There are some differenses in states, yes, but there is logical explanation for that and it should not be a problem. Unfortunately, I am a little bit stacked right now...

[qubvel]

Hey @VladOS95-cyber, thanks for checking. I suppose it's not a blocker, and maybe we can figure it out later. At least inference works for all models, and fine-tuning for bigger models also works fine. I will take a deep dive next week and try to resolve the remaining issues to get this merged.

[VladOS95-cyber]

@qubvel ok, thank you. Basically, all other things are fine. Just there are still couple unresolved problems of modular. It can be easily resolved, I think, we just need to make a decision about that.

[VladOS95-cyber]

Hey @qubvel! I am just curious, I found an issue #36516. Could it be the case that an existing issue with fine-tuning for D-FINE is related to this one?

[qubvel]

Well, it might be an issue, however, other models fine-tuning is OK with this dataset. This model might be a bit more sensitive to noisy labels, let me try to run with a clean one

[qubvel]

run-slow: d_fine, hgnet_v2

[github-actions]

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/d_fine', 'models/hgnet_v2']
quantizations: [] ...

[qubvel]

cc @ArthurZucker for final review

The model is based on RT-DETR. The loss implementation is different, so we added some additional attributes to the RT-DETR output to propagate it through the model and avoid overriding all corresponding forward methods. It's a bit against our philosophy, but I believe it is worth it in order to make the code more standard and modular

[qubvel]

run-slow: d_fine, hgnet_v2

[ArthurZucker]

Happy to merge if you like it @qubvel ! 🤗
LGTM great review and great work!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qubvel]

Merging! @VladOS95-cyber thanks a lot for contributing and iterating on the PR 🤗 Really huge work done!

[VladOS95-cyber]

Merging! @VladOS95-cyber thanks a lot for contributing and iterating on the PR 🤗 Really huge work done!

Thank you very much for your support and review!

[Matvezy]

Hello @VladOS95-cyber @qubvel thanks a lot for adding D-FINE to transformers! I'm interested in fine-tuning and benchmarking your implementation on a custom dataset. I found a fine-tuning notebook example here, however I noticed that it doesn't fully follow the original D-FINE fine-tuning config. I specifically mean not using the same augmentations, optimizer parameters, and not using use EMA like in the original D-FINE configs here.
I wanted to ask if you previously tried fine-tuning with the original D-FINE recipe? I suspect that not having it in place may result in a lower benchmark score, so I wanted to ensure I try fine-tuning with the original recipe as well. Right now I'm just adding those changes myself, but I wanted to see if you may already have anything you tired that I can use?

[qubvel]

Hi @Matvezy! We did not adapt the full fine-tuning recipe for D-FINE. The notebook you mentioned is just a starting point that can be significantly improved with many techniques. We would appreciate any contribution, e.g. if you have some results on your fine-tuning or upgrade the notebook 🤗