add internvl_flash model by Chenhao-Guan · Pull Request #42166 · huggingface/transformers

Chenhao-Guan · 2025-11-12T15:40:53Z

Resolves #41862

Following your guidance in the issue, this PR re-implements the InternVL-Flash model as a completely separate model (instead of using an if flag in the existing InternVL class).

Implementation Details
Created a new, independent model directory: src/transformers/models/internvl_flash/.

Used the transformers add-new-model-like script to scaffold the new model, as you suggested.

Implemented the model logic in modular_internvl_flash.py (including Gating, CrossAttentionPooling, etc.) and converted it using the modular script.

Testing
All local tests are passing:

make fixup (style, quality, and repository consistency checks all pass)

pytest tests/models/internvl_flash/test_modeling_internvl_flash.py

Thank you for the guidance!

Before submitting
[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

[x] Did you read the contributor guideline, Pull Request section?

[x] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case. (Link: #41862)

[x] Did you make sure to update the documentation with your changes? (Added docs/source/en/model_doc/internvl_flash.md and updated _toctree.yml)

[x] Did you write any new necessary tests? (Added tests/models/internvl_flash/test_modeling_internvl_flash.py)

Who can review?
@zucchini-nlp @Rocketknight1

zucchini-nlp · 2025-11-13T13:43:30Z

Taking a look tomorrow-Monday, thanks for making a new model class

zucchini-nlp

Hey @Chenhao-Guan , thanks for making a separate PR for the model!

I have a few major comments:

The model doesn't seem to support batch size > 1 currently. Prob the official release works with a single batch size which is oke. However we need to enable batched inference before merging this PR
I see that you kept 'flash" and "non-flash" paths. We have to delete the "non-flash" code path as it is not needed to run with InternVLFlash released checkpoint
A few minor issues like naming and sticking to transformers standards 👇🏻

zucchini-nlp · 2025-11-13T10:32:22Z

+)
+
+
+class Gating(nn.Module):


InternVLFlashGating or similar naming since it's recommended to have model's name explicit in layer names

zucchini-nlp · 2025-11-13T10:33:51Z

+        if self.use_checkpoint:
+            x = x + cp.checkpoint(self.block1, x)
+            x = x + cp.checkpoint(self.block2, x)
+            x = x + cp.checkpoint(self.block3, x)
+            x = x + cp.checkpoint(self.block4, x)
+        else:


i don't think it, GC can be toggled on by PreTrainedModel if needed

zucchini-nlp · 2025-11-13T10:35:17Z

+        def mlp_block(in_dim, out_dim):
+            return nn.Sequential(
+                nn.Linear(in_dim, out_dim),
+                nn.GELU(),
+                nn.Dropout(dropout),
+                nn.Linear(out_dim, in_dim),
+                nn.Dropout(dropout),
+                nn.LayerNorm(in_dim),
+            )
+


Separate layers are more preferred than a Sequential. Let's create an "nn.Module" with the following and call it smth like InternVLFlashMLP

zucchini-nlp · 2025-11-13T10:35:31Z

+        self.block2 = mlp_block(hidden_size, mid_dim)
+        self.block3 = mlp_block(hidden_size, mid_dim)
+        self.block4 = mlp_block(hidden_size, mid_dim)
+        self.gate = nn.Sequential(nn.LayerNorm(hidden_size), nn.Linear(hidden_size, 2))  # 2 experts


same here, let's have them separated

zucchini-nlp · 2025-11-13T10:35:55Z

+        return probs
+
+
+class CrossAttentionPooling(nn.Module):


same comment for naming

zucchini-nlp · 2025-11-17T13:14:14Z

+        flag_idx = 0
+        for s, e, l, num_blocks in zip(starts.tolist(), ends.tolist(), lengths.tolist(), block_counts):
+            for i in range(num_blocks):
+                block_start = s + i * 256
+                block_end = block_start + 256
+
+                compress = gate_result[flag_idx]
+                flag_idx += 1
+
+                if compress:
+                    keep_mask[block_start + 64 : block_end] = False
+                    delete_flags[block_start + 64 : block_end] = 1
+


could it be vectorized?

zucchini-nlp · 2025-11-17T13:15:03Z

+        mask_idx = mask_idx.squeeze(0)
+        updated_mask_idx = mask_idx - cumulative_deletes[mask_idx.to(cumulative_deletes.device)].to(mask_idx.device)
+        updated_mask_idx = updated_mask_idx.unsqueeze(0)


new attention mask is not used as I can see from current commit, so we have to fix it first. Then, why can't we attention_mask = attention_mask[keep_mask]

zucchini-nlp · 2025-11-17T13:15:23Z

+if is_vision_available():
+    pass


dummy import, can be deleted

zucchini-nlp · 2025-11-17T13:16:32Z

+    @unittest.skip(
+        reason="Failing with `torch._inductor.exc.InductorError: RuntimeError: No valid triton configs. OutOfMemoryError: out of resource: triton_tem_fused_0 Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.`"
+    )
+    def test_flex_attention_with_grads(self):
+        pass


can you delete skip on it? Prob it was failing on your local hardware, might pass with CI runners

zucchini-nlp · 2025-11-17T13:17:01Z

+    @unittest.skip("Skipping compilation test: fails with batch_size=0 reshape error")
+    def test_generate_compile_model_forward_fullgraph(self):
+        pass


we need to support any batch size for the model before merging

Chenhao-Guan · 2025-11-17T13:39:40Z

@zucchini-nlp Thank you for the advice. This is my first time submitting a PR, and Gonna working to resolve the test failures related to batch_size > 1 support. My initial intention in adding non-Flash methods was specifically to bypass these failing tests temporarily. I will continue working to implement a full solution.

Chenhao-Guan · 2025-11-25T07:13:50Z

@zucchini-nlp I've finished the requested modifications. Please let me know if there are any other points to discuss before we merge.

zucchini-nlp · 2025-12-02T11:26:04Z

Thanks, I will review some time this week. It was a bit hectic due to v5 last week

HuggingFaceDocBuilderDev · 2025-12-02T11:34:21Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

zucchini-nlp

Hey, sorry it took long to review. The model arch is more complicated than in other VLMs due to the attention-based gating and different image lengths

I think we can keep for now the approach you have, by slicing out unused image placeholder tokens. I left a few comments on better readability below, let me know if anyrhing isn't clear

zucchini-nlp · 2025-12-08T12:16:45Z

+        return hidden_states
+
+
+class InternvlFlashMLP2(nn.Module):


nit: InternvlFlashMultimodalProjection for explicitness and to not confuse with MLP layer above

zucchini-nlp · 2025-12-08T12:17:21Z

+    def __init__(self, vit_hidden_size, llm_hidden_size, config):
+        super().__init__()
+
+        in_dim = vit_hidden_size * int(1 / config.downsample_ratio) ** 4
+        mid_dim = llm_hidden_size * 2
+        out_dim = llm_hidden_size


vit and llm hidden sizes are available in config, so we dont need to pass those separately

zucchini-nlp · 2025-12-08T12:18:36Z

+class InternvlFlashGating(nn.Module):
+    def __init__(self, hidden_size=2048, expansion_factor=4, dropout=0.1, use_checkpoint=True):
+        super().__init__()
+        self.use_checkpoint = use_checkpoint


attribute not used?

zucchini-nlp · 2025-12-08T12:19:21Z

+        if B == 0:
+            return torch.empty(
+                0, self.query_token.shape[-1], device=self.query_token.device, dtype=self.query_token.dtype
+            )


i think it cannot be 0 unless there is a bug in code?

zucchini-nlp · 2025-12-08T12:53:06Z

+            vision_feature_layer (`int` or `list[int]`):
+                Layer index or list of layer indices to extract features from.


nit: the docstring doesn't match signature

zucchini-nlp · 2025-12-08T13:05:21Z

+        selected_mask = input_ids == self.config.image_token_id
+        if selected_mask.sum() == 0:
+            return inputs_embeds
+        inputs_embeds[selected_mask] = vit_embeds.to(inputs_embeds.device)
+        return inputs_embeds


needs to support input embeddings, similar to other VLMs with gte_placeholder_mask

zucchini-nlp · 2025-12-08T13:15:18Z

+        # 1. Padding
+        max_len = max(t.shape[0] for t in batched_tokens)
+        dtype = self.query_token.dtype
+        padded = torch.zeros(B, max_len, D, dtype=dtype, device=device)
+        padding_mask = torch.ones(B, max_len, dtype=torch.bool, device=device)
+        for i, t in enumerate(batched_tokens):
+            L = t.shape[0]
+            padded[i, :L] = t
+            padding_mask[i, :L] = False
+        # 2. Query token: [B, 1, D]
+        query = self.query_token.unsqueeze(0).expand(B, -1, -1)  # learnable token for each sample
+
+        attention_mask = torch.zeros_like(padding_mask, dtype=query.dtype)
+        min_value = torch.finfo(query.dtype).min
+        attention_mask.masked_fill_(padding_mask, min_value)
+
+        # 3. Adjust Attention Score: [B, Num_Heads, Q_Len, K_Len]
+        attention_mask = attention_mask.unsqueeze(1).unsqueeze(1)


hmm interesting. Ig we could do attention with ragged input though I don't see much difference tbh. Let's just call create_bidirection_mask to get a 4D projected mask (e.g. in BERT-like models they do it)

zucchini-nlp · 2025-12-08T13:21:19Z

+            if attention_mask is not None:
+                if attention_mask.dim() > 2 or attention_mask.numel() != input_ids.numel():
+                    pad_token_id = self.config.pad_token_id if self.config.pad_token_id is not None else 0
+                    attention_mask = input_ids.ne(pad_token_id)


I don't think it is a good idea to reconstruct a mask when a 4D mask is provided. We still can slice it out with keep mask, as long as the mask is broadcasted ,no ?

zucchini-nlp · 2025-12-08T13:24:09Z

+            if isinstance(attention_mask, dict):  # add support for StaticCache
+                attention_mask = attention_mask["full_attention"]


so the model has sliding and full attention, iiuc. In which case we either slice both masks or disallow preparing 4D masks (see Bloom for that, it has special postiion embedding which requires 2D mask)

zucchini-nlp · 2025-12-08T13:24:58Z

+            inputs_embeds, attention_mask = self._reconstruct_batch(
+                inputs_embeds=inputs_embeds,
+                attention_mask=attention_mask,
+                gate_result=gate_result,
+                lengths=lengths,
+                batch_indices=batch_indices,
+                N=N,
+                B=B,
+            )


sorry, not clear to me why we reconstruct the batch again, after it was sliced prev to keep only necessary image placeholder tokens?

the "slicing" earlier removed unnecessary visual tokens within the flattened representation, _reconstruct_batch is needed to reorganize the data back into the proper (B, N, C) batch format the LLM expects, with uniform padding for the new variable lengths.

Chenhao-Guan · 2025-12-09T01:54:25Z

Thank you for your review! Recently working on my final. Hoping to slove this problems ASAP.

zucchini-nlp · 2025-12-09T10:10:15Z

No problem, take your time :)

YanxingLiu · 2025-12-13T11:33:59Z

Thank you for your review! Recently working on my final. Hoping to slove this problems ASAP.

hello, is there any progress? I am the author of the issue #41862. I have free time currently, may I do something for help the PR?

YanxingLiu · 2025-12-13T16:02:46Z

I am tring to convert raw internvl-3.5-flash model to huggingface format by modifing https://github.com/OpenGVLab/InternVL/blob/main/internvl_chat/tools/internvl_custom2hf.py . I find that there are 2 additional params exist in InternvlFlashConfig but not in InternVLConfig. Is it possible to add these two parameters in InternvlFlashConfig (although even without these two parameters, they can still be manually added in config.json to achieve the conversion)?

Chenhao-Guan · 2025-12-15T13:46:19Z

@YanxingLiu Sorry for not checking my mail.I use scripts for internvl to convert the models with some minor changes. Models can be found at( https://huggingface.co/chenhaoguan/InternVL3_5-2B-Flash-hf ). I have mail u my version. I haven't made any changes. Free to ask me if u have any question.

working on batch reconsturction

Chenhao-Guan · 2026-01-12T13:06:38Z

Sorry it takes so long @zucchini-nlp . I’ve addressed most of the issues you pointed out. However, in batch processing, sequences are padded to the maximum length of the batch making it not that efficient, and I feel there must be a more efficient way to handle this.

github-actions · 2026-01-24T00:40:19Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, internvl_flash

Chenhao-Guan and others added 4 commits November 12, 2025 23:34

add internvl_flash model

0fb9f07

Merge branch 'main' into add-internvl-flash-separate

720d54f

Trigger CI rerun

7f2f66b

Trigger CI rerun

80a6f9f

zucchini-nlp reviewed Nov 17, 2025

View reviewed changes

Chenhao-Guan added 15 commits November 25, 2025 13:38

17code base

3784410

1118version

e50779e

18

0a20bd6

add necessary import

3673c52

14errors

0903f1e

to add batch

d2d1f59

btach halfway

e780dba

batch complete

4b0e162

fixup errors

d712485

fix errors

d23246e

fix errors

c5148bc

fix errors

c5ab5cf

fix errors

303509d

fix errors

50241d8

trigger ci

18a6f74

zucchini-nlp reviewed Dec 8, 2025

View reviewed changes

Chenhao-Guan added 2 commits January 12, 2026 17:24

main problem sloved

215f4b4

working on batch reconsturction

remove unnecessary messages

7e9a89f

Chenhao-Guan force-pushed the add-internvl-flash-separate branch from 7f15023 to 7e9a89f Compare January 12, 2026 10:30

Chenhao-Guan and others added 7 commits January 12, 2026 21:06

Merge branch 'main' into add-internvl-flash-separate

c7e0fdb

make style

20e0d11

slove merge problems

6e13f93

make style

e0b105e

chore: trigger test

f825130

chore: trigger test

05354bd

Merge branch 'main' into add-internvl-flash-separate

e900a21

evalstate mentioned this pull request Apr 29, 2026

Cumulative feature and defect updates from recent Transformers PRs evalstate/transformers#42

Open

		vision_feature_layer (`int` or `list[int]`):
		Layer index or list of layer indices to extract features from.

		if isinstance(attention_mask, dict): # add support for StaticCache
		attention_mask = attention_mask["full_attention"]

Uh oh!

Conversation

Chenhao-Guan commented Nov 12, 2025

Uh oh!

zucchini-nlp commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Chenhao-Guan commented Nov 17, 2025

Uh oh!

Chenhao-Guan commented Nov 25, 2025

Uh oh!

zucchini-nlp commented Dec 2, 2025

Uh oh!

HuggingFaceDocBuilderDev commented Dec 2, 2025

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Chenhao-Guan Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Chenhao-Guan commented Dec 9, 2025

Uh oh!

zucchini-nlp commented Dec 9, 2025

Uh oh!

YanxingLiu commented Dec 13, 2025

Uh oh!

YanxingLiu commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Chenhao-Guan commented Dec 15, 2025

Uh oh!

Chenhao-Guan commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jan 24, 2026

zucchini-nlp commented Nov 13, 2025 •

edited

Loading

Chenhao-Guan Jan 12, 2026 •

edited

Loading

YanxingLiu commented Dec 13, 2025 •

edited

Loading

Chenhao-Guan commented Jan 12, 2026 •

edited

Loading