Skip to content

feat(utils): add vision utils for embedding images and getting the hidden size - #40587

Open
AmitMY wants to merge 4 commits into
huggingface:mainfrom
AmitMY:master
Open

feat(utils): add vision utils for embedding images and getting the hidden size#40587
AmitMY wants to merge 4 commits into
huggingface:mainfrom
AmitMY:master

Conversation

@AmitMY

@AmitMY AmitMY commented Sep 1, 2025

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds generic utilities to understand the hidden size of an image encoder (currently need to write complex logic), and generically encode images using an image encoder.

Fixes #40457

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

@qubvel @zucchini-nlp @ArkVex

@Rocketknight1

Copy link
Copy Markdown
Member

cc @zucchini-nlp @molbap @yonigozlan

@molbap molbap left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, could be useful but needs significant expansion around commonly used vision/text models at least.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On this: I think it'd be needed to go through several multimodal models/VLMs. I like the usefulness of it, but I don't want to add an utility (so, added code) that doesn't have enough coverage.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this doesn't have to live in its own file, we have several util files around :)
And same comment, adding the VLMs (at least through config reading) would be needed

@AmitMY

AmitMY commented Oct 28, 2025

Copy link
Copy Markdown
Contributor Author

@molbap this is mainly a utility for VLMs
When building a VLM, you might want to map the dimensions of the images to the hidden dimension of the transformer - so you need to know how to "encode" and what is the "hidden_size"

How do you see it "extended" for VLMs?

@molbap

molbap commented Oct 28, 2025

Copy link
Copy Markdown
Collaborator

As it is it appeared to be mostly for the AutoModelForImageClassification mapped models, right? We would need to have an accompanying documentation then on how to use these utils, with several types of VLMs, also with vision backbones - without the classification head, it's a different class in that case.

And what I meant is that you might want to take an existing VLM and probe for the embedding images/hidden sizes, the documentation should explain how to do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Image Embedding Models (Feature extractors) should have a .hidden_size

3 participants