feat(utils): add vision utils for embedding images and getting the hidden size - #40587
feat(utils): add vision utils for embedding images and getting the hidden size#40587AmitMY wants to merge 4 commits into
Conversation
molbap
left a comment
There was a problem hiding this comment.
Interesting, could be useful but needs significant expansion around commonly used vision/text models at least.
There was a problem hiding this comment.
On this: I think it'd be needed to go through several multimodal models/VLMs. I like the usefulness of it, but I don't want to add an utility (so, added code) that doesn't have enough coverage.
There was a problem hiding this comment.
I think this doesn't have to live in its own file, we have several util files around :)
And same comment, adding the VLMs (at least through config reading) would be needed
|
@molbap this is mainly a utility for VLMs How do you see it "extended" for VLMs? |
|
As it is it appeared to be mostly for the And what I meant is that you might want to take an existing VLM and probe for the embedding images/hidden sizes, the documentation should explain how to do that. |
What does this PR do?
Adds generic utilities to understand the hidden size of an image encoder (currently need to write complex logic), and generically encode images using an image encoder.
Fixes #40457
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@qubvel @zucchini-nlp @ArkVex