Integrate with Hugging Face

[NielsRogge]

This PR fixes #1. It aims to make it possible to load any SAM 2 model directly from the hub.

The checkpoints are already part of the facebook org:

• https://huggingface.co/facebook/sam2-hiera-tiny
• https://huggingface.co/facebook/sam2-hiera-small
• https://huggingface.co/facebook/sam2-hiera-base-plus
• https://huggingface.co/facebook/sam2-hiera-large.

This PR allows to do the following:

from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor.from_pretrained("facebook/sam2-hiera-small")

This also ensures that download stats work, meaning you'll be able to see how many times people actually load one of these checkpoints from the hub. This will also allow us to compare to the download stats of Transformers which will be interesting :)

To do:

• update README to link to HF artifacts
• update notebooks (?)
• support video predictor

[bhack]

Is this also in the to-do list at huggingface/transformers#32394 ?

[haithamkhedr]

@NielsRogge Thanks for supporting ckpt download from HF. Can we make huggingface_hub a soft dependency?

[haithamkhedr]

Thanks for adding support to load the models through HF hub.

[giswqs]

This is great! Is there a plan to make it compatible with transformers, like SAM 1?

https://huggingface.co/docs/transformers/en/tasks/mask_generation

[NielsRogge]

Yes there is, the authors are adding a PR here: huggingface/transformers#32394

[giswqs]

Fantastic! Can't wait for the PR to be merged!

[sueszli]

This is just amazing <3

[leonmkim]

Sorry potentially dumb question, but should this line in the video predictor be changed from return cls(sam_model) to return sam_model? It seems like the former would work for the image predictor based on its constructor but not the video predictor. But I could be wrong :p

[haithamkhedr]

@leonmkim Good catch! Fixed in #205