Unable to find some Unsloth Quants during docker model pull

[akoumjian]

When attempting to download some of the quants provided by unsloth (UD-Q4_K_XL), docker model pull is unable to find / match them.

~ ❯ docker model pull hf.co/unsloth/Kimi-K2-Instruct-0905-GGUF:UD-Q4_K_XL
Retrying download (attempt 1/3) in 1s...
^C%

~ ❯ docker model pull hf.co/unsloth/Kimi-K2-Instruct-0905-GGUF:Q4_K_XL
Retrying download (attempt 1/3) in 1s...
Retrying download (attempt 2/3) in 2s...
^C%

It works fine for other quants in the same repository.

~ ❯ docker model pull hf.co/unsloth/Kimi-K2-Instruct-0905-GGUF:Q4_K_M
5ddfa1832cb1: Downloading [>                                                  ]  397.1MB/44.23GB
5ddfa1832cb1: Downloading [>                                                  ]  401.1MB/44.23GB
124c30f2497a: Downloading [>                                                  ]  482.7MB/49.55GB
eb48051961b5: Downloading [>                                                  ]  337.5MB/46.3GB
...

I think this is most likely due to the prefix used for the folder and/or naming scheme: https://huggingface.co/unsloth/Kimi-K2-Instruct-0905-GGUF/tree/main/UD-Q4_K_XL not working nicely with however docker model runner parses and matches the files in the repo.

Note that HF kind of does the same thing on the model card where it's listing quants, it has it listed without the prefix even though it's not reflected in the folder / file structure.

[ericcurtin]

It's being blocked huggingface side at present because it's a shared gguf:

Error: pull model manifest: 400: {"error":"The specified repository contains sharded GGUF. Ollama does not support this yet. Follow this issue for more info: https://github.com/ollama/ollama/issues/5245"}

@akoumjian how about we upload this model to Docker Hub so you can pull from there?

[akoumjian]

@ericcurtin Okay, so it actually appears that wasn't the issue. Here is the real issue:

The reason I was getting download errors for the UD-Q4_K_XL quant was because I had partially downloaded the model and cancelled, and then when I tried to download it again the resume was failing. This actually happened with the Q4_K_M when I tried resuming after a partial download as well.

~ ❯ docker --debug model pull hf.co/unsloth/Kimi-K2-Instruct-0905-GGUF:Q4_K_M                              aleck@maple
Retrying download (attempt 1/3) in 1s...
Retrying download (attempt 2/3) in 2s...
Retrying download (attempt 3/3) in 4s...
Failed to pull model: failed to download after 3 retries: pulling hf.co/unsloth/kimi-k2-instruct-0905-gguf:q4_k_m failed with status 500 Internal Server Error: error while pulling model: reading model from registry with resume context: failed to pull model "hf.co/unsloth/kimi-k2-instruct-0905-gguf@sha256:0dba654efc95735793e7d98496fa7e3b59ba50e4df43266eabe040f0d5133c10": UNKNOWN - GET https://huggingface.co/v2/unsloth/Kimi-K2-Instruct-0905-GGUF/manifests/sha256:0dba654efc95735793e7d98496fa7e3b59ba50e4df43266eabe040f0d5133c10: unexpected status code 400 Bad Request: {"error":"The specified tag is not a valid quantization scheme. Please use another tag or \"latest\""}

If I run docker model purge and re-attempt, it appears to download fine. So the actual issue with download seems to be with resuming.

Additionally, what was frustrating was that I could not find a way to purge the partially downloaded model. Using docker model rm doesn't work as it hasn't registered the model yet. Instead I had to purge the whole system. I didn't want to attempt locating and finding the individual folder with the partially downloaded file(s), as I wasn't sure if that would remove it all correctly.

When this model completes downloading I will attempt to run it.

[ericcurtin]

I think this fixes it:

#450

[ilopezluna]

Hi @akoumjian this issue has been fixed in #610