Private (gen) model from HF fails running in promptnode

[ArzelaAscoIi]

Describe the bug
(generative) private models on huggingface can't be run, since they append use_auth_token to the model kwargs which is not expected.
That means:

• using private models from HF within the Promptnode does not work
• if you configure an auth token and you have a public model, your will be able to initialize, but not run the pipeline (quite an edge case)

Error message

ValueError: The following `model_kwargs` are not used by the model: ['use_auth_token'] (note: typos in the generate arguments will also show up in this list)

Expected behavior
Send token to HF for download but not inject args as model_kwargs

Additional context
Add any other context about the problem here, like document types / preprocessing steps / settings of reader etc.

To Reproduce

from haystack.nodes import PromptNode, PromptTemplate
from haystack import Document


# Works
pn_works = PromptNode(
    default_prompt_template=PromptTemplate(prompt="deepset/summarization"),
    model_name_or_path="google/flan-t5-base",
)
# retunrs successful
pn_works.run("That works!",documents=[Document(content="asdf")])


# Fails
pn_fails = PromptNode(
    default_prompt_template=PromptTemplate(prompt="deepset/summarization"),
    model_name_or_path="google/flan-t5-base",
    use_auth_token="<any-hf-token>",
)
# retunrs successful
pn_fails.run("Oh no!",documents=[Document(content="asdf")])

FAQ Check

• Have you had a look at our new FAQ page?

System:

• N/A

[ArzelaAscoIi]

Additional information. I tested this with a private flan clone

Soo there are actually two issues now (I think). If you try to use a private model with use_auth_token you are getting the error that the model is not listed (transformers changed the argument to token).
If we take one more step (assuming that we fixed the argument), you will most likely still get the unexpected argument error.

So we need to have two fixes:

• adjust the paramter to token (or map argument to token)
• fix run method to not pass the token to the inference call

[anakin87]

@ArzelaAscoIi I am working on a fix.

I'm experimenting with two use cases:

• use a public model and pass a HF token (that should be ignored)
• use a private model and pass a HF token (that should be used)

from haystack.nodes import PromptNode, PromptTemplate
from haystack import Document

pn_unused_auth_token = PromptNode(
    default_prompt_template=PromptTemplate(prompt="deepset/summarization"),
    model_name_or_path="google/flan-t5-base",
    use_auth_token="UNUSED-HF-TOKEN",
)

res=pn_unused_auth_token.run("Oh no!",documents=[Document(content="Some text to summarize")])


pn_private = PromptNode(
    default_prompt_template=PromptTemplate(prompt="deepset/summarization"),
    model_name_or_path="anakin87/flan-t5-base-clone",
    use_auth_token="REAL-HF-TOKEN",
)

res=pn_private.run("Oh no!",documents=[Document(content="Some text to summarize")])

In case my fix makes the previous code correctly run, can we assume that the bug has been fixed?
Or is there some scenario that I haven't considered?

[ArzelaAscoIi]

Yes. If both scenarios work as expected it would cover all cases we need :) Thanks!

[anakin87]

Fixed in #6533.

The fix will be part of the 1.23.0 release.