Llava runner job failing after HF transformers update

[GregoryComer]

🐛 Describe the bug

The test-llava-runner-linux job has been failing since the dependency update in #9408. We need to investigate and resolve the breakage.

Versions

CI

cc @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng

[jackzhxng]

Probably some update to the Llava transformers model. Maybe would be worth having a separate transformers pin for Llava cc @larryliu0820