fix(models): keep proxied file uploads inside the proxy - #6723
Open
vietnamesekid wants to merge 2 commits into
Open
fix(models): keep proxied file uploads inside the proxy#6723vietnamesekid wants to merge 2 commits into
vietnamesekid wants to merge 2 commits into
Conversation
`litellm.acreate_file` resolves its endpoint independently of the completion call. ADK passed only `custom_llm_provider`, so the upload fell back to the underlying provider's environment variables (`AZURE_API_BASE`, `AZURE_API_KEY`). For a proxied model that splits the request in two: the completion goes to the proxy while the upload goes straight to Azure. Callers who configure only `LiteLlm(model="litellm_proxy/azure/...", api_base=..., api_key=...)` and hold no Azure credentials of their own get `OpenAIError: Missing credentials`, which is the whole reason to front a provider with a proxy. Forward `api_base`, `api_key`, and `api_version` from the completion arguments to the upload when the model is proxied. Direct models forward nothing and keep their existing environment-variable resolution, since their `api_base` already points at the provider. Verified against a local HTTP endpoint: with only proxy credentials set, the upload now reaches the proxy and returns a real file_id, where before it raised before sending anything.
`litellm.acreate_file` resolves its endpoint independently of the completion call, so a proxied upload falls back to the underlying provider's own credentials. That splits the request in two: the completion goes to the proxy while the upload goes somewhere else. For `azure` that fails locally with missing credentials. For `openai` it is worse than a failure: `get_openai_credentials` defaults `api_base` to `https://api.openai.com/v1`, so a stray `OPENAI_API_KEY` in the environment silently sends proxied file content to the public OpenAI API. Three paths reached that fallback: - The proxy configured through `LITELLM_PROXY_API_BASE` / `LITELLM_PROXY_API_KEY` rather than constructor arguments. LiteLLM reads these for the completion call, so the upload has to follow the same resolution. - `USE_LITELLM_PROXY=true`, which routes unprefixed models such as `openai/gpt-4o` through the proxy. Matching only on the `litellm_proxy/` prefix missed those entirely. - No proxy endpoint determinable at all, where the upload previously fell through to the provider default. Fall back to the proxy environment variables, treat the `USE_LITELLM_PROXY` flag as proxied, and raise when the endpoint cannot be determined rather than sending file content to the provider's default endpoint. Note that the last case turns a silently misrouted upload into an explicit error. Callers relying on a stray `OPENAI_API_KEY` to make proxied uploads "work" will now see a failure that names the missing configuration. Verified with dotenv-loaded .env files against a local proxy, driving `LiteLlm(...)` end to end with all outbound requests recorded. Both the upload and the completion reach the proxy, carrying the proxy credentials, with no request to api.openai.com in any scenario.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
Problem:
litellm.acreate_fileresolves its endpoint independently of the completioncall. ADK passes only
custom_llm_provider, so a proxied upload falls back tothe underlying provider's own credential resolution and the request splits in
two: the completion goes to the proxy, the upload goes elsewhere.
For
azurethat is a local failure. Foropenaiit is not a failure at all.get_openai_credentialsdefaultsapi_basetohttps://api.openai.com/v1, soa stray
OPENAI_API_KEYmakes the upload succeed against the public OpenAI APIcarrying proxied file content and the developer's own key.
Three configurations reach that fallback. Each row was run with the repro
script in the linked issue, changing only
.envand the model string.LITELLM_PROXY_API_BASE+LITELLM_PROXY_API_KEYlitellm_proxy/openai/gpt-4oapi.openai.comUSE_LITELLM_PROXY=trueopenai/gpt-4oapi.openai.comOPENAI_API_KEYlitellm_proxy/openai/gpt-4oapi.openai.comCase 1 exists because LiteLLM reads those two variables for the completion call
in
litellm/llms/litellm_proxy/chat/transformation.py, while the upload pathdoes not. Case 2 exists because matching on the
litellm_proxy/prefix alonemisses
USE_LITELLM_PROXY, which routes unprefixed models through the proxy.LiteLLM's own docstring says that flag exists for Google ADK
(BerriAI/litellm#10559).
On 602e58d, the merge of #6578: the provider resolution part of that change is
correct and this PR keeps it. The same commit also set
custom_llm_provider="openai"for proxied uploads, and that is the line thatturns cases 1 and 3 from a local error into a request to
api.openai.com,since
openaiis the provider that defaults to the public endpoint. Onmain,_get_contenttakes onlyparts,providerandmodel, and ADK never setslitellm.api_base, so there is no configuration in which that override couldreach a proxy. This PR keeps the real provider, which is what preserves the
provider specific payload shape, and redirects the endpoint instead.
Solution:
_get_upload_paramsfalls back toLITELLM_PROXY_API_BASEandLITELLM_PROXY_API_KEYwhen the completion call carries no endpoint.Explicit completion arguments still win.
_is_proxied_modelalso treatsUSE_LITELLM_PROXYas proxied, coveringunprefixed models routed through the proxy.
determined, rather than falling through to the provider default.
Direct models are untouched. They forward nothing and keep their existing
environment variable resolution, since their
api_basealready points at theprovider.
One behavior change worth deciding on rather than absorbing silently, item 3
above. Anyone who has
OPENAI_API_KEYset and a proxied model has a workingupload today, to the wrong destination. After this they get a
ValueErrornaming the configuration to set. I think an explicit error beats a silent
misroute, but it is a trade off. If you would rather not take it, items 1 and 2
alone still fix every case where the proxy endpoint is knowable.
Testing Plan
Unit Tests:
The 6 failures are all in
test_anthropic_llm.pyand reproduce identically ona pristine
mainworktree at 370027a, so they are pre-existing and unrelatedto this change.
New tests:
test_get_upload_params_falls_back_to_proxy_env, covering the env fallbackand that explicit completion arguments take precedence over it.
test_is_proxied_model_honors_use_litellm_proxy_flag, covering the flag andthat the explicit prefix still wins regardless of it.
test_get_content_proxied_upload_without_endpoint_raises, which assertsacreate_fileis not called at all when the destination is unknown.I also isolated
test_get_upload_paramsfrom the ambient environment. It wouldotherwise pick up
LITELLM_PROXY_*from a developer or CI machine and pass orfail depending on which shell it ran in.
Each part of the fix was reverted on its own to check the tests fail without it
instead of passing by construction. Reverting the env fallback, the flag check,
or the guard each fails exactly its own test and nothing else.
Manual End-to-End (E2E) Tests:
Driven through
LiteLlm(...)rather than internal helpers, with configurationloaded from
.envbypython-dotenv, a local HTTP server standing in for theproxy, and every outbound request recorded and blocked if it left localhost.
Full script is in the linked issue.
.env:Before, with
model="litellm_proxy/openai/gpt-4o":After:
Both halves of the request reach the proxy, and the proxy sees
Authorization: Bearer sk-proxy-secret, so it is the proxy credentials beingused and not the personal key.
Full matrix, same script throughout:
.envlitellm_proxy/azure/my-deploymentOPENAI_API_KEYlitellm_proxy/openai/gpt-4oapi.openai.comUSE_LITELLM_PROXY=trueopenai/gpt-4oapi.openai.comOPENAI_API_KEYlitellm_proxy/openai/gpt-4oapi.openai.comReproduced on litellm 1.85.7 and on 1.84.0, the floor of the
litellm>=1.84constraint in
pyproject.toml, so this is not specific to one release.I do not have an Azure backed proxy deployment, so none of this is confirmed
against a live endpoint. Everything above is the request ADK builds and where
it actually goes, measured against a local server.
Formatted with
pyink25.12.0 andisort8.0.1, matching the pinned versionsin
.pre-commit-config.yaml.Checklist
Additional context
Known limit, deliberately out of scope. This covers the proxy endpoint when it
comes from constructor arguments or from
LITELLM_PROXY_API_BASEandLITELLM_PROXY_API_KEY. A proxy configured by some other means that ADK cannotsee still lands in case 3 and raises. That is the safe outcome but it is still
a failure, and closing it would mean reimplementing more of LiteLLM's
configuration resolution inside ADK. That felt like your call rather than mine.
The two commits are separable if you would rather take them independently.
7ab75d24forwards constructor supplied endpoints, which is the follow up Iraised on #6578, and
d8643357closes the three cases above.