fix(gateway): skip failover sweep when request context is done#453
Merged
Conversation
When a client disconnects (or the request deadline passes) mid-request, `http.Client.Do` returns `context.Canceled`, which the provider client maps to a 502. `ShouldAttemptFailover` treats any >=500 as retryable and never consulted the context, so the primary failure triggered a full failover sweep over every configured target on an already-dead context. Each doomed attempt failed instantly but still recorded a provider attempt, published a live audit event, and — because the circuit breaker records a failure for any non-nil error — charged a spurious failure to each healthy failover provider's breaker. A burst of client disconnects could therefore trip breakers on providers that were never actually unhealthy. Streaming was always affected. Guard both `tryFailoverResponse` and `tryFailoverStream` on `ctx.Err()`: short-circuit to the primary error before sweeping, and also break out of the loop if the client disconnects partway through the sweep. A failover call on a done context can never succeed, so this loses no working behavior. Adds regression tests that fail without the guard and a positive control confirming a live context still attempts failover. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
📝 WalkthroughWalkthroughThe PR adds context cancellation checks to ChangesFailover context cancellation handling
Estimated code review effort: 2 (Simple) | ~12 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant Gateway
participant tryFailoverResponse
participant FailoverTarget
Client->>Gateway: Request (context)
Gateway->>tryFailoverResponse: primaryErr, ctx
alt context canceled
tryFailoverResponse-->>Gateway: return primaryErr immediately
else context live
loop for each failover target
tryFailoverResponse->>FailoverTarget: attempt failover call
alt context becomes done mid-loop
tryFailoverResponse-->>Gateway: stop, return primaryErr
else success
FailoverTarget-->>tryFailoverResponse: response
tryFailoverResponse-->>Gateway: return response, didFailover=true
end
end
end
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/gateway/failover_test.go`:
- Around line 84-108: Add a symmetric positive-control test for the stream
failover path by creating a new test alongside
TestTryFailoverStreamSkipsWhenContextCanceled that exercises tryFailoverStream
with a live context and verifies failover providers are attempted. Reuse
failoverTestFixture and a mock call function to assert called becomes true, the
returned stream is non-nil, and err is nil when the primary error is a retryable
ProviderError. This should mirror TestTryFailoverResponseAttemptsWhenContextLive
and cover the loop-break behavior in tryFailoverStream.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 7f0707a9-abb9-4732-89ca-f7bc5adafa1d
📒 Files selected for processing (2)
internal/gateway/failover.gointernal/gateway/failover_test.go
Comment on lines
+84
to
+108
| func TestTryFailoverStreamSkipsWhenContextCanceled(t *testing.T) { | ||
| o, workflow := failoverTestFixture() | ||
|
|
||
| ctx, cancel := context.WithCancel(context.Background()) | ||
| cancel() | ||
|
|
||
| primaryErr := core.NewProviderError("openai", http.StatusBadGateway, "context canceled", context.Canceled) | ||
| called := false | ||
| call := func(core.ModelSelector, string, string) (io.ReadCloser, string, string, error) { | ||
| called = true | ||
| return nil, "", "", core.NewProviderError("openai", http.StatusBadGateway, "unexpected failover call", nil) | ||
| } | ||
|
|
||
| stream, _, _, _, _, err := tryFailoverStream(ctx, o, workflow, "openai/gpt-4o", "openai", primaryErr, call) | ||
|
|
||
| if called { | ||
| t.Fatal("tryFailoverStream invoked a failover provider on a canceled context; it must short-circuit") | ||
| } | ||
| if stream != nil { | ||
| t.Fatal("tryFailoverStream returned a stream on a canceled context") | ||
| } | ||
| if !errors.Is(err, context.Canceled) { | ||
| t.Fatalf("err = %v, want the primary error wrapping context.Canceled", err) | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Missing symmetric positive-control test for stream failover.
Response path has both a canceled-context test and a live-context test (TestTryFailoverResponseAttemptsWhenContextLive), but the stream path only has the canceled-context test here. There's no TestTryFailoverStreamAttemptsWhenContextLive verifying tryFailoverStream still sweeps failover targets on a live context — leaving the loop-break guard at failover.go Lines 174-177 unexercised by tests.
✅ Suggested additional test
func TestTryFailoverStreamAttemptsWhenContextLive(t *testing.T) {
o, workflow := failoverTestFixture()
primaryErr := core.NewProviderError("openai", http.StatusInternalServerError, "primary boom", nil)
called := false
call := func(core.ModelSelector, string, string) (io.ReadCloser, string, string, error) {
called = true
return io.NopCloser(strings.NewReader("ok")), "openai", "gpt-5", nil
}
stream, _, _, _, _, err := tryFailoverStream(context.Background(), o, workflow, "openai/gpt-4o", "openai", primaryErr, call)
if !called {
t.Fatal("tryFailoverStream did not attempt failover on a live context")
}
if stream == nil || err != nil {
t.Fatalf("failover result = (stream:%v err:%v), want (non-nil <nil>)", stream, err)
}
}As per path instructions, **/*_test.go: "Add or update tests for behavior changes, and ensure tests cover request translation, response normalization, error handling, default configuration, and provider-specific parameter mapping."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@internal/gateway/failover_test.go` around lines 84 - 108, Add a symmetric
positive-control test for the stream failover path by creating a new test
alongside TestTryFailoverStreamSkipsWhenContextCanceled that exercises
tryFailoverStream with a live context and verifies failover providers are
attempted. Reuse failoverTestFixture and a mock call function to assert called
becomes true, the returned stream is non-nil, and err is nil when the primary
error is a retryable ProviderError. This should mirror
TestTryFailoverResponseAttemptsWhenContextLive and cover the loop-break behavior
in tryFailoverStream.
Source: Path instructions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a correctness bug in the translated-route failover path (surfaced during pre-release testing of #444/#450/#451): a client disconnect or expired deadline triggers a doomed failover sweep that trips healthy providers' circuit breakers.
When the client goes away mid-request,
http.Client.Doreturnscontext.Canceled, which the provider client maps to a 502.ShouldAttemptFailovertreats any>=500as retryable and never consulted the context, so the primary failure kicked off a full failover sweep over every configured target — on an already-dead context. Each attempt failed instantly but still:ProviderAttemptand published a live audit event, andrecordCircuitBreakerCompletionrecords a failure for any non-nil error).So a burst of client disconnects could open circuit breakers on providers that were never actually unhealthy. Streaming is always affected; non-streaming is affected when retries are exhausted/disabled.
Fix
Guard both
tryFailoverResponseandtryFailoverStreamonctx.Err():A failover call on a done context can never succeed, so this removes no working behavior — it only avoids doomed attempts and the circuit-breaker pollution they cause.
User-visible impact
Tests
TestTryFailoverResponseSkipsWhenContextCanceled/TestTryFailoverStreamSkipsWhenContextCanceled— assert no provider call happens on a canceled context (both fail without this patch, verified).TestTryFailoverResponseAttemptsWhenContextLive— positive control: a live context still attempts failover (guards against over-blocking).Verified locally: full build,
go test -race ./internal/gateway/...,golangci-lint(0 issues), and the hot-path perf guard all pass.🤖 Generated with Claude Code
Summary by CodeRabbit