Skip to content

Go SDK: fix operator-precedence bug that could crash the worker process#70141

Open
ColtenOuO wants to merge 2 commits into
apache:mainfrom
ColtenOuO:fix-go-sdk-heartbeat-nil-check
Open

Go SDK: fix operator-precedence bug that could crash the worker process#70141
ColtenOuO wants to merge 2 commits into
apache:mainfrom
ColtenOuO:fix-go-sdk-heartbeat-nil-check

Conversation

@ColtenOuO

Copy link
Copy Markdown
Contributor

Summary

heartbeater.Run in the Go SDK worker checks whether a heartbeat error indicates the server no longer wants the task running (404/409), so it can cancel the task instead of retrying forever:

if resp != nil && resp.StatusCode() == 404 || resp.StatusCode() == 409 {

&& binds tighter than || in Go, so this actually parses as (resp != nil && resp.StatusCode() == 404) || resp.StatusCode() == 409 -- the second disjunct calls resp.StatusCode() unconditionally, bypassing the nil check the resp != nil && clearly intended to guard.

resp here is *api.GeneralHTTPError.Response (*resty.Response), which can be nil for transport-level failures that never got as far as an actual HTTP response (e.g. connection reset, TLS failure). When that happens, resp.StatusCode() panics on a nil pointer -- and this happens inside the heartbeater's own goroutine (go heartbeater.Run(heartbeatCtx) in ExecuteTaskWorkload), which has no recover() of its own. Go's recover() only catches panics in the same goroutine's call stack, so this crashes the entire worker process, not just the one task.

Confirmed by temporarily reverting the fix and running the new test: it panics with SIGSEGV at exactly this line, unrecovered, exactly as described above.

Changes

  • Add parentheses so the nil guard covers both status checks:
    resp != nil && (resp.StatusCode() == 404 || resp.StatusCode() == 409).
  • Add TestTaskHeartbeatNilResponseDoesNotPanic, which returns a
    GeneralHTTPError{Response: nil} from the mocked heartbeat call and
    asserts the task completes normally instead of crashing.

heartbeater.Run checked `resp != nil && resp.StatusCode() == 404 ||
resp.StatusCode() == 409` -- `&&` binds tighter than `||`, so the nil
guard only covered the 404 branch. A GeneralHTTPError with a nil
Response (e.g. a transport-level failure) hit resp.StatusCode() on a
nil pointer in the heartbeater's own goroutine, which has no recover,
crashing the whole worker process instead of just failing the task.
Add parentheses to group both status checks under the nil guard.
Comment thread go-sdk/pkg/worker/runner_test.go Outdated

@Andrushika Andrushika left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, thanks!

…test

Address review feedback: wait on a channel signaled by the mocked
Heartbeat's second call instead of a fixed 250ms sleep, so the test
isn't tied to wall-clock timing that could be flaky under CI load. A
second heartbeat call also positively confirms the heartbeater
survived processing the first nil-Response error.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants