Describe the bug
When a watch connection closes abruptly, the done callback may be invoked multiple times because of reentrancy in the doneCallOnce() function.
Client Version
1.1.2
Server Version
v1.32.2-gke.1182003
To Reproduce
- Use
Watch to start a stream (e.g. watch pods in a namespace).
- Simulate a premature close on the connection (e.g. by destroying the socket on the server side).
- The
done callback is called with "AbortError: The user aborted a request"
- The
done callback is called again, but now with "Error: Premature close"
Expected behavior
The done callback should only be called once per watch termination.
Example Code
The code below can be used to replicate the issue on a local machine.
import http from 'http'
import { KubeConfig, Watch } from '@kubernetes/client-node'
const MOCK_PORT = 8333
const MOCK_URL = `http://localhost:${MOCK_PORT}`
const minimalServer = http.createServer((req, res) => {
res.writeHead(200, {
'Content-Type': 'application/json',
'Transfer-Encoding': 'chunked',
})
res.flushHeaders()
res.destroy() // Prematurely close the connection
})
minimalServer.listen(MOCK_PORT, () => {
const kubeConfig = new KubeConfig()
kubeConfig.loadFromClusterAndUser(
{
name: 'mock-cluster',
server: MOCK_URL,
skipTLSVerify: true,
},
{ name: 'mock-user' }
)
const watch = new Watch(kubeConfig)
watch.watch(
`/api/v1/namespaces/default/pods`,
{},
() => { },
(err) => console.log('done()', err)
)
})
Output:
done() AbortError: The user aborted a request.
at abort (/home/bas/projects/watch-repro/node_modules/node-fetch/lib/index.js:1458:16)
at AbortSignal.abortAndFinalize (/home/bas/projects/watch-repro/node_modules/node-fetch/lib/index.js:1473:4)
at [nodejs.internal.kHybridDispatch] (node:internal/event_target:827:20)
at AbortSignal.dispatchEvent (node:internal/event_target:762:26)
at runAbort (node:internal/abort_controller:449:10)
at abortSignal (node:internal/abort_controller:435:3)
at AbortController.abort (node:internal/abort_controller:468:5)
at PassThrough.doneCallOnce (file:///home/bas/projects/watch-repro/node_modules/@kubernetes/client-node/dist/watch.js:33:28)
at PassThrough.emit (node:events:519:35)
at emitErrorNT (node:internal/streams/destroy:170:8) {
type: 'aborted'
}
done() Error: Premature close
at IncomingMessage.<anonymous> (/home/bas/projects/watch-repro/node_modules/node-fetch/lib/index.js:1748:18)
at Object.onceWrapper (node:events:621:28)
at IncomingMessage.emit (node:events:507:28)
at emitCloseNT (node:internal/streams/destroy:148:10)
at process.processTicksAndRejections (node:internal/process/task_queues:89:21) {
code: 'ERR_STREAM_PREMATURE_CLOSE'
}
Environment (please complete the following information):
- OS: Linux
- Node.js version 23.11.0
- Cloud runtime: GCP
Additional context
This was discovered while debugging a broader issue with ListWatch failing to reconnect on non-410 errors (#2385). In those cases, doneCallOnce emits two errors:
- AbortError: The user aborted a request.
- Error: Premature close
Although doneCallOnce is intended to emit an error only once, it currently calls controller.abort() before setting doneCalled = true. This seems to trigger an AbortError in certain circumstances, which leads to a second invocation of doneCallOnce, since the doneCalled flag hasn’t been set yet.
As a result, both the original error and the AbortError are reported.
|
let doneCalled: boolean = false; |
|
const doneCallOnce = (err: any) => { |
|
if (!doneCalled) { |
|
controller.abort(); |
|
doneCalled = true; |
|
done(err); |
|
} |
|
}; |
I was asked to create a separate issue and am working on a PR to add tests and fix the issue.
Describe the bug
When a watch connection closes abruptly, the
donecallback may be invoked multiple times because of reentrancy in thedoneCallOnce()function.Client Version
1.1.2Server Version
v1.32.2-gke.1182003To Reproduce
Watchto start a stream (e.g. watch pods in a namespace).donecallback is called with "AbortError: The user aborted a request"donecallback is called again, but now with "Error: Premature close"Expected behavior
The
donecallback should only be called once per watch termination.Example Code
The code below can be used to replicate the issue on a local machine.
Output:
Environment (please complete the following information):
Additional context
This was discovered while debugging a broader issue with
ListWatchfailing to reconnect on non-410 errors (#2385). In those cases,doneCallOnceemits two errors:Although
doneCallOnceis intended to emit an error only once, it currently callscontroller.abort()before settingdoneCalled = true. This seems to trigger an AbortError in certain circumstances, which leads to a second invocation ofdoneCallOnce, since thedoneCalledflag hasn’t been set yet.As a result, both the original error and the
AbortErrorare reported.javascript/src/watch.ts
Lines 46 to 53 in 88184dc
I was asked to create a separate issue and am working on a PR to add tests and fix the issue.