e2e tests are unstable

[khassel]

No new problem, our e2e test are failing unrelated to the tested code.

Was already discussed in some PR's so I decided to make an issue for this.

My observations:

• not related to native fetch, was the same with node-fetch before

• most fails are with node v20 (but not exclusive)

• seems to be a timing problem, very hard to reproduce locally

• looks like different tests interfere, maybe the next one is started before the predecessor stopped

• some strange errors are mocked in tests/e2e/helpers/mock-console.js:

  const mockError = (err) => {
    if (
      err.includes("ECONNREFUSED") ||
      err.includes("ECONNRESET") ||
      err.includes("socket hang up") ||
      err.includes("exports is not defined") ||
      err.includes("write EPIPE") ||
      err.includes("AggregateError") ||
      err.includes("ERR_SOCKET_CONNECTION_TIMEOUT")
    ) {
      jest.fn();
    } else {
      console.dir(err);
    }
  };
  
  global.console = {
    log: jest.fn(),
    dir: console.dir,
    error: mockError,
    warn: console.warn,
    info: jest.fn(),
    debug: console.debug
  };

  So removing mockError here will show these errors and may can lead to the real error

[bugsounet]

most fails are with node v20 (but not exclusive)

Maybe it's related to node 20 version, this branch is still young

Just for see: Test suite Passed / Failed node v20.x version:

[Pass]
node v20.7.0

[Failed]
node v20.6.1

[khassel]

yes, since running on v20.7.0 it is more stable.

[bugsounet]

this is not the case with v20.8.0

[bugsounet]

Finally, I went back to node v18. Too many strange problems on my Pi4. but oddly less on my dev PC

[sdetweil]

should this be closed, now on 20.9 and up?

[khassel]

I don't think it is really solved, there are still test fails sometimes. But can be closed for now.