High Impact Suggestions for the Team

[cjihrig]

Hi Test Runner Team,

I realize that everyone here is a volunteer and contributes whatever they want (or doesn't even contribute at all). There are some issues that I think may be flying under your radar that I think could have a big impact on the test runner experience:

• Finish/stabilize code coverage. I've seen this in use a fair amount in the wild, but people always point out that it is still experimental. I think it's worth evaluating the accuracy and considering stabilizing it. The performance would also be good to look at, but should not be considered a blocker since it can be improved without impacting users. Coverage also does not support statement level coverage, but this should not be a blocker because:
  • This is an artifact of V8's coverage output. Outside of changing V8, getting statement level coverage would require an additional parsing pass over all of the source code to map statements to positions in files. This would almost certainly be bad for performance.
  • I believe C8 also does not support statement coverage for V8 output. My understanding is that line coverage is reported as statement coverage because it's good enough.
  • It can always be added in a semver minor change if someone comes up with a good solution.
• Combining test runner flags with npm scripts (--test-name-pattern needing to come before filenames is hostile to npm scripts node#51384) - There are some possible solutions described in that thread. Right now, it's possible to work around this with NODE_OPTIONS, but it's not really a nice solution. That issue appears to be a blocker to jsdom adopting the test runner for example. This would be a massive DX improvement.
• Properly associating console.log() output with its test in the reported output (node:test custom reporters get test:stdout and test:stderr events before test:dequeue node#53103) - One way to solve this could be to add code to Console, similar to what diagnostic channels do. Then, that code could integrate with the test runner's reporter functionality. This would be a massive DX improvement.
• Stabilize module mocking. If module: add clearCache for CJS and ESM node#61767 lands, I think it would make sense to leverage that first. I wanted to do something similar when I originally wrote the mocking code, but I anticipated there being FAR more pushback on breaking spec compliance then there has been in #61767. For ESM in particular, #61767 has the potential to greatly improve the DX of module mocking.

Some other nice to have changes would be:

• Migrate the test runner off of async_hooks. Now that AsyncContextFrame, it may be possible to achieve the same functionality, and I believe the project would like to get rid of async_hooks. Currently, async_hooks are only used to map asynchronous activity back to the test that initiated it. I haven't investigated what it would take to make this change in much detail.
• Give the spec reporter a new paint job. As long as you don't change the programmatic behavior of the reporter, we are free to change the actual output (text, colors, etc.) of the reporters without regard for semver. This has the potential to be a big DX improvement.

[vassudanagunta]

Not a team member, but I'd like to take on nodejs/node#53103. My in-the-works node-test-extra library, which uses a Proxy membrane to augment node:test, has this functionality implemented but it has problems with asynchronous tests executed by the --test runner in subprocesses. I'd rather spend my time implementing it natively in node than figure out a solution in my proxy layer.

I already have a custom spec reporter with a very pretty "paint job" that presents test-specific stdout in a nice way. Along with a PR for the above, I'd be able to offer a PR with a new spec reporter for your consideration.