test: minimize time for child_process benchmark

[Trott]

test-benchmark-child-process sometimes times out on Windows in CI.
Minimize the number of configurations that run so it will (hopefully)
not time out anymore.

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• commit message follows commit guidelines

Affected core subsystem(s)

test child_process benchmark

[Trott]

cc @joyeecheung

[Trott]

I'd like to get this landed sooner than 48 hours because Windows CI is failing a lot because of this right now. @nodejs/testing @nodejs/build

[Trott]

CI: https://ci.nodejs.org/job/node-test-pull-request/7525/

[Trott]

Hmmm....CI still timed out on one node, but also had another odd timeout failure.

Took only 2 seconds to run on a machine where it passed... wonder if we just need to clean up some hosts first...maybe something is just odd with test-azure_msft-win2016-x64-4 or something like that...

Running CI again: https://ci.nodejs.org/job/node-test-pull-request/7527/

[Trott]

Trying with execSync rather than exec to see if that helps at all.

CI: https://ci.nodejs.org/job/node-test-pull-request/7528/

[Trott]

If the above CI jobs don't run cleanly, the thing to do might be to move the test to disabled until we figure out how to make it reliable on Windows. I would have to suspect a bug in the actual benchmark code and not in the test but ¯\(ツ)/¯.

Anyway, hopefully one or both of the above CI runs comes out green and we can at least try this.

[vsemozhetbyt]

@Trott Are we sure we have yes.exe in this machine in PATH? Or can we test its functionality aside?

[Trott]

@Trott Are we sure we have yes.exe in this machine in PATH?

No idea. Guessing not. @nodejs/build?

Guess that may be the problem. Now wondering if this landed without a clean CI run in the first place...

[Trott]

@vsemozhetbyt Took your comment and ran with it: #12326 (comment)

[Trott]

Although I don't think it would hang if that's the case...it would exit with ENOENT, I think.

[Trott]

No real reason to think this will make a huge difference, but set dur=0 because we can. In the likely event this doesn't solve the problem, I'll move the test to disabled or revert the commit that added it.

CI: https://ci.nodejs.org/job/node-test-pull-request/7529/

[Trott]

sigh Windows build failure. Let's try again.

CI: https://ci.nodejs.org/job/node-test-pull-request/7530/

[vsemozhetbyt]

What if we --filter these 5 benchmarks one by one here to detect the source of the problem?

[Trott]

Set dur=0 seems to have fixed it, although a stress test is probably in order.

Also, dur=0.1 really shouldn't break, so doing what @vsemozhetbyt and trying to figure out which benchmark is blowing up is good too. We can do it in CI, but if someone on @nodejs/platform-windows or @nodejs/build happens to have an accessible Windows 2016 machine and figure out which ones are hanging, that would be more awesome and efficient.

[Trott]

Stress test for this PR: https://ci.nodejs.org/job/node-stress-single-test/1156/nodes=win2016/console

Stress test for master (should show failures): https://ci.nodejs.org/job/node-stress-single-test/1157/nodes=win2016/console

Hopefully I didn't make any typos when filling out the Jenkins form...

[Trott]

Also, when it's done waiting for one of the other stress jobs to finish, this should figure out which of the five benchmarks are the issue: https://ci.nodejs.org/job/node-stress-single-test/1159/nodes=win2016/ whoops I had an error in the branch, will try again...

[Trott]

Stress tests are behaving as expected. This branch has run more than 40 times without a problem. Will let it go a bit longer. Master has timed out twice without a successful run so far.

[Trott]

Stress test to identify which of the benchmarks is causing problems: https://ci.nodejs.org/job/node-stress-single-test/1161/nodes=win2016/console

[Trott]

Meantime: Landing to fix CI.

[Trott]

Landed in 32f77ff. CI should be back to green now.

[Trott]

Interestingly, looking at the results on Windows right now for the split out tests, it seems like it may have been a genuine timeout and not anything hanging. Splitting the test into five pieces, they all pass, but take about 30 seconds each to run. Yikes! That seems...odd... https://ci.nodejs.org/job/node-stress-single-test/1161/nodes=win2016/console

Gotta run right now, but maybe if I get some time later, I'll spin up a similar job on another Windows machine and maybe a Linux machine too and compare the run times.

[joyeecheung]

Sorry for being a bit late. I think we can probably make two tests for each set of benchmarks, one setting numeric configurations to 0 (mostly for testing if there are any syntax errors) and one setting those to 1 or a value as small as possible (see if they actually run properly). If we ever have to disable the second one, at least we will have the first one in place...

[gibfahn]

Should land with #12326.