test: make crypto.timingSafeEqual test less flaky by not-an-aardvark · Pull Request #8456 · nodejs/node

not-an-aardvark · 2016-09-08T23:31:27Z

Checklist

make -j4 test (UNIX), or vcbuild test nosign (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

crypto

Description of change

~~WIP; do not merge.~~

The crypto.timingSafeEqual test still seems to be a bit flaky. This makes a few changes to the test:

Separates the basic usage and the benchmarking into different tests
Moves the timing-sensitive benchmark function into a separate module, and reparses the module on every iteration of the loop to avoid shared state between timing measurements.

If this doesn't work, an alternative would be to start a separate child process for each individual timing measurement, which would completely avoid shared state between measurements (although it would also probably make the test much more CPU-intensive).

/cc @Trott

Refs: #8040, #8203, #8304

Trott · 2016-09-08T23:38:47Z

Haven't been able (yet) to force failures via stress test repetition but have seen it coming up on full CI runs, so here's three of those:

CI: https://ci.nodejs.org/job/node-test-pull-request/3975/

CI again: https://ci.nodejs.org/job/node-test-pull-request/3976/

CI one more time: https://ci.nodejs.org/job/node-test-pull-request/3977/

Trott · 2016-09-08T23:40:09Z

test/sequential/test-crypto-timing-safe-equal-benchmarks.js

+const crypto = require('crypto');
+
+const BENCHMARK_FUNC_PATH =
+  '../fixtures/crypto-timing-safe-equal-benchmark-func';


Nit: common.fixturesDir but probably no need to worry about that until we see if this even works.

not-an-aardvark · 2016-09-09T00:47:41Z

3 test timeouts on ARM: 1, 2, 3
1 build failure on Ubuntu 18.04: here

(The test takes a bit longer now because it has to call require() and parse a module 20000 times.)

Trott · 2016-09-09T04:47:54Z

Stress test on debian8-x86 with master shows a 30% failure rate. https://ci.nodejs.org/job/node-stress-single-test/894/nodes=debian8-x86/console

~~Here's a stress test on debian8-x86 against this PR for comparison: https://ci.nodejs.org/job/node-stress-single-test/895/nodes=debian8-x86/console~~

Trott · 2016-09-09T04:50:03Z

Timeout on Raspberry Pi is probably acceptable. A few other tests do this to skip tests on machines with low RAM (like the old Pi devices):

if (!common.enoughTestMem) {
  common.skip(skipMessage);
  return;
}

We can do that on this test too if the Raspberry Pi 1 can't handle it.

not-an-aardvark · 2016-09-09T04:53:51Z

I think this stresstest is running the wrong test; it should be test-crypto-timing-safe-equal-benchmarks for this PR since the test file was split into two.

Trott · 2016-09-09T05:18:43Z

Whoops, yes, let's re-run that stress test but with the correct test this time: https://ci.nodejs.org/job/node-stress-single-test/nodes=debian8-x86/898/console

Trott · 2016-09-09T05:26:04Z

I don't want to monopolize our only debian8-x86 test machine for 14 hours (which is what I'm calculating it would take to run this test 10K times... 10K * 5 seconds = 50K seconds = ~14 hours).

So I'm going to stop it now after ~80 runs without a single failure. Certainly more than enough to feel confident that the failure rate on debian8-x86 will be much less than the 30% we are seeing on current master.

Maybe add the skip code and then we can run regular CI a few more times to see if everything is A-OK?

Trott · 2016-09-09T05:41:32Z

test/sequential/test-crypto-timing-safe-equal-benchmarks.js

+}
+
+if (!common.enoughTestMem) {
+  common.skip('skipping memory-intensive test');


Nit: common.skip() prepends text indicating the test has been skipped so the word 'skipping' is redundant. I'm sure this is actually an issue with a bunch of other tests and is fine to leave as-is (hence the 'Nit' prefix) if you wish, as fixing these throughout the tests would probably make a good first contribution for a newcomer anyway.

Based on a search in the test/ folder it seems like all the other tests actually do this correctly, so I fixed this one to do it correctly as well.

Trott · 2016-09-09T05:42:17Z

CI: https://ci.nodejs.org/job/node-test-pull-request/3982/

Trott · 2016-09-09T06:22:42Z

CI looks great. (Windows failure looks unrelated, will open separate issue to investigate if one isn't already open.)

@nodejs/testing @nodejs/crypto

jasnell · 2016-09-09T13:56:46Z

LGTM

The `crypto.timingSafeEqual` test still seems to be a bit flaky. This makes a few changes to the test: * Separates the basic usage and the benchmarking into different tests * Moves the timing-sensitive benchmark function into a separate module, and reparses the module on every iteration of the loop to avoid shared state between timing measurements. PR-URL: nodejs#8456 Reviewed-By: James M Snell <jasnell@gmail.com>

Trott · 2016-09-12T04:06:38Z

Landed in c678ecb 🎉

The `crypto.timingSafeEqual` test still seems to be a bit flaky. This makes a few changes to the test: * Separates the basic usage and the benchmarking into different tests * Moves the timing-sensitive benchmark function into a separate module, and reparses the module on every iteration of the loop to avoid shared state between timing measurements. PR-URL: #8456 Reviewed-By: James M Snell <jasnell@gmail.com>

test: make crypto.timingSafeEqual test less flaky

30157e8

nodejs-github-bot added the test Issues and PRs related to the tests. label Sep 8, 2016

Trott reviewed Sep 8, 2016
View reviewed changes

not-an-aardvark added 2 commits September 9, 2016 01:30

squash: use common.fixturesDir

c6a2af2

squash: skip tests on Raspberry Pi

5abe8b9

Trott reviewed Sep 9, 2016
View reviewed changes

squash: skip skip()'s 'skipping'

0e2e8e9

not-an-aardvark changed the title ~~WIP: test: make crypto.timingSafeEqual test less flaky~~ test: make crypto.timingSafeEqual test less flaky Sep 9, 2016

Trott mentioned this pull request Sep 9, 2016

util: don't init Debug if it's not needed yet #8452

Closed

2 tasks

not-an-aardvark mentioned this pull request Sep 9, 2016

crypto: re-add crypto.timingSafeEqual #8304

Closed

4 tasks

Trott closed this Sep 12, 2016

not-an-aardvark deleted the fix-more-timing-safe-equal-flakes branch September 12, 2016 04:09

MylesBorins added the dont-land-on-v4.x label Sep 30, 2016

Uh oh!

Conversation

not-an-aardvark commented Sep 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Affected core subsystem(s)

Description of change

Uh oh!

Trott commented Sep 8, 2016

Uh oh!

Trott Sep 8, 2016

Choose a reason for hiding this comment

Uh oh!

not-an-aardvark commented Sep 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Trott commented Sep 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Trott commented Sep 9, 2016

Uh oh!

not-an-aardvark commented Sep 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Trott commented Sep 9, 2016

Uh oh!

Trott commented Sep 9, 2016

Uh oh!

Trott Sep 9, 2016

Choose a reason for hiding this comment

Uh oh!

not-an-aardvark Sep 9, 2016

Choose a reason for hiding this comment

Uh oh!

Trott commented Sep 9, 2016

Uh oh!

Trott commented Sep 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasnell commented Sep 9, 2016

Uh oh!

Trott commented Sep 12, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

not-an-aardvark commented Sep 8, 2016 •

edited

Loading

not-an-aardvark commented Sep 9, 2016 •

edited

Loading

Trott commented Sep 9, 2016 •

edited

Loading

not-an-aardvark commented Sep 9, 2016 •

edited

Loading

Trott commented Sep 9, 2016 •

edited

Loading