Use pure elm for laziness by eeue56 · Pull Request #184 · elm-community/elm-test

eeue56 · 2017-07-01T21:15:43Z

What

This PR does a few things:

switch lazy to be a version written in pure Elm, which only does lazy evaluation (no implicit memoization)
switch lazy-list to use Elm lazy package
switch shrink to use Elm lazy package
switch elm-test to use pure Elm laziness
update the tests to work with the latest runner

Why

In theory, I believe that memoization is actually harmful to large test suites with in terms of memory and performance as a result. As elm-test only visits each node once, there is no need for memoization. I discussed this with @mgold on Slack, who confirmed that this theory could be accurate.

What needs to happen before merging?

I need people to run the benchmarks! As we can't currently run elm-benchmark on node, it would be very useful to have some information on how the benchmarks perform in different browsers (and therefore different GC techniques). My first comment after this PR will be the results that I got, which showed an improvement across the board for the pure-Elm version.

To test:

open benchmarks/new.html in a browser
open benchmarks/old.html in a browser

Future changes

The initial gain over the old approach is decent, but not by a magnitude. If we can confirm that this approach is faster, then there are a couple of optimizations I can do to further improve these results.

eeue56 · 2017-07-01T21:16:52Z

Original approach:

Benchmark Finished

Group: Fuzz

Group: int

Benchmark: generating

mean ops/sec	208,369.55 (stddev: 10,975.32, 5.27%)
mean runtime	4.813ns (stddev: 266.977µs, 5.55%)
total runtime	5.054s
sampling	35 runs of 30,000 calls
Benchmark: shrinking

mean ops/sec	24,374.79 (stddev: 1,139.56, 4.68%)
mean runtime	41.121ns (stddev: 2.028ns, 4.93%)
total runtime	5.058s
sampling	41 runs of 3,000 calls
Group: intRange

Benchmark: generating

mean ops/sec	182,789.21 (stddev: 12,217.25, 6.68%)
mean runtime	5.499ns (stddev: 430.915µs, 7.84%)
total runtime	5.059s
sampling	46 runs of 20,000 calls
Benchmark: shrinking

mean ops/sec	12,062.62 (stddev: 711.1, 5.9%)
mean runtime	83.264ns (stddev: 6.216ns, 7.46%)
total runtime	5.079s
sampling	61 runs of 1,000 calls
Group: string

Benchmark: generating

mean ops/sec	1,965.7 (stddev: 130.09, 6.62%)
mean runtime	511.93ns (stddev: 49.354ns, 9.64%)
total runtime	5.017s
sampling	49 runs of 200 calls
Benchmark: shrinking

mean ops/sec	686.34 (stddev: 44.61, 6.5%)
mean runtime	1.466ms (stddev: 136.001ns, 9.28%)
total runtime	5.027s
sampling	49 runs of 70 calls
Group: float

Benchmark: generating

mean ops/sec	296,874.67 (stddev: 13,341.66, 4.49%)
mean runtime	3.376ns (stddev: 158.472µs, 4.69%)
total runtime	5.063s
sampling	50 runs of 30,000 calls
Benchmark: shrinking

mean ops/sec	21,263.8 (stddev: 1,487.56, 7%)
mean runtime	47.311ns (stddev: 4.083ns, 8.63%)
total runtime	5.015s
sampling	53 runs of 2,000 calls
Group: bool

Benchmark: generating

mean ops/sec	404,458.69 (stddev: 20,036.08, 4.95%)
mean runtime	2.479ns (stddev: 127.5µs, 5.14%)
total runtime	5.057s
sampling	51 runs of 40,000 calls
Benchmark: shrinking

mean ops/sec	74,942.21 (stddev: 5,784.99, 7.72%)
mean runtime	13.443ns (stddev: 1.292ns, 9.61%)
total runtime	5.081s
sampling	54 runs of 7,000 calls
Group: char

Benchmark: generating

mean ops/sec	205,648.59 (stddev: 12,493.51, 6.08%)
mean runtime	4.881ns (stddev: 307.135µs, 6.29%)
total runtime	5.077s
sampling	52 runs of 20,000 calls
Benchmark: shrinking

mean ops/sec	20,204.26 (stddev: 832.03, 4.12%)
mean runtime	49.582ns (stddev: 2.125ns, 4.29%)
total runtime	5.057s
sampling	51 runs of 2,000 calls
Group: list of int

Benchmark: generating

mean ops/sec	3,361.43 (stddev: 218.93, 6.51%)
mean runtime	298.861ns (stddev: 21.238ns, 7.11%)
total runtime	5.021s
sampling	56 runs of 300 calls
Benchmark: shrinking

mean ops/sec	1,645.34 (stddev: 104.77, 6.37%)
mean runtime	610.51ns (stddev: 43.113ns, 7.06%)
total runtime	5.006s
sampling	41 runs of 200 calls
Group: maybe of int

Benchmark: generating

mean ops/sec	140,452.67 (stddev: 9,588.43, 6.83%)
mean runtime	7.161ns (stddev: 608.854µs, 8.5%)
total runtime	5.013s
sampling	35 runs of 20,000 calls
Benchmark: shrinking

mean ops/sec	52,457.21 (stddev: 3,454.4, 6.59%)
mean runtime	19.17ns (stddev: 1.652ns, 8.62%)
total runtime	5.061s
sampling	44 runs of 6,000 calls
Group: result of string and int

Benchmark: generating

mean ops/sec	2,834.92 (stddev: 183.6, 6.48%)
mean runtime	354.531ns (stddev: 27.977ns, 7.89%)
total runtime	5.105s
sampling	48 runs of 300 calls
Benchmark: shrinking

mean ops/sec	1,367.44 (stddev: 74.14, 5.42%)
mean runtime	733.77ns (stddev: 46.287ns, 6.31%)
total runtime	5.063s
sampling	69 runs of 100 calls
Group: map

Benchmark: generating

mean ops/sec	179,094.91 (stddev: 12,517.97, 6.99%)
mean runtime	5.62ns (stddev: 526.297µs, 9.37%)
total runtime	5.058s
sampling	45 runs of 20,000 calls
Benchmark: shrinking

mean ops/sec	20,986.33 (stddev: 1,588.72, 7.57%)
mean runtime	48.009ns (stddev: 4.771ns, 9.94%)
total runtime	5.089s
sampling	53 runs of 2,000 calls
Group: andMap

Benchmark: generating

mean ops/sec	479.79 (stddev: 33.62, 7.01%)
mean runtime	2.097ms (stddev: 186.508ns, 8.89%)
total runtime	5.033s
sampling	48 runs of 50 calls
Benchmark: shrinking

mean ops/sec	158.79 (stddev: 10.71, 6.75%)
mean runtime	6.338ms (stddev: 601.881ns, 9.5%)
total runtime	5.07s
sampling	40 runs of 20 calls
Group: map5

Benchmark: generating

mean ops/sec	1,294.81 (stddev: 80.77, 6.24%)
mean runtime	776.249ns (stddev: 64.516ns, 8.31%)
total runtime	5.046s
sampling	65 runs of 100 calls
Benchmark: shrinking

mean ops/sec	380.67 (stddev: 28.82, 7.57%)
mean runtime	2.647ms (stddev: 269.699ns, 10.19%)
total runtime	5.082s
sampling	48 runs of 40 calls
Group: andThen

Benchmark: generating

mean ops/sec	17,287 (stddev: 1,264.71, 7.32%)
mean runtime	58.258ns (stddev: 5.662ns, 9.72%)
total runtime	5.01s
sampling	43 runs of 2,000 calls
Benchmark: shrinking

mean ops/sec	3,211.39 (stddev: 197.54, 6.15%)
mean runtime	312.789ns (stddev: 22.81ns, 7.29%)
total runtime	5.067s
sampling	54 runs of 300 calls
Group: conditional

Benchmark: generating

mean ops/sec	49,471.98 (stddev: 2,526.8, 5.11%)
mean runtime	20.272ns (stddev: 1.149ns, 5.67%)
total runtime	5.068s
sampling	50 runs of 5,000 calls
Benchmark: shrinking

mean ops/sec	13,531.67 (stddev: 743.86, 5.5%)
mean runtime	74.166ns (stddev: 4.854ns, 6.55%)
total runtime	5.043s
sampling	68 runs of 1,000 calls

New approach:

Benchmark Finished

Group: Fuzz

Group: int

Benchmark: generating

mean ops/sec	233,507.56 (stddev: 8,764.65, 3.75%)
mean runtime	4.289ns (stddev: 168.963µs, 3.94%)
total runtime	5.018s
sampling	39 runs of 30,000 calls
Benchmark: shrinking

mean ops/sec	30,732.62 (stddev: 1,699.79, 5.53%)
mean runtime	32.653ns (stddev: 2.079ns, 6.37%)
total runtime	5.094s
sampling	52 runs of 3,000 calls
Group: intRange

Benchmark: generating

mean ops/sec	200,433.95 (stddev: 13,109.51, 6.54%)
mean runtime	5.014ns (stddev: 381.902µs, 7.62%)
total runtime	5.014s
sampling	50 runs of 20,000 calls
Benchmark: shrinking

mean ops/sec	16,790.86 (stddev: 300.82, 1.79%)
mean runtime	59.576ns (stddev: 1.096ns, 1.84%)
total runtime	5.004s
sampling	42 runs of 2,000 calls
Group: string

Benchmark: generating

mean ops/sec	2,084.66 (stddev: 104.14, 5%)
mean runtime	480.942ns (stddev: 25.089ns, 5.22%)
total runtime	5.098s
sampling	53 runs of 200 calls
Benchmark: shrinking

mean ops/sec	773.9 (stddev: 48.03, 6.21%)
mean runtime	1.3ms (stddev: 122.153ns, 9.4%)
total runtime	5.095s
sampling	49 runs of 80 calls
Group: float

Benchmark: generating

mean ops/sec	314,228.87 (stddev: 11,558.4, 3.68%)
mean runtime	3.187ns (stddev: 121.175µs, 3.8%)
total runtime	5.067s
sampling	53 runs of 30,000 calls
Benchmark: shrinking

mean ops/sec	28,920.49 (stddev: 448.37, 1.55%)
mean runtime	34.586ns (stddev: 539.437µs, 1.56%)
total runtime	5.084s
sampling	49 runs of 3,000 calls
Group: bool

Benchmark: generating

mean ops/sec	446,659.38 (stddev: 11,825.85, 2.65%)
mean runtime	2.24ns (stddev: 59.953µs, 2.68%)
total runtime	5.041s
sampling	45 runs of 50,000 calls
Benchmark: shrinking

mean ops/sec	94,106.22 (stddev: 1,334.11, 1.42%)
mean runtime	10.628ns (stddev: 152.345µs, 1.43%)
total runtime	5.07s
sampling	53 runs of 9,000 calls
Group: char

Benchmark: generating

mean ops/sec	226,048.64 (stddev: 10,470.77, 4.63%)
mean runtime	4.434ns (stddev: 209.759µs, 4.73%)
total runtime	5.054s
sampling	57 runs of 20,000 calls
Benchmark: shrinking

mean ops/sec	23,253.21 (stddev: 1,153.07, 4.96%)
mean runtime	43.114ns (stddev: 2.199ns, 5.1%)
total runtime	5.001s
sampling	58 runs of 2,000 calls
Group: list of int

Benchmark: generating

mean ops/sec	3,600.8 (stddev: 259.92, 7.22%)
mean runtime	279.393ns (stddev: 23.395ns, 8.37%)
total runtime	5.029s
sampling	45 runs of 400 calls
Benchmark: shrinking

mean ops/sec	1,807.83 (stddev: 85.99, 4.76%)
mean runtime	554.396ns (stddev: 26.265ns, 4.74%)
total runtime	5.1s
sampling	46 runs of 200 calls
Group: maybe of int

Benchmark: generating

mean ops/sec	155,852.62 (stddev: 4,892.64, 3.14%)
mean runtime	6.423ns (stddev: 202.38µs, 3.15%)
total runtime	5.01s
sampling	39 runs of 20,000 calls
Benchmark: shrinking

mean ops/sec	57,672.29 (stddev: 2,410, 4.18%)
mean runtime	17.373ns (stddev: 799.769µs, 4.6%)
total runtime	5.108s
sampling	49 runs of 6,000 calls
Group: result of string and int

Benchmark: generating

mean ops/sec	3,019.7 (stddev: 121.82, 4.03%)
mean runtime	331.723ns (stddev: 14.05ns, 4.24%)
total runtime	5.075s
sampling	51 runs of 300 calls
Benchmark: shrinking

mean ops/sec	1,422.01 (stddev: 117.17, 8.24%)
mean runtime	709.351ns (stddev: 74.979ns, 10.57%)
total runtime	5.107s
sampling	36 runs of 200 calls
Group: map

Benchmark: generating

mean ops/sec	194,943.63 (stddev: 10,686.86, 5.48%)
mean runtime	5.147ns (stddev: 321.547µs, 6.25%)
total runtime	5.044s
sampling	49 runs of 20,000 calls
Benchmark: shrinking

mean ops/sec	27,104.66 (stddev: 624.4, 2.3%)
mean runtime	36.915ns (stddev: 903.989µs, 2.45%)
total runtime	5.094s
sampling	46 runs of 3,000 calls
Group: andMap

Benchmark: generating

mean ops/sec	502.26 (stddev: 27.11, 5.4%)
mean runtime	1.998ms (stddev: 121.555ns, 6.09%)
total runtime	5.094s
sampling	51 runs of 50 calls
Benchmark: shrinking

mean ops/sec	182.08 (stddev: 3.98, 2.19%)
mean runtime	5.495ms (stddev: 122.163ns, 2.22%)
total runtime	5.055s
sampling	46 runs of 20 calls
Group: map5

Benchmark: generating

mean ops/sec	1,365.85 (stddev: 67.26, 4.92%)
mean runtime	733.946ns (stddev: 36.658ns, 4.99%)
total runtime	5.064s
sampling	69 runs of 100 calls
Benchmark: shrinking

mean ops/sec	415.43 (stddev: 16.85, 4.06%)
mean runtime	2.412ms (stddev: 109.251ns, 4.53%)
total runtime	5.016s
sampling	52 runs of 40 calls
Group: andThen

Benchmark: generating

mean ops/sec	18,769.09 (stddev: 457.19, 2.44%)
mean runtime	53.312ns (stddev: 1.351ns, 2.53%)
total runtime	5.011s
sampling	47 runs of 2,000 calls
Benchmark: shrinking

mean ops/sec	4,070.96 (stddev: 176.11, 4.33%)
mean runtime	246.165ns (stddev: 12.133ns, 4.93%)
total runtime	5.022s
sampling	51 runs of 400 calls
Group: conditional

Benchmark: generating

mean ops/sec	52,591.72 (stddev: 1,853.36, 3.52%)
mean runtime	19.04ns (stddev: 734.121µs, 3.86%)
total runtime	5.046s
sampling	53 runs of 5,000 calls
Benchmark: shrinking

mean ops/sec	16,536.46 (stddev: 367.71, 2.22%)
mean runtime	60.504ns (stddev: 1.41ns, 2.33%)
total runtime	5.082s
sampling	42 runs of 2,000 calls

tgecho · 2017-07-01T23:37:28Z

I don't know if you're interested in various browsers, but I ran the benchmark in everything installed on my system at the moment: https://gist.github.com/tgecho/ddeed4e4e4cee32ba2cf6476bcf67671

Can do some IE/Edge later if it's of any use. I imagine V8 (because of node) is the focus right now?

eeue56 · 2017-07-02T14:38:19Z

@tgecho Great, thank you! I think just confirming that it is more performant on other people's machines is the most important thing. From your results, it looks consistently faster across the board! good stuff

drathier · 2017-07-18T12:12:11Z

Ran the benchmarks on my old laptop: https://gist.github.com/drathier/bc282ab8e40726c7e15901073ffd2d1d

eeue56 · 2017-07-18T12:34:00Z

Nice! The performance gains make my worries go away. I'll clean this up for review, move those packages to elm-community, then we can merge and be ready for 0.19 😄

mgold · 2017-08-01T03:13:54Z

What's the status of this? Will is result in a MAJOR change?

eeue56 · 2017-08-01T07:42:27Z

@mgold Haven't had time to focus on this yet. Do we want a release with this before the 0.19 alpha? If so I'll prioritize this

rtfeldman · 2017-08-01T13:52:45Z

As I understand it, this should be a PATCH change.

@eeue56 I'd love to release this pre-0.19, both for the perf improvement and so we can confirm there are no issues with it ahead of time. (That way if we hit issues post-0.19, we can rule out this change as a possible culprit since it'll already have been working in 0.18.)

eeue56 · 2017-08-01T14:00:20Z

Okidoki, I'll try to look into it this week

eeue56 · 2017-08-05T22:59:54Z

This is now ready to merge

rtfeldman · 2017-08-06T01:15:54Z

Wonderful, thank you @eeue56! 😻

mgold mentioned this pull request Jul 9, 2017

Planning for 0.19 and 5.0.0 #191

Closed

11 tasks

eeue56 self-assigned this Jul 18, 2017

eeue56 force-pushed the pure-elm-laziness branch from 79d995c to 37f13b4 Compare August 5, 2017 21:49

use the pure elm version of packages

b01f483

eeue56 force-pushed the pure-elm-laziness branch 2 times, most recently from 346ae7b to b01f483 Compare August 5, 2017 22:54

eeue56 changed the title ~~WIP: Use pure elm for laziness~~ Use pure elm for laziness Aug 5, 2017

rtfeldman merged commit d69c73d into elm-community:master Aug 6, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use pure elm for laziness#184

Use pure elm for laziness#184
rtfeldman merged 1 commit intoelm-community:masterfrom
eeue56:pure-elm-laziness

eeue56 commented Jul 1, 2017 •

edited

Loading

Uh oh!

eeue56 commented Jul 1, 2017

Uh oh!

tgecho commented Jul 1, 2017

Uh oh!

eeue56 commented Jul 2, 2017

Uh oh!

drathier commented Jul 18, 2017

Uh oh!

eeue56 commented Jul 18, 2017

Uh oh!

mgold commented Aug 1, 2017

Uh oh!

eeue56 commented Aug 1, 2017

Uh oh!

rtfeldman commented Aug 1, 2017

Uh oh!

eeue56 commented Aug 1, 2017

Uh oh!

eeue56 commented Aug 5, 2017

Uh oh!

rtfeldman commented Aug 6, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

eeue56 commented Jul 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

What needs to happen before merging?

Future changes

Uh oh!

eeue56 commented Jul 1, 2017

Uh oh!

tgecho commented Jul 1, 2017

Uh oh!

eeue56 commented Jul 2, 2017

Uh oh!

drathier commented Jul 18, 2017

Uh oh!

eeue56 commented Jul 18, 2017

Uh oh!

mgold commented Aug 1, 2017

Uh oh!

eeue56 commented Aug 1, 2017

Uh oh!

rtfeldman commented Aug 1, 2017

Uh oh!

eeue56 commented Aug 1, 2017

Uh oh!

eeue56 commented Aug 5, 2017

Uh oh!

rtfeldman commented Aug 6, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

eeue56 commented Jul 1, 2017 •

edited

Loading