Updated benchmarking

[NathanielPrescott]

Hello,

I was just wondering if you had done any benchmarking with mokabench since what is on the readme is over a year old or if you were planning on updating the readme with new benches data?

I noticed that you've updated a few versions since then and mokabench now has tinyufo from Cloudflare and another cache called Stretto. I haven't gotten to try running the mokabench yet but the newer versions of QC are pretty competitive when running them in the TinyUFO benches.

[arthurprs]

I also got curious and re-ran benchmarks. I'll update the readme accordingly.

Since you mentioned TinyUFO benchmarks. Last time I checked, its performance benchmarks were very artificial, as it tested tiny caches with very high hit rates, so I'd take those with a much larger pinch of salt than Mokabench.

Since you also mentioned Stretto. It performs poorly mokabench results (arguably broken). That's why I didn't include it in the README in the past. I suggest looking closely at its claims if you plan to use it.

Let me know if I can clarify anything.

Config { trace_file: S3, ttl: None, tti: None, num_clients: Some([1, 3, 6]), repeat: None, insertion_delay: None, insert_once: false, invalidate: false, invalidate_all: false, invalidate_entries_if: false, iterate: false, eviction_listener: None, size_aware: false, entry_api: false, per_key_expiration: false }

Cache, Max Capacity, Clients, Inserts, Reads, Hit Ratio, Duration Secs
QuickCache Sync Cache, 100000, 1, 14300769, 16407702, 12.841, 2.196
QuickCache Sync Cache, 100000, 3, 14301124, 16407702, 12.839, 1.279
QuickCache Sync Cache, 100000, 6, 14300809, 16407702, 12.841, 0.798
HashLink (LRU w/ Mutex), 100000, 1, 16025830, 16407702, 2.327, 2.422
Stretto, 100000, 1, 16387842, 16407702, 0.121, 8.509
Stretto, 100000, 3, 16388184, 16407702, 0.119, 3.704
Stretto, 100000, 6, 16387857, 16407702, 0.121, 6.631
TinyUFO, 100000, 1, 15685351, 16407702, 4.403, 11.641
TinyUFO, 100000, 3, 15900217, 16407702, 3.093, 8.828
TinyUFO, 100000, 6, 15918936, 16407702, 2.979, 8.104
Mini Moka Sync Cache, 100000, 1, 14695340, 16407702, 10.436, 9.399
Mini Moka Sync Cache, 100000, 3, 14679119, 16407702, 10.535, 8.490
Mini Moka Sync Cache, 100000, 6, 14706822, 16407702, 10.366, 8.064
Moka Sync Cache, 100000, 1, 14694850, 16407702, 10.439, 17.728
Moka Sync Cache, 100000, 3, 14675485, 16407702, 10.557, 15.465
Moka Sync Cache, 100000, 6, 14696116, 16407702, 10.432, 16.445
Moka SegmentedCache(8), 100000, 1, 14756139, 16407702, 10.066, 19.363
Moka SegmentedCache(8), 100000, 3, 14755093, 16407702, 10.072, 12.532
Moka SegmentedCache(8), 100000, 6, 14755440, 16407702, 10.070, 10.119
QuickCache Sync Cache, 400000, 1, 9435745, 16407702, 42.492, 2.537
QuickCache Sync Cache, 400000, 3, 9437141, 16407702, 42.483, 1.323
QuickCache Sync Cache, 400000, 6, 9436549, 16407702, 42.487, 0.899
HashLink (LRU w/ Mutex), 400000, 1, 14432404, 16407702, 12.039, 2.766
Stretto, 400000, 1, 16317450, 16407702, 0.550, 8.110
Stretto, 400000, 3, 16320278, 16407702, 0.533, 4.809
Stretto, 400000, 6, 16322816, 16407702, 0.517, 8.636
TinyUFO, 400000, 1, 11455971, 16407702, 30.179, 14.234
TinyUFO, 400000, 3, 12405638, 16407702, 24.391, 8.695
TinyUFO, 400000, 6, 12551335, 16407702, 23.503, 7.008
Mini Moka Sync Cache, 400000, 1, 9427172, 16407702, 42.544, 8.511
Mini Moka Sync Cache, 400000, 3, 9584914, 16407702, 41.583, 6.624
Mini Moka Sync Cache, 400000, 6, 9656084, 16407702, 41.149, 6.613
Moka Sync Cache, 400000, 1, 9439274, 16407702, 42.470, 19.363
Moka Sync Cache, 400000, 3, 9549851, 16407702, 41.797, 13.332
Moka Sync Cache, 400000, 6, 9670981, 16407702, 41.058, 13.620
Moka SegmentedCache(8), 400000, 1, 9459064, 16407702, 42.350, 21.476
Moka SegmentedCache(8), 400000, 3, 9457995, 16407702, 42.356, 12.234
Moka SegmentedCache(8), 400000, 6, 9455385, 16407702, 42.372, 9.700
QuickCache Sync Cache, 800000, 1, 5184786, 16407702, 68.400, 3.207
QuickCache Sync Cache, 800000, 3, 5185210, 16407702, 68.398, 1.353
QuickCache Sync Cache, 800000, 6, 5185337, 16407702, 68.397, 0.743
HashLink (LRU w/ Mutex), 800000, 1, 7120978, 16407702, 56.600, 2.613
Stretto, 800000, 1, 16221666, 16407702, 1.134, 6.911
Stretto, 800000, 3, 16232009, 16407702, 1.071, 4.396
Stretto, 800000, 6, 16231883, 16407702, 1.072, 8.568
TinyUFO, 800000, 1, 5598215, 16407702, 65.881, 10.043
TinyUFO, 800000, 3, 6007956, 16407702, 63.383, 5.077
TinyUFO, 800000, 6, 6071806, 16407702, 62.994, 3.789
Mini Moka Sync Cache, 800000, 1, 4872358, 16407702, 70.304, 8.574
Mini Moka Sync Cache, 800000, 3, 5012764, 16407702, 69.449, 5.363
Mini Moka Sync Cache, 800000, 6, 5529311, 16407702, 66.301, 4.551
Moka Sync Cache, 800000, 1, 4868066, 16407702, 70.331, 13.347
Moka Sync Cache, 800000, 3, 5035672, 16407702, 69.309, 7.504
Moka Sync Cache, 800000, 6, 5548024, 16407702, 66.186, 7.088
Moka SegmentedCache(8), 800000, 1, 4870652, 16407702, 70.315, 15.225
Moka SegmentedCache(8), 800000, 3, 4867866, 16407702, 70.332, 7.520
Moka SegmentedCache(8), 800000, 6, 4871951, 16407702, 70.307, 5.599

Config { trace_file: Ds1, ttl: None, tti: None, num_clients: Some([1, 3, 6]), repeat: None, insertion_delay: None, insert_once: false, invalidate: false, invalidate_all: false, invalidate_entries_if: false, iterate: false, eviction_listener: None, size_aware: false, entry_api: false, per_key_expiration: false }

Cache, Max Capacity, Clients, Inserts, Reads, Hit Ratio, Duration Secs
QuickCache Sync Cache, 1000000, 1, 37208683, 43704979, 14.864, 9.883
QuickCache Sync Cache, 1000000, 3, 37196302, 43704979, 14.892, 4.546
QuickCache Sync Cache, 1000000, 6, 37196402, 43704979, 14.892, 3.585
HashLink (LRU w/ Mutex), 1000000, 1, 42356290, 43704979, 3.086, 8.901
Stretto, 1000000, 1, 43527783, 43704979, 0.405, 17.927
Stretto, 1000000, 3, 43526414, 43704979, 0.409, 11.678
Stretto, 1000000, 6, 43516034, 43704979, 0.432, 22.269
TinyUFO, 1000000, 1, 42093670, 43704979, 3.687, 60.135
TinyUFO, 1000000, 3, 42203137, 43704979, 3.436, 35.306
TinyUFO, 1000000, 6, 42214889, 43704979, 3.409, 25.440
Mini Moka Sync Cache, 1000000, 1, 37188446, 43704979, 14.910, 25.833
Mini Moka Sync Cache, 1000000, 3, 37332123, 43704979, 14.582, 23.078
Mini Moka Sync Cache, 1000000, 6, 38104018, 43704979, 12.815, 23.571
Moka Sync Cache, 1000000, 1, 37189718, 43704979, 14.907, 74.586
Moka Sync Cache, 1000000, 3, 37643682, 43704979, 13.869, 59.914
Moka Sync Cache, 1000000, 6, 38031651, 43704979, 12.981, 60.272
Moka SegmentedCache(8), 1000000, 1, 37278565, 43704979, 14.704, 85.363
Moka SegmentedCache(8), 1000000, 3, 37265347, 43704979, 14.734, 55.292
Moka SegmentedCache(8), 1000000, 6, 37257361, 43704979, 14.753, 43.601
QuickCache Sync Cache, 4000000, 1, 24156843, 43704979, 44.727, 9.497
QuickCache Sync Cache, 4000000, 3, 24159591, 43704979, 44.721, 4.257
QuickCache Sync Cache, 4000000, 6, 24204044, 43704979, 44.619, 2.777
HashLink (LRU w/ Mutex), 4000000, 1, 34856997, 43704979, 20.245, 10.735
Stretto, 4000000, 1, 43305964, 43704979, 0.913, 16.141
Stretto, 4000000, 3, 43273230, 43704979, 0.988, 10.575
Stretto, 4000000, 6, 43297630, 43704979, 0.932, 22.154
TinyUFO, 4000000, 1, 32607709, 43704979, 25.391, 49.380
TinyUFO, 4000000, 3, 32810383, 43704979, 24.928, 26.440
TinyUFO, 4000000, 6, 32840752, 43704979, 24.858, 20.390
Mini Moka Sync Cache, 4000000, 1, 23863892, 43704979, 45.398, 26.103
Mini Moka Sync Cache, 4000000, 3, 23865870, 43704979, 45.393, 19.896
Mini Moka Sync Cache, 4000000, 6, 25152629, 43704979, 42.449, 17.912
Moka Sync Cache, 4000000, 1, 23863107, 43704979, 45.400, 49.337
Moka Sync Cache, 4000000, 3, 25262589, 43704979, 42.197, 38.292
Moka Sync Cache, 4000000, 6, 27552410, 43704979, 36.958, 41.271
Moka SegmentedCache(8), 4000000, 1, 23901973, 43704979, 45.311, 52.887
Moka SegmentedCache(8), 4000000, 3, 23911198, 43704979, 45.290, 31.638
Moka SegmentedCache(8), 4000000, 6, 23913899, 43704979, 45.283, 24.747
QuickCache Sync Cache, 8000000, 1, 13405437, 43704979, 69.327, 7.703
QuickCache Sync Cache, 8000000, 3, 13406862, 43704979, 69.324, 3.598
QuickCache Sync Cache, 8000000, 6, 13406780, 43704979, 69.324, 2.243
HashLink (LRU w/ Mutex), 8000000, 1, 24896662, 43704979, 43.035, 8.995
Stretto, 8000000, 1, 43184599, 43704979, 1.191, 15.602
Stretto, 8000000, 3, 42922835, 43704979, 1.790, 10.999
Stretto, 8000000, 6, 43002418, 43704979, 1.608, 21.119
TinyUFO, 8000000, 1, 20143204, 43704979, 53.911, 28.245
TinyUFO, 8000000, 3, 20339486, 43704979, 53.462, 14.967
TinyUFO, 8000000, 6, 20364270, 43704979, 53.405, 11.206
Mini Moka Sync Cache, 8000000, 1, 14227354, 43704979, 67.447, 29.082
Mini Moka Sync Cache, 8000000, 3, 13762562, 43704979, 68.510, 17.060
Mini Moka Sync Cache, 8000000, 6, 14926093, 43704979, 65.848, 12.294
Moka Sync Cache, 8000000, 1, 14184272, 43704979, 67.545, 31.866
Moka Sync Cache, 8000000, 3, 14572250, 43704979, 66.658, 18.766
Moka Sync Cache, 8000000, 6, 15705649, 43704979, 64.064, 16.067
Moka SegmentedCache(8), 8000000, 1, 14637051, 43704979, 66.509, 37.606
Moka SegmentedCache(8), 8000000, 3, 14699185, 43704979, 66.367, 19.491
Moka SegmentedCache(8), 8000000, 6, 14543284, 43704979, 66.724, 14.946

Config { trace_file: Spc1LikeRead, ttl: None, tti: None, num_clients: Some([1]), repeat: None, insertion_delay: None, insert_once: false, invalidate: false, invalidate_all: false, invalidate_entries_if: false, iterate: false, eviction_listener: None, size_aware: false, entry_api: false, per_key_expiration: false }

Cache, Max Capacity, Clients, Inserts, Reads, Hit Ratio, Duration Secs
QuickCache Sync Cache, 500000, 1, 36994843, 41351279, 10.535, 7.880
HashLink (LRU w/ Mutex), 500000, 1, 39942249, 41351279, 3.407, 7.969
Stretto, 500000, 1, 41292584, 41351279, 0.142, 16.147
TinyUFO, 500000, 1, 39197208, 41351279, 5.209, 49.395
Mini Moka Sync Cache, 500000, 1, 36231669, 41351279, 12.381, 25.467
Moka Sync Cache, 500000, 1, 36267064, 41351279, 12.295, 72.766
Moka SegmentedCache(8), 500000, 1, 36339280, 41351279, 12.121, 85.703
QuickCache Sync Cache, 2000000, 1, 24123918, 41351279, 41.661, 14.611
HashLink (LRU w/ Mutex), 2000000, 1, 30181071, 41351279, 27.013, 9.041
Stretto, 2000000, 1, 41106147, 41351279, 0.593, 13.126
TinyUFO, 2000000, 1, 27173449, 41351279, 34.286, 46.476
Mini Moka Sync Cache, 2000000, 1, 24032624, 41351279, 41.882, 31.064
Moka Sync Cache, 2000000, 1, 24026202, 41351279, 41.897, 61.780
Moka SegmentedCache(8), 2000000, 1, 24015014, 41351279, 41.924, 69.109
QuickCache Sync Cache, 30000000, 1, 6050363, 41351279, 85.368, 8.170
HashLink (LRU w/ Mutex), 30000000, 1, 6050363, 41351279, 85.368, 7.638
Stretto, 30000000, 1, 37717593, 41351279, 8.787, 11.574
TinyUFO, 30000000, 1, 6050363, 41351279, 85.368, 14.065
Mini Moka Sync Cache, 30000000, 1, 6050363, 41351279, 85.368, 34.452
Moka Sync Cache, 30000000, 1, 6050363, 41351279, 85.368, 22.210
Moka SegmentedCache(8), 30000000, 1, 6050363, 41351279, 85.368, 28.96
``