Program received signal SIGSEGV, Segmentation fault while using producer tokens

[rohitjoshi]

When I use moodycamel::ConcurrentQueuestd::string with 4 consumer and 4 producer threads with initial size 1024*1024.

NOTE:

1. This worked fine while using data type as int.
2. This work fine while using non producer tokens
3. I have undefined MOODYCAMEL_CPP11_THREAD_LOCAL_SUPPORTED
4. Using GCC 4.9
5. Darwin 7831c1c1501a 14.0.0 Darwin Kernel Version 14.0.0: Fri Sep 19 00:26:44 PDT 2014; root:xnu-2782.1.97~2/RELEASE_X86_64 x86_64

Program received signal SIGSEGV, Segmentation fault.
0x000000010000363e in moodycamel::ConcurrentQueue<std::string,     moodycamel::ConcurrentQueueDefaultTraits>::ExplicitProducer::~ExplicitProducer() ()
(gdb) bt
#0  0x000000010000363e in moodycamel::ConcurrentQueue<std::string,     moodycamel::ConcurrentQueueDefaultTraits>::ExplicitProducer::~ExplicitProducer() ()
#1  0x00000001000028b1 in test_1000() ()
#2  0x0000000100008599 in main ()

[cameron314]

Thanks for opening an issue! I'll look into this as soon as I can.

[cameron314]

I added a test case to try to reproduce this, but so far no dice. Would you be able share a short self-contained example program that consistently reproduces the bug for you?

[rohitjoshi]

Let me try that again !!

[cameron314]

Thank you! I really appreciate any help I can get reproducing these issues. Lock-free bugs tend to be elusive.

[rohitjoshi]

I could reproduce consistently but this time seeing different stack trace.

You can see my test project on github repo: https://github.com/rohitjoshi/queue-benchmark
https://github.com/rohitjoshi/queue-benchmark/blob/master/queue_test.h#L77

Let me know if you have any questions. By the way, excellent job on creating this concurrent queue. Performance is amazing.

Program received signal SIGABRT, Aborted.
0x00007fff8a271282 in __pthread_kill () from /usr/lib/system/libsystem_kernel.dylib
(gdb) bt
#0 0x00007fff8a271282 in __pthread_kill () from /usr/lib/system/libsystem_kernel.dylib
#1 0x00007fff88a014c3 in pthread_kill () from /usr/lib/system/libsystem_pthread.dylib
#2 0x00007fff8ffa1b73 in abort () from /usr/lib/system/libsystem_c.dylib
#3 0x00007fff8c1b5937 in free () from /usr/lib/system/libsystem_malloc.dylib
#4 0x0000000100006334 in moodycamel::ConcurrentQueue<std::string, moodycamel::ConcurrentQueueDefaultTraits>::ExplicitProducer::~ExplicitProducer() clone .lto_priv.159
#5 0x0000000100008935 in test_1KQueueSize() ()
#6 0x0000000100008cd9 in main ()
(gdb) f 4
#4 0x0000000100006334 in moodycamel::ConcurrentQueue<std::string, moodycamel::ConcurrentQueueDefaultTraits>::ExplicitProducer::~ExplicitProducer() clone .lto_priv.159

[cameron314]

Awesome, thank you! (And nice project by the way, there's never too many benchmarks.) I'll see if I can reproduce this on my end using a clone of your repo.

[cameron314]

Sadly, I still can't reproduce the crash using your code on either Windows or Linux. Looks to be only (easily) reproducible on Darwin. I'll try running them again on Linux with AddressSanitizer though to see if it picks anything up, just in case.

In the meantime, if could you run your code again with the updated ConcurrentQueue I just pushed, and add #define MOODYCAMEL_QUEUE_INTERNAL_DEBUG before including the header, and email me a crash dump (cameron (at) moodycamel.com), I would be most grateful.

Aside from that, I noticed that your 'KB' version reserves one million elements, and the 'MB' version reserves a billion, so it would seem that the naming is off by a factor of 1024. Note also that a billion std::strings is an 8GB allocation (with my mingw g++), which completely dominates the benchmark due to all the swapping unless you have a lot of free memory available.

[cameron314]

... and AddressSanitizer doesn't find any issues either, sigh.

ThreadSanitizer, on the other hand, finds plenty of races, but I still haven't determined if they are false positives or not. I'll definitely keep looking into this...

[cameron314]

Apparently tsan never reports false positives. So. This is good. It gives me a place to start. I'll keep this issue posted with any developments.

An example output with ThreadSanitizer on:

WARNING: ThreadSanitizer: data race (pid=18658)
  Write of size 4 at 0x7f67ab7fa030 by thread T3:
    #0 enqueue /home/cameron/projects/queue-benchmark/concurrentqueue.h:1682 (exe+0x000000009ab7)
    #1 <null> <null>:0 (libstdc++.so.6+0x003873ab177f)

  Previous read of size 4 at 0x7f67ab7fa030 by thread T4:
    #0 dequeue /home/cameron/projects/queue-benchmark/concurrentqueue.h:1765 (exe+0x00000000f5a4)
    #1 consumer /home/cameron/projects/queue-benchmark/queue_test.h:169 (exe+0x00000000b637)
    #2 <null> <null>:0 (libstdc++.so.6+0x003873ab177f)

  Location is heap block of size 6553600 at 0x7f67ab7fa000 allocated by main thread:
    #0 <null> <null>:0 (libtsan.so.0+0x00000001cb62)
    #1 malloc /home/cameron/projects/queue-benchmark/concurrentqueue.h:218 (exe+0x000000006933)
    #2 main /home/cameron/projects/queue-benchmark/main.cpp:17 (exe+0x000000003662)

  Thread T3 (tid=18667, running) created by main thread at:
    #0 <null> <null>:0 (libtsan.so.0+0x00000001ed0b)
    #1 <null> <null>:0 (libstdc++.so.6+0x003873ab19ce)
    #2 main /home/cameron/projects/queue-benchmark/main.cpp:17 (exe+0x000000003662)

  Thread T4 (tid=18668, running) created by main thread at:
    #0 <null> <null>:0 (libtsan.so.0+0x00000001ed0b)
    #1 <null> <null>:0 (libstdc++.so.6+0x003873ab19ce)
    #2 main /home/cameron/projects/queue-benchmark/main.cpp:17 (exe+0x000000003662)

SUMMARY: ThreadSanitizer: data race /home/cameron/projects/queue-benchmark/concurrentqueue.h:1682 enqueue

[rohitjoshi]

I am glad you are able to reproduce. If you want, I can try to reproduce on Linux. I do need to upgrade my g++ version to support C++11.

Thanks for pointing out the KB vs MB version. Now I understand why my MB version was under performing than KV version. with 1B element each 1000 bytes :)

If you look at my benchmark results, https://github.com/rohitjoshi/queue-benchmark/blob/master/README.md

Why concurrentqueue performs better with 3 or 4 consumers and 3 or 4 producers?
e.g 1 consumer, 1 producer takes 1.49 sec versus 4 consumers, 4 producers takes only 0.81 second.

Any insitghts will be helpful for me as I am building a high performance MQ server in which I have various uses cases.

1. 1 producer/ 1 consumer
2. 1 producer / multiple consumers

with varying size of string payload from 100 bytes to 5K bytes and I would like to optimize based on the max payload size if possible.

moodycamel::ConcurrentQueue: Producer threads:1, Consumer threads:1, producer_counter: 10000000, consumer_counter 10000000. Time Taken:1.49435 seconds.
moodycamel::ConcurrentQueue: Producer threads:2, Consumer threads:2, producer_counter: 10000000, consumer_counter 10000000. Time Taken:1.31297 seconds.
moodycamel::ConcurrentQueue: Producer threads:3, Consumer threads:3, producer_counter: 9999999, consumer_counter 9999999. Time Taken:0.868555 seconds.
moodycamel::ConcurrentQueue: Producer threads:4, Consumer threads:4, producer_counter: 10000000, consumer_counter 10000000. Time Taken:0.810693 seconds.

[cameron314]

On closer inspection, the thread sanitizer race is extremely general (a reused slot is written to after being read from a previous dequeue). I still have to look into it more, but it seems unlikely to be related to the crash you're having.

So, I'm still not able to reproduce the crash -- if you could send me a dump from your environment that'd be great (I could at least inspect the variables at crash time...).

As for the timing you're observing, if I'm reading the code right you seem to be doing the same amount of overall work in each case (iterations / producer_threads), just spreading it over more threads. In this case, depending on your hardware (i.e. core count), the amount of work will complete faster because you're throwing more hardware at it (and it's a trivially parallelizable workload). The scaling isn't completely linear due to the overhead of sharing data (the minimum synchronization that the queue must do to schlep objects around safely), but it's still much faster than doing everything using a single core.

The length of the strings you're moving around should not affect the timing at all, since the strings are moved (not copied) into and out of the queue if possible, and the std::string objects are fixed-size with a pointer to the actual (heap-allocated) string data.

[cameron314]

Seems I spoke too soon -- apparently tsan can report false positives when using low-level lock-free synchronization, which is of course what my entire queue is built on. I'll need to run more (simpler) tests to see if that's what's happening here, and install a recent clang to get the latest tsan version too.

[cameron314]

I've left ThreadSanitizer aside for now, and finally bit the bullet and got the queue as a whole to compile using Relacy (previously components were only simulated in Relacy in isolation). I wrote a few truly integrated tests, and lo-and-behold, Relacy found a bug.

So, this is good news! I have the entire execution history to be able to see exactly how the bug arises, but it will still take some time to analyze (there's over 300 steps involved, and most of them are noise). There's even a chance this is a general bug that's causing the other segfaults in the other issues.