queue.bess can't run with g++6 and -Ofast/-O3 are enabled.

[justinemarie]

When you run queue.bess with g++6 and -Ofast it crashes.

Here's what you see in GDB:
Thread 20 "bessd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffedb27700 (LWP 41252)]
0x000000000062990e in __llring_mp_do_enqueue (behavior=LLRING_QUEUE_VARIABLE, n=32, obj_table=, r=0x7fffe001a5f0) at modules/../kmod/llring.h:433
433 modules/../kmod/llring.h: No such file or directory.

And the backtrace:
#0 0x000000000062990e in __llring_mp_do_enqueue (behavior=LLRING_QUEUE_VARIABLE, n=32, obj_table=, r=0x7fffe001a5f0) at modules/../kmod/llring.h:433
#1 llring_mp_enqueue_burst (n=32, obj_table=, r=0x7fffe001a5f0) at modules/../kmod/llring.h:1009
#2 Queue::ProcessBatch (this=, batch=0x7fffed916770) at modules/queue.cc:109
#3 0x000000000063a6c6 in Module::RunChooseModule (batch=0x7fffed916770, ogate_idx=0, this=) at modules/../module.h:321
#4 Module::RunNextModule (batch=0x7fffed916770, this=) at modules/../module.h:325
#5 Source::RunTask (this=) at modules/source.cc:83
#6 0x00000000005fa085 in Task::Scheduled (this=) at task.cc:36
#7 0x00000000005f8c0e in bess::LeafTrafficClass::RunTasks (this=0x7fffe0026620) at traffic_class.h:440
#8 bess::Scheduler::ScheduleOnce (this=) at scheduler.h:69
#9 bess::Scheduler::ScheduleLoop (this=0x7fffe4210c10) at scheduler.cc:30
#10 0x00000000005fd9e6 in Worker::Run (this=0x7fffed917340, _arg=_arg@entry=0x7ffff64a84e0) at worker.cc:241
#11 0x00000000005fe063 in run_worker (_arg=0x7ffff64a84e0) at worker.cc:254
#12 0x0000000000b00faf in execute_native_thread_routine ()
#13 0x00007ffff749d6ba in start_thread (arg=0x7fffedb27700) at pthread_create.c:333
#14 0x00007ffff71d382d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Unfortunately, many of the variables wind up optimized out so it's not easy to figure out how the code winds up at this point...

[justinemarie]

Also, at first glance, it looks like the problem might be in obj_table, but even if you double-check that obj_table != 0 (from ProcessBatch in Queue, before you pass it over to llring.h) it still freaks out.

[justinemarie]

I'm so grouchy about this Ruben is playing Let it Go for me (from Frozen). '

[apanda]

Hi
I know this was not assigned to me but I was intrigued, sorry. I know what the problem is, and a couple of solution not sure which one you want.

First the problem:

• struct llring is marked as __llring_cache_aligned, i.e., it should be aligned on 64 byte boundaries.
• gcc-6 uses this to infer that after a certain point it can use aligned AVX instructions to implement parts of enqueue_ptrs. In fact it is smart enough to recognize that reading from obj_table requires it to use unaligned access, but writes to ring can be done aligned. This can be seen in the following snippet:

0x000000000062af24 <+964>:   vmovdqu (%r14,%r12,1),%ymm0
0x000000000062af2a <+970>:   add    $0x1,%r15d
=> 0x000000000062af2e <+974>:   vmovdqa %ymm0,0x100(%rdx,%r12,1)

• Unfortunately, queue.cc uses mem_alloc to allocate the llring, which in turn uses malloc. The base address in this case is not appropriately aligned, hence vmovdqa is called on an unaligned destination and segfaults.

One obvious solution would be to add aligned allocation functions in mem_alloc, e.g., using memalign when MEM_ALLOC_PROVIDER is LIBC and passing in the appropriate alignment to rte_zmalloc when using DPDK (this is in mem_alloc.cc) or manually padding the size as appropriate. Not sure which one anyone prefers.

[sangjinhan]

rte_zmalloc() guarantees that allocated pointers are always cacheline-aligned, while malloc() does not. This difference has caused many SIMD-related issues, after the provider was switched to LIBC (0eea1d6).

Added mem_alloc_ex(), which was planned in the header file but never actually implemented. It does not support NUMA-specific memory allocation yet.