Queue allocation space

[GREATwo]

I allocated a size to the queue, but after initializing the queue, there is no space allocated, but the space is allocated only when it is actually used. Is there a way to allocate space immediately

[cameron314]

Example please? Things resize as needed but are generally allocated up front.

[GREATwo]

Example please? Things resize as needed but are generally allocated up front.

tcp_queue[i] = new moodycamel::BlockingConcurrentQueue<struct ether_packet, MyTraits>(3000*32);

I am in such a scenario. A server like udp receives a data service, writes the queue after receiving the data, another thread reads from the queue and sends it to the tcp server. For the first time, the udp server will lose packets After the second time, there will be no packet loss. After initializing, the memory will not change, and the memory will increase with the sending data, so I want to allocate the memory in advance. Do you have a chat account, can you add it

[cameron314]

I still don't understand. Are packets being lost because of latency? How are you using the queue? What memory doesn't change after initialization?

[GREATwo]

I still don't understand. Are packets being lost because of latency? How are you using the queue? What memory doesn't change after initialization?

Sorry for the bad English, the data packet is lost because the first time to enter the queue needs to allocate memory. I initialized the queue. For example, if I set the queue capacity to 1W and a structure with a size of 8k, should the space after creating the queue be 8 * 1024 * 10000, but I found that the process does not take up so much memory

[cameron314]

Ok. The queue does allocate the memory for the elements up front. (There are some other internal data structures that by default start small and may need to grow eventually.) But the total memory taken by newing up a queue of 32*3000 pointers is at least 32*3000*sizeof(void). And the first element enqueued should not need to allocate anything.

How are you measuring memory usage?

Also, I'm still not sure I understand why the first packet is lost. If the queue has to allocate more memory it still won't drop any elements.

[GREATwo]

I use the ps aux command to view the memory of the current process. As the number of data sent increases, the number of enqueues will increase, and the process memory will increase. I send 10G data at a time, and the packet will be lost for the first time. Packet loss, if the queue is not used, the data will be counted when received, and no data will be lost every time. In addition, the queue memory will not be released. I enter the queue faster than the queue, and the memory will continue to grow.

[cameron314]

ps aux doesn't tell the whole picture.

Once the queue allocates memory, it doesn't free it until the queue is destroyed. It recycles its memory internally, however. How many threads are you enqueueing from?

It sounds like you're handling a lot of packets. I suspect the packet loss may be due to reduced throughput at the beginning causing OS buffers to overflow. The queue itself cannot lose elements.

[GREATwo]

Yes, I ’m sure, The queue itself cannot lose elements. I ’m dealing with large amounts of data, 10 Gigabit network card, I have a total of five queues to receive, five threads take data from the queue, the speed of reception is faster than from the queue The speed of fetching data is slightly faster, and then the space of the queue will keep increasing. Is there any good way for you?

[cameron314]

That is a lot of data! The size of the queue should definitely not keep increasing. Perhaps it is filling up faster than it can be emptied?

Maybe a simple ring buffer would be more efficient in this case instead of my queue?

[GREATwo]

The filling speed is slightly faster, is there a better ringbuffer recommended to me, thank you

[cameron314]

Ok, thanks for confirming. My feeling is that at this throughput, the best option will be a custom data structure specific to your needs.

[cameron314]

Although I encourage you to try the bulk methods of the queue with explicit consumer tokens. See the readme.

[GREATwo]

ok ，thank you very much

[GREATwo]

If I ’m a single producer and a single consumer, it ’s better to use this or your single producer consumer queue.

[cameron314]

Definitely use the ReaderWriterQueue if you only have one producer/consumer. Although there are no bulk methods on it.