Efficiently pack network packets

[benaadams]

On Linux we can use the socket option TCP_CORK to get the kernel to only send TCP/IP packets at the full MTU.

• When there is no more input data and no more output processing we remove the cork to send the remaining partial packet.
• The cork also needs to be removed for real-time sends, so Upgrades (WebSockets) and Server Sent Events.

This should improve transmission efficiency where the application writes do not exactly match the ideal network size.

e.g. an example

Given a MTU of 1400, IP+TCP header of 20+20 = 40, ideal write sizes would be in multiples of 1360

If application writes were occurring in 4096 byte chunks; this would be 3 complete packets + 1 packet of 16 bytes (or 1.2% full).

This change instructs the kernel to hold back that final packet and wait for further writes to send when full; making the network transmission more efficient.

This should improve the throughput of any transmission over the network MTU (e.g. 1400 bytes on IPv4); or any sequence of writes where individually they are under the MTU; which should improve MVC, StaticFiles etc and let the kernel worry about the ideal packet size rather than leaking it into application code.

Similar functionality would be needed to use Windows RIO at maximal efficiency

Also don't have an Linux environment to test; have provided info for @sebastienros to provide assistance as to the effects.

[sebastienros]

Before: 876
After: 70

PS: Not a typo

[Tratcher]

Isn't this just a more extreme form of nagling? Don't we already have nagling enabled?

[benaadams]

PS: Not a typo

😢 Might need work... Guess I have to set up a linux environment

Isn't this just a more extreme form of nagling?

No; nagling is whether to send data based on acknowledgement packets received from the client. Cork only sends full packets.

Don't we already have nagling enabled?

Nagling is disabled by default (NoDelay = true)

[benaadams]

From https://yarchive.net/comp/linux/sendfile.html

Now, TCP_CORK is basically me telling David Miller that I refuse to play
games to have good packet size distribution, and that I wanted a way for
the application to just tell the OS: I want big packets, please wait until
you get enough data from me that you can make big packets.

Basically, TCP_CORK is a kind of "anti-nagle" flag. It's the reverse of
"no-nagle". So you'd "cork" the TCP connection when you know you are going
to do bulk transfers, and when you're done with the bulk transfer you just
"uncork" it. At which point the normal rules take effect (ie normally
"send out any partial packets if you have no packets in flight").

This is a much better interface than having to play games with
scatter-gather lists etc. You could basically just do

int optval = 1;

setsockopt(sk, SOL_TCP, TCP_CORK, &optval, sizeof(int));
write(sk, ..);
write(sk, ..);
write(sk, ..);
sendfile(sk, ..);
write(..)
printf(...);
...any kind of output..

optval = 0;
setsockopt(sk, SOL_TCP, TCP_CORK, &optval, sizeof(int));

and notice how you don't need to worry about how you output the data any
more. It will automatically generate the best packet sizes - waiting for
disk if necessary etc.

With TCP_CORK, you can obviously and trivially emulate the HP-UX behaviour
if you want to. But you can just do soo much more.

Imagine, for example, keep-alive http connections. Where you might be
doing multiple sendfile()'s of small files over the same connection, one
after the other. With Linux and TCP_CORK, what you can basically do is to
just cork the connection at the beginning, and then let is stay corked for
as long as you don't have any outstanding requests - ie you uncork only
when you don't have anything pending any more.

(The reason you want to uncork at all, is to obviously let the partial
packets out when you don't know if you'll write anything more in the near
future. Uncorking is important too.

Basically, TCP_CORK is useful whenever the server knows the patterns of
its bulk transfers. Which is just about 100% of the time with any kind of
file serving.

Linus

[tmds]

The issue with CORK is: it doesn't save you system calls. It's better to buffer in userspace until it's meaningful to go to the network. Pipelines should be enabling that.

[benaadams]

The issue with CORK is: it doesn't save you system calls.

No, it increases it by two; and post Meltdown that's particularly bad.

It's better to buffer in userspace until it's meaningful to go to the network. Pipelines should be enabling that.

This has come up several times before... /cc @Drawaes

Need to differentiate between WriteAsync (send at discretion) and FlushAsync (force send). Is more complicated with the other pattern which is GetSpan()->Advance()->FlushAsync() unless a WriteAsync() overload that takes no data is introduced:

ValueTask<FlushResult> WriteAsync(CancellationToken cancellationToken = default(CancellationToken));

For pattern

while (isData)
{
    var memory = pipe.GetMemory();
    // write
    pipe.Advance(written);
    await pipe.WriteAsync();
}
await pipe.FlushAsync()

The two use-cases that come to mind where the downstream has preferred processing chunk sizes are TLS (16kB chunks) and network (MTU chunks, on IPv4 generally 1400 bytes - TCP/IP headers)

The force send would need to ripple through multiple chained pipes; and currently the read side doesn't have a way of differentiating between a force flush and discretionary.

Stream and TextWriter do differentiate between Write and Flush (which is why Flush exists on them).

[Drawaes]

100% agree, this was brought up in the design review as well. We need a flush and a hard flush. Eg here is some data but morenis coming, and here is the last data I will send for a while. With sys calls getting more expensive this has become a bigger factor, but as Ben says for TLS,

There is header generation, then nounce/encryption setup for every packet and a trailer (hash/Mac) if you can pack more data in it's a big win.

[benaadams]

Ah, found the issue; the calls to setsocket option are throwing

[davidfowl]

None of this buffering helps the benchmarks since you get access to the request only, not the connection so buffering outside of the current request is out of your control, at least at the HTTP level in ASP.NET Core.

[Drawaes]

It certainly happens in real life though, where say you serialize a series of objects to the pipeline and you need to write each span out at a time as you don't want to go over the 2k barrier.
Maybe we need some other scenarios added to the benchmarks

[benaadams]

None of this buffering helps the benchmarks since you get access to the request only, not the connection ...

Network Pipe and TLS Pipe are connection level not request level?

Also at the request level where multiple writes occur they are done in random* chunk sizes

*Not matching anything that fits a nice multiple of the lower level; which equally the user-app level can't determine as it depends on length of status and headers etc sent prior to the app data

[davidfowl]

Also at the request level where multiple writes occur they are done in random* chunk sizes

When does that happen in practice? It's usually a single component doing the write. They are rarely scattered across the system (e.g. all of MVC writing to the response).

Network Pipe and TLS Pipe are connection level not request level?

Sure, but I'm talking about this change, not pipelines in general.

[benaadams]

Also at the request level where multiple writes occur they are done in random* chunk sizes

When does that happen in practice? It's usually a single component doing the write.

StaticFiles sends chunks of (64 * 1024) bytes; which is 48 full packets, one 11% full; then 48 packets, one 11% full etc

dotnet new webapp makes two calls to FlushAsync for its 3.1KB Transfer-Encoding: chunked homepage

Adding some Lorem Ipsum to increase it to 74.0KB means that moves up to 6 flushes for the page

And I doubt they are going to be flushing on packet boundaries?

[benaadams]

Added api request for SetRawSocketOption https://github.com/dotnet/corefx/issues/37122

[tmds]

StaticFiles sends chunks of (64 * 1024) bytes; which is 48 full packets, one 11% full; then 48 packets, one 11% full etc

This gets queued in the kernel and then DMAd to the network card. All these copies are very fast.
TCP_CORK will slow you down because of the extra syscalls.

[benaadams]

StaticFiles sends chunks of (64 * 1024) bytes; which is 48 full packets, one 11% full; then 48 packets, one 11% full etc

This gets queued in the kernel and then DMAd to the network card. All these copies are very fast.
TCP_CORK will slow you down because of the extra syscalls.

It sends 64kB; then reads another 64kB from disk then sends that 64kB. The disk read is likely to be a bigger lag between packets than the effect of calling TCP_CORK once at the start of the request?

* though I do agree it would be better to do this at the application layer (in user mode); however it would need more design - unless the ITcpCorkFeature was use to communicate the flushing out-of-band.

[tmds]

It sends 64kB; then reads another 64kB from disk then sends that 64kB. The disk read is likely to be a bigger lag between packets than the effect of calling TCP_CORK once at the start of the request?

Because the copying that happens in the kernel (user to kernel, then kernel to network) is so fast, it is hard to make it faster by packing more 'efficient'. TCP_CORK is adding syscall overhead, and data ends up waiting in the kernel when the network is idle.

To give an example, there is a MSG_ZEROCOPY feature in Linux which allows you to send without copying. It involves some extra syscall overhead to track which memory is still used by the kernel. Due to the overhead, the zerocopy feature is slower for packets smaller than 10k (and that number was before syscalls became more expensive due to meltdown, ...)

though I do agree it would be better to do this at the application layer (in user mode)

+1, I remember some corefx discussions on this.

[davidfowl]

I think we should measure this PR but I don't think we should merge it. There are better ways to buffer during a single request without requiring TCP_CORK

[benaadams]

Syscall cost does make this a wash

[benaadams]

Syscall cost does make this a wash

And the MDS / Zombieload mitigations just made it worse