The combination of TCP auto-tuning and asynchronous I/O event notifications (e.g., epoll) allows the Linux kernel to generally sustain high-volume TCP connections—even for connections with high bandwidth-delay products (high link bandwidth and/or high path latency). However, bufferbloat can quickly become an issue when multiple such connections are in use. In particular, high outbound kernel queuing delays have been observed in the Tor anonymity network, a large distributed system whose relays often manage thousands of sockets—many of which are simultaneously-active, high-volume TCP connections. In this work, we propose a new notification event that triggers when TCP is ready to send data on a socket while seeking to better understand how it can help applications to better manage network I/O and improve performance. The new event supplements and extends the current write event that triggers when a socket buffer has free space, and the difference in semantics allows more precise control over queuing to the application. We describe the problem, detail a proposal for extending epoll to support the new semantics (including a code patch), and show the effect that such a change could have on performance through a small scale simulation.
@inproceedings{epollcwnd-netdev2020,
title = {Reducing Kernel Queuing Delays with TCP Window Space Events},
author = {Goulet, David and Jansen, Rob},
booktitle = {Technical Conference on Linux Networking},
year = {2020},
}