Memory management for user-level network interfaces

M. Welsh, A. Basu, Xun Wilson Huang, T. von Eicken
1998 IEEE Micro  
User-level network interfaces allow applications direct access to the network without operating system intervention on every send and receive. Messages are transferred directly to and from user-space by the network interface while observing the traditional protection boundaries between processes. Current user-level network interfaces limit this message transfer to a per-process region of permanently-pinned physical memory to allow safe DMA. This approach is inflexible in that it requires data
more » ... be copied into and out of this memory region, and does not scale to a large number of processes. This paper presents an extension to the U-Net user-level network architecture (U-Net/MM) allowing messages to be transferred directly to and from any part of an application's address space. This is achieved by integrating a translation look-aside buffer into the network interface and coordinating its operation with the operating system's virtual memory subsystem. This mechanism allows network buffer pages to be pinned and unpinned dynamically. Two implementations of U-Net/MM are described, demonstrating that existing commodity hardware and commercial operating systems can efficiently support the architecture. Recent research in high-speed network interfaces for commodity networks has focused on removing the operating system from the critical path for sending and receiving messages. An effective solution is to provide user-level messaging [1, 2, 3, 9, 11, 13, 14, 15] where the network interface (NI) is virtualized by multiplexing physical network resources among multiple processes. This allows applications to communicate directly with the NI and messages can be sent from and received to user-space without kernel intervention. The communication overheads are thus reduced to the costs of passing commands and data between the main CPU and the NI. Previous work on user-level networking has demonstrated that it can provide low latencies along with full utilization of the available network bandwidth with relatively small messages. One of the main implementation difficulties is managing the mapping between the virtual addresses of message buffers specified by applications and the physical addresses required for actual transmission and reception. This includes two major components: the NI must be able to translate the virtual addresses to physical addresses and the translations must be coordinated with the operating system's virtual memory subsystem. So far, two solutions have been proposed which more or less side-step the issue. Custom designs appropriate for "next generation" systems have integrated the NI into the virtual-address side of the system where it shares or replicates the CPU's TLB [1, 11, 14] . Less aggressive proposals allocate a physically-contiguous buffer region for each application and pin it down to physical memory [2, 3, 12, 15] . The application specifies message buffers using offsets into the buffer region which the network interface can easily bounds-check and translate. This prevents the NI from issuing illegal DMA accesses because the virtual to physical mapping is fixed. The work presented here addresses the memory management issues on commodity processors and networks where the network interfaces reside on the physical-address side of the system (typically the I/O bus) and are independent of the processor architecture. In these systems the NI transfers data to and from message buffers in main memory using DMA which requires physical memory addresses. If applications are to interact with the NI directly, the latter must incorporate some form of address translation. Moreover, for portability reasons, it is desirable that the address translation be independent of the host processor architecture, especially the host's page table representation. This precludes the type of solutions used in parallel machines with custom networks and NIs on the system bus or even integrated on the CPU chip. The major contribution of this paper is two-fold: • It describes a user-level NI design that incorporates a TLB into the NI to overcome the limitations of pinned-down communication buffers. Applications can use arbitrary virtual memory addresses for message sends and receives and only the pages covered by the NI TLB are locked down. The total memory reserved for the network is prima-
doi:10.1109/40.671405 fatcat:fytdjskmkvhwporm2esv7onl5q