High-performance ethernet-based communications for future multi-core processors

Michael Schlansker, Norman P. Jouppi, Nagabhushan Chitlur, Erwin Oertli, Paul M. Stillwell, Linda Rankin, Dennis Bradford, Richard J. Carter, Jayaram Mudigonda, Nathan Binkert
2007 Proceedings of the 2007 ACM/IEEE conference on Supercomputing - SC '07  
Data centers and HPC clusters often incorporate specialized networking fabrics to satisfy system requirements. However, Ethernet's low cost and high performance are causing a shift from specialized fabrics toward standard Ethernet. Although Ethernet's low-level performance approaches that of specialized fabrics, the features that these fabrics provide such as reliable in-order delivery and flow control are implemented, in the case of Ethernet, by endpoint hardware and software. Unfortunately,
more » ... rrent Ethernet endpoints are either slow (commodity NICs with generic TCP/IP stacks) or costly (offload engines). To address these issues, the JNIC project developed a novel Ethernet endpoint. JNIC's hardware and software were specifically designed for the requirements of high-performance communications within future data-centers and compute clusters. The architecture combines capabilities already seen in advanced network architectures with new innovations to create a comprehensive solution for scalable and high-performance Ethernet. We envision a JNIC architecture that is suitable for most in-data-center communication needs. The primary contribution of this work is the design, implementation, and evaluation of a network architecture for Ethernet-based communications in future data centers. The hardware and software architecture is designed with a number of key assumptions. Hardware must be simple for close integration in future multi-core processors. Inexpensive hardware minimizes cost for users with modest communication needs. The architecture uses front-side bus attachment to provide low latency, high bandwidth and efficient hardware/software interaction. The architecture must allow efficient scalable communication within very large data centers. Complex control must be implemented using on-load software that exploits general-purpose processors and memory. Additional general-purpose resources are used to provide increased performance and more complex communication services. Functions such as congestion management, quality of service, and a variety of complex client communication services are incorporated as flexible on-load software to allow the incorporation of new functionality. While most of JNIC's components are similar to components that have been explored within prior work, we believe that JNIC provides a unique and innovative system architecture that improves our understanding of communications architectures for future data centers To understand future system performance, a hardware and software prototype was developed instead of using simulation. This carries benefits and limitations. We integrated and tested all components within a fully-functional system and our complex prototype demonstrates many real-system behaviors. A key limitation of our prototype is modest performance. When work began, we were aware that our 1 Gb Ethernet prototype
doi:10.1145/1362622.1362672 dblp:conf/sc/SchlanskerCOSRBCMBJ07 fatcat:b6nqg7loczb6bifp5cptihxjgy