The RAMCloud Storage System

John Ousterhout, Mendel Rosenblum, Stephen Rumble, Ryan Stutsman, Stephen Yang, Arjun Gopalan, Ashish Gupta, Ankita Kejriwal, Collin Lee, Behnam Montazeri, Diego Ongaro, Seo Jin Park (+1 others)
<span title="2015-08-31">2015</span> <i title="Association for Computing Machinery (ACM)"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/th23p5ixr5d5joiuac47oiqstq" style="color: black;">ACM Transactions on Computer Systems</a> </i> &nbsp;
RAMCloud is a storage system that provides low-latency access to large-scale datasets. To achieve low latency, RAMCloud stores all data in DRAM at all times. To support large capacities (1PB or more), it aggregates the memories of thousands of servers into a single coherent key-value store. RAMCloud ensures the durability of DRAM-based data by keeping backup copies on secondary storage. It uses a uniform logstructured mechanism to manage both DRAM and secondary storage, which results in high
more &raquo; ... formance and efficient memory usage. RAMCloud uses a polling-based approach to communication, bypassing the kernel to communicate directly with NICs; with this approach, client applications can read small objects from any RAMCloud storage server in less than 5μs, durable writes of small objects take about 13.5μs. RAMCloud does not keep multiple copies of data online; instead, it provides high availability by recovering from crashes very quickly (1 to 2 seconds). RAMCloud's crash recovery mechanism harnesses the resources of the entire cluster working concurrently so that recovery performance scales with cluster size. 7:2 J. Ousterhout et al. [Ritchie and Thompson 1974] . Over the past 15 years, the use of DRAM in storage systems has accelerated, driven by the needs of large-scale Web applications. These applications manipulate very large datasets with an intensity that cannot be satisfied by disk and flash alone. As a result, applications are keeping more and more of their long-term data in DRAM. By 2005, all of the major Web search engines kept their search indexes entirely in DRAM, and large-scale caching systems such as memcached [Memcached 2011] have become widely used for applications such as Facebook, Twitter, Wikipedia, and YouTube. Although DRAM's role is increasing, it is still difficult for application developers to capture the full performance potential of DRAM-based storage. In many cases, DRAM is used as a cache for some other storage system, such as a database; this approach forces developers to manage consistency between the cache and the backing store, and its performance is limited by cache misses and backing store overheads. In other cases, DRAM is managed in an application-specific fashion, which provides high performance but at a high complexity cost for developers. A few recent systems such as Redis [2014] and Cassandra [2014] have begun to provide general-purpose facilities for accessing data in DRAM, but their performance does not approach the full potential of DRAMbased storage. This article describes RAMCloud, a general-purpose distributed storage system that keeps all data in DRAM at all times. RAMCloud combines three overall attributes: low latency, large scale, and durability. When used with state-of-the-art networking, RAM-Cloud offers exceptionally low latency for remote access. In our 80-node development cluster with QDR Infiniband, a client can read any 100-byte object in less than 5μs, and durable writes take about 13.5μs. In a large datacenter with 100,000 nodes, we expect small reads to complete in less than 10μs, which is 50 to 1,000 times faster than the storage systems commonly used today. RAMCloud's second attribute is large scale. To support future Web applications, we designed RAMCloud to allow clusters to grow to at least 10,000 servers. RAMCloud aggregates all of their memories into a single coherent key-value store. This allows storage capacities of 1PB or more. The third attribute of RAMCloud is durability. Although RAMCloud keeps all data in DRAM, it also maintains backup copies of data on secondary storage to ensure a high level of durability and availability. This frees application developers from the need to manage a separate durable storage system, or to maintain consistency between in-memory and durable storage. It is our hope that low-latency storage systems such as RAMCloud will stimulate the development of a new class of applications that manipulate large-scale datasets more intensively than is currently possible. Section 2 motivates RAMCloud by showing how the high latency of current storage systems limits large-scale applications, and it speculates about new applications that might be enabled by RAMCloud. Sections 3 through 9 present the RAMCloud architecture from three different angles that address the issues of latency, scale, and durability: Storage management. RAMCloud uses a unified log-structured approach for managing data both in memory and on secondary storage. This allows backup copies to be made efficiently so that RAMCloud can provide the durability of replicated disk and the low latency of DRAM. The log-structured approach also simplifies crash recovery and utilizes DRAM twice as efficiently as traditional storage allocators such as malloc. RAMCloud uses a unique two-level approach to log cleaning, which maximizes DRAM space utilization while minimizing I/O bandwidth requirements for secondary storage. Latency. RAMCloud avoids the overheads associated with kernel calls and interrupts by communicating directly with the NIC to send and receive packets, as
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1145/2806887">doi:10.1145/2806887</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/fg3r5yahbjhxhcor6m2w2q6bxy">fatcat:fg3r5yahbjhxhcor6m2w2q6bxy</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20181126175017/http://pages.cs.wisc.edu:80/~remzi/Classes/739/Fall2018/Papers/ramcloud.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/a1/e8/a1e8f00a6626cd7fd6bb4140fd30741b10fddef2.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1145/2806887"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="external alternate icon"></i> acm.org </button> </a>