TxBox: Building Secure, Efficient Sandboxes with System Transactions

Suman Jana, Donald E. Porter, Vitaly Shmatikov
2011 2011 IEEE Symposium on Security and Privacy  
TXBOX is a new system for sandboxing untrusted applications. It speculatively executes the application in a system transaction, allowing security checks to be parallelized and yielding significant performance gains for techniques such as on-access anti-virus scanning. TXBOX is not vulnerable to TOCTTOU attacks and incorrect mirroring of kernel state. Furthermore, TXBOX supports automatic recovery: if a violation is detected, the sandboxed program is terminated and all of its effects on the host
more » ... are rolled back. This enables effective enforcement of security policies that span multiple system calls. I. INTRODUCTION Secure execution and confinement of untrusted applications is a long-standing problem in system security [35] . We present a new approach to constructing secure, efficient sandboxes based on system transactions. 1 In general, system transactions guarantee that a sequence of updates to system resources satisfies atomicity, consistency, isolation, and durability (ACID). Transactions are increasingly popular for managing concurrency in modern operating systems. Our prototype system, TXBOX, uses transactions for (1) speculative execution of untrusted applications, (2) uncircumventable enforcement of system-call policies, and (3) automatic recovery from the effects of malicious execution. TXBOX consists of a relatively simple, policy-agnostic security monitor running in the OS kernel and a user-level policy manager. The separation of the security policy and the enforcement mechanism facilitates management of policies. Before the transaction associated with a sandboxed program commits, the monitor inspects its effects on the system (conveniently assembled in the transaction's workset and its system-call log) and checks if they satisfy the policy. If so, the transaction is committed and updates become visible to the rest of the system. Otherwise, the transaction is aborted and the system is restored to a good state. TXBOX is suitable for sandboxing "one-shot" execution of unknown, untrusted programs, as well as for model-based enforcement of system-call behavior of known benign programs. Uncircumventability. TXBOX cannot be circumvented by a sandboxed process. Its kernel-based enforcement mechanism prevents exploitation of incorrect mirroring of the kernel 1 System transactions are not transactional memory. System transactions deal with accesses by a user process to system resources such as files and pipes, not with memory accesses. state, TOCTTOU races, and/or other semantic gaps between the security monitor and the OS [21, 58] . Unlike any monitor that infers effects on the OS from the periphery of the kernel, the effects analyzed by the TXBOX monitor when making security decisions are exactly the effects that would take place if execution is permitted. Recoverability. Existing system-call monitors must allow or deny every system call made by the untrusted program before it executes. Once the call is permitted to execute, there is no way to recover. Therefore, they must be able to detect the very first sign of misbehavior since the effects of a malicious execution cannot be "undone." By contrast, TXBOX executes untrusted programs speculatively, inside a transaction. If the monitor determines later that the program has violated a security policy, it aborts the transaction and the system is automatically rolled back to a benign state. All changes made by the violating program to the file system effectively disappear, child processes are stopped, and buffered local inter-process communication is canceled, leaving concurrent updates made by other programs undisturbed. To illustrate the benefits of recoverability, Section VI shows how TXBOX can restore the original state of local files if an untrusted program (e.g., a multimedia converter) attempts to violate the sandboxing policy. Performance. On realistic workloads, the performance overhead of TXBOX is less than 20% including the cost of supporting transactions and less than 5% over untrusted execution in a transactional OS. Note that there is a compelling secular (i.e., unrelated to security) reason for supporting system transactions, namely, managing concurrency. TXBOX can take advantage of multi-core processors. In Section VI, we show how to use TXBOX to execute an antivirus scan in parallel with the speculative execution of an untrusted program. This makes on-access scanning practical for routine use in production systems. Expressive policies. TXBOX can enforce a rich class of practical security policies. This includes all policies supported by system-call interposition tools such as Systrace [49], system-call policies for malware detection [7, 33, 36] , model-based system-call automata [26, 52, 56] , dataflow policies on system-call arguments [5] , and, in general, any policy expressible as a security automaton [16] over system calls and system resources. For system-call policies,
doi:10.1109/sp.2011.33 dblp:conf/sp/JanaPS11 fatcat:thlvosrksjdbfnqn7k6aexkefi