Formal Analysis of Log Files

Howard Barringer, Alex Groce, Klaus Havelund, Margaret Smith
2010 Journal of Aerospace Computing Information and Communication  
Runtime verification as a field faces several challenges. One key challenge is how to keep the overheads associated with its application low. This is especially important in real-time critical embedded applications, where memory and CPU resources are limited. Another challenge is that of devising expressive and yet user-friendly specification languages that can attract software engineers. In this paper, it is shown that for many systems, in-place logging provides a satisfactory basis for
more » ... tem "runtime" verification of logs, where the overhead is already included in system design. Although this approach prevents an online reaction to detected errors, possible with traditional runtime verification, it provides a powerful tool for test automation and debugging-in this case, analysis of spacecraft telemetry by ground operations teams at NASA's Jet Propulsion Laboratory. The second challenge is addressed in the presented work through a temporal specification language, designed in collaboration with Jet Propulsion Laboratory test engineers. The specification language allows for descriptions of relationships between data-rich events (records) common in logs, and is translated into a form of automata supporting data-parameterized states. The automaton language is inspired by the rule-based language of the RULER runtime verification system.A case study is presented illustrating the use of our LOGSCOPE tool by software test engineers for the 2011 Mars Science Laboratory mission. ‡ Laboratory for Reliable Software, § Software System Engineering, 365 BARRINGER ET AL. Consequently, no additional impact on the running software is imposed beyond the already implemented logging. A log is a recorded sequence of events. We argue that systematizing logging can be of additional benefit. Second, we introduce a textual temporal logic-inspired specification language over events for specifying properties of logs. This language supports data-parameterization, essential for monitoring logs (events typically carry data). Specified properties, called patterns, are translated into data-parameterized automata forming an interesting and useful subset of the textual RULER language [1-4]. Users can mix patterns with more expressive automata in a specification. Parameterized automata are visualized using the GRAPHVIZ tool [5]. Our system additionally offers preliminary support for automatically constructing (learning) specifications from example logs. The log analysis framework, LOGSCOPE, is developed to support engineers testing the flight software for NASA's next Mars rover mission, the Mars Science Laboratory (MSL) [6], developed at the Jet Propulsion Laboratory (JPL). In this sense the work represents an instance of the often sought marriage between theory and practice in formal methods. The MSL mission's goal is to put the so-far largest rover (the size of a compact car) on Mars for continued exploration of the red planet. It is scheduled for launch in 2011. A flight software team peaking at approximately 30 programmers develops the software for controlling the rover compute element, which controls all stages of the integrated spacecraft. A testing team of approximately 10 engineers (the Flight software Internal Test, or FIT, team) is responsible for functional testing of the flight software. LOGSCOPE was developed to support this team, and is the result of their requirements as well as our research ideas. The MSL flight software produces rich log information, which is stored in SQL databases (one database per log). A log is in essence a sequence of time-stamped events, where an event can be one of several forms, corresponding to input to and output from the system, as well as internal state transitions and state readings. Each event is in essence a mapping from field names to values (a record). Such logs are usually far too large to be analyzed effectively by humans. Traditionally, these MSL logs have been analyzed by writing scripts, with properties coded up in PYTHON. Such scripts are time consuming to produce and result in difficult-to-read "specifications" that hinder communication, maintenance, specification-sharing, and reuse. Based on our experience with LOGSCOPE, runtime verification of logs using formal specifications has excellent potential as a route to introduce lightweight formal methods into a high-profile NASA flight project. A. Contributions and Related Work The contribution of this work is the design of a simple, user-friendly, and yet for practical purposes sufficiently expressive data-parameterized temporal specification language, with a translation into data-parameterized automata, which are themselves usable for specification. A second contribution is the injection of this technology into a NASA flight mission. The LOGSCOPE system has specifically been influenced by the RULER system [1] [2] [3] [4] . In particular, the automaton language conceptually forms a subset of RULER. LOGSCOPE can be seen as an adaptation of RULER to the specific needs of MSL: adding temporal logic, enhancing usability, and implementing it in PYTHON to integrate with the existing scripting culture. LOGSCOPE has also been influenced by the state machine-based systems RCAT [7] and RMOR [8] . The relationship between LOGSCOPE and RULER is illustrated in [1], whereas the process that led to LOGSCOPE is presented in [9] . Within the last decade, a substantial body of work has been produced on runtime verification [10], e.g., to mention just a few approaches, over and above the work referenced above, consider [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] . Each of these is characterized by several dimensions, such as the specification logic they support, the way in which they connect with the application being monitored, the way in which monitors are generated for the propositional (data-free) case, and the way in which they handle data values. LOGSCOPE offers a user-friendly logic, avoiding some of the pitfalls of LTL (such as the need for nested until-formulas for expressing sequencing). By analyzing log files, the issue of code instrumentation becomes a separate problem. Data parameterization is handled through parameterized automata: a state machine state can carry a vector of data values. The approach is a restricted form of parameterization mechanisms in the RULER system (see Sec. V.D), however, the basic idea is a special case of the monitor state being a formula including data. It is also used in our own EAGLE system [12] , which offers a very sophisticated logic based on minimal and maximal fixpoints. The sophistication of this logic caused the implementation to be highly complex. The HAWK system [14] augmented EAGLE with an event definition language specialized for Java, and monitors were generated as ASPECTJ [22] aspects. JLO [21, 23] generates monitors in a similar way from parameterized LTL formulas. As explained in [23] , the monitor state is a formula including instantiated data values.
doi:10.2514/1.49356 fatcat:y5gj6s5xnncvdbaepgjndg6aum