Empirical Evaluation of Diagrams of the Run-time Structure for Coding Tasks

Nariman Ammar, Marwan Abi-Antoun
2012 2012 19th Working Conference on Reverse Engineering  
With object-oriented design, it is at least as important-possibly more important-to understand the runtime structure, in terms of objects and their relations, as to understand the code structure dealing with source files, classes and packages. Today, many tools and diagrams help developers understand the code structure. Diagrams of the run-time structure, however, are much less mature. One diagram of the run-time structure is a statically extracted, global, hierarchical Ownership Object Graph
more » ... OG). The OOG conveys architectural abstraction by ownership hierarchy by showing architecturally significant objects near the top of the hierarchy and data structures further down. In an OOG, objects are also organized into named, conceptual groups called domains. We posit that types are not enough for object-oriented code comprehension, and that the OOG improves comprehension by giving developers the ability to distinguish the role that an object plays, not only by type, but also by named groups (domains) or by position in the run-time structure (ownership), i.e.,types + ownership + domains. We evaluate, in a controlled experiment, whether an OOG, as a diagram of the run-time structure, improves object-oriented code comprehension by giving developers the ability to distinguish the role that an object plays, not only by type, but also by named groups (domains) or by position in the run-time structure (ownership). We observed 10 participants, for 3 hours each, perform three feature implementation tasks on a framework application. Our experiment identified that, on average, the participants who used information obtained from OOGs, in addition to information obtained from the code structure, i.e., class diagrams and browsing or reading the code in Eclipse, always outperformed the participants who used only information obtained from the code structure, i.e., class diagrams and browsing or reading the code in Eclipse. Our results indicate that, on average, the OOG had a positive effect of varying extents on comprehension that reduced the time spent by 22%-60% and irrelevant code explored by 10%-60%. The difference was significant (p < 0.05) for two of the tasks. Introduction Software maintenance accounts for 50% to 90% of the costs over the life-cycle of a software system. One major activity during maintenance, program comprehension, absorbs around half of the costs [13] . To support comprehension, researchers have produced many tools to visualize the structure of a software system. These tools are based on the widespread belief that diagrams are useful for comprehension. For example, a high-level diagram can help a developer locate where to implement a change. One problem related to diagrams and comprehension is understanding the run-time structure of objectoriented code, in terms of objects and their relations. With object-oriented design, it is at least as important-possibly more important-to understand the run-time structure as to understand the code structure dealing with source files, classes and packages. In object-oriented design patterns, for example, much of the functionality is determined by what instances point to what other instances [23] . For instance, in the Observer design pattern, understanding "what" gets notified during a change notification is crucial for the operation of the system, but "what" does not usually mean a class, "what" means an instance. Moreover, for object-oriented code, the run-time structure is often quite different from the code structure. Thus, a Diagram of the Run-time Structure (DRS) can be highly complementary to a Diagram of the Code Structure (DCS) and can answer several crucial questions to developers while performing code modifications. Currently, there are many widely supported DCS tools, but DRS tools are much less mature. We broadly include in DCS tools various code exploration features found in modern IDEs such as Eclipse, even if they do not display a diagram. For instance, they may show the classes in a project as a tree (Eclipse Package Explorer), or allow searching for strings in files and show the results as a list. But they are still showing code entities, as opposed to run-time entities. Currently, many tools extract DCS, but tools to extract DRS are much less mature. One DRS recently proposed and formalized by Abi-Antoun and Aldrich [4] is a statically extracted, global, hierarchical Ownership Object Graph (OOG). The OOG is a sound approximation of all run-time objects across the entire system with all possible points-to relations between those objects. A DRS assigns different instances of the same type different roles according to the context in which they occur. In particular, on the OOG, the context is expressed using the notion of an ownership domain. A domain is a conceptual grouping of objects, and the domain name conveys design intent in the code based on ownership domain annotations. Due to space limits we describe the annotation language in [2, Appendix A]. In this paper, we deemphasize evaluating the effort of extracting and refining OOGs, which we have recently measured [6] . The interested reader can refer to [12] and [11, Chap.3] to read more about the process of extracting and refining OOGs for this experiment. This paper sheds more light on the inherent difficulties of object-oriented comprehension, and contributes the following: • A theory in comprehension in terms of facts that developers can learn from a DRS; Our theory predicts that the ability to distinguish the role that an instance plays not just by type, but by named groups (domains) or by position in the run-time structure (ownership), enables developers to answer challenging questions about the RS.We illustrate how types are clearly not enough for object-oriented code comprehension, and how information content on a DRS (facts about types + ownership + domains) gives developers a richer language for describing that role than type alone. • A preliminary classification of questions that developers ask about the run-time structure; • A controlled experiment to investigate whether developers can use a DRS to answer questions about the run-time structure during code modifications more easily than developers who do not have access to such a diagram and rely on only type information provided by DCS tools. Outline. In the rest of this paper, we give background on OOGs (Section 2). In Section 3, we explain our theory of comprehension (Section 3.1) and define a classification of facts about the run-time structure based on our theory, and we motivate the need for OOGs to fill gaps in todays diagrams and currently used DCS tools (Section 3.2). Next, we present our controlled experiment. describe our method (Section 4), our analysis (Section 5), and both quantitative and qualitative results (Section 6). We then discuss some limitations, threats to validity and lessons learned (Section 7). Finally, we discuss related work (Section 8) and conclude.
doi:10.1109/wcre.2012.46 dblp:conf/wcre/AmmarA12 fatcat:wihjw2m6orfj3haxwjs7gxzuj4