Semantic Interaction for Visual Analytics: Toward Coupling Cognition and Computation

Alex Endert
2014 IEEE Computer Graphics and Applications  
T he world is becoming increasingly instrumented with sensors, monitoring, and other methods for generating data describing social, physical, and natural phenomena. So, data exist that could be analyzed to uncover, or discover, the phenomena from which they were created. However, as the analytic models leveraged to analyze these data continue to increase in complexity and computational capability, how can visualizations and user interaction methodologies adapt and evolve to continue to foster
more » ... scovery and sensemaking? User interaction is critical to such visual data exploration's success because it lets users test assertions, assumptions, and hypotheses about the information, given their prior knowledge about the world. This cognitive process can be generally called sensemaking. Visual analytics (VA) emphasizes sensemaking of large, complex datasets through interactively exploring visualizations generated through a combination of analytic models. (For more on this, see the related sidebar.) So, a central focus is understanding how to leverage human cognition in concert with powerful computation through usable visual metaphors. My PhD dissertation coined the term semantic interaction in the context of a user interaction methodology for model steering in VA systems. 1 It made three primary contributions. First, it explained the interactions users commonly employ when analyzing text information spatially without computational layout models, and the meaning they externalize into the manually crafted spatial constructs. 2,3 Second, it enabled bidirectionality of spatializations by inverting popular dimension reduction models. 4-6 Finally, it evaluated semantic interaction's impact on sensemaking through the synchronization of the analytic-model parameters, the visualization, and the user's insights in the text analysis domain. 7 Semantic Interaction Semantic interaction aims to enable co-reasoning between the user and the analytic models (coupling cognition and computation) without requiring the user to directly control them. To do this, it utilizes the visual metaphor in two ways: ■ the metaphor through which the insights are obtained (that is, the visualization of information created by computational models) and ■ the interaction metaphor through which hypotheses and assertions are communicated (that is, interaction occurs within the visual metaphor). Users directly manipulate data in visualizations; semantic interaction then captures tacit knowledge of the user and steers the underlying analytic models. These models can be adapted incrementally on the basis of the user's sensemaking process and domain expertise explicated through the user's interaction. (For semantic interaction design guidelines, see the related sidebar.) That is, the visualization's visual constructs expose the underlying analytic models' parameters. On the basis of common visual metaphors (such as the geographic, spatial metaphor in which proximity approximates similarity), we can infer tacit knowledge of the user's reasoning by inverting these analytic models. So, users are shielded from the underlying complexities and can interact with their data through a bidirectional visual medium. The interactions users perform in the visualizations to augment the visual encodings within the metaphor enable the inference of their analytic reasoning, which is systematically applied to the underlying models.
doi:10.1109/mcg.2014.73 pmid:25051565 fatcat:3aslspveqza2tls5l26m72kbom