Nikola Banovic, Tovi Grossman, Justin Matejka, George Fitzmaurice
2012 Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12  
We present Waken, an application-independent system that recognizes UI components and activities from screen captured videos, without any prior knowledge of that application. Waken can identify the cursors, icons, menus, and tooltips that an application contains, and when those items are used. Waken uses frame differencing to identify occurrences of behaviors that are common across graphical user interfaces. Candidate templates are built, and then other occurrences of those templates are
more » ... emplates are identified using a multiphase algorithm. An evaluation demonstrates that the system can successfully reconstruct many aspects of a UI without any prior application-dependant knowledge. To showcase the design opportunities that are introduced by having this additional meta-data, we present the Waken Video Player, which allows users to directly interact with UI components that are displayed in the video.
doi:10.1145/2380116.2380129 dblp:conf/uist/BanovicGMF12 fatcat:ujb2awvrtnh7jkkye7h5cpddoa