Visual Recognition Of Graphical User Interface Components Using Deep Learning Technique

Agyl Ardi Rahmadi, Aris Sudaryanto
2020 Jurnal Ilmu Komputer dan Informasi  
Graphical User Interface (GUI) building in software development is a process which ideally need to go through several steps. Those steps in the process start from idea or rough sketch of the GUI, then refined into visual design, implemented in coding or prototype, and finally evaluated for its function and usability to discover design problem and to get feedback from users. Those steps repeated until the GUI considered satisfactory or acceptable by the user. Computer vision technique has been
more » ... searched and developed to make the process faster and easier; for example generating code for implementation, or automatic GUI testing using component images. But among those techniques, there are still few for usability testing purpose. This preliminary research attempted to make the foundation for usability testing using computer vision technique by built minimalist dataset which has images of various GUI components and used the dataset in deep learning experiment for GUI components visual recognition. The experiment results showed deep learning technique suitable for the intended task, with accuracy of 95% for recognition of two different types of components, and accuracy of 72% for six different types of component.
doi:10.21609/jiki.v13i1.845 fatcat:pbbive4vj5ewzdgbz77wagovjy