Gesture-Based Control of the Scanning Electron Microscope Using a Low-Cost Webcam Sensor

B Wong, BC Breton, DM Holburn, NHM Caldwell
2018 Microscopy and Microanalysis  
This work forms part of our ongoing research into enhancing and improving microscopy and image processing technologies, and was one of a number projects undertaken to explore the potential of gesture-based interface control. This project focused on gesture-based control of a scanning electron microscope (SEM), Carl Zeiss model 1430VP, using a low-cost webcam as the sensor to detect and identify gestures. With the inexorable advances in computing power and adoption of connected devices, there
more » ... been a surge of interest in developing gesture control for many devices and apparatus in a wide range of applications. A number of these have already achieved commercial success. For example, the Microsoft Xbox and the Nintendo Wii both use gesture controls for video gaming to enhance the user experience. The objective of this project was to investigate the feasibility and practicality of developing a hand gesture control system for a scanning electron microscope and develop a proof of concept system. Apart from the on-site usage of the SEM by using a keyboard, mouse and joystick, there is nowadays the possibility of remote access to an SEM, where the user remotely connects to the instrument and controls it using standard mouse and keyboard commands, with images shown on a remote screen. Interacting remotely with a SEM through gestures may be more intuitive and can offer a better user experience. Up to now, most implementations of gesture control have made use of hardware custom-designed for the gaming industry, such as the Microsoft Kinect and the Leap Motion Sensor. These two devices target different styles of interaction. Kinect focuses on capturing body pose, but there is limited software support for developers. Leap is a short-range hand-gesture capture device using a stereo infrared camera, with a range of programming interfaces available to developers. Cater et al have described how the Leap Motion Sensor can be used to recognise a selection of gestures, and hence to allow control of a Scanning Electron Microscope [1]. However, they identified a trend in the industry towards the deprecation of "gestures" in the most recent Leap Motion software releases; this may mean that gesture control of software and instrumentation could be limited to emulation of touchscreen gestures scaled up from smartphone or tablet computers for the foreseeable future. This project has taken a different approach and explores the possibility of developing a real time hand gesture-control system for SEM or similar applications using low-cost hardware, based on an ordinary webcam of the type commonly used for web communications, video conferencing or security, coupled with computer vision technology implemented using an open-source vision software library. The programming approach taken was based on the use of the OpenCV computer vision library [2], to discriminate the subject's hand from the background, and to recognise gestures by extracting features of the hand and fingers using digital image processing techniques. In the first stage, to locate the subject's hand, skin segmentation is carried out. This is a challenging exercise, since skin appearance is affected
doi:10.1017/s1431927618002970 fatcat:chfou5dyera4deihmqbcsjol3y