ClearShot: Eavesdropping on Keyboard Input from Video

Davide Balzarotti, Marco Cova, Giovanni Vigna
2008 IEEE Symposium on Security and Privacy : Proceedings  
Eavesdropping on electronic communication is usually prevented by using cryptography-based mechanisms. However, these mechanisms do not prevent one from obtaining private information through side channels, such as the electromagnetic emissions of monitors or the sound produced by keyboards. While extracting the same information by watching somebody typing on a keyboard might seem to be an easy task, it becomes extremely challenging if it has to be automated. However, an automated tool is needed
more » ... ated tool is needed in the case of long-lasting surveillance procedures or long user activity, as a human being is able to reconstruct only a few characters per minute. This paper presents a novel approach to automatically recovering the text being typed on a keyboard, based solely on a video of the user typing. As part of the approach, we developed a number of novel techniques for motion tracking, sentence reconstruction, and error correction. The approach has been implemented in a tool, called ClearShot, which has been tested in a number of realistic settings where it was able to reconstruct a substantial part of the typed information.
doi:10.1109/sp.2008.28 dblp:conf/sp/BalzarottiCV08 fatcat:opq6mjr7ujbhneqvb63vlpgkba