Camera phones are ubiquitous, and consumers have been adopting them faster than any other technology in modern history. When connected to a network, though, they are capable of more than just picture taking: Suddenly, they gain access to the power of the cloud. We exploit this capability by providing a series of image-based personal advisory services. These are designed to work with any handset over any cellular carrier using commonly available Multimedia Messaging Service (MMS) and Short

more »

... e Service (SMS) features. Targeted at the unsophisticated consumer, these applications must be quick and easy to use, not requiring download capabilities or preplanning. Thus, all application processing occurs in the back-end system (i.e., as a cloud service) and not on the handset itself. Presenting an image to an advisory service in the cloud, a user receives information that can be acted upon immediately. Two of our examples involve color assessment -selecting cosmetics and home décor paint palettes; the third provides the ability to extract text from a scene. In the case of the color imaging applications, we have shown that our service rivals the advice quality of experts. The result of this capability is a new paradigm for mobile interactions -image-based information services exploiting the ubiquity of camera phones. Cameras to the Cloud For many consumers, a cell phone is simply a device used for spoken communication. What makes it special, however, is that it is mobile and is evolving to contain a plethora of sensors such as camera, GPS, accelerometer, proximity and light detector, and microphone [4]. While equipped with increasing numbers of sensing modalities, we feel that none has quite the potential of the camera, because of the capability of computing advisory and other services linked to its images. Such image-based computation could be performed on the handset, but our view is that the most interesting opportunity comes in accessing the cloud. In short, we can take a picture, request expert analysis, and receive feedback and advice of immediate value. This in situ measurement capability is unparalleled. The convenient and easy-to-use nature of the device greatly facilitates its use by consumers. Snap, Analyze, and Go is a novel operational model. An additional driving factor for using the cloud as the application server is that with the large number of cell phone brands, operating systems, technologies employed, and service providers out there, it would be impractical to develop a cell phone application supported across all these variants. Using the cloud lets us bypass this difficulty; we simply rely 2 on the mobile client being connected to cloud services, and let the network be the unifier. In this paper we describe three applications built around this notion of using any camera-equipped network-connected cell phone as the front end for accessing personalized image processing services. A New Paradigm: Image-based Queries In demonstrating these applications, we also argue that this presents a new paradigm for mobile applications; imagebased queries. If a picture is worth a thousand words, we suggest that a picture may be worth a thousand key strokes. Applications would be much easier to use if the only requirement were taking and sending a picture -about the simplest possible interface. If we tried to rely on the usual speech or text for such an application, the process would be much more difficult. We have created specific-interpretation image-based systems for cosmetics selection, home décor, and text extraction, and carried out initial explorations in healthcare, language translation, and other areas. Note that the use of cell phones as the imaging sensor gives immediate ubiquity. Cell phone figures on a world-wide basis make it clear that, for many populations, this will be their single communication and computation device. Cell phone penetration in the US is already at 89% and continuing up [5] . For many in the developing world it is their only computing platform. Consider as well that many of the engines of innovation revolve around commercial applications, bringing an accelerating effect to the injection of technology to this space. The three examples we demonstrate have identical interfaces-up to the destination number keyed. MMS is used for the outgoing, and SMS brings back the response. In what follows, we discuss fundamental challenges in this approach, requirements of the individual applications, our means and methods in performance evaluation, and plans for future developments. Color Assessment from Camera Phones Despite the ease of taking a picture, objective color assessment remains difficult, particularly with the variable quality of devices generally found in cell phones. The same scene imaged with different cameras can result in significant aesthetic and quantitative differences, primarily due to uneven illuminant compensation, in-camera processing, and varying imager characteristics. It is impossible to accurately assign a color from an uncalibrated digital imager -both the camera and the scene present unknown influences [6] . Color properties of objects are fully characterized by their reflectance spectra, i.e. the percentage of light reflected by the object's surface at each wavelength and incident angle. In many applications, however, it is sufficient to retrieve only tristimulus values, and these can be acquired from an RGB camera. Several approaches using calibrated trichromatic imaging systems have been presented ([7,8]). Our method requires just a simple calibrated reference target to be present in the scene, providing a cheap alternative to using an expensive calibrated device (that would only provide for the camera side of the unknowns, anyhow). The object of interest is imaged together with the reference target, its reference values observed, and a transform derived, allowing color correction of images independently of both the imaging device and the scene illuminant. The next three sections detail the applications and our solutions.