69 Hits in 0.92 sec

Seeing Tree Structure from Vibration [article]

Tianfan Xue, Jiajun Wu, Zhoutong Zhang, Chengkai Zhang, Joshua B. Tenenbaum, William T. Freeman
2018 arXiv   pre-print
Xue, T., Rubinstein, M., Liu, C., Freeman, W.T.: A computational approach for obstruction-free photography. ACM TOG 34(4), 79 (2015) 48.  ... 
arXiv:1809.05067v1 fatcat:tainnq5arzfhtlp27hc6fyvake

Unprocessing Images for Learned Raw Denoising [article]

Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, Jonathan T. Barron
2018 arXiv   pre-print
Machine learning techniques work best when the data used for training resembles the data used for evaluation. This holds true for learned single-image denoising algorithms, which are applied to real raw camera sensor readings but, due to practical constraints, are often trained on synthetic image data. Though it is understood that generalizing from synthetic to real data requires careful consideration of the noise properties of image sensors, the other aspects of a camera's image processing
more » ... line (gain, color correction, tone mapping, etc) are often overlooked, despite their significant effect on how raw measurements are transformed into finished images. To address this, we present a technique to "unprocess" images by inverting each step of an image processing pipeline, thereby allowing us to synthesize realistic raw sensor measurements from commonly available internet photos. We additionally model the relevant components of an image processing pipeline when evaluating our loss function, which allows training to be aware of all relevant photometric processing that will occur after denoising. By processing and unprocessing model outputs and training data in this way, we are able to train a simple convolutional neural network that has 14%-38% lower error rates and is 9x-18x faster than the previous state of the art on the Darmstadt Noise Dataset, and generalizes to sensors outside of that dataset as well.
arXiv:1811.11127v1 fatcat:oyshl7ssefgcjlh3vy36s4fsja

Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks [article]

Tianfan Xue, Jiajun Wu, Katherine L. Bouman, William T. Freeman
2016 arXiv   pre-print
We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose a novel approach that models future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. Future frame synthesis is challenging, as it involves low- and high-level image
more » ... motion understanding. We propose a novel network structure, namely a Cross Convolutional Network to aid in synthesizing future frames; this network structure encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-wold videos. We also show that our model can be applied to tasks such as visual analogy-making, and present an analysis of the learned network representations.
arXiv:1607.02586v1 fatcat:oubrsjq4qrgtloqm3kiowhtr74

MarrNet: 3D Shape Reconstruction via 2.5D Sketches [article]

Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, William T Freeman, Joshua B Tenenbaum
2017 arXiv   pre-print
3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenges for learning-based approaches, as 3D object annotations are scarce in real images. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from domain adaptation when tested on real data. In this work, we propose MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches
more » ... nd 3D object shape. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, 2.5D sketches are much easier to be recovered from a 2D image; models that recover 2.5D sketches are also more likely to transfer from synthetic to real data. Second, for 3D reconstruction from 2.5D sketches, systems can learn purely from synthetic data. This is because we can easily render realistic 2.5D sketches without modeling object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shape to 2.5D sketches; the framework is therefore end-to-end trainable on real images, requiring no human annotations. Our model achieves state-of-the-art performance on 3D shape reconstruction.
arXiv:1711.03129v1 fatcat:7nex5ycjmjhlvagd257okg64zq

Real-time Localized Photorealistic Video Style Transfer [article]

Xide Xia, Tianfan Xue, Wei-sheng Lai, Zheng Sun, Abby Chang, Brian Kulis, Jiawen Chen
2020 arXiv   pre-print
We present a novel algorithm for transferring artistic styles of semantically meaningful local regions of an image onto local regions of a target video while preserving its photorealism. Local regions may be selected either fully automatically from an image, through using video segmentation algorithms, or from casual user guidance such as scribbles. Our method, based on a deep neural network architecture inspired by recent work in photorealistic style transfer, is real-time and works on
more » ... y inputs without runtime optimization once trained on a diverse dataset of artistic styles. By augmenting our video dataset with noisy semantic labels and jointly optimizing over style, content, mask, and temporal losses, our method can cope with a variety of imperfections in the input and produce temporally coherent videos without visual artifacts. We demonstrate our method on a variety of style images and target videos, including the ability to transfer different styles onto multiple objects simultaneously, and smoothly transition between styles in time.
arXiv:2010.10056v1 fatcat:n2pv3dtg3zfifnnj6tm3mge6w4

Stereoscopic Dark Flash for Low-light Photography [article]

Jian Wang, Tianfan Xue, Jonathan T. Barron, Jiawen Chen
2019 arXiv   pre-print
Tianfan Xue is a researcher at Google, working on computational photography, computer vision, and machine learning. He received his Ph.D. in EECS from MIT, working with William T. Freeman.  ... 
arXiv:1901.01370v2 fatcat:z2xkud4kufge3l3lqeievegfge

Learned Dual-View Reflection Removal [article]

Simon Niklaus and Xuaner Cecilia Zhang and Jonathan T. Barron and Neal Wadhwa and Rahul Garg and Feng Liu and Tianfan Xue
2020 arXiv   pre-print
Traditional reflection removal algorithms either use a single image as input, which suffers from intrinsic ambiguities, or use multiple images from a moving camera, which is inconvenient for users. We instead propose a learning-based dereflection algorithm that uses stereo images as input. This is an effective trade-off between the two extremes: the parallax between two views provides cues to remove reflections, and two views are easy to capture due to the adoption of stereo cameras in
more » ... es. Our model consists of a learning-based reflection-invariant flow model for dual-view registration, and a learned synthesis model for combining aligned image pairs. Because no dataset for dual-view reflection removal exists, we render a synthetic dataset of dual-views with and without reflections for use in training. Our evaluation on an additional real-world dataset of stereo pairs shows that our algorithm outperforms existing single-image and multi-image dereflection approaches.
arXiv:2010.00702v1 fatcat:wgosnumnxre5jmiio76jriycyu

Best-Buddies Similarity - Robust Template Matching using Mutual Nearest Neighbors [article]

Shaul Oron, Tali Dekel, Tianfan Xue, William T. Freeman, Shai Avidan
2016 arXiv   pre-print
We propose a novel method for template matching in unconstrained environments. Its essence is the Best-Buddies Similarity (BBS), a useful, robust, and parameter-free similarity measure between two sets of points. BBS is based on counting the number of Best-Buddies Pairs (BBPs)--pairs of points in source and target sets, where each point is the nearest neighbor of the other. BBS has several key features that make it robust against complex geometric deformations and high levels of outliers, such
more » ... s those arising from background clutter and occlusions. We study these properties, provide a statistical analysis that justifies them, and demonstrate the consistent success of BBS on a challenging real-world dataset while using different types of features.
arXiv:1609.01571v1 fatcat:s7yu4h3qpzgxbklaf533fe6iva

Symmetric piecewise planar object reconstruction from a single image

Tianfan Xue, Jianzhuang Liu, Xiaoou Tang
2011 CVPR 2011  
Recovering 3D geometry from a single view of an object is an important and challenging problem in computer vision. Previous methods mainly focus on one specific class of objects without large topological changes, such as cars, faces, or human bodies. In this paper, we propose a novel single view reconstruction algorithm for symmetric piecewise planar objects that are not restricted to some object classes. Symmetry is ubiquitous in manmade and natural objects and provides rich information for 3D
more » ... reconstruction. Given a single view of a symmetric piecewise planar object, we first find out all the symmetric line pairs. The geometric properties of symmetric objects are used to narrow down the searching space. Then, based on the symmetric lines, a depth map is recovered through a Markov random field. Experimental results show that our algorithm can efficiently recover the 3D shapes of different objects with significant topological variations.
doi:10.1109/cvpr.2011.5995405 dblp:conf/cvpr/XueLT11 fatcat:jv7wgizytvg7dke7vbpnixusmi

A computational approach for obstruction-free photography

Tianfan Xue, Michael Rubinstein, Ce Liu, William T. Freeman
2015 ACM Transactions on Graphics  
Tianfan Xue is supported by Shell Research and ONR MURI 6923196.  ... 
doi:10.1145/2766940 fatcat:dodui3xpqjgj3cd2mujipoeski

Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling [article]

Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B. Tenenbaum, William T. Freeman
2018 arXiv   pre-print
We study 3D shape modeling from a single image and make contributions to it in three aspects. First, we present Pix3D, a large-scale benchmark of diverse image-shape pairs with pixel-level 2D-3D alignment. Pix3D has wide applications in shape-related tasks including reconstruction, retrieval, viewpoint estimation, etc. Building such a large-scale dataset, however, is highly challenging; existing datasets either contain only synthetic data, or lack precise alignment between 2D images and 3D
more » ... s, or only have a small number of images. Second, we calibrate the evaluation criteria for 3D shape reconstruction through behavioral studies, and use them to objectively and systematically benchmark cutting-edge reconstruction algorithms on Pix3D. Third, we design a novel model that simultaneously performs 3D reconstruction and pose estimation; our multi-task learning approach achieves state-of-the-art performance on both tasks.
arXiv:1804.04610v1 fatcat:wirkhr4ppfhlxlrv6aoa57bjvm

Example-based 3D object reconstruction from line drawings

Tianfan Xue, Jianzhuang Liu, Xiaoou Tang
2012 2012 IEEE Conference on Computer Vision and Pattern Recognition  
Recovering 3D geometry from a single 2D line drawing is an important and challenging problem in computer vision. It has wide applications in interactive 3D modeling from images, computer-aided design, and 3D object retrieval. Previous methods of 3D reconstruction from line drawings are mainly based on a set of heuristic rules. They are not robust to sketch errors and often fail for objects that do not satisfy the rules. In this paper, we propose a novel approach, called example-based 3D object
more » ... econstruction from line drawings, which is based on the observation that a natural or man-made complex 3D object normally consists of a set of basic 3D objects. Given a line drawing, a graphical model is built where each node denotes a basic object whose candidates are from a 3D model (example) database. The 3D reconstruction is solved using a maximum-a-posteriori (MAP) estimation such that the reconstructed result best fits the line drawing. Our experiments show that this approach achieves much better reconstruction accuracy and are more robust to imperfect line drawings than previous methods.
doi:10.1109/cvpr.2012.6247689 dblp:conf/cvpr/XueLT12 fatcat:5c3mniqap5ebtf22g2farhviga

Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image [article]

Shumian Xin, Neal Wadhwa, Tianfan Xue, Jonathan T. Barron, Pratul P. Srinivasan, Jiawen Chen, Ioannis Gkioulekas, Rahul Garg
2021 arXiv   pre-print
We present a method that takes as input a single dual-pixel image, and simultaneously estimates the image's defocus map -- the amount of defocus blur at each pixel -- and recovers an all-in-focus image. Our method is inspired from recent works that leverage the dual-pixel sensors available in many consumer cameras to assist with autofocus, and use them for recovery of defocus maps or all-in-focus images. These prior works have solved the two recovery problems independently of each other, and
more » ... en require large labeled datasets for supervised training. By contrast, we show that it is beneficial to treat these two closely-connected problems simultaneously. To this end, we set up an optimization problem that, by carefully modeling the optics of dual-pixel images, jointly solves both problems. We use data captured with a consumer smartphone camera to demonstrate that, after a one-time calibration step, our approach improves upon prior works for both defocus map estimation and blur removal, despite being entirely unsupervised.
arXiv:2110.05655v1 fatcat:gh4vmrmhwna2tem2ww5nxdmxni

The aperture problem for refractive motion

Tianfan Xue, Hossein Mobahi, Fredo Durand, William T. Freeman
2015 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)  
Xue et al. [31] estimate the fluid motion by tracking the distortion of the observed sequence over time. Alterman [7] proposed to recover turbulence strength field using linear tomography.  ... 
doi:10.1109/cvpr.2015.7298960 dblp:conf/cvpr/XueMDF15 fatcat:hmuv3jpmijayxccbvyondb3jky

MoSculp: Interactive Visualization of Shape and Time [article]

Xiuming Zhang, Tali Dekel, Tianfan Xue, Andrew Owens, Qiurui He, Jiajun Wu, Stefanie Mueller, William T. Freeman
2018 arXiv   pre-print
We present a system that allows users to visualize complex human motion via 3D motion sculptures---a representation that conveys the 3D structure swept by a human body as it moves through space. Given an input video, our system computes the motion sculptures and provides a user interface for rendering it in different styles, including the options to insert the sculpture back into the original video, render it in a synthetic scene or physically print it. To provide this end-to-end workflow, we
more » ... troduce an algorithm that estimates that human's 3D geometry over time from a set of 2D images and develop a 3D-aware image-based rendering approach that embeds the sculpture back into the scene. By automating the process, our system takes motion sculpture creation out of the realm of professional artists, and makes it applicable to a wide range of existing video material. By providing viewers with 3D information, motion sculptures reveal space-time motion information that is difficult to perceive with the naked eye, and allow viewers to interpret how different parts of the object interact over time. We validate the effectiveness of this approach with user studies, finding that our motion sculpture visualizations are significantly more informative about motion than existing stroboscopic and space-time visualization methods.
arXiv:1809.05491v1 fatcat:2fkf32opvvfgngxe3vm5jqs3v4
« Previous Showing results 1 — 15 out of 69 results