In this paper, we propose a method that separates specular reflections from polarized color images. With line constraints in RGB color space and spatial domain information, we used a Variational energy minimization approach to restore diffuse reflections. With anisotropic smoothing, our method preserves the shape of textures and successfully separates specular reflections. Also, saturation of specular reflections is considered using a simple inpainting method.