We consider the problem of reconstructing the features of a weak anisotropic background potential by the trajectories of vortex dipoles in a nonlinear Gross-Pitaevskii equation. At leading order, the dynamics of vortex dipoles are given by a Hamiltonian system. If the background potential is sufficiently smooth and flat, the background can be reconstructed using ideas from the boundary and the lens rigidity problems. We prove that reconstructions are unique, derive an approximate reconstruction formula, and present numerical examples.