Using video-microscopy imaging we demonstrate the existence of a short-ranged equilibrium attraction between heavy silica colloids diffusing on soft surfaces of cross-linked polymer gels. The inter-colloid potential can be tuned by changing the gel stiffness or by coating the colloids with a polymer layer. On sufficiently soft substrates, the interaction induced by the polymer matrix leads to large-scale colloidal aggregation. We correlate the in-plane interaction with a colloid-surface attraction.