In this work, we aim to synthesize images from scene description with retrieved patches as reference. We propose a differentiable retrieval module.  ...  We conduct extensive quantitative and qualitative experiments to demonstrate that the proposed method can generate realistic and diverse images, where the retrieved patches are reasonable and mutually  ...  With the proposed differentiable retrieval design, the proposed RetrieveGAN is capable of retrieving image patches that 1) considers the surrogate image generation quality, and 2) are mutually compatible  ...