Visual content-based product retrieval has become increasingly important for e-commerce. Fashion retrieval, in particular, is a challenging problem owing to a wide range of visual distortions in their product images. In this paper, we propose a Grid Search Network (GSN) for learning feature embeddings for fashion retrieval. The proposed approach posits the training procedure as a search problem, focused on locating matches for a reference query image in a grid containing both positive and

more »

... ve images w.r.t the query. The proposed framework significantly outperforms existing state-of-the-art methods on benchmark fashion datasets. We also utilize a reinforcement learning based strategy to learn a specialized transformation function which further improves retrieval performance when applied over the feature embeddings. We also extend the reinforcement learning based strategy to learn custom kernel functions for SVM based classification over FashionM-NIST and MNIST datasets, showing improved performance. We highlight the generalization capabilities of this search strategy by showing performance improvement in domains beyond fashion.