After a critical review of the established techniques for Data Parallel training from both a distributed computing and deep learning perspective, a novel approach based on nearest-neighbour communications  ...  Moreover, distributed training of deep learning models will happen more and more outside the well-known borders of cloud and HPC infrastructure and will move to edge and mobile platforms.  ...  Galeazzi, "Hpc4ai, an ai-on- demand federated platform endeavour", in ACM Computing Frontiers, Ischia, Italy, May 2018 Conference on Parallel Distributed and network-based Processing (PDP), Cambridge  ...