Cognitive communications have emerged as a promising solution to enhance, adapt, and invent new tools and capabilities that transcend conventional wireless networks. Deep learning (DL) is critical in enabling essential features of cognitive systems because of its fast prediction performance, adaptive behavior, and model-free structure. These features are especially significant for multi-antenna wireless communications systems, which generate and handle massive data. Multiple antennas may

more »

... multiplexing, diversity, or antenna gains that, respectively, improve the capacity, bit error rate, or the signal-to-interference-plus-noise ratio. In practice, multi-antenna cognitive communications encounter challenges in terms of data complexity and diversity, hardware complexity, and wireless channel dynamics. DL solutions such as federated learning, transfer learning and online learning, tackle these problems at various stages of communications processing, including multi-channel estimation, hybrid beamforming, user localization, and sparse array design. This article provides a synopsis of various DL-based methods to impart cognitive behavior to multi-antenna wireless communications for improved robustness and adaptation to the environmental changes while providing satisfactory spectral efficiency and computation times. We discuss DL design challenges from the perspective of data, learning, and transceiver architectures. In particular, we suggest quantized learning models, data/model parallelization, and distributed learning methods to address the aforementioned challenges.