This paper investigates the security performance of two relay selection schemes for cooperative non-orthogonal multiple access (NOMA) systems, where K randomly distributed relays are employed with either decode-and-forward (DF) or amplify-and-forward (AF) protocols. More particularly, two-stage relay selection (TRS) and optimal relay selection (ORS) schemes are taken into consideration. To characterize the secrecy behaviors of these RS schemes, new closed-form expressions of both exact and

more »

... totic secrecy outage probability (SOP) are derived. We confirm that the SOP of the TRS scheme is equal to that of the ORS scheme for DF/AF-based NOMA systems. Based on the analytical results, the secrecy diversity orders achieved by the pair of RS schemes for the DF/AF-based NOMA systems are K , which are equal to the number of relays. It is shown that the secrecy diversity orders for the cooperative NOMA systems are determined by the number of the relays. The numerical results are presented to demonstrate that: 1) the secrecy performance of the AF-based NOMA system outperforms that of the DF-based NOMA system, when not all DF relays successfully decode the received information; 2) with the number of relays increasing, the SOP of these RS schemes for the DF-/AF-based NOMA systems becomes lower, and; 3) the TRS/ORS schemes are capable of achieving better secrecy outage behaviors compared with random RS and orthogonal multiple access-based RS schemes. INDEX TERMS Amplify-and-forward, decode-and-forward, non-orthogonal multiple access, physical layer security, relay selection. The associate editor coordinating the review of this manuscript and approving it for publication was Yuanwei Liu. multiple users through appropriate power allocation scheduling [9], [10]. NOMA has been extended to cooperative communications [11], [12] , where the users with better channel conditions were selected as decode-and-forward (DF) relays to deliver information and improve transmission reliability of users with poor channel conditions. Inspired by this, the authors of [13] have analyzed the outage performance of DF-based NOMA system with full-duplex (FD) and halfduplex (HD) scenarios, respectively. From the perspective of enhancing spectrum efficiency and energy efficiency, simultaneous wireless information and power transfer has been applied to cooperative NOMA [14] , [15] , in which the user relays harvest energy from base station (BS). Specifically, in [15] , the authors comprehensively analyzed the system performance under considering DF and amplify-and-forward (AF) protocols. On the other hand, cooperative NOMA schemes with dedicated relays have been widely investigated. The cooperative NOMA systems with AF relays are obviously superior to cooperative OMA in terms of coding gain, outage performance and system throughput in [16]-[18]. 86274 This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/ VOLUME 7, 2019 School of Communication and Information Engineering, Shanghai University. Her research interests include 5G networks, the Internet of Things, cooperative networks, and non-orthogonal multiple access. ZHANGYOU PENG received the M.S. degree in communication engineering from Tsinghua University, in 1992, and the Ph.D. degree from Shanghai University, in 2009, where he has been a Professor with the School of Communication and Information Engineering, since 2002. His research interests include satellite and radar signal processing, weak signal processing, and wireless communication.