In this paper, a transmit antenna selection (TAS)based incremental hybrid decode-amplify-forward (IHDAF) scheme is proposed to enhance physical layer security in cooperative relay networks. Specifically, TAS is adopted at the source in order to reduce the feedback overhead. In the proposed TAS-based IHDAF scheme, the network transmits signals to the destination adaptive select direction transmission mode, AF mode or DF mode depending on the capacity of the source-relay link and source-relay

more »

... . In order to fully examine the benefits of the proposed TAS-based IHDAF scheme, we first derive its secrecy outage probability (SOP) in a closed-form expression. We then conduct asymptotic analysis on the SOP, which reveals the secrecy performance floor of the proposed TAS-based IHDAF scheme when no channel state information is available at the source. Theoretical analysis and simulation results demonstrate that the proposed TAS-based IHDAF scheme outperforms the selective decode-and-forward (SDF), the incremental decodeand-forward (IDF), and the noncooperative direction transmission (DT) schemes in terms of the SOP and effective secrecy throughout, especially when the relay is close to the destination. Furthermore, the proposed TAS-based IHDAF scheme offer a good trade-off between complexity and performance compare with using all antennas at the source. Index Terms-Physical layer security, cooperative communications, transmit antenna selection, secrecy outage probability.