In the underlay mode of cognitive radio, secondary users can transmit when the primary is transmitting, but under tight interference constraints, which limit the secondary system performance. Antenna selection (AS)-based multiple antenna techniques, which require less hardware and yet exploit spatial diversity, help improve the secondary system performance. In this paper, we develop the optimal transmit AS rule that minimizes the symbol error probability (SEP) of an average

more »

... ed secondary system that operates in the underlay mode. We show that the optimal rule is a non-linear function of the power gains of the channels from secondary transmit antenna to primary receiver and secondary transmit antenna to secondary receive antenna. The optimal rule is different from the several ad hoc rules that have been proposed in the literature. We also propose a closed-form, tractable variant of the optimal rule and analyze its SEP. Several results are presented to compare the performance of the closed-form rule with the ad hoc rules, and interesting inter-relationships among them are brought out.