<div>Abstract—In this research, we have proposed a new scheme to detect and extract the activity of an unknown smooth template in presence of white Gaussian noise with unknown variance. In this regard, the problem is considered a binary hypothesis test, and it is solved employing the generalized likelihood ratio (GLR) method. GLR test uses the maximum likelihood (ML) estimation of unknown parameters under each hypothesis. The ML estimation of the desired signal yields an optimization problem

more »

... h smoothness constraint which is in the form of a conventional least square error estimation problem and can be solved optimally. The proposed detection scheme is studied for P300 elicitation from the background electroencephalography signal. In addition, to assume the P300 smoothness, two prior knowledge are considered in terms of positivity and approximate occurrence time of P300. The performance of the method is assessed on both real and synthetic datasets in different noise levels and compared to a conventional signal detection scheme without considering smoothness priors, as well as state-of-theart linear and quadratic discriminant analysis. The results are illustrated in terms of detection probability, false alarm rate, and accuracy. The proposed method outperforms the counterparts in low signal-to-noise ratio situations.</div>