Based on the fact that most of the algorithms assume that the camera is fixed and the changing background is learned in the training period, a robust algorithm is proposed for complex background where a shaking camera, changing background and shadows are presented. It combines a new improved mixture of Gaussians model and a square neighborhood matching algorithm to eliminate shadows and reduce false positive detections caused by camera motion and changing background. Experiments results demonstrate the efficiency and accuracy of this algorithm.