We study the behavior of the fermion determinant at finite temperature and chemical potential, as a function of the Polyakov loop. The phase of the determinant is correlated with the imaginary part of the Polyakov loop. This correlation and its consequences are considered in static QCD, in a toy model of free quarks in a constant A_0 background, and in simulations constraining the imaginary part of the Polyakov loop to zero.