Polyakov loop eigenvalues and their N-dependence are studied in 2 and 4 dimensional SU(N) YM theory. The connected correlation function of the single eigenvalue distributions of two separated Polyakov loops in 2D YM is calculated and is found to have a structure differing from the one of corresponding hermitian random matrix ensembles. No large N non-analyticities are found for two point functions in the confining regime. Suggestions are made for situations in which large-N phase transitions involving Polyakov loops might occur.