In this note we consider several maps that occur naturally between modular Shimura varieties, Hilbert-Blumenthal varieties and the moduli spaces of polarized abelian varieties when forgetting certain endomorphism structures. We prove that, up to birational equivalences, these forgetful maps coincide with the natural projection by suitable abelian groups of Atkin-Lehner involutions.