One of the most surprising phenomena in frustrated magnets is the order-by-disorder mechanism: thermal and/or quantum fluctuations lift a macroscopic accidental degeneracy of the classical ground state of frustrated magnets. The resulting magnetically ordered-by-(thermal/quantum)disorder state often breaks a real-space symmetry of the crystal lattice. In this talk, we use symmetry arguments and Monte Carlo simulations to investigate the effects of impurities (vacancies, chemical doping, etc.). Because inhomogeneities generically break a real-space symmetry, it couples to the order-parameter as a conjugate field. As a consequence, inhomogeneities non-perturbatively destabilize an ordered-by-disorder state at sufficiently low dimensions. As an example, we apply our theory to the J1-J2 Heisenberg model on a square lattice  and to the easy-plane pyrochlores .  M. Miranda, I. C. Almeida, E. C. Andrade, J. A. Hoyos, Phase diagram of a frustrated Heisenberg model: From disorder to order and back again, Phys. Rev. B 104, 054201 (2021).  E. C. Andrade, J. A. Hoyos, S. Rachel, and M. Vojta, Cluster-Glass Phase in Pyrochlore XY Antiferromagnets with Quenched Disorder, Phys. Rev. Lett. 120, 097204 (2018).