In contrast to Yb/Al-doped fibers, the influence of very low Tm2O3 concentrations (≥ 0.1 mol-ppm) on photodarkening (PD) is clearly detectable in Yb/P-doped fibers that are known to show little degradation effects. For Tm2O3 additions of more than 50 mol-ppm, the measured PD loss is even similar to Yb/Al-doped fibers with comparable rare earth concentrations. Our work reveals the risk of color center generation by pumping at wavelengths of 915 nm or 976 nm even in Al-free Yb-doped fibers and emphasizes the importance of high purity of raw materials for the preparation of Yb laser fibers with expected very low PD.