We carefully examined the fluorescence properties of Yb-doped fibers before and after pump-induced photodarkening (PD) and found a significant reduction of Yb3+ fluorescence intensity and lifetime due to the degradation process, obviously correlated with the measured PD loss. To explain these results, a non-radiative energy transfer from excited Yb3+ ions to the atomic defects (color centers) in close vicinity of the Yb3+ ions is suggested and discussed quantitatively as a function of PD loss generated in fibers with different Yb concentrations, by various pump powers or at different photodarkening states. All modifications of fluorescence properties proved to be reversed by thermal bleaching of the PD-induced color centers.