Asignificant challenge in the operation of alkali vapor cells is the unwanted obstruction of their optical windows by films of alkali metal or remains fromcell fabrication.We present a method for cleaning those windows using pulsed femtosecond laser rear-side ablation. The effectiveness of this cleaning process is demonstrated by comparing the optical transmission of microfabricated Cs vapor cells before and after treatment. We confirm the preserved hermeticity of the cells by analyzing the spectral position of the Cs D1 absorption line. By eliminating the need for separate alkali vapor filling structures, this method enables more compact cell geometries, increasing the yield of microfabricated cells per wafer batch and enabling further miniaturization of quantum sensors and devices. Laser rear-side ablation at pulse durations from femtoseconds to nanoseconds is discussed briefly in the context of the ablation thresholds of common glasses and metals.