The propagation properties of optical fibers can be significantly influenced by intrinsic stress. These effects are often undesired but in some cases essential for certain applications, e.g. in polarization maintaining (PM) fibers. In this paper, we present systematic studies on the influence of the fiber drawing process on the generated stress and demonstrate an approach to significantly increase the stress induced birefringence of PM-fibers. It is shown that the thermal stress caused by the material composition is superimposed with the mechanical stress caused by the fiber fabrication process. This intrinsic stress has a strong effect on the optical and mechanical properties of the glass and thus influences the fiber stability and modal behavior. By applying a thermal annealing step, the mechanical stress due to the fiber drawing process can be canceled. It is shown that this annealing step compensates the stress reducing influence of the drawing process on the birefringence of PM-fibers with panda structure. The comparison of the intrinsic stress states after fabrication with the state after the additional high temperature annealing step clearly shows that it is possible to improve the overall birefringence of panda fibers using appropriate preparation steps.