Theory of four-wave mixing for bound and leaky modes

We present a comprehensive theory of four-wave mixing in waveguide geometries, providing a rigorous description of the dynamics of both bound as well as leaky modes within a single theoretical framework. Our approach is based on the resonant-state expansion including analytical mode normalization. For bound modes, our approach agrees with the previous theory, while it results in modified nonlinear pulse propagation for leaky modes. For instance, it predicts a more efficient generation of Stokes and anti-Stokes bands with an earlier onset than expected from the previous theory for bound modes. These effects have been demonstrated numerically for a gas-filled hollow-core annulus fiber that supports leaky modes, rendering conventional bound-mode theory inappropriate for such systems. Moreover, we find that leaky modes provide modulation instability, not only in the anomalous but also in the normal dispersion regime. The modulation instability can occur for all frequencies, which is a fundamental difference to bound modes.