A digital SQUID magnetometer measures magnetic fields by counting integer magnetic flux quanta within its superconducting input loop. Although resolution is limited in comparison to analog SQUID systems, the digital SQUID is able to outrange its analog counterpart with regard to parameters such as slew rate and dynamic range. In this work we evaluate the performance of a digital SQUID based on a three-level logic. Due to this basic principle, we face a combination of two comparator grayzones leading to hysteretic behavior of the sensor that produces a "dead zone" in the signal reversal point. The dependence of the comparator threshold on design parameters is investigated by simulation studies and reconfirmed by experimental results. Furthermore, the influence of the shifted threshold on the total harmonic distortion of the sensor is analyzed. With the current digital SQUID design, we were able to reach a total dynamic range of more than 53000 flux quanta (about 16 bit) with a linearity error of about 5 bit due to the mentioned hysteretic behavior. We discuss the results of our investigations and provide guidelines to extend dynamic range and linearity for future sensor designs.