Sample size planning for classification models

in: Analytica Chimica Acta (2013)
Beleites, Claudia; Neugebauer, Ute; Bocklitz, Thomas W.; Krafft, Christoph; Popp, Jürgen
In biospectroscopy, suitably annotated and statistically independent samples (e. g. patients, batches, etc.) for classifier training and testing are scarce and costly. Learning curves show the model performance as function of the training sample size and can help to determine the sample size needed to train good classifiers. However, building a good model is actually not enough: the performance must also be proven. We discuss learning curves for typical small sample size situations with 5 - 25 independent samples per class. Although the classication models achieve acceptable performance, the learning curve can be completely masked by the random testing uncertainty due to the equally limited test sample size. In consequence, we determine test sample sizes necessary to achieve reasonable precision in the validation and find that 75 - 100 samples will usually be needed to test a good but not perfect classifier. Such a data set will then allow refined sample size planning on the basis of the achieved performance. We also demonstrate how to calculate necessary sample sizes in order to show the superiority of one classifier over another: this often requires hundreds of statistically independent test samples or is even theoretically impossible. We demonstrate our findings with a data set of ca. 2550 Raman spectra of single cells (ve classes erythrocytes, leukocytes and three di erent tumour cell lines) as well as by an extensive simulation that allows precise determination of the actual performance of the models in question.

Third party cookies & scripts

This site uses cookies. For optimal performance, smooth social media and promotional use, it is recommended that you agree to third party cookies and scripts. This may involve sharing information about your use of the third-party social media, advertising and analytics website.
For more information, see privacy policy and imprint.
Which cookies & scripts and the associated processing of your personal data do you agree with?

You can change your preferences anytime by visiting privacy policy.