Thanks to its noninvasive, nondestructive, and label-free features, Raman spectroscopy has gained intensive attention in biomedical applications over the last two decades. In combination with machine learning techniques, the rich molecular fingerprints contained in Raman spectra bring huge possibilities to early diagnosis and deeper understanding of biomedical processes. To bring it to real-world applications, however, remains a challenge mainly because spectral variations of interest are easily overwhelmed by replicate-to-replicate, patient-to-patient, and device-to-device variations. Machine learning models that are robust to these variations are in definite need. To fill this gap, we developed an approach based on graph neural networks (GNNs), which represent Raman spectra with graphs and achieve classification at the graph level. The method was tested on four Raman datasets, three from single-cell bacteria of four species measured on three devices, and one from oil of eight types. The GNN demonstrated robustness to unwanted variations and achieved promising across-device transferability, with significantly improved balanced accuracy compared to both linear discriminant analysis (LDA) and RamanNet. In addition, graph representation not only brings intrinsic advantage of model interpretation but also provides a new dimension that supports better spectral distinguishment. All the results and features endorsed the proposed method to highly benefit Raman-based biomedical applications.