Radiation therapy (RT) is widely used for cancer treatment but is found with side effects of radiation dermatitis and fibrosis thereby calling for timely assessment. Nevertheless, current clinical assessment methods are found to be subjective, prone to bias, and accompanied by variability. There is, therefore, an unmet clinical need to explore a new assessment technique, ideally portable and affordable, making it accessible to less developed regions too. We developed an affordable (16764 CNY) and portable high-resolution ((3.91 µm) darkfield polarization-sensitive multispectral imaging (PS-MSI) microscope. The implementation of the Monte Carlo simulation on the PS multi spectra allows the quantitative analysis of physiological parameters (i.e., blood volume fraction (BVF) and oxygen saturation of hemoglobin) at different skin layers for the dermatitis assessment. Further derivation of the degree of linear polarization (DOLP) reflects randomly distributed collagen fibers associated with fibrosis for the fibrosis assessment. PS-MSI microscope developed revealed a significant decrease (p < 0.001, analysis of variance, ANOVA) in the DOLP associated with fibrosis like scar tissue, and significant (p < 0.001, ANOVA) increases in BVF and oxygen saturation of hemoglobin accompanying artificially induced dermatitis. One-dimensional convolutional neural network implemented on the DOLP and multiple spectra achieved accuracies of 96% and 92.2%, respectively, for the classification of the artificially induced skin dermatitis and fibrosis like scar, demonstrating the potential of the affordable PS-MSI microscope developed for objective, unbiased and consistent assessment of radiation dermatitis and fibrosis in the clinics.