The antibiotic resistance of bacterial pathogens has become one of the most serious global health issues due to misusing and overusing of antibiotics. Recently, different technologies were developed to determine bacteria susceptibility towards antibiotics; however, each of these technologies has its advantages and limitations in clinical applications. In this contribution, we aim to assess and automate the detection of bacterial susceptibilities towards three antibiotics; i.e. ciprofloxacin, cefotaxime and piperacillin using a combination of image processing and machine learning algorithms. Therein, microscopic images were collected from different E. coli strains, then the convolutional neural network U-Net was implemented to segment the areas showing bacteria. Subsequently, the encoder part of the trained U-Net was utilized as a feature extractor, and the U-Net bottleneck features were utilized to predict the antibiotic susceptibility of E. coli strains using a one-class support vector machine (OCSVM). This one-class model was always trained on images of untreated controls of each bacterial strain while the image labels of treated bacteria were predicted as control or non-control images. If an image of treated bacteria is predicted as control, we assume that these bacteria resist this antibiotic. In contrast, the sensitive bacteria show different morphology of the control bacteria; therefore, images collected from these treated bacteria are expected to be classified as non-control. Our results showed 83% area under the receiver operating characteristic (ROC) curve when OCSVM models were built using the U-Net bottleneck features of control bacteria images only. Additionally, the mean sensitivities of these one-class models are 91.67% and 86.61% for cefotaxime and piperacillin; respectively. The mean sensitivity for the prediction of ciprofloxacin is only 59.72% as the bacteria morphology was not fully detected by the proposed method.