This paper proposes a novel approach to detect and classify defects in fabric. Fabric defects may cause a roll of fabric to be graded as second or worse in quality checks, and may affect sales. Traditionally, fabrics are inspected by skilled workers at an inspection platform, which is a long and tiring process, and error-prone. Automated detection of defects in fabric has the potential to eliminate human errors and improve accuracy; therefore it has been an area of research over the last decade. This paper proposes a novel model to detect and classify defects in fabric by training and evaluating our model using the AITEX data set. In the proposed model, the images of fabrics are first fed into U-Net, which is a convolutional neural network (CNN), to determine whether the fabric is defect-free or not. VGG16 and random forest are then used to classify the defects in the fabrics. The training settings of the model were chosen as initial learning rate = 0.001, β1 = 0.9 and β2 =.999. The proposed approach achieved accuracy of 99.3% to detect defects, with high accuracy to classify sloughed filling (100%), broken pick (97%), broken yarn (80%) and fuzzy ball (74%), but low for nep (12.5%) and cut selvage (0%).