Abstract: This article presented an automatic recognition method to identify the fault of railway truck brake shoe key, based on the technology of digital image analysis of Trouble of moving Freight car Detection System(TFDS). Regional localization for the original image, histogram equalization and image de-noising were taken in image pretreatment.Through the histograms of oriented gradients to extract the feature vector of the brake shoe bolt, the extract features vector was imputed to the trained classifier linearly separable support vector machine (SVM) model for fault diagnosis and automatic recognition. Experimental results demonstrated that the proposed algorithms of automatic fault recognizing for brake shoe key were available, stability and reliability.