Trouble of moving Freight car Detection System (TFDS) constitutes an important part of the railway safety warning systems. With TFDS, dynamic images of freight cars are captured by high-speed cameras, timely transmitted to the train inspection center by special optical fiber network and finally observed for fault recognition. In this paper, an automatic fault recognition method for side frame key in TFDS is proposed based on open source computer vision library (OpenCV) to overcome the disadvantages of manual fault recognition. At first, image preprocessing and segmentation are applied to eliminate the impact of the surrounding environment and further simplify images. Then the axle and through-hole are calibrated through Hough circle transformation and the side frame key is located indirectly according to the geometric relationship among the axle, through-hole and side frame key. Finally, the difference of mean gray values in the region of interest (ROI) is analyzed to judge whether the side frame key is missing or not. Featured by the high efficiency, reliability and practicability, the proposed method lays the foundation for engineering applications in automatic fault recognition of freight cars.