Abstract: To ensure the safety of railway transport and prevent pedestrians, livestock, wild animals, and other objects from invading the railway, this paper proposed a method of using deep learning technology to detect railway foreign object intrusion on monitoring video along the railway. In response to the reality of the lack and difficulty in collecting image dataset for railway foreign object intrusion limits, the paper constructed a dedicated foreign object intrusion limit dataset for railway scenes through various means, and introduced various data augmentation techniques to expand the dataset. This not only enhanced the diversity of the samples, but also effectively avoided overfitting during the training stage; The paper focused on the particularity of railway scenes and made some adaptive improvements to the YOLO (You Only Look Once) v5 deep learning model structure. It was used as a railway foreign object intrusion detection model and trained and tested on self-made dataset samples. The test results show that the detection accuracy of this model reaches over 88%, and it can be used for detecting foreign object intrusion in railway sites.