Abstract: This paper proposed an online detection method for high-speed railway perimeter intrusion based on video sequence inference to address the issues of high modeling difficulty and small differences between sample classes caused by event temporal dependency in the detection of high-speed railway perimeter intrusion events. The paper utilized gated loop units to capture the dynamic features of video sequences, distinguished differences between different events, constructed global temporal dependencies of video sequences through Transformer encoders, understood the global context of events, and used the prediction results of Transformer decoders as supplementary information to assist in the detection of current events. The experimental results show that compared with traditional detection methods, this method can more accurately detect high-speed rail perimeter intrusion events and has promotional value.