Abstract: In order to strengthen the safety management of large-scale construction machinery in railway engineering and help construction management units achieve overall control of large-scale construction machinery on construction sites, this paper designed a large-scale construction machinery supervision system, introduced its overall architecture, and focused on the design of an intelligent video analysis model. This model was based on the YOLOv6 model, combined with various deep learning techniques such as transfer learning, imbalanced learning, and data augmentation, to implement rapid positioning and classification of large-scale railway construction machinery. The average macro accuracy of the model can reach 94.0%, mAP can reach 0.956, and the detection frame rate per second can reach 84. Its accuracy and real-time performance meet practical application requirements.