Abstract: To address the problems of too many errors and omissions, high labor intensity and high cost in manual inspection, an aided inspection system for railway based on deep learning and VR remote control technology is designed. With the combination of open source datasets and customized datasets, the inspection system can identify the position and category of foreign objects reliably by adopting deep learning algorithms and uses server push stream technology to interact with the terminals to achieve control over remote devices. And VR + mobile App technology is used on the client to achieve immersive visual effects. The results of actual tests prove that the system is practical, economical, and accurate, providing a good solution as reference for the inspection system of railway.