Abstract: The traditional fault diagnostic methods such as principal component analysis and BP neural network are with poor generalization capability and low fault identification accuracy in complex nonlinear systems. The isolation forest (iForest) algorithm uses the idea of isolation tree partition to identify the abnormal data, which is applicable to the state monitoring of nonlinear systems. The Adaboost algorithm is a boosting algorithm based on the idea of combined classification, and the overall algorithm has better generalization capability through the superposition of multiple weak classifiers. Therefore, the isolation forest algorithm and Adaboost algorithm are used in this paper to establish the iForest-Adaboost primary loop fault diagnostic system for nuclear power plants, and the GSE real-time simulation platform and the simulation data in unit 1 of Fuqing Nuclear Power plant are used for the test. The test results show that the isolation forest algorithm can identify the system anomalies faster than the principal component analysis and QTA threshold algorithm. The Adaboost algorithm has a higher fault identification accuracy than BP neural network and support vector machine algorithm.