Abstract: To overcome the disadvantages in the efficiency and applicability of the traditional shielding optimization method based on the Monte Carlo method, in this paper, we studied the reactor radiation shielding optimization method by the non-dominated sorting genetic algorithm (NSGA-Ⅱ) based on the elitist strategy and the mini-batch gradient descent (MBGD), and improved the adaptive mutation rate operator of the genetic algorithm to enhance the global optimization ability of the genetic algorithm. A multi-objective optimization model of the reactor secondary shielding is constructed for comparing the output of normalized neutron transmittance between Monte Carlo method and neural network prediction method, which has verified the accuracy of MBGD. Through the coupling of neural network and NSGA-Ⅱ algorithm, the Pareto front of the radiation shielding design model can be found quickly, which can be applied to the multi-objective optimization engineering design of reactor radiation shielding.