Coordinated Optimization Method for Feeder Container Ship Route Planning and Stowage Based on DQN Algorithm

Given the unique features of feeder container shipping, including varying feeder port numbers and inconsistent berthing conditions, as well as the divers' types of container fleets, this research investigates the coordinated optimization for route planning and stowage in feeder container shipping considering their close connection in the actual transportation process. A two-stage hierarchical method is employed to study the route planning and container stowage problems. Multiple ports, different ship types with their respective bays and stack combinations, and containers of various sizes are included in the study. The fundamental relationships among these elements are established to achieve integrity and continuity of the two-stage optimization process. The first stage involves establishing a ship route planning model with the objective of minimizing the total operational cost. The second stage focuses on optimizing the stowage from the perspective of primary bay planning. The correspondence between containers and stacks is determined, and a ship stowage model is developed with the objective of minimizing the number of mixed container stacks. The stowage model ensures that the ship's stability meets the requirements throughout the route, while reducing the number of mixed stacks to improve port operation efficiency. To efficiently solve the proposed models, a Markov process corresponding to route planning and stowage decision-making is designed based on the Deep Q-learning Network (DQN) algorithm from deep reinforcement learning. The intelligent agent's state space, action space, and reward function are designed based on the problem's characteristics to construct the two-stage hierarchical DQN algorithm. Experimental results demonstrate that as the number of ships and the ship loading rate increase, the time required for accurate model solution significantly rises. Some cases cannot be solved within 600 seconds, while the DQN algorithm achieves rapid solutions in all examples. Compared with traditional models and the Particle Swarm Optimization (PSO) algorithm, the DQN algorithm efficiently solves cases of different scales. The maximum solving time for large-scale cases is 31.40 s, with an average time of less than 30 s, indicating good solution efficiency. Further calculations indicate that under different feeder port numbers, the average standard deviation of solving time for the DQN algorithm is only 1.74, showing better robustness compared to the average standard deviation of 11.20 for the PSO algorithm. Overall, the DQN algorithm exhibits less fluctuation in solving time with changing problem scales, showcasing stable solving performance and efficient optimization capabilities.