1.School of Electrical Engineering, Southeast University, Nanjing 210096, China;2.Suzhou Power Supply Company of State Grid Jiangsu Electric Power Co., Ltd., Suzhou 215004, China

To achieve the maximum benefit on the demand side, a hierarchical energy scheduling method based on deep reinforcement learning (DRL) that can cope with complex environments is proposed. Firstly, a two-tier framework for the home energy management system (HEMS) is established. By changing the charging and discharging power of the second-tier energy storage system, the power over-limit of the first-tier is solved caused by the concentration of the load to the low-electricity-price period for meeting the power demand of users and reducing the electricity bill. Then, electrical appliances are classified and modeled according to their load characteristics. Secondly, Markov decision process (MDP) is used to model the energy management problem. The reward function is employed to replace objective functions and constraints. Moreover, Rainbow algorithm is introduced to optimize the strategy with the goal of maximizing the long-term benefits and achieving online scheduling economically and efficiently. Finally, a simulation is performed on a residential house, which includes solar panels, an energy storage system, multiple electrical appliances, and an electric vehicle, to verify the effectiveness and superiority of the proposed method in dealing with the uncertain problems.