A framework of hierarchical deep Q-network for portfolio management

Reinforcement Learning algorithms and Neural Networks have diverse applications in many domains, e.g., stock market prediction, facial recognition and automatic machine translation. The concept of modeling the portfolio management through a reinforcement learning formulation is novel, and the Deep Q-Network has been successfully applied to portfolio management recently. However, the model does not take into account of commission fee for transaction. This paper introduces a framework, based on the hierarchical Deep Q-Network, that addresses the issue of zero commission fee by reducing the number of assets assigned to each Deep Q-Network and dividing the total portfolio value into smaller parts. Furthermore, this framework is flexible enough to handle an arbitrary number of assets. In our experiments, the time series of four stocks for three different time periods are used to assess the efficacy of our model. It is found that our hierarchical Deep Q-Network based strategy outperforms ten other strategies, including nine traditional strategies and one reinforcement learning strategy, in profitability as measured by the Cumulative Rate of Return. Moreover, the Sharpe ratio and Max Drawdown metrics both demonstrate that the risk of policy associated with hierarchical Deep Q-Network is the lowest among all ten strategies.