TY - GEN
T1 - Diversity-Based Trajectory and Goal Selection with Hindsight Experience Replay
AU - Dai, Tianhong
AU - Liu, Hengyan
AU - Arulkumaran, Kai
AU - Ren, Guangyu
AU - Bharath, Anil Anthony
N1 - Publisher Copyright:
© 2021, Springer Nature Switzerland AG.
PY - 2021
Y1 - 2021
N2 - Hindsight experience replay (HER) is a goal relabelling technique typically used with off-policy deep reinforcement learning algorithms to solve goal-oriented tasks; it is well suited to robotic manipulation tasks that deliver only sparse rewards. In HER, both trajectories and transitions are sampled uniformly for training. However, not all of the agent’s experiences contribute equally to training, and so naive uniform sampling may lead to inefficient learning. In this paper, we propose diversity-based trajectory and goal selection with HER (DTGSH). Firstly, trajectories are sampled according to the diversity of the goal states as modelled by determinantal point processes (DPPs). Secondly, transitions with diverse goal states are selected from the trajectories by using k-DPPs. We evaluate DTGSH on five challenging robotic manipulation tasks in simulated robot environments, where we show that our method can learn more quickly and reach higher performance than other state-of-the-art approaches on all tasks.
AB - Hindsight experience replay (HER) is a goal relabelling technique typically used with off-policy deep reinforcement learning algorithms to solve goal-oriented tasks; it is well suited to robotic manipulation tasks that deliver only sparse rewards. In HER, both trajectories and transitions are sampled uniformly for training. However, not all of the agent’s experiences contribute equally to training, and so naive uniform sampling may lead to inefficient learning. In this paper, we propose diversity-based trajectory and goal selection with HER (DTGSH). Firstly, trajectories are sampled according to the diversity of the goal states as modelled by determinantal point processes (DPPs). Secondly, transitions with diverse goal states are selected from the trajectories by using k-DPPs. We evaluate DTGSH on five challenging robotic manipulation tasks in simulated robot environments, where we show that our method can learn more quickly and reach higher performance than other state-of-the-art approaches on all tasks.
KW - Deep reinforcement learning
KW - Determinantal point processes
KW - Hindsight experience replay
UR - http://www.scopus.com/inward/record.url?scp=85119277258&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-89370-5_3
DO - 10.1007/978-3-030-89370-5_3
M3 - Conference Proceeding
AN - SCOPUS:85119277258
SN - 9783030893699
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 32
EP - 45
BT - PRICAI 2021
A2 - Pham, Duc Nghia
A2 - Theeramunkong, Thanaruk
A2 - Governatori, Guido
A2 - Liu, Fenrong
PB - Springer Science and Business Media Deutschland GmbH
T2 - 18th Pacific Rim International Conference on Artificial Intelligence, PRICAI 2021
Y2 - 8 November 2021 through 12 November 2021
ER -