TY - JOUR
T1 - Offline Reinforcement Learning with Implicit Policy Constraints and Energy-Based Modeling
AU - Daniel Kim
AU - Jihoon Park
AU - Min-seo Kang
AU - Hyun-woo Jung
JF - Journal of Trustworthy and Explainable Intelligent Systems
JO - J Trustworthy Explain Intell Syst
Y1 - 2023
VL - 2
IS - 2
SP - 34
N2 - Offline reinforcement learning (RL) focuses on training policies exclusively from passively gathered datasets, establishing it as a data-driven decision-making framework. A major obstacle in offline RL is the phenomenon of distribution shift, which leads the algorithm to encounter out-of-distribution (OOD) samples. While constraining the divergence between the target policy and the behavior policy can alleviate distribution shift, this strategy can unduly restrict the target policy and degrade algorithmic performance, because it fails to explicitly differentiate between in-distribution and OOD samples. Furthermore, accurately learning and representing a multi-modal behavior policy is highly challenging when datasets are compiled from multiple distinct behavior policies. To resolve these limitations, the authors tackle the distribution shift problem through implicit policy constraints utilizing energy-based models (EBMs), bypassing the explicit modeling of the behavior policy. EBMs offer a powerful framework for characterizing complex multi-modal distributions while effectively distinguishing between in-distribution and OOD samples. Experimental outcomes demonstrate that this approach substantially surpasses explicit policy constraint methods and other baseline techniques. Furthermore, the trained energy model serves as an effective indicator of OOD encounters, providing alerts for potential failures.
UR - https://tipublications.com/p490067642
ER -