Search

Offline reinforcement learning (RL) focuses on training policies exclusively from passively gathered datasets, establishing it as a data-driven decision-making framework. A major obstacle in offline RL is the phenomenon of distribution shift, which leads the algorithm to encounter out-of-distribution (OOD) samples. While constraining the divergence between the target policy and the behavior policy can alleviate distribution shift, this strategy can unduly restrict the target policy and degrade algorithmic performance, because it fails to explicitly differentiate between in-distribution and OOD samples. Furthermore, accurately learning and representing a multi-modal behavior policy is highly challenging when datasets are compiled from multiple distinct behavior policies. To resolve these limitations, the authors tackle the distribution shift problem through implicit policy constraints utilizing energy-based models (EBMs), bypassing the explicit modeling of the behavior policy. EBMs offer a powerful framework for characterizing complex multi-modal distributions while effectively distinguishing between in-distribution and OOD samples. Experimental outcomes demonstrate that this approach substantially surpasses explicit policy constraint methods and other baseline techniques. Furthermore, the trained energy model serves as an effective indicator of OOD encounters, providing alerts for potential failures.