Search

Offline reinforcement learning (RL) focuses on training policies exclusively from passively gathered datasets, establishing it as a data-driven decision-making framework. A major obstacle in offline RL is the phenomenon of distribution shift, which leads the algorithm to encounter out-of-distribution (OOD) samples. While constraining the divergence between the target policy and the behavior policy can alleviate distribution shift, this strategy can unduly restrict the target policy and degrade algorithmic performance, because it fails to explicitly differentiate between in-distribution and OOD samples. Furthermore, accurately learning and representing a multi-modal behavior policy is highly challenging when datasets are compiled from multiple distinct behavior policies. To resolve these limitations, the authors tackle the distribution shift problem through implicit policy constraints utilizing energy-based models (EBMs), bypassing the explicit modeling of the behavior policy. EBMs offer a powerful framework for characterizing complex multi-modal distributions while effectively distinguishing between in-distribution and OOD samples. Experimental outcomes demonstrate that this approach substantially surpasses explicit policy constraint methods and other baseline techniques. Furthermore, the trained energy model serves as an effective indicator of OOD encounters, providing alerts for potential failures.

Laminectomy represents one of the most frequently performed posterior spinal procedures. Because the lamina is situated in close proximity to critical neural structures, any operative damage can induce profound complications, potentially culminating in paralysis. To mitigate these risks and associated adverse events, surgical robotics and ultrasonic bone scalpels have been integrated into spinal laminectomy procedures, prompting extensive academic investigation into methods for detecting bone tissue states. At present, virtually all state-recognition methodologies rely on sensor signals acquired via high-precision instrumentation integrated into the end-effectors of surgical robots. Nevertheless, previous approaches have struggled to achieve highly accurate identification of spinal bone tissue states. In an innovative departure, this study frames the identification of bone tissue states as a time-series classification problem. The LSTM-FCN classification algorithm was deployed to analyze multi-sensor fusion signals combining force and cutting depth data, thereby enabling precise categorization of the laminal bone tissue state. Furthermore, experimental validation demonstrated that this approach achieves an accuracy of 98.85% in identifying bone tissue states during porcine spinal laminectomy. Crucially, the maximum penetration depth can be restricted to within 0.6 mm, confirming its safety and viability for practical clinical application.