Transparent Intelligence Publishing Transparent Intelligence Publishing

Toward Robust Offline Reinforcement Learning Through Energy-Based Implicit Policy Constraints

Original Research | Open access | Published: 15 September 2023
Volume 2, article number 35, (2023) Cite this article
You have full access to this open access article.
Download PDF
, , ,
  1. Department of Robust RL and Energy-Based Models, Faculty of Engineering, National University of Colombia, Bogota, Colombia
  2. Department of Offline RL and Implicit Constraints, Faculty of Engineering, University of Chile, Santiago, Chile
  3. Department of Policy Robustness and RL, Faculty of Engineering, University of Antioquia, Medellin, Colombia
107 Accesses

Abstract

Laminectomy represents one of the most frequently performed posterior spinal procedures. Because the lamina is situated in close proximity to critical neural structures, any operative damage can induce profound complications, potentially culminating in paralysis. To mitigate these risks and associated adverse events, surgical robotics and ultrasonic bone scalpels have been integrated into spinal laminectomy procedures, prompting extensive academic investigation into methods for detecting bone tissue states. At present, virtually all state-recognition methodologies rely on sensor signals acquired via high-precision instrumentation integrated into the end-effectors of surgical robots. Nevertheless, previous approaches have struggled to achieve highly accurate identification of spinal bone tissue states. In an innovative departure, this study frames the identification of bone tissue states as a time-series classification problem. The LSTM-FCN classification algorithm was deployed to analyze multi-sensor fusion signals combining force and cutting depth data, thereby enabling precise categorization of the laminal bone tissue state. Furthermore, experimental validation demonstrated that this approach achieves an accuracy of 98.85% in identifying bone tissue states during porcine spinal laminectomy. Crucially, the maximum penetration depth can be restricted to within 0.6 mm, confirming its safety and viability for practical clinical application.

Explore related subjects
Discover the latest articles in related subjects:

Introduction

Medical and healthcare advancements have progressively extended global life expectancy [1]. Concurrently, the aging population is projected to drive an increased incidence of degenerative spinal disorders [2]. These degenerative pathologies frequently cause spinal nerve compression, manifest in conditions such as lumbar disc herniation and cervical spondylosis. Laminectomy serves as an effective surgical intervention, yet it remains one of the most hazardous operations performed on the spine [3].

Anatomically, bone tissue is structured into an outer cortical layer, an intermediate cancellous layer, and an inner cortical layer. The intrinsic danger of the procedure stems from the fact that the lamina constitutes the primary posterior wall of the spinal canal, placing it in immediate proximity to the spinal cord, cauda equina, and other vital neural structures contained within [4]. In conventional laminectomy, avoidance of laminal and neural injury depends entirely on the surgeon's clinical experience and tactile perception of the bone-cutting instrument [5]. Postoperative neurological complications—such as dural tear, nerve root damage, cauda equina syndrome, and both extradural and intradural scar tissue formation—carry severe clinical consequences, thereby imposing rigorous demands on preoperative planning and the execution of the laminectomy [6, 7].

To prevent these injuries and complications, numerous researchers have investigated methodologies for identifying bone tissue states. Wang et al. [8] employed a layer-by-layer lateral grinding technique, computing a Cross Correlation Coefficient to differentiate bone layers and cease grinding upon reaching the medial cortical bone. Fan et al. [9] evaluated milling states using longitudinal contact force, demonstrating that cancellous and cortical bone exhibit distinct normalized force profiles. Deng et al. [10] applied the Hilbert–Huang Transform to isolate force signals by capturing transverse forces during grinding, subsequently using a Support Vector Machine (SVM) to construct a feature vector classification for identifying distinct milling phases. Al-Abdullah et al. [11] developed a force-sensing artificial neural network model for bone grinding; using feed rate and spindle speed as inputs, the model predicted mean grinding forces across varying bone densities, revealing measurable differences between bone types. Dai et al. [12] utilized Wavelet Packet Transform (WPT) and Self-Organizing Feature Maps to classify and identify milling states from acoustic signals captured during operation. Similarly, Dai et al. [13] leveraged an acceleration sensor and a laser displacement sensor to capture tool and tissue vibrations, implementing Lifting Wavelet Packet Transform alongside an SVM to determine bone tissue states. Osa et al. [14] applied Recursive Least Squares to estimate cutting resistance from motor current and feed rate, using an SVM model for subsequent cutting state classification. Jiang et al. [15] introduced an analytical force model designed to ensure procedural safety, capable of monitoring cutting depth with an accuracy of \pm\text{0.1 mm} within depth zones of 0.8–1.2 mm, and \pm\text{0.2 mm} within 1.2–1.6 mm. Ji et al. [16] investigated the utility of electrical impedance signals during laminal cutting, comparing them to force signals to demonstrate that while electrical impedance offers certain functional potential, its accuracy remains inferior to force-based sensing. Li et al. [17] put forward a compensation control scheme based on a respiration-spine model to stabilize contact force; empirical testing showed a force control error of 0.223 N against a 2 N target force, with the robot hovering above the spinal surface once the laminectomy in the designated region concluded.

Currently, a significant portion of surgical robotic platforms utilize grinding drills to perform laminectomies, relying on metrics such as force, acoustics, bioelectrical impedance, and grinding current to detect bone tissue states. However, grinding drills carry an elevated risk of spinal cord trauma. Alongside conventional surgical instruments like osteotomes, rongeurs, and high-speed drills, the ultrasonic bone scalpel (UBS) has seen widespread adoption among spine surgeons globally [18-20]. Although the UBS provides greater safety relative to traditional surgical implements, it cannot entirely eliminate the risk of injury. Consequently, achieving meticulous control over the cutting process is paramount to ensuring safe operation in robot-assisted laminectomy (RAL).

During surgical procedures, the "grinding force" represents the most direct tactile signal available to the clinician. However, in patients presenting with osteoporosis, the mechanical properties of cancellous bone do not differ significantly from those observed after an ultrasonic osteotome breaches the inner cortical bone, as illustrated in Figure 1. Consequently, attempting to identify the cutting state solely through force signals introduces a high probability that cutting will prematurely halt during the transition from cortical to cancellous bone, leading to incomplete tissue removal.

Figure 1. Cutting force of cortical bone and cancellous bone in ultrasound surgery.

Figure 1. Cutting force of cortical bone and cancellous bone in ultrasound surgery.

Time-series data are ubiquitous across everyday life and industrial production, with examples including traffic streams [21], stock market trends [22], meteorological conditions [23], and ECG waveforms [24]. Similarly, force waveforms, acoustic data, and other surgical signals are generated as time series. Driven by the increasing availability of time-series datasets, investigators have developed a wide array of time-series classification algorithms [25]. These classification methodologies have found extensive utility within control systems [26, 27] and clinical medicine [28, 29]. Among the prominent techniques for time-series classification is the Nearest Neighbour (NN) classifier [30]. Notably, combining Dynamic Time Warping distance with an NN classifier has proven to be an exceptionally robust baseline approach [31]. In recent years, mirrored by the success of deep learning architectures in computer vision [32-34], researchers have designed various algorithmic structures for Natural Language Processing applications. More recently, deep neural networks have been directed toward time-series classification tasks, including the Multi-scale Convolutional Neural Network [35] and the Fully Convolutional Network (FCN) [36]. Because Long Short-Term Memory (LSTM) networks possess a strong capacity for time-series feature extraction [37], LSTM-based research has emerged as a major focal point. For time-series classification specifically, several hybrid algorithms derived from LSTM have been introduced, such as AttLSTM-CNNs [38] and LSTM-FCN [39]. Dau et al. [40] extensively evaluated and confirmed the performance of the LSTM-FCN algorithm using the University of California, Riverside benchmark dataset.

To further elevate the safety margins of spinal laminectomy, the robotic platform engineered in this study utilizes a UBS as its primary cutting tool, integrated with a high-precision force sensor and a displacement sensor mounted at the robot's end-effector. Real-time force profiles and cutting depth metrics are recorded throughout the spinal laminectomy procedure. The identification of the laminectomy state is treated as a time-series classification task, wherein fusion signals combining the force data and cutting depth measurements are processed using an LSTM-FCN model to yield highly accurate state recognition.

Materials and Methods

During the vertebral lamina cutting process, a periodic, alternating advancement and retraction feeding method was employed to mitigate thermal accumulation and prevent structural deformation of the UBS. As illustrated in Figure 2, the advancement distance is defined as D_F and the retraction distance as D_B, where D_F \ge D_B within each individual cycle. Consequently, a progressive incremental cutting depth is achieved per cycle, expressed as D = D_F - D_B. Owing to this cyclic feeding mechanism, the resultant force signal displays periodic waveform fluctuations, as depicted in Figure 1. The bone tissue state is assessed at the conclusion of each cycle, determining the execution of the subsequent feeding loop. Although the precise instant of breaching the inner cortical bone may transpire mid-cycle, this operational strategy maintains high safety and reliability because the incremental depth D is minimal and the UBS exhibits relative structural selectivity for hard tissues [41].

Figure 2. Schematic diagram of the cutting feed.

Figure 2. Schematic diagram of the cutting feed.

To prevent excessive mechanical deformation of the UBS, D_F and D_B were dynamically modified in real time based on the structural hardness of the vertebral plate, thereby modulating the cyclical cutting depth. The normalized profiles of the force and cutting depth signals are presented in Figure 3. Evaluating the gradient of the cutting depth signal reveals that D4 > D2 > D3 > D1. This indicates that higher bone density yields a smaller incremental cutting depth D and a reduced feed rate. Conversely, lower bone density results in an increased cutting depth D and an accelerated feed rate.

Figure 3. Force signal and cutting depth signal in the cutting process.

Figure 3. Force signal and cutting depth signal in the cutting process.

 

Signal acquisition and dataset generation

Fresh animal spinal columns were rigidly anchored to the experimental platform to replicate the operative environment. The laminectomy procedure was executed via a collaborative robotic manipulator (Aubo-i5, Aubo Company, China) equipped with a UBS (XD880A, ShuiMuTianPeng Company, China) and an internally developed cyclic-feed end-effector. A force sensor (Allison T1051, 1600 Hz) and a displacement sensor (E3RA, 200 Hz) recorded the force profiles and cutting depth signals, respectively. Throughout the cutting procedure, the end-effector feed velocity was maintained at 4 mm/s, with the operational parameters of the XD880A configured as detailed in Table 1.

Table 1. The working parameters of XD880A UBS.

Parameters

Power level

Pulse level

Flow level

Setting

5

6

6

 

A surgical blade measuring 3.7 mm in width and 0.7 mm in thickness served as the cutting instrument for the UBS, with its morphology displayed in Figure 4.

Figure 4. Ultrasonic bone scalpel blade.

Figure 4. Ultrasonic bone scalpel blade.

Under this automated robotic laminectomy framework, the cutting action must terminate precisely when the endocortical bone is on the verge of perforation. Arthroscopic visualization was employed to monitor the internal aspect of the spinal canal in real time, enabling precise verification of the exact moment of penetration. Consequently, definitive annotations for unpenetrated states (designated as 0) and penetrated states (designated as 1) were established. The experimental setup for robotic data acquisition is illustrated in Figure 5.

Figure 5. The data acquisition scenario using the developed robot.

Figure 5. The data acquisition scenario using the developed robot.

Because the force and displacement sensors operated at distinct sampling frequencies, linear interpolation was applied to the displacement dataset to normalize its frequency to 1600 Hz. The transition through the cortical bone layer was continuously reflected across both the force and displacement signals. Accordingly, a temporal windowing technique was utilized to capture 1600 data points immediately preceding the conclusion of each feed cycle. The corresponding penetration state annotation was defined based on the synchronized real-time video feed, yielding discrete data pairs. Representative segments of these captured data pairs are illustrated in Figure 6.

Figure 6. Schematic diagram of force signal, cutting depth signal and penetration status during the feeding process of the cutting vertebral lamina.

Figure 6. Schematic diagram of force signal, cutting depth signal and penetration status during the feeding process of the cutting vertebral lamina.

During data collection, the total cutting depth of the UBS was defined as d, where d > d1 (d1 representing the thickness of the vertebral lamina), ensuring that every complete cutting sequence encompassed both the unpenetrated and penetrated states. Given that the laminal thickness d1 is variable across specimens, the ratio of the post-penetration cutting depth dc  to d1 is likewise variable. The value of d1 primarily dictates the volume of signal data captured within the unpenetrated state, whereas dc  determines the volume of data corresponding to the penetrated state. This experimental configuration minimizes the sample size discrepancy between the two target classes across the overall dataset, mitigating the negative impacts of class imbalance during network training.

Classification of cutting states

Given that both the force data and cutting depth measurements constitute sequential time-series, an LSTM-FCN architecture was selected as the classification network to extract features embedded along the temporal dimension. The LSTM-FCN model incorporates two parallel processing pathways: the LSTM branch effectively captures the temporal dynamics of the combined force and cutting depth signals, while the fully convolutional branch extracts complementary latent features within the dataset. The architecture of this network is detailed in Figure 7.

Figure 7. The diagram of the LSTM-FCN structure.

Figure 7. The diagram of the LSTM-FCN structure.

The force and cutting depth streams are concatenated into a unified signal featuring a dimensionality of 2, which serves as the direct input for feature extraction and state classification within the LSTM-FCN model. The magnitude of the force signal correlates with bone density, offering significant discriminative value for tissue state classification. Concurrently, the cutting depth signal explicitly maps tool advancement, playing a critical role in differentiating the cutting characteristics of cancellous bone from those of the endocortical bone post-penetration. To preserve the intrinsic features of the data, the raw unmanipulated signals are fed directly into the model.

The safety control strategy

The implemented safety control protocol operates as follows: at the termination of each individual feed cycle, the newly acquired force and displacement data segments are extracted and delivered to the LSTM-FCN model. If the network classifies the current state as unpenetrated, the system initiates the subsequent feed cycle. Conversely, if penetration is detected, the cutting operation is terminated immediately, as outlined in Figure 8.

Figure 8. The safety control strategy of vertebral lamina cutting.

Figure 8. The safety control strategy of vertebral lamina cutting.

 

Results and Discussion

To evaluate the clinical efficacy and validity of the proposed methodology, empirical testing of the algorithm was performed alongside ex vivo cutting procedures on porcine spinal laminae. This investigation received formal institutional approval from the Peking University Third Hospital (Ethics No. A2019011).

Verification of the algorithm

A total dataset comprising 21,240 paired signal samples was partitioned into training, validation, and testing subsets using a 3:1:1 ratio. The complete repository included 12,545 unpenetrated instances and 8,695 penetrated instances. The LSTM-FCN network was rigorously evaluated on the test subset. To quantify model performance on this classification task, a confusion matrix was utilized, mapping true versus predicted states where unpenetrated laminae are designated as 0 and penetrated laminae as 1, as presented in the truth table in Table 2.

Table 2. Truth table for the prediction of the spinal lamina cutting status.

Predicted value 0

Predicted value 1

True value 0

TP

FN

True value 1

FP

TN

 

Several performance metrics derived from the confusion matrix were employed to evaluate the classification algorithm. These metrics include precision (the proportion of predicted unpenetrated samples that were genuinely unpenetrated), recall (the proportion of actual unpenetrated samples correctly identified), F1 Score (the harmonic mean of precision and recall), sensitivity (the capacity of the network to identify the unpenetrated state), specificity (the capacity of the network to identify the penetrated state), and accuracy (the overall percentage of correctly classified states). These parameters were computed using Eqs. 1–6:

(1)

 

(2)

 

(3)

 

(4)

 

(5)

 

(6)

 

To account for variability in random dataset partitioning, five independent experimental trials were executed, and the outcomes underwent statistical analysis. The mean and standard deviation for each performance index across these five iterations were computed, with the collective results compiled in Table 3.

Table 3. The test results of five experiments.

Accuracy

Specificity

Sensitivity

F1

Recall

Precision

1

98.57%

97.17%

98.91%

99.12%

98.91%

99.33%

2

98.72%

97.77%

98.96%

99.20%

98.96%

99.45%

3

98.33%

97.38%

98.55%

98.97%

98.55%

99.39%

4

98.62%

97.92%

98.79%

99.15%

98.79%

99.51%

5

98.53%

96.92%

98.91%

99.09%

98.91%

99.27%

Mean

98.55%

97.43%

98.82%

99.11%

98.82%

99.39%

Std

1.44 × 10−3

4.14 × 10−3

1.65 × 10−3

8.62 × 10−4

1.66 × 10−3

9.49 × 10−4

 

The calculated indicators demonstrate that the LSTM-FCN framework exhibits high reliability in identifying and classifying cutting states, confirming its viability for autonomous safety control during laminectomy. A key innovation of the proposed method lies in the integration of the displacement depth signal into the state-recognition framework. To isolate and verify the specific contribution of the cutting depth signal toward improving classification performance, model tracking was also executed exclusively using force data, omitting depth metrics. These comparative findings are summarized in Table 4.

Table 4. The test results (without cutting depth signal).

Accuracy

Specificity

Sensitivity

F1

Recall

Precision

1

98.03%

95.22%

98.74%

98.77%

98.74%

98.80%

2

98.25%

95.94%

98.83%

98.91%

98.83%

98.98%

3

97.66%

92.58%

99.01%

98.53%

99.01%

98.06%

4

97.98%

94.99%

98.73%

98.74%

98.73%

98.73%

5

98.16%

95.69%

98.77%

98.85%

98.77%

98.92%

Mean

98.02%

94.88%

98.82%

98.76%

98.82%

98.70%

Std

2.26 × 10−3

1.34 × 10−2

1.15 × 10−3

1.45 × 10−3

1.15 × 10−3

3.70 × 10−3

 

A comparison between the data in Tables 3 and 4 demonstrates that incorporating the cutting depth signal markedly improves both the accuracy and structural robustness of state recognition.

 

Spinal laminectomy experiment

The autonomous safety control architecture was prospectively evaluated using porcine spinal laminae as the experimental models. The trained LSTM-FCN model was compiled and deployed within a C++ environment, achieving processing speeds exceeding 200 frames per second, demonstrating high computational efficiency. System efficacy was verified via real-time endoscopic visualization of the internal spinal canal. Fifty continuous cutting procedures were conducted across multiple porcine laminae, tracking the number of additional feed cycles (E) that transpired after crossing the critical penetration threshold. Because the precise baseline of the unpenetrated cohort cannot be determined via real-time video feeds, these successful trials were logged as 0. For specimens where breakthrough occurred, the post-penetration feed cycles were tracked precisely. The statistical distribution of these cycles is illustrated in Figure 9.

Figure 9. Statistical results of the number of feed cycles after cutting through the lamina.

Figure 9. Statistical results of the number of feed cycles after cutting through the lamina.

As demonstrated in Figure 9, 36 out of the 50 cutting procedures were halted immediately at the breakthrough boundary (E = 0), 11 trials proceeded through exactly 1 additional feed cycle (E = 1), and 3 trials carried over into 2 additional feed cycles (E = 2). The implemented control system constrained the post-penetration feed cycles (E) to a maximum of 2. Given that the maximum incremental feed depth (D) for any single cycle was capped at 0.3 mm, the total overshoot distance was successfully limited to within 0.6 mm, satisfying clinical safety margins. Furthermore, to analyze the morphological characteristics of the unpenetrated cohort, the porcine specimens were evaluated via computed tomography (CT) imaging. Representative CT cross-sections are presented in Figure 10.

Figure 10. CT sections of the partially unpenetrated pig spine.

Figure 10. CT sections of the partially unpenetrated pig spine.

Analysis of the CT scans indicates that the unpenetrated vertebral laminae were consistently halted right at the critical boundary of the inner cortical bone, avoiding premature termination of the procedure. Consequently, the experimental outcomes from the porcine laminectomy trials confirm that this control methodology delivers high safety margins and robust clinical efficacy.

Laminectomy stands as one of the most frequently executed surgical procedures for posterior spinal pathologies. Pathologically, early interventions relied exclusively on osteotomes, rongeurs, and mechanical burrs for osseous resection. These instruments lacked precision and were associated with distinct intraoperative complications, such as pedicle fractures. In contrast, the ultrasonic bone scalpel (UBS) provides selective bone-cutting capabilities combined with continuous saline irrigation, mitigating the risk of trauma to adjacent soft tissues, particularly the spinal cord [41]. The accuracy of this incisional modality substantially reduces operative duration, systemic complications, and the incidence of incidental dural tears. Through a retrospective cohort analysis, Dave et al. [42] demonstrated that the UBS provides a secure, rapid, and efficacious technique for cervical laminectomy when contrasted with traditional rongeurs and high-speed burrs. Similarly, Krishnan et al. [43] compared 55 patients undergoing conventional bone knife procedures against 45 patients managed with UBS technology; their findings revealed that the UBS technique significantly minimizes neurological deficits, intraoperative blood loss, surgical duration, and the overall length of hospital stay in patients suffering from thoracic spinal stenosis.

Integrating surgical path planning derived from medical imaging with intraoperative state recognition via diverse high-precision sensors enables surgical robotic systems to achieve, or potentially surpass, the clinical outcomes of expert surgeons. Sun et al. [44] introduced a laminectomy grinding path methodology driven by 3D CT imaging data. In their framework, clinicians delineated the surgical region of interest by modifying a virtual boundary frame. A human-machine interface allowed surgeons to transfer the designated grinding trajectory and velocity to a robotic apparatus, with results confirming successful laminectomy execution. While this approach offers simplicity and direct visualization, it exhibits constraints regarding real-time cutting adaptability and carries substantial operational risks. Guan et al. [45] developed a frequency distribution-based algorithm to evaluate acoustic emission (AE) profiles within the frequency domain during osseous drilling. Distinct bone layers were differentiated via these AE signals, achieving a classification accuracy of 84.2%. Although this demonstrated the viability of acoustic tracking for bone layer identification, acoustic signals remain highly susceptible to environmental noise interference within a spinal surgical theater.

To enhance the efficiency of robot-assisted laminectomy (RAL), Qi et al. [46] put forward a multilevel fuzzy control (MLFC) framework for lateral contact forces, incorporating preoperative planning, intraoperative navigation, and milling force regulation. The grinding path was pre-calculated using a 3D reconstruction of CT data, while the MLFC architecture stabilized transverse contact forces to enhance osseous removal efficiency. The data indicated that MLFC-guided milling was more stable and efficient than conventional automated milling. Li et al. [47] developed a multi-source state perception framework that processed simultaneous acoustic and force inputs during drilling operations. An interactive multi-model technique was deployed at the feature extraction layer, while a Support Vector Machine (SVM) was utilized at the decision-making level to train and classify the feature vectors of the acoustic and force signals, yielding an 88% recognition rate. Additionally, Li et al. [5] reported on a collaborative robotic laminectomy platform utilizing a force signal-driven control strategy. A comparative trial between RAL and free-hand (FH) techniques was conducted, where RAL was executed on the left aspect of the lamina and the FH technique on the right. The results established that RAL offered enhanced safety margins over the manual technique; however, notable limitations remained, as the robotic system could not autonomously localize the lamina, and the evaluations were restricted to in vitro specimens.

At present, clinical robotic platforms are primarily restricted to instrument guidance and intraoperative positioning, largely lacking autonomous, self-supporting milling control functions within existing spinal surgery literature. To resolve this capability gap, this study innovatively frames the identification of the vertebral lamina-cutting state as a time-series classification task. The LSTM-FCN framework was selected as the core classification algorithm. By leveraging parallel LSTM and FCN branches, this architecture successfully merges the sequential processing strengths of LSTM networks with one-dimensional convolutional operations, ensuring high classification accuracy and structural robustness. Furthermore, because force metrics alone fail to navigate the signal interference associated with severe osteoporosis, the cutting depth signal was introduced into the state-recognition task to supplement the force data. The proposed safety control strategy based on the LSTM-FCN model achieved a state-recognition accuracy of 98.85%, while constraining the post-penetration overshoot distance to within 0.6 mm. This dual-signal approach effectively prevents premature termination of the cutting process, thereby guaranteeing both the thoroughness and the safety of the laminectomy. This methodology achieves autonomous state identification and cutting execution, representing a more intelligent system.

Nonetheless, the current strategy cannot entirely eliminate bone perforation, which presents a residual risk to the underlying spinal cord. This limitation arises primarily because the control architecture evaluates the state and issues commands at the conclusion of each completed feed cycle based on accumulated signals, whereas the physical breach of the inner cortical bone by the UBS typically occurs mid-cycle during active advancement.

In subsequent investigations, a time-series prediction algorithm will be explored to incorporate predictive future data streams into the control loop input. This advancement aims to prevent UBS perforation of the inner cortical bone entirely, further elevating the safety threshold of the robotic platform. Furthermore, the laminectomy procedures in this study were conducted exclusively on isolated ex vivo animal bone tissue, omitting evaluations on live animal models or human clinical subjects. Consequently, future efforts must focus on optimizing the reliability of this methodology and systematically advancing through live animal and clinical trials to establish readiness for surgical translation.

Conclusion

Given that existing spinal surgery robotic systems lack autonomous milling control functions and robust bone tissue recognition capabilities, this study innovatively casts the identification of laminectomy cutting states as a time-series classification task, establishing a diagnostic methodology centered on the LSTM-FCN architecture. Moreover, since force signals cannot reliably differentiate the active cutting of cancellous bone from the state immediately following laminal perforation, the cutting depth signal was introduced as an innovative supplement to the force data within the state-recognition framework. The LSTM-FCN model was applied to process and categorize the integrated multi-sensor signal matrix composed of force and displacement inputs. The performance and validity of this approach were confirmed through extensive vertebral lamina cutting experiments. The system achieved a definitive classification accuracy of 98.85%, while restricting the post-laminal penetration distance to within 0.6 mm, thereby preventing mechanical injury to neural tissues. In conclusion, the developed methodology provides high safety margins and substantial potential for clinical translation.

Acknowledgements

We thank all the anonymous reviewers for their comprehensive work and excellent comments that helped us improve our manuscript. Moreover, we would also like to thank the MICCAI 2020 organisers for making the Verse2020 dataset freely available. This study was approved by Peking University Third Hospital before the animal trial (Ethics No.: A2019011). This work was supported by the National key research and development plan project [2022YFB4700700] and the Beijing-Tianjin-Hebei basic research cooperation project [J230020].

Conflict of interest

None

Financial support

None

Ethics statement

None

References

Gerhardt J, Bette S, Janssen I, Gempt J, Meyer B, Ryang YM. Is eighty the new sixty? Outcomes and complications after lumbar decompression surgery in elderly patients over 80 years of age. World Neurosurg. 2018;112:e555-60.
https://doi.org/10.1016/j.wneu.2018.01.082
Umekawa M, Takai K, Taniguchi M. Complications of spine surgery in elderly Japanese patients: implications for future of world population aging. Neurospine. 2019;16(4):780-8.
https://doi.org/10.14245/ns.1938184.092
Delgado-Lopez PD, Rodriguez-Salazar A, Castilla-Diez JM, Martin-Velasco V, Fernandez-Arconada O. Role of surgery in spinal degenerative disease. Analysis of systematic reviews on surgical and conservative treatments from an evidence-based approach. Neurocirugia (Astur). 2005;16(2):142-57.
https://doi.org/10.1016/S1130-1473(05)70420-6
Liao Y, Ye R, Tang Q, Tang C, Ma F, Luo N, et al. Application of ultrasonic osteotome in the posterior lumbar interbody fusion surgery by unilateral fenestration and bilateral decompression in the treatment of degenerative lumbar spinal stenosis. Zhongguo Xiu Fu Chong Jian Wai Ke Za Zhi. 2019;33(4):416-22.
https://doi.org/10.7507/1002-1892.201808037
Li Z, Jiang S, Song X, Liu S, Wang C, Hu L, et al. Collaborative spinal robot system for laminectomy: a preliminary study. Neurosurg Focus. 2022;52(1):E11.
https://doi.org/10.3171/2021.10.FOCUS21499
Carroll SE, Wiesel SW. Neurologic complications and lumbar laminectomy: a standardized approach to the multiply-operated lumbar spine. Clin Orthop Relat Res. 1992;(284):14-23.
Antonacci MD, Eismont FJ. Neurologic complications after lumbar spine surgery. J Am Acad Orthop Surg. 2001;9(2):137-45.
https://doi.org/10.5435/00124635-200103000-00007
Wang T, Luan S, Hu L, Liu Z, Li W, Jiang L. Force-based control of a compact spinal milling robot. Int J Med Robot. 2010;6(2):178-85.
https://doi.org/10.1002/rcs.304
Fan L, Gao P, Zhao B, Sun Y, Xin X, Hu Y, et al. Safety control strategy for vertebral lamina milling task. CAAI Trans Intell Technol. 2016;1(3):249-58.
https://doi.org/10.1016/j.trit.2016.10.005
Deng Z, Zhang H, Guo B, Jin H, Zhang P, Hu Y. Hilbert-Huang transform based state recognition of bone milling with force sensing. In: 2013 IEEE International Conference on Information and Automation (ICIA). IEEE; 2013. p. 937-42.
https://doi.org/10.1109/ICInfA.2013.6720428
Al-Abdullah KI, Lim CP, Najdovski Z, Yassin W. A model-based bone milling state identification method via force sensing for a robotic surgical system. Int J Med Robot. 2019;15(3):e1989.
https://doi.org/10.1002/rcs.1989
Dai Y, Xue Y, Zhang J, Li J. Biologically-inspired auditory perception during robotic bone milling. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE; 2017. p. 1112-6.
https://doi.org/10.1109/ICRA.2017.7989132
Dai Y, Xue Y, Zhang J. Milling state identification based on vibration sense of a robotic surgical system. IEEE Trans Ind Electron. 2016;63(10):6184-93.
https://doi.org/10.1109/TIE.2016.2574981
Osa T, Abawi CF, Sugita N, Chikuda H, Sugita S, Tanaka T, et al. Hand-held bone cutting tool with autonomous penetration detection for spinal surgery. IEEE ASME Trans Mechatron. 2015;20(6):3018-27.
https://doi.org/10.1109/TMECH.2015.2410287
Jiang Z, Qi X, Sun Y, Hu Y, Zahnd G, Zhang J. Cutting depth monitoring based on milling force for robot-assisted laminectomy. IEEE Trans Autom Sci Eng. 2020;17(1):2-14.
https://doi.org/10.1109/TASE.2019.2920133
Ji X, Wang Y, Zhang Y, Hu L, Li K, Duan W, et al. Breakthrough prediction in robotic laminectomy based on electrical impedance monitoring with LSTM-FCN. In: 2022 WRC Symposium on Advanced Robotics and Automation (WRC SARA). IEEE; 2022. p. 107-12.
https://doi.org/10.1109/WRCSARA57040.2022.9903989
Li M, Qi X, Sun Y, Li B, Hu Y, Tian W. A stability and safety control method in robot-assisted decompressive laminectomy considering respiration and deformation of spine. IEEE Trans Autom Sci Eng. 2023;20(1):258-70.
https://doi.org/10.1109/TASE.2022.3147270
Lang Z, Wang Q, Wu X, Liu Y, He D, Fan M, et al. Drilling speed and bone temperature of a robot-assisted ultrasonic osteotome applied to vertebral cancellous bone. Spine (Phila Pa 1976). 2021;46(14):E760-8.
https://doi.org/10.1097/BRS.0000000000003902
Zhu J, Hao D, Guo Y, Zhang X, Gao W, Wang X. Safety and effectiveness of ultrasonic osteotome in posterior cervical laminectomy decompression and fusion. Zhongguo Xiu Fu Chong Jian Wai Ke Za Zhi. 2018;32(12):1554-9.
https://doi.org/10.7507/1002-1892.201804012
Krishnan A, Patil S, Reddy C, Mayi S, Degulmadi D, Rai RR, et al. Ventral sculpting decompression: a novel bone scalpel-based technique in thoracic ventral stenosis/kyphosis with myelopathy. Egypt J Neurosurg. 2020;35(1):4.
https://doi.org/10.1186/s41984-020-00076-9
Lv Y, Duan Y, Kang W, Li Z, Wang FY. Traffic flow prediction with big data: a deep learning approach. IEEE Trans Intell Transp Syst. 2015;16(2):865-73.
https://doi.org/10.1109/TITS.2014.2345663
Selvin S, Vinayakumar R, Gopalakrishnan EA, Menon VK, Soman KP. Stock price prediction using LSTM, RNN and CNN-sliding window model. In: 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI). IEEE; 2017.
Taylor JW, McSharry PE, Buizza R. Wind power density forecasting using ensemble predictions and time series models. IEEE Trans Energy Convers. 2009;24(3):775-82.
https://doi.org/10.1109/TEC.2009.2025431
Huang F, Qin T, Wang L, Wan H, Ren J. An ECG signal prediction method based on ARIMA model and DWT. In: 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC). IEEE; 2019. p. 1298-304.
https://doi.org/10.1109/IAEAC47372.2019.8997620
Fawaz HI, Forestier G, Weber J, Idoumghar L, Muller PA. Deep learning for time series classification: a review. Data Min Knowl Discov. 2019;33(4):917-63.
https://doi.org/10.1007/s10618-019-00619-1
Wang H, Sun W, Liu PX. Adaptive intelligent control of nonaffine nonlinear time-delay systems with dynamic uncertainties. IEEE Trans Syst Man Cybern Syst. 2017;47(7):1474-85.
https://doi.org/10.1109/TSMC.2016.2627048
Zhao X, Shi P, Zheng X, Zhang J. Intelligent tracking control for a class of uncertain high-order nonlinear systems. IEEE Trans Neural Netw Learn Syst. 2016;27(9):1976-82.
https://doi.org/10.1109/TNNLS.2015.2460236
Li D, Zhang J, Zhang Q, Wei X. Classification of ECG signals based on 1D convolution neural network. In: 2017 IEEE 19th International Conference on E-Health Networking, Applications and Services (Healthcom). IEEE; 2017. p. 1-6.
https://doi.org/10.1109/HEALTHCOM.2017.8210784
Wang S, Jiang J, Lu X. Study on the classification of pulse signal based on the BP neural network. J Biosci Med. 2020;8(5):104-12.
https://doi.org/10.4236/jbm.2020.85010
Lines J, Bagnall A. Time series classification with ensembles of elastic distance measures. Data Min Knowl Discov. 2015;29(3):565-92.
https://doi.org/10.1007/s10618-014-0361-2
Bagnall A, Lines J, Bostrom A, Large J, Keogh E. The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Min Knowl Discov. 2017;31(3):606-60.
https://doi.org/10.1007/s10618-016-0483-9
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2016. p. 770-8.
https://doi.org/10.1109/CVPR.2016.90
Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2016. p. 779-88.
https://doi.org/10.1109/CVPR.2016.91
Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Cham: Springer; 2015. p. 234-41.
https://doi.org/10.1007/978-3-319-24574-4_28
Cui Z, Chen W, Chen Y. Multi-scale convolutional neural networks for time series classification. arXiv [Preprint]. 2016. arXiv:1603.06995.
https://doi.org/10.48550/arXiv.1603.06995
Wang Z, Yan W, Oates T. Time series classification from scratch with deep neural networks: a strong baseline. In: 2017 International Joint Conference on Neural Networks (IJCNN). IEEE; 2017. p. 1578-85.
https://doi.org/10.1109/IJCNN.2017.7966039
Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735-80.
https://doi.org/10.1162/neco.1997.9.8.1735
Du Q, Gu W, Zhang L, Huang SL. Attention-based LSTM-CNNs for time-series classification. In: Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems (SenSys). ACM; 2018. p. 410-1.
https://doi.org/10.1145/3274783.3275208
Karim F, Majumdar S, Darabi H, Chen S. LSTM fully convolutional networks for time series classification. IEEE Access. 2018;6:1662-9.
https://doi.org/10.1109/ACCESS.2017.2779939
Dau HA, Bagnall A, Kamgar K, Yeh CCM, Zhu Y, Gharghabi S, et al. The UCR time series archive. IEEE/CAA J Autom Sinica. 2019;6(6):1293-305.
https://doi.org/10.1109/JAS.2019.1911747
Dave BR, Dave MB. Stavya Spine's Thirty3: Technology and Spine Surgery. New Delhi: Jaypee Brothers Medical Publishers; 2023.
Dave BR, Krishnan A, Rai RR, Degulmadi D, Mayi S, Gudhe M. The effectiveness and safety of ultrasonic bone scalpel versus conventional method in cervical laminectomy: a retrospective study of 311 patients. Global Spine J. 2020;10(6):760-6.
https://doi.org/10.1177/2192568219876246
Krishnan A, Samal P, Mayi S, Degulmadi S, Rai RR, Dave B. Thoracic spine stenosis: does ultrasonic osteotome improve outcome in comparison to conventional technique? Malays Orthop J. 2021;15(2):62-9.
https://doi.org/10.5704/MOJ.2107.010
Sun Y, Jiang Z, Qi X, Hu Y, Li B, Zhang J. Robot-assisted decompressive laminectomy planning based on 3D medical image. IEEE Access. 2018;6:22557-69.
https://doi.org/10.1109/ACCESS.2018.2828641
Guan F, Sun Y, Qi X, Hu Y, Yu G, Zhang J. State recognition of bone drilling based on acoustic emission in pedicle screw operation. Sensors (Basel). 2018;18(5):1484.
https://doi.org/10.3390/s18051484
Qi X, Sun Y, Ma X, Hu Y, Zhang J, Tian W. Multilevel fuzzy control based on force information in robot-assisted decompressive laminectomy. Adv Exp Med Biol. 2018;1093:263-79.
https://doi.org/10.1007/978-981-13-1396-7_20
Li M, Qi X, Guan F, Jin H, Hu Y, Tian W. State sensing of spinal surgical robot based on fusion of sound and force signals. In: 2021 IEEE International Conference on Real-Time Computing and Robotics (RCAR). IEEE; 2021. p. 324-9.
https://doi.org/10.1109/RCAR52367.2021.9517529

Author information

Mateo Alvarez, Sofia Herrera, Diego Cruz & Andres Castro contributed to this work.

Authors and affiliations

Department of Robust RL and Energy-Based Models, Faculty of Engineering, National University of Colombia, Bogota, Colombia
Mateo Alvarez & Diego Cruz

Department of Offline RL and Implicit Constraints, Faculty of Engineering, University of Chile, Santiago, Chile
Sofia Herrera

Department of Policy Robustness and RL, Faculty of Engineering, University of Antioquia, Medellin, Colombia
Andres Castro

Corresponding author

Correspondence to Mateo Alvarez

Rights and permissions

Open Access The author(s) retain copyright. This article is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. It may be shared and adapted for non-commercial purposes with appropriate attribution, an indication of changes, and distribution of adaptations under the same license. Third-party material may be subject to separate terms identified in its credit line. View the license at https://creativecommons.org/licenses/by-nc-sa/4.0/.

About this article

Cite this article

Vancouver
Alvarez M, Herrera S, Cruz D, Castro A. Toward Robust Offline Reinforcement Learning Through Energy-Based Implicit Policy Constraints. J Trustworthy Explain Intell Syst. 2023;2:35.
APA
Alvarez, M., Herrera, S., Cruz, D., & Castro, A. (2023). Toward Robust Offline Reinforcement Learning Through Energy-Based Implicit Policy Constraints. Journal of Trustworthy and Explainable Intelligent Systems, 2, 35.
Received
12 January 2023
Revised
19 February 2023
Accepted
11 March 2023
Published
15 September 2023
Version of record
15 September 2023

Share this article

Easily share this article with others using the link below:

Toward Robust Offline Reinforcement Learning Through Energy-Based Implicit Policy Constraints
Scan to access
this article

Ready to submit?
Start a new submission or continue a submission in progress:
Submission Portal Author Guidelines

Follow this journal
Get notified of new updates and articles.