Cheima Bouden

Comparative Exploration of the Contribution of Reinforcement Learning in Robotic Surgery

PDF (768KB), PP.37-53

Views: 0 Downloads: 0

Author(s)

1. University of Constantine 2 – Abdelhamid Mehri/department of Fundamental Computer Science and its Applications, Constantine, Algeria

* Corresponding author.

DOI: https://doi.org/10.5815/ijem.2024.04.04

Received: 12 Apr. 2024 / Revised: 14 May 2024 / Accepted: 20 Jun. 2024 / Published: 8 Aug. 2024

Index Terms

Artificial Intelligence, Reinforcement Learning, Surgical Robot, Autonomy, Simulation

Abstract

Problem: The precision, efficiency, and safety of surgical procedures need significant improvements. Traditional methods are limited by human capabilities, and existing robotic systems lack the advanced adaptability required for complex surgical tasks. The integration of reinforcement learning (RL) into robotic surgery represents a potential revolution in the medical field.
Methods: This comparative review synthesizes recent progress in RL applications for robotic surgery. We highlight innovative methodologies and successful applications of RL, focusing on advanced simulations to train RL agents and the importance of human demonstrations in the learning process.
Results: Emerging trends such as the effective use of simulations and human demonstrations to enhance RL in robotic surgery are identified. The review also discusses challenges associated with RL applications, emphasizing the need for clinical validation and ensuring patient safety.
Conclusion: The transformative potential of RL in robotic surgery is evident, though challenges remain. Future work should prioritize clinical validation, patient safety, and interdisciplinary collaboration.

Cite This Paper

Cheima Bouden, "Comparative Exploration of the Contribution of Reinforcement Learning in Robotic Surgery", International Journal of Engineering and Manufacturing (IJEM), Vol.14, No.4, pp. 37-53, 2024. DOI:10.5815/ijem.2024.04.04

Reference

[1]Bouden, C., & Mezioud, C. (2023, August). An Intelligent Decision Support System for the Surgical Preoperative Phase: An Approach Based on Machine Learning. In International conference on WorldS4 (pp. 483-494). Singapore: Springer Nature Singapore.
[2]Bouden, C., & Mezioud, C. (2023). Towards an Intelligent Surgical Preoperative Decision Support System: An Approach Based on Decision Trees and the Random Forest. Intelligent Methods In Engineering Sciences, 2(3), 67-74.
[3]Bouden, C., & Mezioud, C. (2023, April). Towards an Intelligent Preoperative Surgical Decision Support System: A machine learning-based approach. In Proceedings of International Conference on Intelligent Systems and New Applications (Vol. 1, pp. 19-26).
[4]L. P. Kaelbling, M. L. Littman, & A. W. Moore, ( 1996). Reinforcement learning: A survey. Journal of artificial intelligence research, 4, 237-285.
[5]R. S. Sutton, & A. G. Barto (1999). Reinforcement learning. Journal of Cognitive Neuroscience, 11(1), 126-134.
[6]P. Dayan, & CJCH Watkins, (1992). Q-learning. Machine Learning, 8(3), 279-292.
[7]Li, Y. (2017). Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274.
[8]M. Roderick, J. MacGlashan, & S. Tellex (2017). Implementing the deep q-network. arXiv preprint arXiv:1711.07478.
[9]J. Peters, & S. Schaal (2008). RL of motor skills with policy gradients. Neural networks, 21(4), 682-697.
[10]V. Konda, & J. Tsitsiklis (1999). Actor-critic algorithms. Advances in neural information processing systems, 12.
[11]B. Armstrong, O. Khatib, & J. Burdick (1986, April). The explicit dynamic model and inertial parameters of the PUMA 560 arm. In Proceedings. 1986 IEEE International Conference on Robotics and Automation (Vol. 3, pp. 510-518).
[12]S. Tsuda, D. Oleynikov, J. Gould, D. Azagury, B. Sandler, M. Hutter & R. Satava (2015). SAGES TAVAC safety and effectiveness analysis: da Vinci® surgical system (Intuitive Surgical…). Surgical endoscopy, 29, 2873-2884.
[13]www.maxongroup.ch/maxon/view/application/Am-OP-Tisch-steht-ein-Roboterchirurg?isoCode=fr
[14]J. D. Sires, J. D. Craik, & C. J. Wilson (2019). Accuracy of bone resection in MAKO total knee robotic-assisted surgery. The journal of knee surgery, 34(07), 745-748.
[15]www.docmartinortho.vegas/mako-smartrobotics/
[16]C. Batailler, D. Hannouche, F. Benazzo, & S. Parratte (2021). Concepts and techniques of a new robotically-assisted technique for total knee arthroplasty: the ROSA knee system. Archives of Orthopedic and Trauma Surgery, 141(12), 2049-2058.
[17]www.zimmerbiomet.com/en/products-and-solutions/zb-edge/robotics/rosa-brain.html
[18]B. C. Thomas, M. Slack, M. Hussain, N. B, A. P, E. D, & G.D.S (2021). Preclinical evaluation of the versus surgical system, a new robot-assisted surgical device for use in minimal access renal and prostate surgery. European urology focus, 7(2), 444-452.
[19]www.maxhealthcare.in/technology/versius-next-generation-robotic-surgery
[20]T. deBeche -Adams, W. S. Eubanks, & SG de la Fuente, ( 2019). Early experience with the Senhance ®-laparoscopic/robotic platform in the US. Journal of Robotics Surgery, 13(2), 357-359.
[21]www.medicaldevice-network.com/projects/asensus-surgicals-senhance-surgical-system-usa/?cf-view&cf-closed
[22]E.B. Wilson, (2009). The evolution of robotic general surgery. Scandinavian Journal of Surgery, 98(2), 125-129.
[23]A. A. Shah, J. Bandari, D. Pelzman, B. J. Davies, & B. L. Jacobs (2021). Diffusion and adoption of the surgical robot in urology. Translational andrology and urology, 10(5), 2151.
[24]A. Gioè, G. Monterossi, S. Gueli Alletti, G. Panico, G. Campagna, B. Costantini & G. Scambia, (2024). The new robotic system HUGO RAS for gynecologic surgery: First European experience from Gemelli Hospital. International Journal of Gynecology & Obstetrics.
[25]P. Q. Thuan, PT V. Chuong, NHNam & NH Dinh (2023). Coronary Artery Bypass Surgery: Evidence-Based Practice. Cardiology in Review, 10-1097.
[26]D, K., & Koutserimpas, C. (2024). Pitfalls with the MAKO Robotic-Arm-Assisted Total Knee Arthroplasty. Medicine, 60(2), 262.
[27]O. Khanna, R.D. Beasley, F, & S. DM (2021). The path to surgical robotics in neurosurgery. Operative Neurosurgery, 20(6), 514-520.
[28]J. Chen, H. Y. Lau, W. Xu, & H. Ren (2016, February). Towards transferring skills to flexible surgical robots with demonstration and reinforcement learning programming. In 2016 Eighth International Conference on Advanced Computational Intelligence (ICACI) (pp. 378-384). IEEE.
[29]Guru, Khurshid, Mohd Raashid Sheikh, Syed Johar Raza, Andrew Stegemann and John Nyquist. “New knotting technique for robot-assisted surgery” The Canadian Journal of Urology 19 4 (2012): 6401-3.
[30]V. Varier, VM, Rajamani, DK, Goldfarb, N., Tavakkolmoghaddam, F., Munawar, A., & Fischer, GS (2020, August). Collaborative suturing: A reinforcement learning approach to automate hand-off tasks in suturing for surgical robots. In 2020, 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) (pp. 1380-1386). IEEE.
[31]www.youtube.com/watch?v=u33fOYOjCZ8
[32]RC Jackson and MC Cavusoglu, “Needle path planning for autonomous robotic surgical suturing,” in 2013 IEEE International Conference on Robotics and Automation, Karlsruhe, Germany: IEEE, May 2013, p. 1669 1675. doi: 10.1109/ICRA.2013.6630794.
[33]F. Gharagozloo and M. Meyer, “Robotic Transthoracic First Rib Resection for Neurogenic Thoracic Outlet Syndrome,” WJCS, vol. 12, no. 01, p. 1 11, 2022, doi: 10.4236/wjcs.2022.121001.
[34]B. Thananjeyan, A. Garg, S. Krishnan, C. Chen, L. Miller, and K. Goldberg, "Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning," 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 2017, pp. 2371-2378, doi: 10.1109/ICRA.2017.7989275.
[35]R. Bogue, “Cutting robots: a review of technologies and applications”, Industrial Robot: An International Journal, vol. 35, no. 5, p. 390 396, August 2008, doi: 10.1108/01439910810893554.
[36]M. Jacob, Y.-T. Li, G. Akingba, and JP Wachs, “ Gestonurse: a robotic surgical nurse for handling surgical instruments in the operating room,” J Robotic Surg, vol. 6, no. 1, p. 53 63, March 2012, doi: 10.1007/s11701-011-0325-0.
[37]X. Tan, C. B. Chng, Y. Su, K. B. Lim, & C. K. Chui (2019). Robot-assisted training in laparoscopy using deep reinforcement learning. IEEE Robotics and Automation Letters, 4(2), 485-492.
[38]J. Sun, D. Zhou, Y. Liu, J. Deng, and S. Zhang, “Development of a Minimally Invasive Surgical Robot Using Self-Helix Twisted Artificial Muscles,” IEEE Trans. Ind. Electron., vol. 71, no. 2, p. 1779 1789, Feb. 2023, doi: 10.1109/TIE.2023.3257374.
[39]W. Chi, J. Liu, M. E. Abdelaziz, G. Dagnino, C. Riga, C. Bicknell, & G. Z. Yang (2018, October ). Trajectory optimization of robot-assisted endovascular catheterization with reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 3875-3881). IEEE.
[40]A. Bourdillon, A. Garg, H. Wang, Y. J. Woo, M. Pavone, & J. Boyd (2023). Integration of reinforcement learning in a virtual robotic surgical simulation. Surgical Innovation, 30(1), 94-102.
[41]A. Kienzlen, F. Jaensch, A. Verl and L. Cheng, "Concept for a Reinforcement Learning Approach to Navigate Catheters Through Blood Vessels," 2022 28th International Conference on Mechatronics and Machine Vision in Practice (M2VIP), Nanjing, China, 2022, p. 1-4, doi: 10.1109/M2VIP55626.2022.10041096.
[42]H. Gao et al., " GESRsim: Gastrointestinal Endoscopic Surgical Robot Simulator," 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 2022, pp. 9542-9549, doi: 10.1109/IROS47612.2022.9982138.
[43]AA Shahkoo and AA Abin, "Autonomous Tissue Manipulation via Surgical Robot Using Deep Reinforcement Learning and Evolutionary Algorithms," in IEEE Transactions on Medical Robotics and Bionics, vol. 5, no. 1, pp. 30-41, Feb. 2023, doi: 10.1109/TMRB.2023.3237772.
[44]AASHahkoo, AAAbin “Deep reinforcement learning in continuous action space for autonomous robotic surgery,” Int J CARS 18, 423–431 (2023). https://doi.org/10.1007/s11548-022-02789-8
[45]Y. Long, W. Wei, T. Huang, Y. Wang, and Q. Dou, "Human-in-the-Loop Embodied Intelligence With Interactive Simulation Environment for Surgical Robot Learning,” in IEEE Robotics and Automation Letters, vol. 8, no. 8, pp. 4441-4448, Aug. 2023, doi: 10.1109/LRA.2023.3284380.
[46]R. Bendikas, V. Modugno, D. Kanoulas, F. Vasconcelos and D. Stoyanov, “Learning Needle Pick-and-Place Without Expert Demonstrations,” in IEEE Robotics and Automation Letters, vol. 8, no. 6, pp. 3326-3333, June 2023, doi: 10.1109/LRA.2023.3266720.
[47]G. Ji, Q. Gao, M. Sun, G. Mi, X. Hu, and Z. Sun, "Surgical Continuum Manipulator Control Using Multi-agent Team Deep Q Learning," 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Sydney, Australia, 2023, pp. 1-5, doi: 10.1109/EMBC40787.2023.10340943.
[48]Y. Ou and M. Tavakoli, "Towards Safe and Efficient Reinforcement Learning for Surgical Robots Using Real-Time Human Supervision and Demonstration," 2023 International Symposium on Medical Robotics (ISMR), Atlanta, GA, USA, 2023, pp. 1-7, doi: 10.1109/ISMR57123.2023.10130214.
[49]G. Xinlan, L. Tao and Z. Jian, "Trajectory Planning and Obstacle Avoidance Behavior of Manipulator Based on Q-learning Algorithm," 2023 IEEE International Conference on Control, Electronics and Computer Technology (ICCECT), Jilin, China, 2023, pp. 1235-1241, doi: 10.1109/ICCECT57938.2023.10140814.
[50]K. Iyengar, S. Spurgeon and D. Stoyanov, "Deep Reinforcement Learning for Concentric Tube Robot Path Following," in IEEE Transactions on Medical Robotics and Bionics, vol. 6, no. 1, pp. 18-29, Feb. 2024, doi: 10.1109/TMRB.2023.3310037.

International Journal of Engineering and Manufacturing (IJEM)