Digitalization of surgical features improves surgical accuracy via surgeon guidance and robotization

Cataract surgery video and image collection

A total of 26,466 surgical videos from patients (26,466 eyes) who underwent routine phacoemulsification combined with IOL implantation at the Zhongshan Ophthalmic Center (ZOC, Guangzhou, China) of Sun Yat-Sen University from January 2016 to March 2019 were collected for capsulorhexis evaluation. A senior cataract clinician conducted strict screening of the video quality. Videos with the following conditions were excluded: (1) blurry focus; (2) incomplete exposure of the eyeball within the field of view of the video; (3) complicated cataract surgeries, including congenital cataract, traumatic cataract, concomitant corneal diseases and uveitis; (4) poor pupil mydriasis (<6 mm) affecting capsulorhexis evaluation; and (5) occurrence of surgical complications. Finally, 17,538 videos of cataract surgery met the criteria and were included in the study. The screenshots of the IOL and capsulorhexis were obtained for evaluating the quality of the CO after centering the IOL at the end of the operation.

Quality analysis of traditional experience-based capsulorhexis

All IOL and capsulorhexis images obtained from surgical videos were independently analyzed by two professionally trained graders from the ZOC image reading center (Guangzhou, China) using ImageJ image analysis software. The horizontal diameter, vertical diameter, and area of the CO were measured using the diameter of the IOL optical surface (6 mm) as the reference, and then the average values of the two graders were obtained. In addition, the IOL edge coverage of the CO was assessed as 360°, 270 to 360°, 180 to 270°, 90 to 180°, or <90° of coverage and recorded. A capsulorhexis diameter between 5.0 and 5.5 mm and complete coverage of the edge of the IOL was considered ideal capsulorhexis. A CO diameter larger than 5.5 mm was considered too large, while a diameter <4.5 mm was considered too small. When there was a discrepancy between the two graders, a cataract specialist with over 10 years of experience made the final decision.

Image labeling

All capsulorhexis images were further classified into three grades according to their quality, including the capsulorhexis-IOL overlap, centricity, and circularity, as follows: (1) ideal capsulorhexis: centered and round CO, covering the edge of the IOL optical surface continuously for 360 degrees with a diameter between 5.0 and 5.5 mm; (2) acceptable capsulorhexis: smaller diameter (4.5–5.0 mm) or off-center CO completely covering the edge of the IOL optical surface, or IOL overlap of more than 180° but <360°; and (3) poor capsulorhexis: excessively large CO with IOL overlap of <180° or an insufficient opening size (diameter < 4.5 mm). Two cataract specialists who had more than 5 years of experience were recruited to label the images. For disputed images, arbitration was performed by another senior cataract specialist with over 10 years of experience.

Development of the evaluation module of MetaS

A flow chart showing the development of the evaluation and feature extraction modules of our intelligent MetaS system is shown in Fig. 2. A total of 17,538 screenshots of IOLs and COs from videos of cataract surgery were collected and labeled by cataract specialists as ideal, acceptable, or poor capsulorhexis. Afterwards, they were then applied to train the evaluation module of MetaS. Image standardization was performed prior to deep learning algorithm training. All images were resized to 512 × 512 pixels, and the pixel values were normalized to an interval between 0 and 1. Data augmentation was used to increase the image heterogeneity of the training dataset and thus reduce the chance of overfitting during the deep learning process. The new samples were obtained through simple transformations of the original images and corresponded to “real-world” acquisition conditions. Random horizontal and vertical flipping, random rotations of up to 90° around the center of the image, and random brightness shifts within the range of 0.8 to 1.2 were applied to the images in the training set in real time during training.

The training set contained 14,400 images, and the validation set contained 3138 images. No individual images overlapped among these sets. A state-of-the-art deep CNN architecture, InceptionResNetV2, which mimics the architectural features of two previous state-of-the-art CNNs (Residual Network and Inception Network), was used to train the evaluation model¹⁸. Weights pretrained for ImageNet classification were applied to initialize the CNN architectures. The model was trained for up to 500 epochs. In the training process, validating ResNet on loss was assessed using the validation set after each epoch and applied as a reference for model selection. Early stopping was employed, such that when the validation loss did not improve over 120 consecutive epochs, the training process would stop. The model state with the lowest validation loss was saved as the final state of the model.

Development of the feature extraction module of MetaS

Furthermore, we improved and optimized a feature extraction algorithm, Mask R-CNN, to establish an extraction module capable of recognizing and extracting the path and features of ideal capsulorhexis procedure²⁰. As an instance segmentation algorithm model, Mask R-CNN can perform pixel-level object segmentation and target recognition simultaneously.

For the extraction of the capsulorhexis path, 2931 videos of ideal capsulorhexis were decomposed into individual image frames. From these, 1063 pre- and post-capsulorhexis images were used for model training. The CL, pupil margin, and their respective centers were manually annotated in the pre-capsulorhexis images using the annotation tool LabelMe ( Additionally, the edges of CO and the IOL optical surface were annotated in the post-capsulorhexis images. The annotated dataset was then utilized to train the feature extraction module of MetaS. Given the irregular circular shape of the CO, multiple points along its edge were marked to enhance the model’s accuracy in identifying the opening.

The remaining 1868 pre- and post-capsulorhexis images were used to extract the trajectory of the ideal capsulorhexis. First, the model was applied to identify the CL before and after capsulorhexis, and their pixel areas were measured as SCL1 and SCL2, respectively. The CO was also identified, and its pixel area was measured as SCO2. Then, the translational and scaling relationships before and after capsulorhexis were determined by the ratio of SCL1/SCL2. According to this ratio, SCO2 could be shifted and scaled into the pre-capsulorhexis image, and its pixel area was measured as SCO1. By calculating the area ratio of SCO1/ SCL1 across all images, the radius ratio between the CO and the CL was determined, leading to the conclusion that the radius of the ideal CO (RC) was CL radius (RL) × 0.58. The detailed calculation of RC is presented in Eq. (1):

Therefore, by identifying the CL and pupil margin, the path of ideal capsulorhexis can be determined as a circle centered at the pupil center with a radius equal to 0.58×RL.

Furthermore, the feature extraction module of MetaS was also used to extract the specific characteristics of the ideal capsulorhexis from the images of the IOL and CO. MetaS could automatically measure the CO diameter and area, the linear distance between the capsulorhexis center and the cornea center, and the distances between different points along the capsulorhexis edge and the CL (3, 6, 9, and 12 o’clock positions) in batches using the diameter of the IOL optical surface (6 mm) as the reference.

Lens caliper-assisted capsulorhexis versus conventional experience-based capsulorhexis

A total of 9498 patients (9498 eyes) who planned to undergo phacoemulsification with IOL implantation from April 2019 to October 2021 at the Zhongshan Ophthalmic Center were consecutively enrolled. Ethics committee approval was obtained from the Zhongshan Ophthalmic Center (Guangzhou, China) Institutional Review Board (No.: 2019 KYPJ033), and all study procedures adhered to the principles of the Declaration of Helsinki. Patients provided written informed consent prior to enrollment in the study and were not offered any compensation or incentives. The exclusion criteria were the same as those used in the quality analysis for traditional manual capsulorhexis. All the surgeries were performed by the same seven experienced cataract surgeons in the traditional experience-based group.

Before performing capsulotomy, with the pupil center as the center of capsulorhexis, the LC was used to measure and locate the position of capsulorhexis with 5.3 mm diameter. The surgeon could gently use the blunt head of the LC to further make the key markers for capsulorhexis path on the lens anterior capsule, and then carried out capsulotomy according to the marks. After capsulorhexis, hydro-delination and hydro-dissection, phacoemulsification, irrigation/aspiration of the remaining lens cortex, and IOL implantation were performed as usual. After adjusting the position of the IOL to the center, a screenshot is taken for the evaluation of capsulorhexis quality. We further compared the performance for capsulorhexis assisted by the LC with that for traditional experience-based capsulorhexis.

Development of the MetaS real-time guidance module for cataract surgery

In the guidance module of MetaS, we compared the effectiveness of Model 1 (GhostNet backbone) and Model 2 (GhostNet backbone integrated with FPN Neck). Model 1 employed the GhostNet Backbone. Due to its unique Ghost Module design, it can remarkably reduce the number of parameters and computational load while maintaining high accuracy^45,46. This enables the model to run at a faster speed, making it suitable for application scenarios that demand quick responses, such as real-time video analysis or continuous image set processing⁴⁷. It can also support real-time tracking during surgical procedures.

Model 2 was formed by superimposing the GhostNet Backbone and the FPN Neck. By fusing the Feature Map to a quarter of the original image size, it substantially reduced the computational burden and enhanced the system’s real-time processing capabilities^48,49. It was more adept at handling continuous detection tasks in dynamically changing environments, like real-time tracking during surgery. Moreover, it further improved the accuracy.

To evaluate the performance of two models, we adopted several common statistical methods in this paper, including accuracy (Acc), sensitivity (Sen), precision (Pre), Matthew’s correlation coefficient (MCC), F1 score, and the area under ROC curves (AUC). Detailed definitions and calculation methods are referred to in previous studies⁵⁰.

During cataract surgery, the model can automatically identify the eyeball structure, including the CL, iris, pupil, and lens. Then, the model automatically provides an ideal capsulorhexis path (8 points) with the radius of CL*0.58 as radius at the 1.5, 3, 4.5, 6, 7.5, 9, 10.5, and 12 o’clock positions on the anterior capsule in real-time to guide the surgeon to perform capsulorhexis (Fig. 4a, b). To maintain a stable display and automatically track eyeball movement during the operation, an unsupervised video object segmentation algorithm was also used. Finally, to assess the quality of capsulorhexis at the end of the operation, a deep residual network (ResNet) model was also introduced to increase the depth and width of the convolutional network and improve the accuracy. It can measure the actual diameter of the CO and evaluate the corresponding IOL edge coverage after adjusting the IOL position to the center.

Real-time MetaS-guided capsulorhexis versus conventional experience-based capsulorhexis

To evaluate the performance of MetaS-guided capsulorhexis, a total of 53 consecutive patients (53 eyes) with age-related cataract who planned to undergo cataract surgery were recruited between July 2021 and December 2021 to receive MetaS-guided capsulorhexis. Another 52 consecutive patients (52 eyes) were recruited to undergo conventional capsulorhexis without any assistance. All operations were performed by the top cataract surgeon (Y.L.) at ZOC following a standardized procedure. Our real-time guidance module was installed on an external computer and captured the surgical video through a high-definition multimedia interface (HDMI) port. The surgeon could control the guidance system with the connected footswitch during surgery and perform capsulorhexis under system guidance. After capsulorhexis, the following procedures were performed as usual. After adjusting the position of the IOL, our model automatically measured the capsulorhexis diameter and offered quality evaluation results. We compared the performance for capsulorhexis guided by the digital features of ideal capsulorhexis using the real-time guidance module of MetaS with that for traditional experience-based capsulorhexis.

Capsulorhexis by an autonomous surgical robot using the digital features extracted by MetaS

Finally, the digital features of ideal capsulorhexis were used to guide a robot to perform autonomous capsulorhexis in ex vivo porcine eyes. The detailed procedures are described below:

Step 1: The robot arm holds the diathermic capsulorhexis tip and starts at the initial position.

Step 2: The movement path of diathermic capsulorhexis tip is planned from the initial position to the target position in the porcine eye. As shown in Fig. 4b, the diathermic capsulorhexis tip moves from the initial position to the corneal incision along the vertical line of the corneal incision. The distance between the initial position of the diathermic capsulorhexis tip and the corneal incision is defined as S. The detailed calculation of S is presented in Eq. (1):

R is the corneal radius; r is the target radius of capsulorhexis (5.3 mm); and a is the maximum RCM angle.

After entering the corneal incision, the diathermic capsulorhexis tip moves 1 mm vertically upward and then moves to the target position. The distance that the diathermic capsulorhexis device moves within the anterior chamber is calculated according to the diameters of the cornea and target CO. Force feedback technology is used to ensure that the device does not damage intraocular tissue.

Step 3: Perform diathermic capsulorhexis. After the diathermic capsulorhexis tip reaches the target position, it moves downward under impedance control until it touches the lens anterior capsule with a 20mN micro force. Under the control of the RCM, the surgical robot controls the diathermic capsulorhexis device to draw a circular capsulorhexis trajectory and complete the procedure autonomously.

As illustrated in Fig. 4c, the motion track of the motor was calculated according to the capsulorhexis trajectory, the maximum RCM angle a (±12°), and the radius of the CO (2.65 mm).

Step 4: Upon completion of the electronic capsulorhexis, the robot safely exits according to its original entry path.

Statistical analysis

Statistical analysis was performed using StataSE15 (version 15.0, Stata Corp LP, TX, USA). A P value less than 0.05 was considered statistically significant. All continuous variables are expressed as the mean ± standard deviation, and categorical variables are expressed as counts and percentages. To validate the performance of the evaluation module of MetaS, the area under the receiver operating characteristic curve (AUC), sensitivity, and specificity of the capsulorhexis evaluation were calculated according to the reference standard. The 95% confidence intervals (95% CIs) were estimated for all performance metrics. Independent t tests were performed to assess the differences in the capsulorhexis diameter and area between experience-based or verification-guided capsulorhexis and MetaS-guided capsulorhexis. The Pearson chi-square test was performed to assess differences in categorical variables between the two groups.