How 3D Scanning Boosts 3D Simulation for the Application of Endoscopic Robots

A research group in the field of biomedical engineering at Boğaziçi University in Istanbul, Turkey used EinScan Pro 2X to digitize intestines for 3D simulation. Biomedical Engineering is concerned with the application of engineering technology and scientific methods for the analysis of biological, physiological and health care problems.

The Institute’s objective is to provide specialized academic training and research opportunity infrastructure for graduate students in specific areas of Biomedical Engineering.

Current areas of research in the Institute´s laboratories include biomedical instrumentation and bioelectronics, biomechanics, neurosignal analysis, biophotonics, medical imaging, cellular imaging and electrophysiology, robotics, medical device design and testing and psychophysics.

Dr. Mehmet Turan has received new funding through the TÜBİTAK 2232 International Fellowship for Outstanding Researchers Program to work in the Institute. The project is entitled “Magnetically Actuated Al-Powered Endoscopic Capsule Robot for Targeted Drug Delivery and Multiple Biopsy Operations”.

The project aims to make major scientific and technological innovations in the mechatronics design, remote magnetic control, and localization and mapping algorithms for wireless capsule endoscope (WCE) robots using recent advances in artificial intelligence. Deep learning techniques are promising to improve dense topography reconstruction and pose estimation, as well as simultaneous localization and mapping (SLAM). Currently available datasets do not support effective quantitative benchmarking. In the project, a comprehensive endoscopic SLAM dataset containing both capsule and standard endoscopy recordings is introduced. A Panda robotic arm, EinScan Pro 2X, two different commercially available capsule endoscopes with different camera properties and two different conventional endoscopy cameras were employed to collect data from eight ex-vivo porcine gastrointestinal (GI)-tract organs.

The ground truth geometries of the organs were acquired by EinScan Pro 2x Multifunctional 3D Scanner. The 3D-scanned models obtained for six organs are fixed to scaffolds that were cut in O, Z and L shapes to mimic the GI-tract path through the ascending colon to the transverse colon. Gathering point cloud data from two colons, one intestine and three stomachs from different individuals makes the dataset appropriate for transfer learning algorithms. Besides, the algorithm performance on the tissue with various texture details for the same organ type can be tested.

As this was the first trial of dataset shooting the research team faced some difficulties:

• It was not possible to scan small, light and dark areas without markers

Answer from SHINING 3D:
“The Handheld mode works very well for objects larger than 10 cm whereas scanning smaller objects is quite difficult. We recommend the fixed scan mode, which can capture object sizes up to 2-3cm. This mode is especially designed for the acquisition of small objects and delivers very good results. If areas are very dark, the light is absorbed. The brightness of the scanner can be increased but it happens that the scanner does not “see” anything. Medical scanning spray, as it is used in dentistry can solve this problem. The spray turns the areas white and makes them easier to scan. These medical sprays are not harmful or toxic.”

• To acquire a full 3D model, we had to rotate the scaffolds which sometimes caused irregularities in the positioning of the adjacent surfaces

Answer from SHINING 3D:
“When turning objects to be scanned, one needs to make sure that all the parts of the objects are fixed because otherwise it become difficult for the software to detect and stitch the acquired data together.”

After the organs (stomach, colon and small intestine) were sewn in L, O and Z form, the 3D scanning started. The picture below shows the scanning of a stomach. The scanning was done in the handheld rapid scan mode. For this purpose, scanning was started at a corner of the organ and continued alongside one another to the last corner. The middle area of the stomach contains the most important structure. Therefore, brightness had to be adjusted to be able to capture all the the tiny details. The brightness was easily adjusted with the + and – button settings on the 3D Scanner. The brightness is shown at the top of the left small screen. The red color is an alert that it is too bright to continue the scanning process. However, the main focus was on the internal structure of the stomach, which is shown as green on the screen. The EinScan Pro 2X Multifunctional Handheld 3D  Scanner was used to scan the organ surface in detail. The stomach had a large surface so it could be scanned without any problem. In contrary, the colon had a small area, and the white background of the scaffold made the scanning difficult. Therefore, the research team used markers to facilitate the capturing of the structure. The markers were placed close together on the white surface of the scaffold. In general, using markers helps scanning details in small areas.

The scanning process was similar for all organs. The team looked for a corner that the EinScan Pro 2X  recognized and followed the instructions on the user interface.

After being satisfied with the coverage of the scanned organ, the mesh model was created. During post processing the holes in the Mesh were filled with the auto hole filling application, and the unnecessary areas eliminated with the CTRL + right mouse selection. Finally, the team saved the the 3D model in .ply, .obj and .txt.

The acquired .ply file was edited in Blender to create a heatmap in order to obtain the depth values of the models.

The final scan data was used for training purposes for deep learning algorithms. The deep learning training process with the 3D scan model is shown in the image below. The team from Boğaziçi University used the recorded capsule endoscopy images as input. The localization was determined from the images by using the 3D model.

The project aim was to imitate the organ structure in the smallest detail. This was successfully achieved with the EinScan Pro 2X Multifunctional Handheld 3D Scanner. The project team was able to record all structures, corners and depths with high precision. There was no significant change from the original structure.

In a previous research project, another 3D Scanner was used. This 3D scanner had also achieved good results, but there exist technical differences in the resolution and point accuracy. Point accuracy was one of the most important factors in our project to achieve the deep learning process which is why the team decided to count on the capacities of the EinScan Pro 2X for this project.

EinScan Pro 2X Multifunctional Handheld 3d Scanner met the needs of the Boğaziçi University research team. SHINING 3D is happy to support interesting scanning applications. If you have an exciting project, contact us and we will evaluate possibilities to cooperate with you!