Which segmentation tool should you use?
A detailed evaluation of five well-known tools
A nyone who annotates medical image data for the development of AI applications in medical technology knows that there is a wide range of segmentation tools to choose from. Therefore a common question in these types of projects is: what is the best segmentation tool out there?
In this article, we evaluate five different segmentation tools: 3D Slicer, MITK Workbench, ITK Snap, ImFusion Labels and QuPath. We used each of these tools extensively with an impressive total of around a thousand hours of segmentation work. Based on our experience in different projects, we share with you what we learned about the five tools.
What to expect from the article:
- Which criteria a great segmentation tool should meet
- Our experience with five different tools and what we used them for
- The strengths and weaknesses of each segmentation tool
- Which segmentation tool we would recommend to you
The criteria we used to evaluate the segmentation tools
There are two important areas when it comes to using segmentation tools: the technical standard and the usability of the tool. Not only should the tool have certain technical functionalities that work well, but it should also be user-friendly and have learning resources available. We evaluate the tools based on the following five criteria that cover both the technical standard and usability of a segmentation tool:
1. Manual segmentation
Manual segmentation means that a user of the tool annotates the different areas of the image by hand without any automation. A good tool has user-friendly functions for manual segmentation and allows the user to get precise results.
2. Automatic segmentation
Automatic segmentation means that the segmentation tool automatically segments the image based on previous input or parameters that were set. A good tool delivers precise results after setting up the appropriate parameters..
3. Thresholding
Thresholding means that the segmentation tool selects areas of the medical image based on a certain threshold value. A good tool delivers precise results, can be applied to similar images (e.g. different tissue types in automatic tissue differentiation) and has enough settings to generate the desired outcomes.
4. User interface
The user interface is what the segmentation tool looks like when you’re working with it. A good tool is easy to operate, has all essential functions within reach, can be configured easily and has intuitive buttons.
5. Learning resources
It is important that there are enough learning resources available, so you can adequately learn to work with a new segmentation tool. Tools with tutorials, user manuals, help menus and additional support by the manufacturer score extra points on user-friendliness.
The segmentation tools & why we are qualified to evaluate them
In the course of two years, we have carried out various projects in the field of medical image data annotation. In this period, we invested around a thousand hours of work into the five segmentation tools that we’re evaluating in this article:
3D Slicer
We used 3D slicer in a dermatology project, where we segmented squamous and basalcellcarcinoma in confocal micorscopy images. We also used the tool for a project in spine surgery, where we annotated and segmented diverse pathologies varying from fractures to degenerative inflammatory diseases on X-ray, CT and MRI images.
MITK Workbench
We used MITK Workbench for urology projects, in which we segmented prostate carcinomas on different MRI maps (T2, DCE, DWI, ADC).
ITK Snap
For our Covid-19 project in the field of pneumology, we used ITK Snap to segment high-resolution CT images of Covid-19 pneumonias to make disease-typical lesions identifiable using AI.
ImFusion Labels
We also used ImFusion Labels in the Covid-19 project mentioned above.
QuPath
We used QuPath to segment tumor masses in squamous cell carcinomas and basal cell carcinomas.
Because we did a variety of projects and spent many hours using the tools, we are now able to compare the technical capabilities as well as the usability of the four segmentation tools.
The evaluation of the segmentation tools
Before we go into the evaluation of the tools themselves, we want to give a short disclaimer. These evaluations are based on our own experiences with the tools in very specific projects. It is important to note that every project is different and has its own requirements. Therefore, it is impossible to give a general recommendation for or against any of the tools.
3D Slicer
The most commonly used tool
3D Slicer is one of the most common tools for segmenting and annotating medical image data. As for manual segmentation, it has a wide range of functions, reliable performance in storage and processing as well as precise results. 3D Slicer has a wide range of possible use scenarios and even very special use cases can be mapped. It is particularly helpful that exact regions can be selected, so that even two neighbouring structures are segmented very precisely. Using the great thresholding function as a mask for manual segmentation in the marked areas can save you a lot of time.
For automatic segmentation extensions are needed. For example, we found an extension in the library for rare image formats that enabled us to read confocal image data in a dermatology project. An extension from graphics manufacturer Nvidia enabled us to perform auto-segmentation of soft tissues and internal organs, which otherwise requires a relatively large amount of manual segmentation work. For some body structures these extensions work well, but for others the results are not very precise.
The user interface of 3D Slicer has a certain complexity and therefore the tool requires a learning curve. Luckily, there is a lively user community in which a lot of information and learning resources have become available over time. This makes it easier for beginners to get started with the tool.
Manual segmentation ★★★
Automatic segmentation ★★☆
Thresholding ★★★
User interface ★★☆
Learning resources ★★★
MITK Workbench
A clear user interface for beginners
MITK Workbench has a very comprehensive range of functions. One feature we have come to particularly appreciate is the automatic 3D interpolation. After manually segmenting a selection of slices at regular intervals, this function automatically completes the slices in between. This way, a considerable amount of time can be saved. In a urology project, we were able to divide the extent of the prostate into 25 slices, of which we only segmented ten manually. The remaining slices were completed by interpolation. While some other tools offer similar features, with MITK Workbench we found the calculated result most convincing.
MITK Workbench has an hierarchical arrangement of the objects, which allows for quick orientation. The layout of the tool appears clear and tidy, which supports focused work. Only the number of settings for thresholding and manual segmentation is somewhat lower than in other tools. If the still very extensive features are sufficient for your application, you will appreciate the user interface and good interpolation results. There are some learning resources available, such as instructional videos on YouTube and there are written instructions. These are harder to work with as you miss the visual aspect and unfortunately the videos are not available in a structured way.
Manual segmentation ★★☆
Automatic segmentation ★★☆
Thresholding ★★☆
User interface ★★★
Learning resources ★★☆
ITK Snap
A tool for fast results
ITK Snap served us well in segmenting part of the image data in the Covid-19 project. One of its biggest advantages is the so-called ‘snake’ function, which allows large structures to be segmented quickly and reliably. This tool is based on thresholding and looks at the pixel information in the image. Since time was a particularly important factor in this project, using ITK Snap was worthwhile. In the future, we will probably mainly use the tool for projects where we need results fast, but where a multitude of settings is not important.
However, in our view, the user interface is rather confusing compared to other tools. In addition, manually created segmentations can easily be overwritten by mistake. This may make sense in certain projects, but for our project this feature was a disadvantage. We also missed the possibility to set editable areas with this tool. There are good instruction videos available on the Snake functions, but for other functions you have to rely on the written instructions.
Manual segmentation ★☆☆
Automatic segmentation ★★★
Thresholding ★★☆
User interface ★☆☆
Learning resources ★★☆
ImFusion Labels
A very customer-oriented tool
In this evaluation, ImFusion Labels is the only tool from a commercial supplier that is not free of charge. However, there is a free trial version, which allows you to test it properly before you buy it. What we particularly like about ImFusion Labels is the user-friendly, clear interface and the performance of automatic segmentation. The latter also compensates for the smaller range of functions of the manual segmentation.
The tool, which is still quite new and under constant development, has not yet formed an extensive user community and therefore not many learning resources are available on the internet. However, the team behind the tool is very customer-oriented and we were impressed by the enthusiasm with which ImFusion Labels responds to customer queries. When we didn’t know what to do, we were helped in a competent way and the team even programmed an extension especially for us. At the moment there are not as many preset options available for thresholding as in 3D Slicer, but it is possible to get a personalised segmentation tool based on the needs in your project.
Manual segmentation ★★☆
Automatic segmentation ★★★
Thresholding ★★☆
User interface ★★★
Learning resources ★☆☆
QuPath
The best tool for data with one slice
QuPath is our favourite tool for segmenting histological data with only one slice. The tool is very reliable and produces precise results when using the semi-automatic “wand tool” function that looks at pixel information. Automatic annotations can be made in searched regions by, for example, cell shape and cell size. By using QuPath, we managed to reduce the annotation time from six to three hours, which is an impressive reduction of 50%.
It is important to note that the tool is great for histological data with one slice, but it is not meant for volumetric data. To edit file formats, you can use the linked ImageJ. This requires a certain amount of prior knowledge, as QuPath isn’t as intuitive as other tools. To just save your data, you already need scripts for AI applications that are provided by developers. The learning resources that are available are structured and easy to understand.
Manual segmentation ★★☆
Automatic segmentation ★★★
Thresholding ★★★
User interface ★★☆
Learning resources ★★★
The results: comparing the four segmentation tools
This is how the segmentation tools compare to each other on the five established criteria:
For beginners, a tool such as MITK Workbench could be a good choice, as it has a very clear user interface and is intuitive in its use. Also 3D Slicer could be a good place to start, as it has many learning resources available, which is beneficial especially for beginners. For the specific use of segmenting histological data with one slice, QuPath is our favourite choice given that you have some prior knowledge with segmentation tools. For projects that need fast results, but that don’t require many settings, ITK Snap is very suitable. If you’re in need of a more personalised tool with a customer-oriented approach, then it could be worthwhile looking into ImFusion Labels.
So, what is the best tool to use? As you can tell, there is not one clear winner in our evaluation. Our conclusion is that there is not one tool that performs best in all situations. Each tool has its own advantages and disadvantages and all five have proven themselves in specific situations.
A better question to ask is: which tool is the best choice for my project? Before choosing a tool, you should consider whether you need automatic segmentation, manual segmentation or both. Another important factor is your knowledge level of segmentation tools. This determines whether or not you will need a clear user-interface and learning resources. The best way to choose a segmentation tool is to first determine the specific needs of your project and to then find a tool that matches with your requirements.
Do you need help to find the right tool for your project?
We are happy to advise you on the design of your medical AI project and to help you find the right segmentation tool for your project.
