The following workflow will demonstrate how to use transfer learning in KNIME for image segmentation and prediction in a brain MRI scan dataset.
Brain MRI Segmentation: https://www.kaggle.com/mateuszbuda/lgg-mri-segmentation
MRI Scan Segmentation & Prediction Workflow: https://tinyurl.com/yhrfs8y8
This is what the dataset looks like. Each MRI Scan contains an associated tumour mask.
We first read all the tif image files using the image reader node. For the image pre-processing, we are mainly doing the 4 following things:
- Appending each mask next to their MRI scan so we can view them per row
- Filtering out the images that do not have tumours
- Resizing the images to 256x256
- Normalizing the images
Although a bit tedious, we have to use a combination of string manipulation, row filter, column rename and joiner nodes to append the mask of each MRI scan next to each other, as they are read as individual pictures the first time. There’s no correct way to do this, as long as we arrive at the same output as seen below.
After appending the mask images in the adjacent column as so, we can then use the image features node to filter out the images that have a blank (black) mask, indicating there are no tumours. Under the column selection, select IMAGE = IMAGE_MASK and under features, tick FIRST ORDER STATISTICS. Select MAX under features and click ok.
Drag and drop a row filter after the image feature node and filter out any rows that contain a lower bound of 255.0. This will filter out all the rows where the labeling mask is empty. Before we move on to #3 and #4, we need to convert the mask to a labeling so that KNIME can superimpose the tumour mask on top of the Brain MRI for viewing. Use the image to labeling node and include the IMAGE MASK within the green box. Under the options tab, tick USE BACKGROUND VALUE AS BACKGROUND and select BACKGROUND VALUE = 0. You can now append the labeling to the same row as the brain MRI scan, and you can use the interactive segmentation view to see the tumours, superimposed on top of the original image.
Now use the image calculator and normalize both the original MRI scan and the mask. You can do this by including the following expression in the expressions column.
The RESULT PIXEL TYPE = FLOAT TYPE. We then resize the image to 256x256x3 for the original brain scan, and 256x256x1 for the mask. Use a joiner to combine these 2 tables back together.
For partitioning we are going to use a 70-30 split. First we partition the test and non-test data set. Then from the non-test data set, we partition the training and validation set.
We are going to use the DL Python Network Creator to import the Unet architecture. You would need to import it using segmentation_models, and this has to also be installed on your virtual environment. We are also going to implement LeakyReLU in the architecture, which is an advanced loss function. Follow the figure below to put in the code for this node.
The DL Python Network Editor is used to implement the LeakyReLU in the architecture, by simply using the following code.
The Keras Network Learner will be configured similarly to our previous workflows, other than the epoch of which we will change to 80. After testing, a lower epoch count has been shown to sometimes give a blank prediction for the output mask for smaller tumours which leads to lower accuracy. Increasing the epochs has been shown to solve the issue, so 80 epochs are the recommended minimum. The Keras Network Writer node is optional, as this will save your trained model to a .h5 file to be used in other workflows or projects if needed.
The Keras Network Executor will be configured as follows,
We will first convert the predictions (which we named sigmoid) and the masks to an unsigned byte type using the image converter node. We then normalize both items using the following configuration in the image normalizer node,
Then, by using the image calculator node we can then subtract the predicted mask from the original label so that we can see the difference in the predictions visually. This is more of a subjective measure on how to visually see the difference, and more can be done in order to numerically calculate the accuracy if needed. We then convert the image converter to convert the images to bit type and use the compare segments node to see the pixel agreement between the original mask and the predicted. You should receive an output such as this at the end,
