Calculating ROIs based off of labels

[ajc3xc]

When I was working on my plant phenotyping Capstone project for MarsFarm, when I was trying to calculate rois automatically for a diverse set of images I developed a program to automatically calculate roi points and roi sizes based off of plant labeled masks. I used the midpoint of each label as the points of the rois, and I used a special algorithm to calculate roi size. Below is a repository containing an example running this program, where calculate_rois.py is the program used for calculating the rois. Here is the repository.

[HaleySchuhl]

Hi @ajc3xc thank you very much for opening this issue and contributing this new feature. It seems helpful! Are you interested in drafting a Pull Request yourself or would you prefer someone on our team take the lead on the PR?

Also, I ran into an error while testing on the example image provided in your repository. I haven't tried troubleshooting this error yet. Have you been able to resolve this in your own environment?

TypeError                                 Traceback (most recent call last)
Cell In[2], line 42
     39 labeled_image, number_of_plants = pcv.create_labels(mask=cleaned_mask)
     41 #get the centers and optimal radius size
---> 42 centers, optimal_radius_size = auto_roi(labeled_image)
     44 #now that this is calculated, use pcv.roi.multi to automatically generate the rois
     45 rois = pcv.roi.multi(img=img, coord=centers, radius=optimal_radius_size)

TypeError: cannot unpack non-iterable auto_roi object

• the workflow I'm using

from plantcv import plantcv as pcv
import cv2
import numpy as np
from skimage import morphology
from calculate_rois import auto_roi

image_path = "plant_image.jpg"

img, _, _ = pcv.readimage(image_path)

#mask the image with an example mask. Filter out small areas
# Convert to HSV and LAB color spaces
hsv_image = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
lab_image = cv2.cvtColor(img, cv2.COLOR_BGR2LAB)

# Define HSV range for filtering using OpenCV
# OpenCV uses 0-180 for Hue, so the values are halved
hsv_min = np.array([int(28/2), int(20/100*255), int(20/100*255)])
hsv_max = np.array([int(144/2), 255, 255])
hsv_mask = cv2.inRange(hsv_image, hsv_min, hsv_max)

# Define LAB range for filtering using OpenCV
# OpenCV uses 0-255 for L, a*, and b*
# Note: 'a' and 'b' ranges need to be shifted from [-128, 127] to [0, 255]
# L is scaled from [0, 100] in LAB to [0, 255] in OpenCV
lab_lower = np.array([int(10/100*255), 0, 132])
lab_upper = np.array([int(90/100*255), 124, 255])
lab_mask = cv2.inRange(lab_image, lab_lower, lab_upper)

# Combine the masks (logical AND) and apply to the original image
combined_mask = cv2.bitwise_and(hsv_mask, lab_mask)

#remove small holes in the image so object detection won't be as bad
# Remove small objects
cleaned_mask = morphology.remove_small_objects(combined_mask, min_size=300)

#generate automatic labels from denoised mask
#create labels for masks that may have plants, count number of objects
labeled_image, number_of_plants = pcv.create_labels(mask=cleaned_mask)

#get the centers and optimal radius size
centers, optimal_radius_size = auto_roi(labeled_image)

#now that this is calculated, use pcv.roi.multi to automatically generate the rois
rois = pcv.roi.multi(img=img, coord=centers, radius=optimal_radius_size)from plantcv import plantcv as pcv
import cv2
import numpy as np
from skimage import morphology
from calculate_rois import auto_roi

image_path = "plant_image.jpg"

img, _, _ = pcv.readimage(image_path)

#mask the image with an example mask. Filter out small areas
# Convert to HSV and LAB color spaces
hsv_image = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
lab_image = cv2.cvtColor(img, cv2.COLOR_BGR2LAB)

# Define HSV range for filtering using OpenCV
# OpenCV uses 0-180 for Hue, so the values are halved
hsv_min = np.array([int(28/2), int(20/100*255), int(20/100*255)])
hsv_max = np.array([int(144/2), 255, 255])
hsv_mask = cv2.inRange(hsv_image, hsv_min, hsv_max)

# Define LAB range for filtering using OpenCV
# OpenCV uses 0-255 for L, a*, and b*
# Note: 'a' and 'b' ranges need to be shifted from [-128, 127] to [0, 255]
# L is scaled from [0, 100] in LAB to [0, 255] in OpenCV
lab_lower = np.array([int(10/100*255), 0, 132])
lab_upper = np.array([int(90/100*255), 124, 255])
lab_mask = cv2.inRange(lab_image, lab_lower, lab_upper)

# Combine the masks (logical AND) and apply to the original image
combined_mask = cv2.bitwise_and(hsv_mask, lab_mask)

#remove small holes in the image so object detection won't be as bad
# Remove small objects
cleaned_mask = morphology.remove_small_objects(combined_mask, min_size=300)

#generate automatic labels from denoised mask
#create labels for masks that may have plants, count number of objects
labeled_image, number_of_plants = pcv.create_labels(mask=cleaned_mask)

#get the centers and optimal radius size
centers, optimal_radius_size = auto_roi(labeled_image)

#now that this is calculated, use pcv.roi.multi to automatically generate the rois
rois = pcv.roi.multi(img=img, coord=centers, radius=optimal_radius_size)

[ajc3xc]

Hi @ajc3xc thank you very much for opening this issue and contributing this new feature. It seems helpful! Are you interested in drafting a Pull Request yourself or would you prefer someone on our team take the lead on the PR?

Also, I ran into an error while testing on the example image provided in your repository. I haven't tried troubleshooting this error yet. Have you been able to resolve this in your own environment?

TypeError                                 Traceback (most recent call last)
Cell In[2], line 42
     39 labeled_image, number_of_plants = pcv.create_labels(mask=cleaned_mask)
     41 #get the centers and optimal radius size
---> 42 centers, optimal_radius_size = auto_roi(labeled_image)
     44 #now that this is calculated, use pcv.roi.multi to automatically generate the rois
     45 rois = pcv.roi.multi(img=img, coord=centers, radius=optimal_radius_size)

TypeError: cannot unpack non-iterable auto_roi object

• the workflow I'm using

from plantcv import plantcv as pcv
import cv2
import numpy as np
from skimage import morphology
from calculate_rois import auto_roi

image_path = "plant_image.jpg"

img, _, _ = pcv.readimage(image_path)

#mask the image with an example mask. Filter out small areas
# Convert to HSV and LAB color spaces
hsv_image = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
lab_image = cv2.cvtColor(img, cv2.COLOR_BGR2LAB)

# Define HSV range for filtering using OpenCV
# OpenCV uses 0-180 for Hue, so the values are halved
hsv_min = np.array([int(28/2), int(20/100*255), int(20/100*255)])
hsv_max = np.array([int(144/2), 255, 255])
hsv_mask = cv2.inRange(hsv_image, hsv_min, hsv_max)

# Define LAB range for filtering using OpenCV
# OpenCV uses 0-255 for L, a*, and b*
# Note: 'a' and 'b' ranges need to be shifted from [-128, 127] to [0, 255]
# L is scaled from [0, 100] in LAB to [0, 255] in OpenCV
lab_lower = np.array([int(10/100*255), 0, 132])
lab_upper = np.array([int(90/100*255), 124, 255])
lab_mask = cv2.inRange(lab_image, lab_lower, lab_upper)

# Combine the masks (logical AND) and apply to the original image
combined_mask = cv2.bitwise_and(hsv_mask, lab_mask)

#remove small holes in the image so object detection won't be as bad
# Remove small objects
cleaned_mask = morphology.remove_small_objects(combined_mask, min_size=300)

#generate automatic labels from denoised mask
#create labels for masks that may have plants, count number of objects
labeled_image, number_of_plants = pcv.create_labels(mask=cleaned_mask)

#get the centers and optimal radius size
centers, optimal_radius_size = auto_roi(labeled_image)

#now that this is calculated, use pcv.roi.multi to automatically generate the rois
rois = pcv.roi.multi(img=img, coord=centers, radius=optimal_radius_size)from plantcv import plantcv as pcv
import cv2
import numpy as np
from skimage import morphology
from calculate_rois import auto_roi

image_path = "plant_image.jpg"

img, _, _ = pcv.readimage(image_path)

#mask the image with an example mask. Filter out small areas
# Convert to HSV and LAB color spaces
hsv_image = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
lab_image = cv2.cvtColor(img, cv2.COLOR_BGR2LAB)

# Define HSV range for filtering using OpenCV
# OpenCV uses 0-180 for Hue, so the values are halved
hsv_min = np.array([int(28/2), int(20/100*255), int(20/100*255)])
hsv_max = np.array([int(144/2), 255, 255])
hsv_mask = cv2.inRange(hsv_image, hsv_min, hsv_max)

# Define LAB range for filtering using OpenCV
# OpenCV uses 0-255 for L, a*, and b*
# Note: 'a' and 'b' ranges need to be shifted from [-128, 127] to [0, 255]
# L is scaled from [0, 100] in LAB to [0, 255] in OpenCV
lab_lower = np.array([int(10/100*255), 0, 132])
lab_upper = np.array([int(90/100*255), 124, 255])
lab_mask = cv2.inRange(lab_image, lab_lower, lab_upper)

# Combine the masks (logical AND) and apply to the original image
combined_mask = cv2.bitwise_and(hsv_mask, lab_mask)

#remove small holes in the image so object detection won't be as bad
# Remove small objects
cleaned_mask = morphology.remove_small_objects(combined_mask, min_size=300)

#generate automatic labels from denoised mask
#create labels for masks that may have plants, count number of objects
labeled_image, number_of_plants = pcv.create_labels(mask=cleaned_mask)

#get the centers and optimal radius size
centers, optimal_radius_size = auto_roi(labeled_image)

#now that this is calculated, use pcv.roi.multi to automatically generate the rois
rois = pcv.roi.multi(img=img, coord=centers, radius=optimal_radius_size)

The bug was due to an error in the code. I made some changes to the code to address the issue. I also added a jupyter notebook to make a visual example of its functionality with an example image.

I don't have much experience making a pull request. Would I want to base my changes off of main? Would there be a specific group of functions that this class would go under? I would be more than fine doing it, I just don't know too much about how they work. Thanks in advance for any information.

[HaleySchuhl]

Hi @ajc3xc apologies for the delay in response. This is great, I'm happy to help you get this code into a Pull Request. Yes, you'll want to make a new branch based on the main branch, and the location for the new code can go into plantcv.plantcv.roi.roi_methods.py. I haven't looked too deeply at your source code yet but I'm curious if you think the function could be refactored without a class or whether that structure is necessary for this tool. We've got a doc page about contributing but please don't hesitate to ask questions here that we don't cover in documentation.

I'll also make sure @nfahlgren has added you as a contributor to the repository so you can open a branched PR rather than a fork. Thanks in advance!

[ajc3xc]

@HaleySchuhl I wasn't able to add a new branch so I made a fork to add my code into plantcv.plantcv.roi.roi_methods.py. I converted the class into a function so that it would be consistent with the rest of the code. There are subfunctions within the main function, so it is a bit hard to follow. I have tested out the code and it seemed to work. Here is the fork.

Also, the location and example image where I tested out this code on one image is here

[HaleySchuhl]

This is fantastic @ajc3xc thank you. Since this message I have reached out to Noah and you should now have permissions to branch and copy those changes in plantcv.plantcv.roi.roi_methods.py over. Then you can publish the branch here to GitHub, and from there, I'm happy to help you with the other components of a pull request (namely the init files, adding unit test(s), adding a documentation page and including the new doc page in the table of contents mkdocs.yml). This is outlined here but I'm also happy to setup a quick zoom call to talk through the process, just let me know if this would be helpful.

[ajc3xc]

Ok, the changes should be under the branch label_rois. Glad to contribute to this project!

[HaleySchuhl]

@ajc3xc this is great thank you! I went ahead and just opened a Pull Request from your branch, and I will pull down the changes that were merged into PlantCV this past Friday. I aim to review the new function tomorrow and will ping you with any comments.