ECSE_415_Project_Winter_2023 - Copy

pdf

School

McGill University *

*We aren’t endorsed by this school

Course

415

Subject

Computer Science

Date

Jan 9, 2024

Type

pdf

Pages

7

Uploaded by ElderDanger102077

Report
ECSE 415 Introduction to Computer Vision Final Project: Segmentation in Medical Imaging Due: April 12, 2023, 11:59 PM February 20, 2023 1 Introduction The goal of this project is to implement a complete pipeline to segment and detect nuclei in medical images. The pipeline will consist of a number of steps: (1) a dataset pipeline (10 points) (2) an image segmentor (60 points), and (4) a detector (30 points). For this project, you will form a team of 4-5 people. The objective of the project is to permit your team to explore a sequence of concepts described in class. Projects should be submitted electronically via myCourses. All submitted materials are expected to be the students’ own work or appropriately cited (see Academic Integrity guide- lines in the Syllabus). Software packages (e.g. scikit-learn, MATLAB toolboxes, Keras, PyTorch, etc.) that implement machine learning-related algorithms (SVM, RF, k-fold cross- validation, etc.) are allowed. Appropriate citations are expected. The use of external code without citation will be treated as plagiarism. Projects received after the deadline up to 24 hours late will be penalized by 30%. Projects received up to 24 hours late will not be graded. The project will be graded out of a total 100 points. 2 Submission Instructions 1. Submit (i) report in pdf format, (ii) all code 2. Comment your code appropriately. 3. Make sure that the submitted code is running without error. Add a README file if required. 4. The report should be comprehensive but concise. It should not exceed 10 pages ex- cluding references. 5. If external libraries were used in your code, please specify them by name and version in the requirement.txt file. 6. Submissions that do not follow the format will be penalized 10%. 1
3 Dataset [10 points] The complete dataset is available at the link- click here. The organization folder after unzipping the downloaded data will be: ECSE 415 csv mask rgb Details: rgb: input images. All images are at 0.2 microns-per-pixel. All coordinate values in tables are pixel units at 0.2 microns-per-pixel. csv: annotation coordinates mask: masks for the input images. The first channel in each mask image encodes the class labels. The product of the second and third channels encode the unique instance label for each nucleus. The fov area (gray) is included in the first channel of the mask. The naming convention in all the files follows the fomat TCGA-A1-A0SP-DX1 id-[..] left-[a] top-[b] bottom-[c] right-[d] . In this particular example A1 is the hospital name, A0SP-DX1 is the unique patient id, and a , b , c and d are the coordinates. So a combination of the first 4 strings i.e, TCGA-A1-A0SP-DX1 is unique for a patient. This dataset contains multiple samples of the same subject i.e, many input image can belong to the same patient which can be identified the unique id TCGA-A1-A0SP-DX1 in the filename . Prepare a data pipeline to perform the following tasks: 1. Plot three images for a sample subject - (i) an input image, (ii) the corresponding mask, and (iii) overlay the periphery of the mask onto the image to draw bounding boxes around different types of nuclei and label them. Sample output is shown in Figure 1 [5 points] Figure 1: Visualization of the original Image (left image), mask (middle image) and nuclei labels (right image). 2
2. Loop over all the image files to obtain unique subjects and then create a train- ing and validation split, i.e., put 80% of patients in training and 20% of data in the validation. Save training and validation splits as train subjects.csv and validate subjects.csv files. The file organization structures should look as be- low: ( Hint: Your training data will have approximately 100 unique subjects ). [5 points]: ECSE 415 csv mask rgb splits train subjects.csv validate subjects.csv Please note: Teams are not allowed to use any pre-trained models (except in the bonus question) or extra data apart from what is being provided. 4 Segmentation [60 points] The first part of the project is to segment the medical images from the background. For each image, there is a corresponding binary ground truth set of labels. Your task is to generate a binary mask for the given RGB image. The use of deep learning algorithms including CNNs is strictly prohibited for this task. Use at least two different segmentation methods. Optimize the hyperparameters of your segmentation algorithms. 4.1 Segmentation - Unsupervised Using K-mean clustering, segment all input images for subjects in train subjects.csv , and answer the following questions: 1. Find an optimized value of K. Show the results for K=3 and K= 5. [4 points] 2. How did you decide the optimized value of K? Explain your answer in the context of computing DICE coefficients. [4 points] 3. Show (i) the original image, (ii) the segmented image for the optimized value of K. (iii) the ground truth segmentation, available in the csv file corresponding to the image you have chosen. [4 points] 4. Do your segmentation results improve if you use a grayscale image compared to an RGB image? Show 3 relevant examples to support your answer. [6 points] 5. You have studied a number of other unsupervised segmentation methods in class. Pick one and compare the results to the K-means results (for the optimal K found above). Run your code on the RGB image. [6 points] 3
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
4.2 Segmentation - Supervised Given that the mask for different nuclei groups (e.g. tumor , fibroblast ) are available, these labels are provided at every pixel for all the images. Supervised learning segmentation can then be performed by casting the problem into a supervised classification framework (i.e. a class assigned to every pixel). In this question, you will now perform supervised segmentation by training on the labelled training images. You are free to use any supervised learning classification method (e.g. random forest, SVM) you learned in class. The first step would be to convert the input RGB images to grayscale (the input images for this should be the same as that in section 4.1). Please answer the following questions: 1. Describe the supervised approach you are using. [4 points] 2. Report the performance of your supervised segmentor on the training images. [6 points] 3. Report the performance of your supervised segmentor on the validation images (images for the subjects in val subjects.csv ). [6 points] 4. Select a few samples (at least 3) from the validation images and compare the perfor- mance of the supervised and unsupervised methods (i) qualitatively and (ii) quanti- tatively. Discuss which method (supervised or unsupervised) has better performance. [10 points] 5. For a given subject, which nuclei group can be segmented easily? Discuss by showing some sample visualizations. [6 points] 6. Why do you think the nuclei you mentioned in the previous question are easy to segment? [4 points] 5 Detecting and Counting Tumors [30 points] For this section, you are required to implement a supervised learning framework to detect and then count tumors. First, your will extract features from the tumor and non-tumor masked pathological regions. You are free to use any computer vision features discussed in class or others you have found online (e.g. SIFT, SURF, HoG, Bag of Words, etc.). Keep in mind that multiple feature extractors can be combined. The use of deep learning, including CNNs, is strictly prohibited for this task. Once features are extracted, train a Support Vector Machine (SVM) classifier using your computed features and the ground truth labels to detect tumors from non-tumors. Optimize the hyperparameters (e.g., number of features, thresholds, SVM kernel, etc.) for the feature extraction stage and for the detection/classification framework you are designing. You are expected to provide a count of the number of instances of tumors for a given input image. 5.1 Preparing the data Follow the steps mentioned below to build a supervised detector/classifier. 4
1. Convert input images and masks to grayscale. 2. Multiply the mask and the original image to obtain the regions of interest. ( Note: It is possible that size of image and mask differ by a number, to avoid this slice either the image or the mask so that both are of the same size ) 3. Remove the background noise in the resultant image. For the purpose of better under- standing, a sample image after these operations are shown in Fig. 2. 4. Extract features of the resultant image that has different pathological structures (blobs). 5. Using these features, build a detection model to segregate tumor pathologies from non-tumor pathologies. 6. The target label for tumors and non-tumors can be obtained from the corresponding csv file of the image. 7. The count for this image would be the number of tumor in the given image i.e, count the number of instances of tumor for the given image. The ground truth count can be obtained by reporting the number of instances of tumor from the corresponding csv file. 8. Repeat the steps mentioned above, to loop over all the images for the subjects in train subjects.csv to construct a python data frame with X train as features and Y train as the target label of tumors and non-tumors. Similarly, create X val and Y val for the images of subjects in val subjects.csv . Figure 2: Steps for preparing the input data for detection. The number of tumors detected in this case is 14. 5.2 Classification Framework Train a supervised classification model to label each of the masked pathological structures in the images and then identify (detect) the tumors in each image. After training is done, you are required to detect the tumors and then count them on a new image. Report the following metrics, based on the regression model you have created: 5
1. Report performance metrics of the detection model for both training and validation images. [10 points] 2. After detecting the tumor for all validation images, count the number of detector tumors in the image. Report the root mean squared error (RMSE) for this task of counting tumors. [5 points] 3. Is there a difference in the performance if you use the ‘original image’ in Fig. 2 (i.e. the unmasked image (Step 1)) instead of only classifying each of the provided pathological regions (i.e. the masked regions)? Provide a qualitative and quantitative discussion of your results. ( Hint: To answer this you will need to recreate X train , Y train , X val , and Y val . You are required to use the features from the original image for this task. ) [10 points] 4. Based on the features you used in the detection model, what pathological structures are easier to segregate from tumors? Justify your answer using appropriate metrics. [5 points] 6 Bonus question [10 points] For this section, you are required to perform a segmentation task on a single subject. Con- sider the patient with hospital id and subject id as TCGA-OL-A66P-DX1 with 58 samples. 1. Supervised Learning: Using any pre-trained deep learning model, fine-tune it to segment any random 50 samples. Report the performance on training data. Include key details of the implementation. [7 points] 2. Inference: Report the performance of this segmentor on the remaining 8 samples. [3 points] Appendix Definitions 1. Accuracy: The number of correct predictions divided by the total predictions. TP + TN TP + TN + FP + FN 2. DICE Coefficient: Intersection over union for predicted and true labels; 2 TP TP + TN + FP + FN . See figure 3. 3. Precision: True positives divided by true positives and false positives: TP TP + FP “Of the predictions made for class C, what fraction was correct?” 4. Recall: True positives divided by true positives and false negatives: TP TP + FN “Of the predictions made for class C, how many were correctly predicated?” 6
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
Figure 3: Schematic illustration of measuring the segmentation errors for calculating the Dice similarity coefficient (DSC). 7