FALL2023_HW3_Student_Bharat

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 1/71 Fall 2023 CS4641/CS7641 A Homework 3 Instructor: Dr. Mahdi Roozbahani Deadline: Friday, November 10th, 11:59 pm EST No unapproved extension of the deadline is allowed. Submission past our 48-hour penalized acceptance period will lead to 0 credit. Discussion is encouraged on Ed as part of the Q/A. However, all assignments should be done individually. Plagiarism is a serious offense . You are responsible for completing your own work. You are not allowed to copy and paste, or paraphrase, or submit materials created or published by others, as if you created the materials. All materials submitted must be your own.</font> All incidents of suspected dishonesty, plagiarism, or violations of the Georgia Tech Honor Code will be subject to the institute’s Academic Integrity procedures. If we observe any (even small) similarities/plagiarisms detected by Gradescope or our TAs, WE WILL DIRECTLY REPORT ALL CASES TO OSI , which may, unfortunately, lead to a very harsh outcome. Consequences can be severe, e.g., academic probation or dismissal, grade penalties, a 0 grade for assignments concerned, and prohibition from withdrawing from the class. </font> Instructions for the assignment This assignment consists of both programming and theory questions. Unless a theory question explicitly states that no work is required to be shown, you must provide an explanation, justification, or calculation for your answer. To switch between cell for code and for markdown, see the menu -> Cell -> Cell Type You can directly type Latex equations into markdown cells. If a question requires a picture, you could use this syntax <img src="" style="width: 300px;"/> to include them within your ipython notebook. Your write up must be submitted in PDF form. You may use either Latex, markdown, or any word processing software. We will **NOT** accept handwritten work. Make sure that your work is formatted correctly, for example submit instead of \text{sum_{i=0} x_i} When submitting the non-programming part of your assignment, you must correctly map pages of your PDF to each question/subquestion to reflect where they appear. **Improperly mapped questions may not be graded correctly and/or will result in point deductions for the error.**

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 2/71 All assignments should be done individually, and each student must write up and submit their own answers. Graduate Students : You are required to complete any sections marked as Bonus for Undergrads Using the autograder Grads will find three assignments on Gradescope that correspond to HW3: "Assignment 3 Programming", "Assignment 3 - Non-programming" and "Assignment 3 Programming - Bonus for all". Undergrads will find an additional assignment called "Assignment 3 Programming - Bonus for Undergrads". You will submit your code for the autograder in the Assignment 3 Programming sections. Please refer to the Deliverables and Point Distribution section for what parts are considered required, bonus for undergrads, and bonus for all. We provided you different .py files and we added libraries in those files please DO NOT remove those lines and add your code after those lines. Note that these are the only allowed libraries that you can use for the homework. You are allowed to make as many submissions until the deadline as you like. Additionally, note that the autograder tests each function separately, therefore it can serve as a useful tool to help you debug your code if you are not sure of what part of your implementation might have an issue. For the "Assignment 3 - Non-programming" part, you will need to submit to Gradescope a PDF copy of your Jupyter Notebook with the cells ran. See this EdStem Post for multiple ways on to convert your .ipynb into a .pdf file. Please refer to the Deliverables and Point Distribution section for an outline of the non- programming questions. When submitting to Gradescope, please make sure to mark the page(s) corresponding to each problem/sub-problem. The pages in the PDF should be of size 8.5" x 11", otherwise there may be a deduction in points for extra long sheets. Using the local tests For some of the programming questions we have included a local test using a small toy dataset to aid in debugging. The local test sample data and outputs are stored in .py files in the local_tests_folder . The actual local tests are stored in localtests.py . There are no points associated with passing or failing the local tests, you must still pass the autograder to get points. It is possible to fail the local test and pass the autograder since the autograder has a certain allowed error tolerance while the local test allowed error may be smaller. Likewise, passing the local tests does not guarantee passing the autograder. You do not need to pass both local and autograder tests to get points, passing the Gradescope autograder is sufficient for credit.

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 4/71 RMSE [5pts] Construct Poly Features 1D [2pts] Construct Poly Features 2D [3pts] Prediction [5pts] Linear Fit Closed Form [5pts] Ridge Fit Closed Form [5pts] Ridge Cross Validation [5pts] Linear Gradient Descent [5pts] Bonus for Undergrad Linear Stochastic Gradient Descent [5pts] Bonus for Undergrad Ridge Gradient Descent [5pts] Ridge Bonus for Undergrad Ridge Stochastic Gradient Descent [5pts] Ridge Bonus for Undergrad 3.2 About RMSE [3 pts] non-programming 3.3 Testing: General Functions and Linear Regression [5 pts] non-programming 3.4 Testing: Ridge Regression Ridge [5 pts + 2 pts Bonus for All] non-programming 3.5 Cross Validation [7 pts] non-programming 3.6 Noisy Input Samples in Linear Regression [10 pts] non-programming BONUS FOR ALL Q4: Naive Bayes and Logistic Regression [35pts] Deliverables: logistic_regression.py and Written portion 4.1 Llama Breed Problem using Naive Bayes [5 pts] non-programming 4.2 News Data Sentiment Classification Using Logistic Regression [30 pts] - programming sigmoid [2 pts] bias_augment [3 pts] predict_probs [5 pts] predict_labels [2 pts] loss [3 pts] gradient [3 pts] accuracy [2 pts] evaluate [5 pts]

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 5/71 fit [5 pts] Q5: Noise in PCA and Linear Regression [15pts] Deliverables: Written portion 5.1 Slope Functions [5 pts] non-programming 5.2 Error in Y and Error in X and Y [5 pts] non-programming 5.3 Analysis [5 pts] non-programming Q6: Feature Reduction.py [25pts Bonus for All] Deliverables: feature_reduction.py and Written portion 6.1 Feature Reduction [18 pts] - programming forward_selection [9pts] backward_elimination [9pts] 6.2 Feature Selection - Discussion [7 pts] non-programming Q7: Movie Recommendation with SVD [10pts Bonus for All] Deliverables: svd_recommender.py and Written portion 7.1 SVD Recommender recommender_svd [5pts] predict [5pts] 7.2 Visualize Movie Vectors [0pts] 0 Set up This notebook is tested under python 3. . , and the corresponding packages can be downloaded from miniconda . You may also want to get yourself familiar with several packages: jupyter notebook numpy matplotlib sklearn There is also a VS Code and Anaconda Setup Tutorial on Ed under the "Links" category Please implement the functions that have raise NotImplementedError , and after you finish the coding, please delete or comment out raise NotImplementedError .

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 7/71 <matplotlib.image.AxesImage at 0x10772b450> transform = fig . transFigure , fontsize = EO_SIZE , color = EO_COLOR , alpha = EO_ALPHA , fontname = EO_FONT , ha = "center" , va = "center" , rotation = EO_ROT , ) plt . imshow ( image ) Out[ ]: In [ ]: ############################### ### DO NOT CHANGE THIS CELL ### ############################### def rgb2gray ( rgb ): return np . dot ( rgb [ ... , : 3 ], [ 0.299 , 0.587 , 0.114 ]) fig = plt . figure ( figsize = ( 10 , 10 )) if not STUDENT_VERSION : fig . text ( 0.5 , 0.5 , EO_TEXT ,

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 8/71 <matplotlib.image.AxesImage at 0x14e299050> 1.1 Image compression [20pts] **[P]** SVD is a dimensionality reduction technique that allows us to compress images by throwing away the least important information. Higher singular values capture greater variance and, thus, capture greater information from the corresponding singular vector. To perform image compression, apply SVD on each matrix and get rid of the small singular values to compress the image. The loss of information through this process is negligible, and the difference between the images can be hardly spotted. transform = fig . transFigure , fontsize = EO_SIZE , color = EO_COLOR , alpha = EO_ALPHA , fontname = EO_FONT , ha = "center" , va = "center" , rotation = EO_ROT , ) # plot several images plt . imshow ( rgb2gray ( image ), cmap = plt . cm . bone ) Out[ ]:

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 10/71 1.1.2 Local Tests for Image Compression Color Case [No Points] You may test your implementation of the functions contained in imgcompression.py in the cell below. Feel free to comment out tests for functions that have not been completed yet. See Using the Local Tests for more details. UnitTest passed successfully for "SVD calculation - color images"! UnitTest passed successfully for "Image compression - color images"! UnitTest passed successfully for "SVD reconstruction - color images"! UnitTest passed successfully for "Compression ratio - color images"! UnitTest passed successfully for "Recovered variance proportion - color images"! 1.2.1 Black and white [5 pts] **[W]** This question will use your implementation of the functions from Q1.1 to generate a set of images compressed to different degrees. You can simply run the below cell without making any changes to it, assuming you have implemented the functions in Q1.1. Make sure these images are displayed when submitting the PDF version of the Juypter notebook as part of the non-programming submission of this assignment. In [ ]: ############################### ### DO NOT CHANGE THIS CELL ### ############################### from utilities.localtests import TestImgCompression unittest_ic = TestImgCompression () unittest_ic . test_svd_color () unittest_ic . test_compress_color () unittest_ic . test_rebuild_svd_color () unittest_ic . test_compression_ratio_color () unittest_ic . test_recovered_variance_proportion_color () In [ ]: ############################### ### DO NOT CHANGE THIS CELL ### ############################### from imgcompression import ImgCompression imcompression = ImgCompression () bw_image = rgb2gray ( image ) U , S , V = imcompression . svd ( bw_image ) component_num = [ 1 , 2 , 5 , 10 , 20 , 40 , 80 , 160 , 256 ] fig = plt . figure ( figsize = ( 18 , 18 )) # plot several images i = 0 for k in component_num : U_compressed , S_compressed , V_compressed = imcompression . compress ( U , S , V , k ) img_rebuild = imcompression . rebuild_svd ( U_compressed , S_compressed , V_compresse c = np . around ( imcompression . compression_ratio ( bw_image , k ), 4 ) r = np . around ( imcompression . recovered_variance_proportion ( S , k ), 3 ) ax = fig . add_subplot ( 3 , 3 , i + 1 , xticks = [], yticks = []) ax . imshow ( img_rebuild , cmap = plt . cm . bone ) ax . set_title ( f"{ k } Components" ) if not STUDENT_VERSION : ax . text (

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 11/71 1.2.2 Black and White Compression Savings [No Points] This question will use your implementation of the functions from Q1.1 to compare the number of bytes required to represent the SVD decomposition for the original image to the compressed image using different degrees of compression. You can simply run the below cell without making any changes to it, assuming you have implemented the functions in Q1.1. 0.5 , 0.5 , EO_TEXT , transform = ax . transAxes , fontsize = EO_SIZE / 2 , color = EO_COLOR , alpha = EO_ALPHA , fontname = EO_FONT , ha = "center" , va = "center" , rotation = EO_ROT , ) ax . set_xlabel ( f"Compression: { c },\nRecovered Variance: { r }" ) i = i + 1

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 13/71 HINT 2: Try performing SVD on the individual color channels and then stack the individual channel , , matrices. HINT 3: You may need separate implementations for a color or grayscale image in the same function. In [ ]: ############################### ### DO NOT CHANGE THIS CELL ### ############################### from imgcompression import ImgCompression imcompression = ImgCompression () image_rolled = np . moveaxis ( image , - 1 , 0 ) U , S , V = imcompression . svd ( image_rolled ) component_num = [ 1 , 2 , 5 , 10 , 20 , 40 , 80 , 160 , 256 ] fig = plt . figure ( figsize = ( 18 , 18 )) # plot several images i = 0 for k in component_num : U_compressed , S_compressed , V_compressed = imcompression . compress ( U , S , V , k ) img_rebuild = np . clip ( imcompression . rebuild_svd ( U_compressed , S_compressed , V_compressed ), 0 , 1 ) img_rebuild = np . moveaxis ( img_rebuild , 0 , - 1 ) c = np . around ( imcompression . compression_ratio ( image_rolled , k ), 4 ) r = np . around ( imcompression . recovered_variance_proportion ( S , k ), 3 ) ax = fig . add_subplot ( 3 , 3 , i + 1 , xticks = [], yticks = []) ax . imshow ( img_rebuild ) ax . set_title ( f"{ k } Components" ) if not STUDENT_VERSION : ax . text ( 0.5 , 0.5 , EO_TEXT , transform = ax . transAxes , fontsize = EO_SIZE / 2 , color = EO_COLOR , alpha = EO_ALPHA , fontname = EO_FONT , ha = "center" , va = "center" , rotation = EO_ROT , ) ax . set_xlabel ( f"Compression: { np . around ( c , 4 ) },\nRecovered Variance: R: { r [ 0 ] } G: { r [ 1 ] } ) i = i + 1

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 14/71 1.3.2 Color Compression Savings [No Points] This question will use your implementation of the functions from Q1.1 to compare the number of bytes required to represent the SVD decomposition for the original image to the compressed image using different degrees of compression. You can simply run the below cell without making any changes to it, assuming you have implemented the functions in Q1.1. Running this cell is primarily for your own understanding of the compression process. In [ ]: ############################### ### DO NOT CHANGE THIS CELL ### ############################### from imgcompression import ImgCompression imcompression = ImgCompression () U , S , V = imcompression . svd ( image_rolled ) component_num = [ 1 , 2 , 5 , 10 , 20 , 40 , 80 , 160 , 256 ] # Compare the memory savings of the color image i = 0

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 16/71 This means two important things for us: The matrix , often referred to as the right singular vectors of , is equivalent to the eigenvectors of . is equivalent to the eigenvalues of . So the first -principal components are obtained by projecting by the first vectors from . Similarly, gives a measure of the variance retained. 2.1 Implementation [10 pts] **[P]** Implement PCA. In the pca.py file, complete the following functions: fit : You may use np.linalg.svd . Set full_matrices=False . Hint 1 may be useful. transform transform_rv : You may find np.cumsum helpful for this function. Assume a dataset is composed of N datapoints, each of which has D features with D < N. The dimension of our data would be D. However, it is possible that many of these dimensions contain redundant information. Each feature explains part of the variance in our dataset, and some features may explain more variance than others. HINT 1: Make sure you remember to first center your data by subtracting the mean of each feature. 2.1.1 Local Tests for PCA [No Points] You may test your implementation of the functions contained in pca.py in the cell below. Feel free to comment out tests for functions that have not been completed yet. See Using the Local Tests for more details. UnitTest passed successfully for "PCA fit"! UnitTest passed successfully for "PCA transform"! UnitTest passed successfully for "PCA transform with recovered variance"! 2.2 Visualize [5 pts] **[W]** PCA is used to transform multivariate data tables into smaller sets so as to observe the hidden trends and variations in the data. It can also be used as a feature extractor for images. Here you will visualize two datasets using PCA, first is the iris dataset and then a dataset of masked and unmasked images. In [ ]: ############################### ### DO NOT CHANGE THIS CELL ### ############################### from utilities.localtests import TestPCA unittest_pca = TestPCA () unittest_pca . test_pca () unittest_pca . test_transform () unittest_pca . test_transform_rv ()

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 17/71 In the pca.py , complete the following function: visualize : Use your implementation of PCA and reduce the datasets such that they contain only two features. Using Plotly's Express make a 2d and 3d scatterplot of the data points using these features. Make sure to differentiate the data points according to their true labels using color. We recommend converting the data to a pandas dataframe before plotting. The datasets have already been loaded for you in the subsequent cells. NOTE: Here, we won't be testing for accuracy. Even with correct implementations of PCA, the accuracy can differ from the TA solution. That is fine as long as the visualizations come out similar. Iris Dataset In [ ]: ############################### ### DO NOT CHANGE THIS CELL ### ############################### # Use PCA for visualization of iris dataset from pca import PCA iris_data = load_iris ( return_X_y = True ) X = iris_data [ 0 ] y = iris_data [ 1 ] fig_title = "Iris Dataset with Dimensionality Reduction" PCA () . visualize ( X , y , fig_title )

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 19/71 2.3 PCA Reduced Facemask Dataset Analysis [5 pts] **[W]** Facemask Dataset The masked and unmasked dataset is made up of grayscale images of human faces facing forward. Half of these images are faces that are completely unmasked, and the remaining images show half of the face covered with an artificially generated face mask. The images have already been preprocessed, they are also reduced to a small size of 64x64 pixels and then reshaped into a feature vector of 4096 pixels. Below is a sample of some of the images in the dataset. Iris Dataset with Dimensionality Reduction (3D) In [ ]: ############################### ### DO NOT CHANGE THIS CELL ### ############################### X = np . load ( "./data/smallflat_64.npy" ) y = np . load ( "./data/masked_labels.npy" ) . astype ( "int" ) i = 0 fig = plt . figure ( figsize = ( 18 , 18 )) for idx in [ 0 , 1 , 2 , 150 , 151 , 152 ]: ax = fig . add_subplot ( 6 , 6 , i + 1 , xticks = [], yticks = []) image = ( np . rot90 ( X [ idx ] . reshape ( 64 , 64 ), k = 1 ) if idx % 2 == 1 and idx < 150 else X [ idx ] . reshape ( 64 , 64 ) )

11/10/23, 11:49 PM FALL2023_HW3_Student_Bharat file:///C:/Users/Arpit/Downloads/FALL2023_HW3_Student_Bharat.html 20/71 m_status = "Unmasked" if idx < 150 else "Masked" ax . imshow ( image , cmap = "gray" ) ax . set_title ( f"{ m_status } Image at i = { idx }" ) i += 1 In [ ]: ############################### ### DO NOT CHANGE THIS CELL ### ############################### # Use PCA for visualization of masked and unmasked images X = np . load ( "./data/smallflat_64.npy" ) y = np . load ( "./data/masked_labels.npy" ) fig_title = "Facemask Dataset Visualization with Dimensionality Reduction" PCA () . visualize ( X , y , fig_title ) print ( "*In this plot, the 0 points are unmasked images and the 1 points are masked im ) 4 6 8 Facemask Dataset Visualization with Dimensionality Reduct