homework2

f) [2 pts.] Give a few bullet points explaining the pros and cons of these algorithms and when and why we use logistic regression over linear regres- sion. 3 PCA - Dimensionality Reduction The original dataset contains 28 * 28 = 784 features. Therefore, we will try to reduce the dimension by using PCA. a) [1 pt.] Perform PCA decomposition, initially using all principal com- ponents . Before performing PCA you usually need to mean-center the data, see here why, which means you have to calculate the mean of each variable (column) and subtract it from the respective column. However, many libraries perform this step implicitly, so consult the documentation of the library you are going to use (e.g. PCA of Sklearn). b) [2 pts.] Plot the CDF of the explained variance as a function of the number of principal components. c) [1 pt.] Choose a number of principal components to use by arguing why your choice is reasonable as a trade-off between the number of com- ponents used and classification performance. Afterwards, train a kNN classifier (choose a k that gave you the best results in Problem 2c.) and report train and test accuracy . d) [5 pts.] For this part we will perform the following experiment: First , ran- domly sample a part of your dataset (using a fixed k and all features), of size ranging from 3,000 to 21,000 (the whole dataset) in increments of 3,000. Fit a kNN classifier and plot the running time. Now, use a fixed k and all samples of your dataset, but fit a kNN clas- sifier using a varying number of Principal Components , ranging from 50 to 750 in increments of 100. Plot the running time on the same plot as above . Describe the plot. What seems to affect -as a trend- the fitting time more? Number of samples used for trying, or the dimensions of the data? -Python’s time package may be useful for this problem. e) [3 pts.] Using your results from the previous question, produce the most accurate model you can on the condition that it is faster than 50% of the models you tested. List the values chosen for: k , number of samples, and number of principal components. How does this model compare to your most accurate model (the best model you created when time wasn’t a factor)? f) [1 pt.] Bonus point : Plot the images of the 10 first Principal Compo- nents. That is, plot the image approximation using each principle compo- nent. 3