Project2

Project 2 - Binary Classification Comparative Methods For this project we're going to attempt a binary classification of a dataset using multiple methods and compare results. Our goals for this project will be to introduce you to several of the most common classification techniques, how to perform them and tweek parameters to optimize outcomes, how to produce and interpret results, and compare performance. You will be asked to analyze your findings and provide explanations for observed performance. DEFINITIONS </u> Binary Classification: In this case a complex dataset has an added 'target' label with one of two options. Your learning algorithm will try to assign one of these labels to the data. Supervised Learning: This data is fully supervised, which means it's been fully labeled and we can trust the veracity of the labeling. Submission Details Project is due May 17th at 12:00 pm (Wednesday Noon). To submit the project, please save the notebook as a pdf file and submit the assignment via Gradescope. In addition, make sure that all figures are legible and su ff iciently large. For best pdf results, we recommend downloading Latex and print the notebook using Latex. Loading Essentials and Helper Functions In [ ]: #Here are a set of libraries we imported to complete this assignment. #Feel free to use these or equivalent libraries for your implementation import numpy as np # linear algebra import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv) import matplotlib.pyplot as plt # this is used for the plot the graph import matplotlib import os import time #Sklearn classes from sklearn.model_selection import train_test_split , cross_val_score , GridSearchCV , KFo from sklearn import metrics from sklearn.svm import SVC #SVM classifier from sklearn.linear_model import LogisticRegression from sklearn.neighbors import KNeighborsClassifier from sklearn.metrics import confusion_matrix import sklearn.metrics.cluster as smc from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler , OneHotEncoder , Normalizer , MinMaxScale from sklearn.compose import ColumnTransformer , make_column_transformer Loading [MathJax]/extensions/Safe.js

from matplotlib import pyplot import itertools % matplotlib inline #Sets random seed import random random . seed ( 42 ) In [ ]: # Helper function allowing you to export a graph def save_fig ( fig_id , tight_layout = True , fig_extension = "png" , resolution = 300 ): path = os . path . join ( fig_id + "." + fig_extension ) print ( "Saving figure" , fig_id ) if tight_layout : plt . tight_layout () plt . savefig ( path , format = fig_extension , dpi = resolution ) In [ ]: # Helper function that allows you to draw nicely formatted confusion matrices def draw_confusion_matrix ( y , yhat , classes ): ''' Draws a confusion matrix for the given target and predictions Adapted from scikit-learn and discussion example. ''' plt . cla () plt . clf () matrix = confusion_matrix ( y , yhat ) plt . imshow ( matrix , interpolation = 'nearest' , cmap = plt . cm . YlOrBr ) plt . title ( "Confusion Matrix" ) plt . colorbar () num_classes = len ( classes ) plt . xticks ( np . arange ( num_classes ), classes , rotation = 90 ) plt . yticks ( np . arange ( num_classes ), classes ) fmt = 'd' thresh = matrix . max () / 2. for i , j in itertools . product ( range ( matrix . shape [ 0 ]), range ( matrix . shape [ 1 ])): plt . text ( j , i , format ( matrix [ i , j ], fmt ), horizontalalignment = "center" , color = "white" if matrix [ i , j ] > thresh else "black" ) plt . ylabel ( 'True label' ) plt . xlabel ( 'Predicted label' ) plt . tight_layout () plt . show () In [ ]: def heatmap ( data , row_labels , col_labels , figsize = ( 20 , 12 ), cmap = "YlGn" , cbar_kw = {}, cbarlabel = "" , valfmt = " {x:.2f} " , textcolors = ( "black" , "white" ), threshold = None ): """ Create a heatmap from a numpy array and two lists of labels. Taken from matplotlib example. Parameters ---------- data A 2D numpy array of shape (M, N). row_labels A list or array of length M with the labels for the rows. col_labels A list or array of length N with the labels for the columns. ax Loading [MathJax]/extensions/Safe.js

if isinstance ( valfmt , str ): valfmt = matplotlib . ticker . StrMethodFormatter ( valfmt ) # Loop over the data and create a `Text` for each "pixel". # Change the text's color depending on the data. texts = [] for i in range ( data . shape [ 0 ]): for j in range ( data . shape [ 1 ]): kw . update ( color = textcolors [ int ( im . norm ( data [ i , j ]) > threshold )]) text = im . axes . text ( j , i , valfmt ( data [ i , j ], None ), ** kw ) texts . append ( text ) In [ ]: def make_meshgrid ( x , y , h = 0.02 ): """Create a mesh of points to plot in Parameters ---------- x: data to base x-axis meshgrid on y: data to base y-axis meshgrid on h: stepsize for meshgrid, optional Returns ------- xx, yy : ndarray """ x_min , x_max = x . min () - 1 , x . max () + 1 y_min , y_max = y . min () - 1 , y . max () + 1 xx , yy = np . meshgrid ( np . arange ( x_min , x_max , h ), np . arange ( y_min , y_max , h )) return xx , yy def plot_contours ( clf , xx , yy , ** params ): """Plot the decision boundaries for a classifier. Parameters ---------- ax: matplotlib axes object clf: a classifier xx: meshgrid ndarray yy: meshgrid ndarray params: dictionary of params to pass to contourf, optional """ Z = clf . predict ( np . c_ [ xx . ravel (), yy . ravel ()]) Z = Z . reshape ( xx . shape ) out = plt . contourf ( xx , yy , Z , ** params ) return out def draw_contour ( x , y , clf , class_labels = [ "Negative" , "Positive" ]): """ Draws a contour line for the predictor Assumption that x has only two features. This functions only plots the first two col """ X0 , X1 = x [:, 0 ], x [:, 1 ] xx0 , xx1 = make_meshgrid ( X0 , X1 ) plt . figure ( figsize = ( 10 , 6 )) plot_contours ( clf , xx0 , xx1 , cmap = "PiYG" , alpha = 0.8 ) scatter = plt . scatter ( X0 , X1 , c = y , cmap = "PiYG" , s = 30 , edgecolors = "k" ) plt . legend ( handles = scatter . legend_elements ()[ 0 ], labels = class_labels ) Loading [MathJax]/extensions/Safe.js

Example Project In this part, we will go over how to perform a Binary classification task using a variety of models. We will provide examples of how to train and evaluate these models. Dataset Description Healthcare is an important industry that uses machine learning to aid doctors in diagnosing many different kinds of illnesses and diseases. For this example project, we will be using the Breast Cancer Wisconsin Dataset to determine whether a mass found in a body is benign or malignant. Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image. Feature Information: Column 1: ID number Column 2: Diagnosis (M = malignant, B = benign) Ten real-valued features are computed for each cell nucleus: 1. radius (mean of distances from center to points on the perimeter) 2. texture (standard deviation of gray-scale values) 3. perimeter 4. area 5. smoothness (local variation in radius lengths) 6. compactness (perimeter^2 / area - 1.0) 7. concavity (severity of concave portions of the contour) 8. concave points (number of concave portions of the contour) 9. symmetry 10. fractal dimension ("coastline approximation" - 1) Due to the statistical nature of the test, we are not able to get exact measurements of the previous values. Instead, the dataset contains the mean and standard error of the real-valued features. Columns 3-12 present the mean of the measured values Columns 13-22 present the standard error of the measured values Load and Analyze the dataset plt . xlim ( xx0 . min (), xx0 . max ()) plt . ylim ( xx1 . min (), xx1 . max ()) In [ ]: #Load Data data = pd . read_csv ( 'datasets/breast_cancer_data.csv' ) Loading [MathJax]/extensions/Safe.js

<class 'pandas.core.frame.DataFrame'> RangeIndex: 569 entries, 0 to 568 Data columns (total 22 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 id 569 non-null int64 1 diagnosis 569 non-null object 2 radius_mean 569 non-null float64 3 texture_mean 569 non-null float64 4 perimeter_mean 569 non-null float64 5 area_mean 569 non-null float64 6 smoothness_mean 569 non-null float64 7 compactness_mean 569 non-null float64 8 concavity_mean 569 non-null float64 9 concave points_mean 569 non-null float64 10 symmetry_mean 569 non-null float64 11 fractal_dimension_mean 569 non-null float64 12 radius_se 569 non-null float64 13 texture_se 569 non-null float64 14 perimeter_se 569 non-null float64 15 area_se 569 non-null float64 16 smoothness_se 569 non-null float64 17 compactness_se 569 non-null float64 18 concavity_se 569 non-null float64 19 concave points_se 569 non-null float64 20 symmetry_se 569 non-null float64 21 fractal_dimension_se 569 non-null float64 dtypes: float64(20), int64(1), object(1) memory usage: 97.9+ KB While .info shows that every entry has 569 non-null and there are 569 entries, it is good to explicitly check for nulls. id 0 diagnosis 0 radius_mean 0 texture_mean 0 perimeter_mean 0 area_mean 0 smoothness_mean 0 compactness_mean 0 concavity_mean 0 concave points_mean 0 symmetry_mean 0 fractal_dimension_mean 0 radius_se 0 texture_se 0 perimeter_se 0 area_se 0 smoothness_se 0 compactness_se 0 concavity_se 0 concave points_se 0 symmetry_se 0 fractal_dimension_se 0 dtype: int64 Awesome! No need for imputation! While we are looking at the dataset, we shall remove the "id" column. In [ ]: data . isnull () . sum () Out[ ]: Loading [MathJax]/extensions/Safe.js

Looking at the target labels For this project, we wish to classify the diagnosis column. 0 M 1 M 2 M 3 M 4 M .. 564 M 565 M 566 M 567 M 568 B Name: diagnosis, Length: 569, dtype: object We need to transform this column into numerical column so that we may use them in our models. To do this, we will employ the LabelEncoder to automatically transform all the target label. ['B' 'M'] 0 1 1 1 2 1 3 1 4 1 .. 564 1 565 1 566 1 567 1 568 0 Name: diagnosis, Length: 569, dtype: int64 Let's look at a histogram of the full dataset. Its always good to get a global view of your datasets by looking at their histograms. You might see some interesting trends. In [ ]: data = data . drop ([ "id" ], axis = 1 ) In [ ]: data [ "diagnosis" ] Out[ ]: In [ ]: from sklearn.preprocessing import LabelEncoder le = LabelEncoder () data [ 'diagnosis' ] = le . fit_transform ( data [ 'diagnosis' ]) print ( le . classes_ ) In [ ]: data [ 'diagnosis' ] Out[ ]: In [ ]: data . hist ( figsize = ( 20 , 15 )) Loading [MathJax]/extensions/Safe.js

From the histograms, we can see some interesting trends. Possible observations: Many of the _se columns indicate a heavy skewness towards low values and have large tails Many of the _mean columns look more Gaussian in shape There is a large disparity between the ranges of certain features. For example, radius mean can go from 0 to 25 while smoothness_mean is in the range [0.050,0.150]. This indicates we will have to normalize or standardize the features if the models are sensitive to such measures. Looking at the correlation matrix to get an idea about which features are important In [ ]: correlations = data . corr () columns = list ( data ) #Creates the heatmap heatmap ( correlations . values , columns , columns , figsize = ( 20 , 12 ), cmap = "hsv" ) In [ ]: #Let's specifically look at the correlations of our target feature correlations [ "diagnosis" ] . sort_values ( ascending = False ) Loading [MathJax]/extensions/Safe.js

diagnosis 1.000000 concave points_mean 0.776614 perimeter_mean 0.742636 radius_mean 0.730029 area_mean 0.708984 concavity_mean 0.696360 compactness_mean 0.596534 radius_se 0.567134 perimeter_se 0.556141 area_se 0.548236 texture_mean 0.415185 concave points_se 0.408042 smoothness_mean 0.358560 symmetry_mean 0.330499 compactness_se 0.292999 concavity_se 0.253730 fractal_dimension_se 0.077972 symmetry_se -0.006522 texture_se -0.008303 fractal_dimension_mean -0.012838 smoothness_se -0.067016 Name: diagnosis, dtype: float64 We can see that there is a lot of correlation between the features and the target label. Thus, we can expect to learn something from the data When doing classification, check if classes are heavily imbalanced. It is important that the dataset does not prefer one class over any others. Otherwise, it may bias the model to not learn the minority classes well. Lets use a histogram and count the number of elements in each class. 0 357 1 212 Name: diagnosis, dtype: int64 Out[ ]: In [ ]: data [ 'diagnosis' ] . hist ( bins = 2 , figsize = ( 5 , 5 )) data [ 'diagnosis' ] . value_counts () Out[ ]: Loading [MathJax]/extensions/Safe.js

0 285 1 170 Name: diagnosis, dtype: int64 0 72 1 42 Name: diagnosis, dtype: int64 Out[ ]: In [ ]: #Testing classes target_test . hist ( bins = 2 , figsize = ( 5 , 5 )) target_test . value_counts () Out[ ]: Loading [MathJax]/extensions/Safe.js

We can see that the class balance is about the same as before the split. In fact, we can see that if a classifier just guessed class 0, it would have an accuracy of $100 * \frac{72}{72+42} = 63.15\%$. We can consider this the baseline accuracy to compare against. Models for Classification: KNN For our first model, we will use KNN classfication. This is a model we have seen many times throughout the course and it would be interesting to see how well it performs. Simple KNN classification with K = 3 Let us try KNN on the raw data with simply 3 nearest neighbors. We use the sklearn metric library to calculate the measures of interest. In this case, we focus on accuracy. Accuracy: 0.877193 We can see that there is already a huge improvement in accuracy on comparison to the baseline of 63.15%. Let's see the effect that standardizing the input features would have on the KNN performance. Affect of pre-processing on KNN Accuracy: 0.921053 We can see that with pre-processing we were able to get a much better classification accuracy. Here we only used StandardScaler. Lets see if other pre-processing techniques could have also worked. As such, lets look at MinMaxScaler and Normalizer: In [ ]: # k-Nearest Neighbors algorithm knn = KNeighborsClassifier ( n_neighbors = 3 ) knn . fit ( train_raw , target ) predicted = knn . predict ( test_raw ) In [ ]: print ( " %-12s %f " % ( 'Accuracy:' , metrics . accuracy_score ( target_test , predicted ))) In [ ]: #Since all features are real-valued, we only have one pipeline pipeline = Pipeline ([ ( 'scaler' , StandardScaler ()) ]) #Transform raw data train = pipeline . fit_transform ( train_raw ) test = pipeline . transform ( test_raw ) #Note that there is no fit calls In [ ]: # k-Nearest Neighbors algorithm knn = KNeighborsClassifier ( n_neighbors = 3 ) knn . fit ( train , target ) testing_result = knn . predict ( test ) predicted = knn . predict ( test ) In [ ]: print ( " %-12s %f " % ( 'Accuracy:' , metrics . accuracy_score ( target_test , predicted ))) In [ ]: preprocessors = [ StandardScaler (), MinMaxScaler (), Normalizer () ] Loading [MathJax]/extensions/Safe.js

Loading [MathJax]/extensions/Safe.js

We can see that as k gets larger, the decision boundary gets smoother. Models for Classification: Logistic Regression While KNN is a very powerful model, it does come with a few issues such as Loading [MathJax]/extensions/Safe.js

Parameters for Logistic Regression In Sci-kit Learn, the following are just some of the parameters we can pass into Logistic Regression: penalty: {‘l1’, ‘l2’, ‘elasticnet’, ‘none’} default="l2" Specifies the type of regularization to use. Not all penalties work for each solver. C: positive float, default=1 Inverse of the regularization strength. You can treat C as $\frac{1}{\lambda}$ as shown in lecture. Thus, as C gets smaller, the regularization strength increases. solver: {‘newton-cg’, ‘lbfgs’, ‘liblinear’, ‘sag’, ‘saga’}, default=’lbfgs’ Algorithm to use in the optimization problem. Each algorithms solves logistic regression using different iterative methods that are based on the gradient. Read the sci-kit learn documentation for more information. max_iter: int, default=100 Maximum number of iterations taken for the solvers to converge. Each parameter has a different effect on the model. Let's look at how the choose of max_iter affects the model performance on the raw data and the standardized dataset. Raw Data Accuracy: 0.938596 Preprocessed Data Accuracy: 0.947368 We see that the accuraccies are pretty close to each other. Lets see what happens when we decrease the max_iter. In [ ]: #Since all features are real-valued, we only have one pipeline preprocesser = Pipeline ([ ( 'scaler' , StandardScaler ()) ]) #Transform raw data train = preprocesser . fit_transform ( train_raw ) test = preprocesser . transform ( test_raw ) #Note that there is no fit call In [ ]: log_reg = LogisticRegression ( penalty = "l2" , max_iter = 1000 , solver = "lbfgs" , C = 0.01 ) #Train raw is the data before preprocessing log_reg . fit ( train_raw , target ) predicted = log_reg . predict ( test_raw ) print ( " %-12s %f " % ( 'Raw Data Accuracy:' , metrics . accuracy_score ( target_test , predicted )) #Train is the data after preprocessing (using Standard scalar) log_reg . fit ( train , target ) predicted = log_reg . predict ( test ) print ( " %-12s %f " % ( 'Preprocessed Data Accuracy:' , metrics . accuracy_score ( target_test , pr In [ ]: log_reg = LogisticRegression ( penalty = "l2" , max_iter = 70 , solver = "lbfgs" , C = 0.01 ) #Train raw is the data before preprocessing log_reg . fit ( train_raw , target ) predicted = log_reg . predict ( test_raw ) print ( " %-12s %f " % ( 'Raw Data Accuracy:' , metrics . accuracy_score ( target_test , predicted )) Loading [MathJax]/extensions/Safe.js

Raw Data Accuracy: 0.921053 /Users/kunalpatil/anaconda3/lib/python3.10/site-packages/sklearn/linear_model/_logistic. py:458: ConvergenceWarning: lbfgs failed to converge (status=1): STOP: TOTAL NO. of ITERATIONS REACHED LIMIT. Increase the number of iterations (max_iter) or scale the data as shown in: https://scikit-learn.org/stable/modules/preprocessing.html Please also refer to the documentation for alternative solver options: https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression n_iter_i = _check_optimize_result( Ooops! The model did not seem to converge. Its seem that the scale of the features strongly affects the convergence speed of the iterative algorithm. As suggested, we can fix this issue by increaing the max_iter, re-scaling the data, or using a different solver. Preprocessed Data Accuracy: 0.947368 Cross Validation for Logistic Regression Let us do a little experiment using cross validation to see how each term affects the logistic regression. We will perform this example on the standardized data. In [ ]: #Train is the data after preprocessing (using Standard scalar) log_reg . fit ( train , target ) predicted = log_reg . predict ( test ) print ( " %-12s %f " % ( 'Preprocessed Data Accuracy:' , metrics . accuracy_score ( target_test , pr In [ ]: #You may even do Cross validation for classification from sklearn.model_selection import GridSearchCV #Note that this a list of dict #Each dict describes the combination of parameters to check parameters = [ { "penalty" : [ "l2" ], "C" : [ 0.01 , 1 , 100 ], "solver" : [ "lbfgs" , "liblinear" ]}, #These solvers support penalty = "l2" { "penalty" : [ "none" ], "C" : [ 1 ], #Specified to prevent error message "solver" : [ "lbfgs" , "newton-cg" ]}, #These solvers support penalty = "none" ] #instantiate model #Implementing cross validation k = 3 kf = KFold ( n_splits = k , random_state = None ) log_reg = LogisticRegression ( penalty = "none" , max_iter = 1000 , solver = "lbfgs" ) #will c grid = GridSearchCV ( log_reg , parameters , cv = kf , scoring = "accuracy" ) grid . fit ( train , target ) Loading [MathJax]/extensions/Safe.js