Tutorial_W12 solution (3)

function. 3. The following table consists of training data from an employee database. The data have been generalized. For a given row entry, the count represents the number of data tuples having the values for the department, status, age, and salary given in that row. Design a multilayer feed-forward neural network for the given data. Label the nodes in the input and output layers. 4. The Neural network training is accomplished by repeatedly passing the training data through the network while a. Individual network weights are modified. b. Training instance attribute values are modified. c. The ordering of the training instances is modified. d. Individual network nodes have the coefficients on their corresponding functional parameters modified. 5. What happens when a neural network is over-trained? 6. Why Neural Networks are called as Universal Approximator? Exercise 3: Comparison of algorithms 1. Discuss how will you control overfitting in algorithms such as decision tree, neural network, and logistic regression. 2. Compare the 3 classification algorithms, Decision tree, Neural network, and Logistic regression. Identify specific features of the dataset/learning such as the presence of missing

In sklearn, if a neural network does not achieve convergence before maximum iteration, it will raise a "convergence is not reached" warning message. This first neural network raised the convergence warning. If a warning message is raised, the max_iter hyperparameter of the neural network should be increased. However, if the network fails to reach convergence even with the very large number of maximum iteration, this might indicate a problem with error computation. The following code demonstrates the status of MLP classifier training with 700 max_iter . Train accuracy: 0.9960176991150442 Test accuracy: 0.534755677907777 precision recall f1-score support 0 0.53 0.53 0.53 1453 1 0.53 0.54 0.54 1453 accuracy 0.53 2906 macro avg 0.53 0.53 0.53 2906 weighted avg 0.53 0.53 0.53 2906 MLPClassifier(max_iter=700, random_state=10) The neural network after setting the iteration of 700 performed with much higher accuracy on the training dataset (99%). However, the test accuracy remains the same (53%). This clearly indicates overfitting to the training data. Next, we will use the GridSearch tuning and dimensionality reduction techniques to control the overfitting of the model to the training data. Solver and activation function Finding the best combination of weights in neural networks is a significant search problem. The algorithm used to find this optimal weight set is called solver in sklearn, and the most common one is Gradient Descent. Gradient descent starts with a set of randomly generated weights. In each iteration of gradient descent, predictions are made on X_train and the error value (cost) is computed. The weight set is then altered to reduce this error value. Each iteration is called an epoch. To stop gradient descent iterations, a strategy such as maximum iterations , minimum error threshold or convergence reached (error is not improved over a certain number of epochs) is used. A combination of max iterations and convergence is the most commonly used criterion. Observe the MLPClassifier object hyperparameters printed out before. You should see that the solver hyperparameter is set by default to adam (stands for adaptive moment estimation). Adam is an extension of gradient descent, designed to speed up the training process and be In [4]: model_2 = MLPClassifier ( max_iter = 700 , random_state = rs ) model_2 . fit ( X_train , y_train ) print ( "Train accuracy:" , model_2 . score ( X_train , y_train )) print ( "Test accuracy:" , model_2 . score ( X_test , y_test )) y_pred = model_2 . predict ( X_test ) print ( classification_report ( y_test , y_pred )) print ( model_2 )

Total number of models: 5 Now, plot the mean train and test scores of all runs. Total number of models: 5 In [8]: import matplotlib.pyplot as plt train_result = result_set [ 'split0_train_score' ] test_result = result_set [ 'split0_test_score' ] print ( "Total number of models: " , len ( test_result )) # plot hidden layers hyperparameter values vs training and test accuracy score plt . plot ( range ( 0 , len ( train_result )), train_result , 'b' , range ( 0 , len ( test_result )), plt . xlabel ( 'Hyperparameter Hidden_layers\nBlue = training acc. Red = test acc.' ) plt . xticks ( range ( 0 , len ( train_result )), range ( 5 , 86 , 20 )) plt . ylabel ( 'score' ) plt . show () In [9]: ### Enter your code train_result = result_set [ 'mean_train_score' ] test_result = result_set [ 'mean_test_score' ] print ( "Total number of models: " , len ( test_result )) # plot hidden layers hyperparameter values vs training and test accuracy score plt . plot ( range ( 0 , len ( train_result )), train_result , 'b' , range ( 0 , len ( test_result )), plt . xlabel ( 'Hyperparameter Hidden_layers\nBlue = training acc. Red = test acc.' ) plt . xticks ( range ( 0 , len ( train_result )), range ( 5 , 86 , 20 )) plt . ylabel ( 'score' ) plt . show () In [10]:

Train accuracy: 0.6128318584070797 Test accuracy: 0.5684790089470062 precision recall f1-score support 0 0.57 0.57 0.57 1453 1 0.57 0.57 0.57 1453 accuracy 0.57 2906 macro avg 0.57 0.57 0.57 2906 weighted avg 0.57 0.57 0.57 2906 {'alpha': 0.001, 'hidden_layer_sizes': (7,)} The RFE selected feature set showed improvements over the original data set. However, compared with the previous best model ( cv_2 ), the model performed slightly worse. This is an indication that elimination with logistic regression did not produce the feature set suitable for neural network. 4.2. Selecting using decision tree Lastly, we will use the decision tree model to perform feature selection for neural network modelling. To start, we need to tune a decision tree with GridSearchCV. {'criterion': 'entropy', 'max_depth': 5, 'min_samples_leaf': 10} GiftCnt36 : 0.2786889679489785 DemMedHomeValue : 0.16527761052028647 GiftAvgLast : 0.12080196710144986 GiftCntAll : 0.06936170003476945 GiftTimeLast : 0.06852490725467034 StatusCatStarAll : 0.039760012572680054 DemAge : 0.0395320434572485 PromCnt36 : 0.034873357233237874 GiftCntCardAll : 0.030230871689451603 X_test_rfe = rfe . transform ( X_test ) # step = int((X_train_rfe.shape[1] + 5)/5); params = { 'hidden_layer_sizes' : [( 3 ,), ( 5 ,), ( 7 ,), ( 9 ,)], 'alpha' : [ 0.01 , 0.001 , 0.00 rfe_cv = GridSearchCV ( param_grid = params , estimator = MLPClassifier ( random_state = rs ), c rfe_cv . fit ( X_train_rfe , y_train ) print ( "Train accuracy:" , rfe_cv . score ( X_train_rfe , y_train )) print ( "Test accuracy:" , rfe_cv . score ( X_test_rfe , y_test )) y_pred = rfe_cv . predict ( X_test_rfe ) print ( classification_report ( y_test , y_pred )) print ( rfe_cv . best_params_ ) In [15]: import pickle with open ( 'DT.pickle' , 'rb' ) as f : dt_best , roc_index_dt_cv , fpr_dt_cv , tpr_dt_cv = pickle . load ( f ) print ( dt_best . best_params_ ) In [16]: from dm_tools import analyse_feature_importance analyse_feature_importance ( dt_best . best_estimator_ , X . columns )

Based on the ROC curve, the neural network model with decision tree selected features, with the ROC index of 0.594, performs marginally better than other models. Now, your task is to compare the best performing neural network with the best performing decision tree and logistic regression. Identify which model performs the best on this dataset. import matplotlib.pyplot as plt plt . plot ( fpr_nn_1 , tpr_nn_1 , label = 'NN_default {:.3f}' . format ( roc_index_nn_1 ), color plt . plot ( fpr_nn_2 , tpr_nn_2 , label = 'NN with relu {:.3f}' . format ( roc_index_nn_2 ), col plt . plot ( fpr_cv_1 , tpr_cv_1 , label = 'NN cv_1 {:.3f}' . format ( roc_index_cv_1 ), color = 'b plt . plot ( fpr_cv_2 , tpr_cv_2 , label = 'NN cv_2 {:.3f}' . format ( roc_index_cv_2 ), color = 'y plt . plot ( fpr_cv_3 , tpr_cv_3 , label = 'NN cv_3 {:.3f}' . format ( roc_index_cv_3 ), color = 'c plt . plot ( fpr_rfe_cv , tpr_rfe_cv , label = 'NN rfe_cv {:.3f}' . format ( roc_index_rfe_cv ), plt . plot ( fpr_cv_sel_model , tpr_cv_sel_model , label = 'NN with cv_sel_model {:.3f}' . for plt . plot ([ 0 , 1 ], [ 0 , 1 ], color = 'navy' , lw = 0.5 , linestyle = '--' ) plt . xlim ([ 0.0 , 1.0 ]) plt . ylim ([ 0.0 , 1.0 ]) plt . xlabel ( 'False Positive Rate' ) plt . ylabel ( 'True Positive Rate' ) plt . title ( 'Receiver operating characteristic example' ) plt . legend ( loc = "lower right" ) plt . show () In [21]: ### Enter your code import pickle with open ( 'DT.pickle' , 'rb' ) as f : dt_best , roc_index_dt_cv , fpr_dt_cv , tpr_dt_cv = pickle . load ( f ) with open ( 'LR.pickle' , 'rb' ) as f : lr_best , roc_index_lr_cv , fpr_lr_cv , tpr_lr_cv = pickle . load ( f ) plt . plot ( fpr_dt_cv , tpr_dt_cv , label = 'DT {:.3f}' . format ( roc_index_dt_cv ), color = 'red plt . plot ( fpr_lr_cv , tpr_lr_cv , label = 'LR {:.3f}' . format ( roc_index_lr_cv ), color = 'gre plt . plot ( fpr_cv_sel_model , tpr_cv_sel_model , label = 'NN with cv_sel_model {:.3f}' . for plt . plot ([ 0 , 1 ], [ 0 , 1 ], color = 'navy' , lw = 0.5 , linestyle = '--' ) plt . xlim ([ 0.0 , 1.0 ]) plt . ylim ([ 0.0 , 1.0 ]) plt . xlabel ( 'False Positive Rate' ) plt . ylabel ( 'True Positive Rate' ) plt . title ( 'Receiver operating characteristic example' )