Ridge and Lasso Regression Analysis for Predictive Modeling

Module 4 Assignment College of Professional Studies, Northeastern University ALY6015, 21626 Harpreet Sharma February 5 th , 2024 Table of Contents Introduction ................................................................................................................................ 3

Analysis ...................................................................................................................................... 3 Ridge Regression .................................................................................................................... 3 Figure 1 .............................................................................................................................. 3 Ridge Regression with Cross-validation ............................................................................. 3 Figure 2 .............................................................................................................................. 4 The plot of Cross-validation result of Ridge Regression .................................................... 4 Figure 3 .............................................................................................................................. 5 Coefficients of the lambda min model ................................................................................. 5 Figure 4 .............................................................................................................................. 5 Coefficients of the lambda 1se model ................................................................................. 5 Lasso Regression .................................................................................................................... 6 Figure 5 .............................................................................................................................. 6 Lasso Regression with Cross-validation ............................................................................. 6 Figure 6 .............................................................................................................................. 7 The plot of the Cross-validation result of the Lasso Regression ......................................... 7 Figure 7 .............................................................................................................................. 8 Coefficients of the lambda min model ................................................................................. 8 Figure 8 .............................................................................................................................. 8 Coefficients of the lambda 1se model ................................................................................. 8 Conclusion/Interpretation ......................................................................................................... 10 References ................................................................................................................................ 11 Appendices ............................................................................................................................... 12 2

Introduction This report focuses on building regularization models using Ridge and Lasso regression techniques on the College dataset from the ISLR library which comprises 777 records and 18 variables. To address the problem of multicollinearity and overfitting in predictive modeling, regularization methods such as Ridge and Lasso are used. The objective is to use different predictor variables in the data set to predict graduation rates. Analysis 1. The dataset is split into a training set and a testing set (see Appendix A). This splitting is crucial for evaluating the performance of the models on unseen data. Ridge Regression 2. Ridge regression with cross-validation is performed on the training data to find the optimal regularization parameter (see Appendix B). Figure 1 Ridge Regression with Cross-validation As shown in Figure 1, Lambda min (1.775) minimizes MSE for better predictive accuracy but can lead to a more complex model. Lambda 1se (16.558) offers a slightly more regularized model within one standard error, striking a balance between simplicity and accuracy. 3

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

3. Plot of the Results Figure 2 The plot of Cross-validation result of Ridge Regression As shown in Figure 2, The x-axis displays the log of λ, and the y-axis represents the mean-squared error. The figures above the plot indicate the number of variables (with non-zero coefficients). The two dashed lines represent two lambda values: lambda min on the far left and lambda 1se on the right. 4. Fitting a Ridge Regression Model A regression model is fit against the training set and the following coefficients are obtained (see Appendix C). 4

Figure 3 Coefficients of the lambda min model Figure 4 Coefficients of the lambda 1se model As shown in Figures 3 and 4, what seems interesting is that the ridge regression models illustrate the balance between reducing coefficients towards zero and retaining all features, highlighting the compromise between model complexity and predictive accuracy. 5. Performance of Fit Model against the Ridge Training Set by RMSE The Ridge regression model with lambda min has an RMSE of approximately 12.54 on the training set, while the model with lambda 1se has an RMSE of 5

approximately 13.05 (see Appendix D). The lower RMSE of the model with lambda min indicates slightly better predictive accuracy on the training data compared to the model with lambda.1se. 6. Performance of Fit Model against the Ridge Test Set by RMSE The RMSE for the Ridge regression model with lambda min on the test set is approximately 13.02, while for the model with lambda 1se, it is approximately 12.97 (see Appendix E). This indicates that the model with lambda 1se performs slightly better in terms of predictive accuracy on the test data compared to the model with lambda min. The model does not appear to be overfit as the test set has similar or slightly lower RMSE values than the training set. This indicates that the model generalizes well to unseen data. Lasso Regression 7. Lasso regression with cross-validation is performed on the training data to find the optimal regularization parameters (see Appendix F). Figure 5 Lasso Regression with Cross-validation As shown in Figure 5, Lambda min (0.0734) minimizes MSE for better predictive accuracy but can lead to a more complex model. Lambda 1se (1.3122) offers a slightly more regularized model within one standard error, striking a balance between simplicity and accuracy. 6

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

8. Plot of the Results Figure 6 The plot of the Cross-validation result of the Lasso Regression As shown in Figure 6, the variables at the top of the plot illustrate the number of predictors retained in the model. As the number of predictors decreases, the mean- squared error tends to increase. The dashed line on the right indicates that, with lambda 1se, the optimal model contains 9 variables, whereas lambda min suggests an optimal model with 15 variables. 9. Fitting a Lasso Regression Model A regression model is fit against the training set and the following coefficients are obtained (see Appendix G). 7

Figure 7 Coefficients of the lambda min model For the Lasso regression model with lambda min, the coefficients for the variables F. Undergrad, and Books are reduced to zero as denoted by the dots (.). Figure 8 Coefficients of the lambda 1se model For the Lasso regression model with Lambda 1se, the coefficients for the variables Accept, Enroll, F. Undergrad, Books, Ph.D., Terminal, S.F. Ratio, and Expend are reduced to Zero as denoted by the dots (.). 10. Performance of Fit Lasso Model against the Training Set by RMSE 8

The Lasso regression model with Lambda min has a lower RMSE (12.49) on the training set compared to the model with Lambda 1se (13.04) (see Appendix H), indicating slightly better predictive accuracy. 11. Performance of Fit Lasso Model against the Test Set by RMSE The RMSE for the LASSO regression model with lambda min on the test set is approximately 13.17, while for the model with lambda 1se, it is approximately 12.99. The model with lambda 1se exhibits slightly better predictive accuracy on the test data compared to the model with lambda min. The model does not appear to be overfit as the test set has similar or slightly lower RMSE values than the training set. This indicates that the model generalizes well to unseen data. 12. Assessing Model Performance: Comparison and Expectations The Lasso model outperformed the Ridge model on both the training and test sets, which was expected. By selectively choosing important features and reducing model complexity, the Lasso model achieved slightly better predictive accuracy. This outcome aligns with the anticipated behavior of the Lasso regularization technique. 13. Comparative Performance and Preference of Feature Selection Methods: Stepwise Selection vs. Ridge Regression and Lasso The stepwise selection procedure resulted in a best model with an AIC of 3965.4 (see Appendix J), which includes 11 variables. It has an RMSE of 12.63. Comparing these three models based on the RMSE and their ability to generalize to new data, the Lasso regression model is preferred. It strikes a balance between predictive accuracy, mitigating overfitting, offering feature selection, and robustness in making predictions on unseen data. 9

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

Conclusion/Interpretation In conclusion, the analysis demonstrated the effectiveness of Ridge and Lasso regression techniques in modeling graduation rates on the College dataset. Lasso outperformed Ridge, highlighting its feature selection capability and ability to achieve a balance between model complexity and predictive accuracy. When compared with stepwise selection, Lasso regression also outperforms it because it mitigates overfitting, offers feature selection, and is robust in making predictions on unseen data it. These results emphasize the importance of regularization methods in improving model performance and generalization in complex datasets. 10

References Bhattacharyya, S. (2018, September 26). Ridge and Lasso Regression: L1 and L2 Regularization . Medium; Towards Data Science. https://towardsdatascience.com/ridge- and-lasso-regression-a-complete-guide-with-python-scikit-learn-e20e34bcbf0b Bose, P. (2023, August 25). Guide to Lasso and Ridge Regression Techniques with Use Cases . Blogs & Updates on Data Science, Business Analytics, AI Machine Learning. https://www.analytixlabs.co.in/blog/lasso-and-ridge-regression/ How. (2017, March 14). How to interpret the results when both ridge and lasso separately perform well but produce different coefficients . Cross Validated. https://stats.stackexchange.com/questions/267345/how-to-interpret-the-results-when- both-ridge-and-lasso-separately-perform-well-b Regularization methods in R . (2024). Fu-Berlin.de. https://www.geo.fu-berlin.de/en/v/soga- r/Advances-statistics/Multivariate-approaches/Multiple-linear-regression/ Regularization-methods/Regularization-methods-in-R/index.html RPubs - Stepwise model selection . (2012, November 27). Rpubs.com. https://rpubs.com/kaz_yos/stepwise 11

Appendices Appendix A Train and test set As shown in Appendix A, the R code details how the dataset is split into a train and test set. Appendix B Ridge Regression Appendix B details the R code used to estimate the lambda min and lambda 1se values. Appendix C Ridge Regression Model Appendix C details the R code used to fit a ridge regression model for the lambda min and lambda 1se training set. 12

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

Appendix D RMSE of Fit Ridge Model against Training Set Appendix D details the R code used to determine the performance of the fit model against the training set (Lambda min and Lambda 1se) by RMSE. Appendix E RMSE of Fit Ridge Model against Test Set Appendix D details the R code used to determine the performance of the fit model against the test set (Lambda min and Lambda 1se) by RMSE. Appendix F Lasso Regression Appendix F details the R code used to estimate the lambda min and lambda 1se values for the Lasso regression. 13

Appendix G Lasso Regression Model Appendix G details the R code used to fit a Lasso regression model for the training set. Appendix H RMSE of Fit Lasso Model against Training Set Appendix D details the R code used to determine the performance of the fit model against the training set (Lambda min and Lambda 1se) by RMSE. Appendix I RMSE of Fit Lasso Model against Test Set Appendix D details the R code used to determine the performance of the fit model against the test set (Lambda min and Lambda 1se) by RMSE. 14

Appendix J Stepwise selection Appendix J details the R code used to fit model for and perform stepwise selection. 15

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

Module4Assignment - Copy

Related Documents