Introduction
Statistical analysis aims to produce conclusions and solutions that are more robust by minimizing human error. This assignment proposes to address the fundamentals of machine learning using training and test datasets, to evaluate the model’s performance on data not used for training (Cody et al., 2022). Two popular machine learning tools will be used in the analysis, namely the Least Absolute Shrinkage and Selection Operator (LASSO) and Ridge Regression.
Firstly, LASSO is used for regression analysis to minimize the sum of the squares of the model errors and the sum of the absolute values of the coefficients multiplied by the regularization parameter lambda (Kennedy & Ward, 2020). Simply put, LASSO removes variables that are not statistically significant from the model, retaining only significant predictors. In this context, the parameter lambda represents the degree of coefficient retention: the higher the lambda, the fewer non-zero predictors remain in the regression model.
Second, Ridge Regression is also a regression analysis that aims to reduce the influence of irrelevant variables in the model by preventing overfitting (Hoerl, 2020). Unlike LASSO, Ridge Regression does not remove predictors; instead, it penalizes coefficients to control for redundancy, keeping only the statistically significant ones. This model also uses lambda, which has a similar practical significance to that presented in LASSO. This analysis proposes using the college dataset from the ISLR library to build regularization models.
Analysis
Data Partitioning
After importing the College data into the workspace, the task was to split the dataset into training and test sets. As the data partitioning guidelines suggested, 70% of the dataset was assigned to the training set, and the remaining 30% to the test set. The partitioning was randomized, and seed() was used to create reproducibility (Zach, 2022). This implies that the data were randomly split into sets, with the split kept under an index to allow for future reproducibility.
Computation of Lambdas
To conduct LASSO and Ridge regression analyses, it was first necessary to determine the lambda values that would form the models. An array of lambdas is generated by automatically splitting the data using k-fold cross-validation, and the one that yields the best model performance is then selected (Rodrigues et al., 2022). Thus, the minimum lambda was 110.2443, and the one standard error away lambda was 120.993.
A regression model built with the calculated minimum lambda minimizes cross-validation error and is more adaptable to the training dataset. In contrast, a model with a lambda of one standard error allows one to generate a model that prevents overtraining while maintaining accuracy on test data. The goals at hand determine the choice between the two lambdas: if a more robust model is required, then using the one-standard-error lambda is preferable; at the same time, if there is a risk of overtraining, then using the minimal lambda is more appropriate.
Figure 1 shows how the error varies with lambda. The minimum of this curve, characterized by the red vertical line, refers to the minimum lambda, while the blue straight line is the lambda level of one standard error. This graph shows that the lambdas chosen for the analysis yielded minimal model errors and, consequently, improved regression accuracy.
Ridge
Ridge regression was performed on the training set to obtain coefficient estimates for each of the 18 predictors in the original data set. First, despite the low coefficient values, none were zero because, as previously stated, this method does not set them to zero (Hoerl, 2020). Second, the relatively low values may be due to multicollinearity among the predictors, so running a conventional multiple regression would likely have yielded larger coefficient values. It was also shown that the variables Top10perc (Pct. new students from top 10% of H.S. class), S.F.Ratio (Student/faculty ratio), and Grad. The rate (Graduation rate) had significant positive contributions, while some predictors had negative coefficients.
The calculated root mean square error (RMSE) was 232.3503, which is sufficiently far from zero to indicate a perfect model. However, this RMSE is of little practical use without context, as there is no reference value to compare and contrast. When using Ridge regression on the test set, the RMSE grew to 308.7293 units. An increase in RMSE on the test data may indicate overtraining, in which the predictive model memorizes the training data and cannot generalize to the test data (Hodson, 2022). Another reason could be a shortage of test data, which can bias the RMSE.
LASSO
First, the lambda parameters in this analysis were similar to those used in Ridge. Interestingly, when applying the LASSO model to the training data, only Apps (Number of applications received) was statistically significant, and all other coefficients were set to 0. Since none of the predictors other than Apps had a numerical coefficient, this indicated that these variables did not contribute to the model. For this model, the RMSE on the training data was 110.2443, and on the test data, 128.7271. As with Ridge, an increase in RMSE could indicate risks of overtraining the model or a lack of data, leading to distortion.
Comparison
The results are consistent with initial expectations: in the Ridge regression, no predictors were zeroed out, whereas in the LASSO model, they were. It was unexpected that 17 of the 18 predictors were removed in LASSO, and only Apps made a meaningful contribution to the model. When comparing the models, the RMSEs for LASSO were almost half as high and closer to zero than those for Ridge. This may indicate that LASSO is a more accurate model as it had reduced error and increased accuracy compared to the other regression models. On the other hand, LASSO nullified almost 94% of the predictors, indirectly indicating increased model complexity and limitations of the analysis.
After creating an additional stepwise regression model, there were three models, among which one, the most optimal, had to be selected. According to the manual, AIC and BIC values could be used for comparison. Since LASSO and Ridge were implemented in the glmnet library, the AIC() and BIC() functions were not available for them; the formulas for their calculations were entered in-house. The results showed that for LASSO, Ridge, and Stepwise, the AIC and BIC values were obtained, as displayed in Table 1. As this comparison suggests, Ridge regression was the best choice because it resulted in the lowest AIC and BIC values.
Table 1. AIC and BIC computational results for the three regression models.
Conclusion
In this paper, regression analysis was conducted using machine learning fundamentals. After dividing the College data set into training (70%) and test (30%) and calculating the lambdas, two regression analyses, LASSO and Ridge, were performed. Based on the RMSE results, LASSO was the more efficient model, producing fewer errors. After conducting Stepwise regression and AIC and BIC, it was evident that Ridge was the better-performing model, as its AIC and BIC values were minimal.
References
Cody, T., Lanus, E., Doyle, D. D., & Freeman, L. (2022). Systematic training and testing for machine learning using combinatorial interaction testing.
Hodson, T. O. (2022). Root-mean-square error (RMSE) or mean absolute error (MAE): When to use them or not. Geoscientific Model Development, 15(14), 5481-5487.
Hoerl, R. W. (2020). Ridge regression: A historical context. Technometrics, 62(4), 420-425.
Kennedy, C., & Ward, R. (2020). Greedy variance estimation for the lasso. Applied Mathematics & Optimization, 82, 1161-1182.
Rodrigues, S., Huggins, R., & Liquet, B. (2022). Central subspaces review: Methods and applications. Statistic Surveys, 16, 210-237.
Zach. (2022). How (and when) to use set.seed in R. Statology.
Appendix A
#Data Import
install.packages(“ISLR”)
library(ISLR)
summary(College)
data(“College”)
#Data Separation
install.packages(“caret”)
library(caret)
install.packages(“ggplot2”)
library(ggplot2)
install.packages(“lattice”)
library(lattice)
install.packages(c(“caret”, “ipred”))
set.seed(2810)
index <- createDataPartition(College$Apps, p = 0.7, list = FALSE, times = 1)
traindata <- College[index, ]
testdata <- College[-index, ]
#Lambdas
install.packages(“glmnet”)
library(glmnet)
xtrain <- as.matrix(traindata[, -1])
ytrain <- traindata$Apps
xtest <- as.matrix(testdata[-index, -1])
ytest <- testdata[-index, “Apps”]
cv_fit <- cv.glmnet(xtrain, ytrain)
bestlambda_min <- cv_fit$lambda.min
print(bestlambda_min)
bestlambda_1se <- cv_fit$lambda.1se
print(bestlambda_1se)
plot(cv_fit)
title(“Cross-Validation Plot”)
xlabel(“log(lambda)”)
ylabel(“Mean Squared Error”)
abline(v = log(bestlambda_min), col = “red”)
abline(v = log(bestlambda_1se), col = “blue”)
legend(“topright”, legend = c(“lambda.min”, “lambda.1se”), col = c(“red”, “blue”), lty = 2, lwd = 2)
#Ridge
ridge_model <- glmnet(xtrain, ytrain, alpha = 0, lambda = bestlambda_min)
coef(ridge_model)
predvalues <- predict(ridge_model, s = bestlambda_min, newx = xtrain)
rmse <- sqrt(mean((ytrain – predvalues)^2))
print(rmse)
predvalues_test <- predict(ridge_model, s = bestlambda_min, newx = xtest)
rmse_test <- sqrt(mean((ytest – predvalues_test)^2))
print(rmse_test)
#LASSO
lasso <- cv.glmnet(xtrain, ytrain, alpha = 1)
coef(lasso)
predtrain <- predict(lasso, newx = xtrain, s = “lambda.min”)
rmsetrain <- sqrt(mean((ytrain – predtrain)^2))
print(rmsetrain)
predtest <- predict(lasso, newx = xtest, s = “lambda.min”)
rmsetest <- sqrt(mean((ytest – predtest)^2))
print(rmsetest)
#Stepwise
initial_model <- lm(Apps ~., data = College)
final_model <- step(initial_model, direction = ‘both’, trace = 1)
summary(final_model)
#AIC & BIC
print(lasso)
print(ridge_model)
print(final_model)
x <- model.matrix(Apps ~., data = College)[, -1]
y <- College$Apps
n <- length(y)
aiclasso <- n * log(sum((predict(lasso, newx = x, s = bestlambda_min) – y)^2)/n) + 2 * (length(lasso$beta) + 1)
biclasso <- n * log(sum((predict(lasso, newx = x, s = bestlambda_min) – y)^2)/n) + log(n) * (length(lasso$beta) + 1)
print(aiclasso)
print(biclasso)
aicridge <- n * log(sum((predict(ridge_model, newx = x, s = bestlambda_min) – y)^2)/n) + 2 * (length(ridge_model$beta) + 1)
bicridge <- n * log(sum((predict(ridge_model, newx = x, s = bestlambda_min) – y)^2)/n) + log(n) * (length(ridge_model$beta) + 1)
print(aicridge)
print(bicridge)
AIC(final_model)
BIC(final_model)