The blog post introduces an important concept of regularization in machine learning.

Ridge Regression
In an extreme case where there are only two training data points, we can always draw a straight line that has zero residuals and achieve zero bias.
However, as the above shows, having zero bias often leads to high variance and overfitting due to the bias-variance tradeoff. (If you are not sure about the bias-variance tradeoff, check out the last article on cross-validation.) How can we avoid overfitting? One way is to use Ridge Regression or L2 Regularization. In Ridge regression, we add a ridge regression penalty to the cost function.
, where are the slopes of the linear function. By adding the square of the slope, we can penalize it when slope is steep (even if the slope is negative), effectively making the prediction less sensitive to the training data. is the regularization rate, which allows us to control how much penality should be added. This is one of the hyperparameters that we can tune with cross-validation.
Lasso Regression
Instead of adding squares, we can use the absolute value so that we can penalize it when the slope is steep.
This is called Lasso Regression or L1 Regularization.
L1 vs L2 Regularization
You might think it does not matter which regularization technique to use, but there is a reason that I introduced both here. When training the model, we compute the gradient and subtract that graident from the parameters. Hence, let's compute the gradient for slopes in Ridge and Lasso regression. For Ridge regression (L2), the gradient looks like this:
For Lasso regression (L1), it looks like the following:
They are very similar, but we can see that the penalty term is different between L1 and L2 regularization. In L1 regularization, we subtract regardless of the value of the slope. Hence, it encourages the slope to be zero. However, in L2 regularization, we subtract , meaning the penalty term gets smaller as the value of the slope gets smaller. Thus, while L2 regularization encourages slopes to be close to zero, it does not make the slope zero.
Therefore, when we are sure that some of the explanatory variables are completely useless and want to automatically avoid them from the equation, we can use Lasso regression or L1 regularization, as it can make the slopes for those meaningless variables zero. However, when we know that all of the variables are meaningful, we can use Ridge regression or L2 regularization to keep all of the slopes non-zero values while preventing overfitting.
Elastic-Net Regression
While the distinction between L1 and L2 regularization can help us choose which one to use sometimes, there are some cases where we are not even sure if there are some explanatory variables that are redundant or not due to the sheer number of features. In that case, we can use Elastic-Net Regression that combines them into one like below.
By combining both L1 and L2, we can eliminate unnecessary explanatory variables while encoruaging the slopes to be smaller for other useful variables. The downside to this approach is that we have more hyperparameters to tune and more computation to perform.
Coding Implementation
Let's incorporate regularization feature in LinearRegressionGD
.
class LinearRegressionGD():
def __init__(self, lr=0.01, regularization="l1", alpha=0.01, beta=0.01):
self.W = np.zeros(X.shape[1])
self.b = 0
self.lr = lr # Learning rate
self.history = [] # History of loss
self.regularization = regularization
self.alpha = alpha # lambda (lambda_1 for elastic-net)
self.beta = beta # lambda_2 for elastic-net
def predict(self, X):
return np.sum(self.W*X + self.b, axis=1)
def gradient(self, X, y, pred):
diff = pred - y
grad_W = np.sum((1/n)*diff[:, np.newaxis]*X, axis=0)
grad_b = np.sum((1/n)*diff)
# Regularization
if (self.regularzation == "l1"):
grad_W += self.alpha
elif (self.regularization == "l2"):
grad_W += 2 * self.alpha * self.W
elif (self.regularization == "elastic-net"):
gradW += self.alpha + 2 * self.beta * self.W
return grad_W, grad_b
def fit(self, X, y, epochs=100):
for i in range(epochs):
pred = self.predict(X)
n = len(y)
self.history.append(mean_squared_error(y, pred))
grad_W, grad_b = gradient(X, y, pred)
self.W -= self.lr * grad_W
self.b -= self.lr * grad_b
return self.history
Using the above, you can use Ridge, Lasso, and Elastic-Net regressions like below.
l1 = LinearRegressionGD(regularization="l1", alpha=0.01)
l2 = LinearRegressionGD(regularization="l2", alpha=0.01)
en = LinearRegreessionGD(regularization="elastic-net", alpha=0.01, beta=0.01)
Then, you can use cross-validation for choosing the best model with lowest variance. I recommend you to
try incorporate them in LogisticRegressionGD
and SoftmaxRegressionGD
too as coding exercises.
[Fun Fact] Bayesian Interpretation
I was fortunate to discover the video from ritvikmath (2021) titled Bayesian Linear Regression : Data Science Concepts on YouTube, which presents an interesting interpretation of Ridge and Lasso regression in terms of Bayesian statistics. In short, the video explains that MSE corresponds to the negative log-likelihood of observing data given the parameters, and the regularization terms for Ridge and Lasso regression correspond to the Gaussian prior and Laplacian prior, respectively. It is a brilliant video, and I highly recommend checking it out.
Resources
- StatQuest with Josh Starmer. 2019. Regularization Part 1: Ridge (L2) Regression. YouTube.
- StatQuest with Josh Starmer. 2019. Regularization Part 2: Lasso (L1) Regression. YouTube.
- StatQuest with Josh Starmer. 2019. Regularization Part 3: Elastic Net Regression. YouTube.