Once I discover my mannequin is overfitting, I usually assume, “It’s time to regularize”. However how do I resolve which regularization methodology to make use of (L1, L2) and what parameters to decide on? Usually, I carry out hyperparameter optimization by the use of a grid search to pick the settings. Nonetheless, what occurs if the impartial variables have totally different scales or various ranges of affect? Can I design a hyperparameter grid with totally different regularization coefficients for every variable? Is this kind of optimization possible in high-dimensional areas? And are there alternative routes to design regularization? Let’s discover this with a hypothetical instance.
My fictional instance is a binary classification use case with 3 explanatory variables. Every of those variables is categorical and has 7 totally different classes. My reproducible use case is on this notebook. The perform that generates the dataset is the next:
import numpy as np
import pandas as pddef get_classification_dataset():
n_samples = 200
cats = ["a", "b", "c", "d", "e", "f"]
X = pd.DataFrame(
information={
"col1": np.random.selection(cats, dimension=n_samples),
"col2": np.random.selection(cats, dimension=n_samples),
"col3": np.random.selection(cats, dimension=n_samples),
}
)
X_preprocessed = pd.get_dummies(X)
theta = np.random.multivariate_normal(
np.zeros(len(cats) * X.form[1]),
np.diag(np.array([1e-1] * len(cats) + [1] * len(cats) + [1e1] * len(cats))),
)
y = pd.Sequence(
information=np.random.binomial(1, expit(np.dot(X_preprocessed.to_numpy(), theta))),
index=X_preprocessed.index,
)
return X_preprocessed, y
For data, I intentionally selected 3 totally different values for the theta covariance matrix to showcase the advantage of the Laplace approximated bayesian optimization methodology. If the values had been one way or the other related, the curiosity can be minor.
Together with a easy baseline mannequin that predicts the imply noticed worth on the coaching dataset (used for comparability functions), I opted to design a barely extra complicated mannequin. I made a decision to one-hot encode the three impartial variables and apply a logistic regression mannequin on high of this primary preprocessing. For regularization, I selected an L2 design and aimed to seek out the optimum regularization coefficient utilizing two strategies: grid search and Laplace approximated bayesian optimization, as you will have anticipated by now. Lastly, I evaluated the mannequin on a take a look at dataset utilizing two metrics (arbitrarily chosen): log loss and AUC ROC.
Earlier than presenting the outcomes, let’s first take a better have a look at the bayesian mannequin and the way we optimize it.
Within the bayesian framework, the parameters are not fastened constants, however random variables. As an alternative of maximizing the probability to estimate these unknown parameters, we now optimize the posterior distribution of the random parameters, given the noticed information. This requires us to decide on, usually considerably arbitrarily, the design and parameters of the prior. Nonetheless, it is usually potential to deal with the parameters of the prior as random variables themselves — like in Inception, the place the layers of uncertainty hold stacking on high of one another…
On this examine, I’ve chosen the next mannequin:
I’ve logically chosen a bernouilli mannequin for Y_i | θ, a traditional centered prior equivalent to a L2 regularization for θ | Σ and eventually for Σ_i^{-1}, I selected a Gamma mannequin. I selected to mannequin the precision matrix as an alternative of the covariance matrix as it’s conventional within the literature, like in scikit be taught consumer information for the Bayesian linear regression [2].
Along with this written mannequin, I assumed the Y_i and Y_j are conditionally (on θ) impartial in addition to Y_i and Σ.
Chance
In response to the mannequin, the probability can consequently be written:
With a purpose to optimize, we have to consider almost all the phrases, aside from P(Y=y). The phrases within the numerators may be evaluated utilizing the chosen mannequin. Nonetheless, the remaining time period within the denominator can not. That is the place the Laplace approximation comes into play.
Laplace approximation
With a purpose to consider the primary time period of the denominator, we are able to leverage the Laplace approximation. We approximate the distribution of θ | Y, Σ by:
with θ* being the mode of the mode the density distribution of θ | Y, Σ.
Despite the fact that we have no idea the density perform, we are able to consider the Hessian half because of the next decomposition:
We solely have to know the primary two phrases of the numerator to judge the Hessian which we do.
For these serious about additional rationalization, I counsel half 4.4, “The Laplace Approximation”, from Sample Recognition and Machine Studying from Christopher M. Bishop [1]. It helped me rather a lot to know the approximation.
Laplace approximated probability
Lastly the Laplace approximated probability to optimize is:
As soon as we approximate the density perform of θ | Y, Σ, we might lastly consider the probability at no matter θ we wish if the approximation was correct in every single place. For the sake of simplicity and since the approximation is correct solely near the mode, we consider approximated probability at θ*.
Right here after is a perform that evaluates this loss for a given (scalar) σ²=1/p (along with the given noticed, X and y, and design values, α and β).
import numpy as np
from scipy.stats import gammafrom module.bayesian_model import BayesianLogisticRegression
def loss(p, X, y, alpha, beta):
# computation of the loss for given values:
# - 1/sigma² (named p for precision right here)
# - X: matrix of options
# - y: vector of observations
# - alpha: prior Gamma distribution alpha parameter over 1/sigma²
# - beta: prior Gamma distribution beta parameter over 1/sigma²
n_feat = X.form[1]
m_vec = np.array([0] * n_feat)
p_vec = np.array(p * n_feat)
# computation of theta*
res = reduce(
BayesianLogisticRegression()._loss,
np.array([0] * n_feat),
args=(X, y, m_vec, p_vec),
methodology="BFGS",
jac=BayesianLogisticRegression()._jac,
)
theta_star = res.x
# computation the Hessian for the Laplace approximation
H = BayesianLogisticRegression()._hess(theta_star, X, y, m_vec, p_vec)
# loss
loss = 0
## first two phrases: the log loss and the regularization time period
loss += baysian_model._loss(theta_star, X, y, m_vec, p_vec)
## third time period: prior distribution over sigma, written p right here
out -= gamma.logpdf(p, a = alpha, scale = 1 / beta)
## fourth time period: Laplace approximated final time period
out += 0.5 * np.linalg.slogdet(H)[1] - 0.5 * n_feat * np.log(2 * np.pi)
return out
In my use case, I’ve chosen to optimize it by the use of Adam optimizer, which code has been taken from this repo.
def adam(
enjoyable,
x0,
jac,
args=(),
learning_rate=0.001,
beta1=0.9,
beta2=0.999,
eps=1e-8,
startiter=0,
maxiter=1000,
callback=None,
**kwargs
):
"""``scipy.optimize.reduce`` appropriate implementation of ADAM -
[http://arxiv.org/pdf/1412.6980.pdf].
Tailored from ``autograd/misc/optimizers.py``.
"""
x = x0
m = np.zeros_like(x)
v = np.zeros_like(x)for i in vary(startiter, startiter + maxiter):
g = jac(x, *args)
if callback and callback(x):
break
m = (1 - beta1) * g + beta1 * m # first second estimate.
v = (1 - beta2) * (g**2) + beta2 * v # second second estimate.
mhat = m / (1 - beta1**(i + 1)) # bias correction.
vhat = v / (1 - beta2**(i + 1))
x = x - learning_rate * mhat / (np.sqrt(vhat) + eps)
i += 1
return OptimizeResult(x=x, enjoyable=enjoyable(x, *args), jac=g, nit=i, nfev=i, success=True)
For this optimization we’d like the spinoff of the earlier loss. We can not have an analytical type so I made a decision to make use of a numerical approximation of the spinoff.
As soon as the mannequin is educated on the coaching dataset, it’s essential to make predictions on the analysis dataset to evaluate its efficiency and evaluate totally different fashions. Nonetheless, it isn’t potential to instantly calculate the precise distribution of a brand new level, because the computation is intractable.
It’s potential to approximate the outcomes with:
contemplating:
I selected an uninformative prior over the precision random variable. The naive mannequin performs poorly, with a log lack of 0.60 and an AUC ROC of 0.50. The second mannequin performs higher, with a log lack of 0.44 and an AUC ROC of 0.83, each when hyperoptimized utilizing grid search and bayesian optimization. This means that the logistic regression mannequin, which contains the dependent variables, outperforms the naive mannequin. Nonetheless, there is no such thing as a benefit to utilizing bayesian optimization over grid search, so I’ll proceed with grid seek for now. Thanks for studying.
… However wait, I’m pondering. Why are my parameters regularized with the identical coefficient? Shouldn’t my prior rely upon the underlying dependent variables? Maybe the parameters for the primary dependent variable might take greater values, whereas these for the second dependent variable, with its smaller affect, needs to be nearer to zero. Let’s discover these new dimensions.
To this point now we have thought of two strategies, the grid search and the bayesian optimization. We are able to use these similar strategies in greater dimensions.
Contemplating new dimensions might dramatically improve the variety of nodes of my grid. For this reason the bayesian optimization is smart in greater dimensions to get the perfect regularization coefficients. Within the thought of use case, I’ve supposed there are 3 regularization parameters, one for every impartial variable. After encoding a single variable, I assumed the generated new variables all shared the identical regularization parameter. Therefore the full regularization parameters of three, even when there are greater than 3 columns as inputs of the logistic regression.
I up to date the earlier loss perform with the next code:
import numpy as np
from scipy.stats import gammafrom module.bayesian_model import BayesianLogisticRegression
def loss(p, X, y, alpha, beta, X_columns, col_to_p_id):
# computation of the loss for given values:
# - 1/sigma² vector (named p for precision right here)
# - X: matrix of options
# - y: vector of observations
# - alpha: prior Gamma distribution alpha parameter over 1/sigma²
# - beta: prior Gamma distribution beta parameter over 1/sigma²
# - X_columns: listing of names of X columns
# - col_to_p_id: dictionnary mapping a column title to a p index
# as a result of many column names can share the identical p worth
n_feat = X.form[1]
m_vec = np.array([0] * n_feat)
p_list = []
for col in X_columns:
p_list.append(p[col_to_p_id[col]])
p_vec = np.array(p_list)
# computation of theta*
res = reduce(
BayesianLogisticRegression()._loss,
np.array([0] * n_feat),
args=(X, y, m_vec, p_vec),
methodology="BFGS",
jac=BayesianLogisticRegression()._jac,
)
theta_star = res.x
# computation the Hessian for the Laplace approximation
H = BayesianLogisticRegression()._hess(theta_star, X, y, m_vec, p_vec)
# loss
loss = 0
## first two phrases: the log loss and the regularization time period
loss += baysian_model._loss(theta_star, X, y, m_vec, p_vec)
## third time period: prior distribution over 1/sigma² written p right here
## there's now a sum as p is now a vector
out -= np.sum(gamma.logpdf(p, a = alpha, scale = 1 / beta))
## fourth time period: Laplace approximated final time period
out += 0.5 * np.linalg.slogdet(H)[1] - 0.5 * n_feat * np.log(2 * np.pi)
return out
With this method, the metrics evaluated on the take a look at dataset are the next: 0.39 and 0.88, that are higher than the preliminary mannequin optimized by the use of a grid search and a bayesian method with solely a single prior for all of the impartial variables.
The use case may be reproduced with this notebook.
I’ve created an instance as an example the usefulness of the method. Nonetheless, I’ve not been capable of finding an appropriate real-world dataset to totally reveal its potential. Whereas I used to be working with an precise dataset, I couldn’t derive any vital advantages from making use of this method. In case you come throughout one, please let me know — I’d be excited to see a real-world software of this regularization methodology.
In conclusion, utilizing bayesian optimization (with Laplace approximation if wanted) to find out the perfect regularization parameters could also be various to conventional hyperparameter tuning strategies. By leveraging probabilistic fashions, bayesian optimization not solely reduces the computational value but additionally enhances the probability of discovering optimum regularization values, particularly in excessive dimension.
- Christopher M. Bishop. (2006). Sample Recognition and Machine Studying. Springer.
- Bayesian Ridge Regression scikit-learn consumer information: https://scikit-learn.org/1.5/modules/linear_model.html#bayesian-ridge-regression
