September 19, 2022
Weight Decay parameter for SGD optimizer in PyTorch
PyTorchPragatiL2 regularization is also referred to as weight decay. The reason for this name is that thinking about SGD and backpropagation, the negative gradient of the L2 regularization term with respect to a parameter w_i is – 2 * lambda * w_i, where lambda is the aforementioned hyperparameter, simply named weight decay in PyTorch.