Category Archives: Deep Learning
Differences between Learning Rate and Weight Decay Hyperparameters in Neural networks.
The amount of regularization must be balanced for each dataset and architecture. Recognition of this principle permits the general use of super-convergence. Reducing other forms of regularization and regularizing with very large learning rates makes training significantly more efficient.