Knowledge Transfer

It is for the neural network to learn both deep patterns using the deep path and simple rules through the short path. In contrast, regular MLP forces all the data to flow through the entire stack of layers. These simple patterns in the data may end up being distorted by this sequence of transformations.

The stack function serves the same role as append in lists. It concatenates the sequence of tensors along a new dimension. It doesn’t change the original vector space but instead adds a new index to the new tensor, so you retain the ability to get the original tensor you added to the list by indexing in the new dimension.

A module can have one or more Parameters (its weights and bise) instances as attributes, which are tensors. A module can also have one or more submodules (subclasses of nn.Module) as attributes, and it will also be able to track their parameters.

Another popular learning rate schedule used with deep learning models is systematically dropping the learning rate at specific times during training.Decays the learning rate by gamma every step_size epochs.

Adam does not generalize as well as SGD with momentum when tested on a diverse set of deep learning tasks such as image classification, character-level language modeling, and constituency parsing. Adam lies in its dysfunctional implementation of weight decay.

Dimensions along which they encode probabilities and others in which they don’t, nn.Softmax requires us to specify the dimension along which the softmax function is applied. Let’s test it on an input vector

Choosing the best value for the num_workers argument depends on your hardware, characteristics of your training data (such as its size and shape), the cost of your transform function, and what other processing is happening on the CPU at the same time. A simple heuristic is to use the number of available CPU cores.

The amount of regularization must be balanced for each dataset and architecture. Recognition of this principle permits the general use of super-convergence. Reducing other forms of regularization and regularizing with very large learning rates makes training significantly more efficient.

L2 regularization is also referred to as weight decay. The reason for this name is that thinking about SGD and backpropagation, the negative gradient of the L2 regularization term with respect to a parameter w_i is – 2 * lambda * w_i, where lambda is the aforementioned hyperparameter, simply named weight decay in PyTorch. 

When we call loss.backward(), PyTorch traverses this graph in the reverse direction to compute the gradients and accumulate their values in the grad attribute of those tensors (the leaf nodes of the graph).