Knowledge Transfer

When the user specifies model.eval() and the model contains a batch normalization module, the running estimates are frozen and used for normalization. To unfreeze running estimates and return to using the minibatch statistics, we call model.train(), just as we did for dropout.

We can save tensors quickly this way but if we want to load them with the file format itself is not interoperable. We can’t read the tensor with software other than PyTorch. Depending on the use case, this may or may not be a limitation, but we should learn how to save tensors interoperably.

It is for the neural network to learn both deep patterns using the deep path and simple rules through the short path. In contrast, regular MLP forces all the data to flow through the entire stack of layers. These simple patterns in the data may end up being distorted by this sequence of transformations.

The stack function serves the same role as append in lists. It concatenates the sequence of tensors along a new dimension. It doesn’t change the original vector space but instead adds a new index to the new tensor, so you retain the ability to get the original tensor you added to the list by indexing in the new dimension.

A module can have one or more Parameters (its weights and bise) instances as attributes, which are tensors. A module can also have one or more submodules (subclasses of nn.Module) as attributes, and it will also be able to track their parameters.

Another popular learning rate schedule used with deep learning models is systematically dropping the learning rate at specific times during training.Decays the learning rate by gamma every step_size epochs.

Adam does not generalize as well as SGD with momentum when tested on a diverse set of deep learning tasks such as image classification, character-level language modeling, and constituency parsing. Adam lies in its dysfunctional implementation of weight decay.

Dimensions along which they encode probabilities and others in which they don’t, nn.Softmax requires us to specify the dimension along which the softmax function is applied. Let’s test it on an input vector

Choosing the best value for the num_workers argument depends on your hardware, characteristics of your training data (such as its size and shape), the cost of your transform function, and what other processing is happening on the CPU at the same time. A simple heuristic is to use the number of available CPU cores.

The amount of regularization must be balanced for each dataset and architecture. Recognition of this principle permits the general use of super-convergence. Reducing other forms of regularization and regularizing with very large learning rates makes training significantly more efficient.