Knowledge Transfer

Dimensions along which they encode probabilities and others in which they don’t, nn.Softmax requires us to specify the dimension along which the softmax function is applied. Let’s test it on an input vector

Choosing the best value for the num_workers argument depends on your hardware, characteristics of your training data (such as its size and shape), the cost of your transform function, and what other processing is happening on the CPU at the same time. A simple heuristic is to use the number of available CPU cores.

The amount of regularization must be balanced for each dataset and architecture. Recognition of this principle permits the general use of super-convergence. Reducing other forms of regularization and regularizing with very large learning rates makes training significantly more efficient.

L2 regularization is also referred to as weight decay. The reason for this name is that thinking about SGD and backpropagation, the negative gradient of the L2 regularization term with respect to a parameter w_i is – 2 * lambda * w_i, where lambda is the aforementioned hyperparameter, simply named weight decay in PyTorch. 

When we call loss.backward(), PyTorch traverses this graph in the reverse direction to compute the gradients and accumulate their values in the grad attribute of those tensors (the leaf nodes of the graph).

In working with images, it is good practice to compute the mean and standard deviation on all the training data in advance and then subtract and divide by these fixed, precomputed quantities.

PyTorch offers a package called Torchvision that includes many commonly used transforms for image processing. In PyTorch, we can achieve this using the Normalize Transform.

If you observe that the model is overfitting, you can increase the dropout rate. Conversely, you should try decreasing the dropout rate if the model underfits the training set.

Inside your data are subsets of data that represent different groups  If you plot the price values for each subset, you might get a bunch of different shapes.

torch.gather() creates a new tensor from the input tensor by taking the values from each row or column along the input dimension. The index values are passed as tensors, specifying which value to take from each ‘row’ or ‘column’.