Category Archives: PyTorch

When creating an np.array from torch.tensor or vice versa, both object reference the same underlying storage in memory. Since np.ndarray does not store the computational graph associated with the array, this graph should be explicitly removed using detach() when sharing both numpy and torch wish to reference the same tensor.

If backward is called again as in any training loop then the loss is evaluated again, and the gradient at each leaf is accumulated on top.

Using the designated settings for training model.train() and evaluation model.eval() will automatically set the mode for the dropout layer and batch normalization layers and rescale appropriately so that we do not have to worry about that at all.

Access an individual batch from the DataLoader, we first pass the DataLoader object to Python’s iter() built-in function, which returns an object representing a stream of data.

Computing the binary loss by providing the logits, and not the class probabilities is usually preferred due to numerical stability.

you could just sum it and calculate the mean after the epoch finishes or at the end of the epoch, we divide by the number of steps(dataset size). It gives you the correct average sample loss for this particular epoch. This training loss is used to see, “how well your model performs on the training dataset”.

The workaround is to use log probability instead of probability, which takes care to make the calculation numerically stable. The reformulated version allows us to evaluate softmax with only small numerical errors even when z contains extremely large or extremely negative numbers.

Contiguous tensors are convenient because we can visit them efficiently in order without jumping around in the storage. It improves data locality and improves performance because of the way memory access works on modern CPUs. This advantage of course depends on the way algorithms visit.

The saturated neurons can kill gradients if we’re too positive or too negative of an input. They’re also not zero-centered and so we get these, this inefficient kind of gradient update. The third problem is an exponential function. This is a little bit computationally expensive.

we want to make a copy of a tensor and ensure that any operations are done with the cloned tensor to ensure that the gradients are propagated to the original tensor, we must use clone(). We should use detach() when we don’t want to include a tensor in the resulting computational graph.