Knowledge Transfer

The saturated neurons can kill gradients if we’re too positive or too negative of an input. They’re also not zero-centered and so we get these, this inefficient kind of gradient update. The third problem is an exponential function. This is a little bit computationally expensive.

You can evaluate the relationship between each feature and target using a correlation and selecting those features that have the strongest relationship with the target variable. Such as methods that remove redundant variables using correlation.

You can keep adding plt.plot as many times as you like. As for multiple scatter plot, you need to specify the color, marker, edgecolor, alpha so you can differentiate them.

We’ll look at a real-world example with data that I’ll load from a CSV file and also we’re going to learn how to draw overlapping histograms using Pandas and Matplotlib. Before we go and set up how we create the chart let’s look at what the end results would be.

we want to make a copy of a tensor and ensure that any operations are done with the cloned tensor to ensure that the gradients are propagated to the original tensor, we must use clone(). We should use detach() when we don’t want to include a tensor in the resulting computational graph.

The PyTorch model is mutable if we change any of the two models this action will have a direct impact on the other model too, as they both point to the same object reference in memory.

The weight matrix defined the linear function this basically demonstrates how the network mapping changes as the weights are updated during the training process when we update the weight we are changing the function.

We want to convert this to a tensor of size bs x ch, so we take the average over the last two dimensions and flatten the trailing 1×1 dimension as we did in our previous model. 

The total number of parameters is shown at the end, which is equal to the number of trainable and non-trainable parameters. In this model, all the layers are trainable. 

Create a custom Dataset and DataLoader to preprocess the time series like data into a matrix-like shape. We reshape the data in that way to just illustrate the point.