The accuracy of our model on the validation data would peak after training for a number of epochs, and would then stagnate or start decreasing. If you train for too long though, the model will start to overfit. We need to strike a balance. In other words, our model would overfit to the training data.

Learning how to save the best model when overfitting is important. Although it’s often possible to achieve high accuracy in the middle of the training model, what we really want is to save high-accuracy models that generalize well to a testing set.

In this tutorial, we understand how to save the best model during the training. Using two different callbacks ModelCheckpoint and EarlyStopping.

ModelCheckpoint

If you want to save the best model during training, you have to use the ModelCheckpoint callback class. It has options to save the model weights at given times during the training and will allow you to keep the weights of the model at the end of the epoch specifically where the validation loss was at its minimum. 

You can set the filepath by using certain dynamic variables available during training so that the filenames have some useful information:

checkpoint_filepath = 'weights.{epoch:02d}-{val_loss:.2f}.h5'

If you don’t use save_best_only, the default behavior is to save the model at the end of every epoch. This is selected using the save_best_only parameter. use it like this:

model_checkpoint_callback = keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_filepath,
    monitor='val_accuracy',
    mode='max',
    save_best_only=True)

The  ‘save_best_only’ parameter is used to whether to only keep the model that has achieved the “best performance” so far, or whether to save the model at the end of every epoch regardless of performance. Definition of ‘best’ which quantity to monitor and whether it should be maximized or minimized.

history=model.fit(x_train, y_train, batch_size=batch_size, validation_data=(x_test,y_test),epochs=epochs,callbacks=[model_checkpoint_callback], 
          validation_split=0.1)

You can conjunction with model.fit() to save a model or weights in a checkpoint file, so the model or weights can be loaded later to continue the training from the state saved.

EarlyStopping

The EarlyStopping callback will restore the best weights only if you initialized with the parameters restore_best_weights to True. For example, if a model is set to train for 50 epochs, uses early stopping with the patience of 5, and the best weights occur in epoch 25, then the best weights are restored from epoch 25. The final weights will not be saved(the weights where your patience parameter is triggered). 

The class can be initialized with the parameters restore_best_weights, as seen below. 

callback=keras.callbacks.EarlyStopping(
    monitor='val_loss', min_delta=0, patience=2, verbose=2, mode='auto',
    baseline=None, restore_best_weights=True)

It does this by tracking a chosen metric and comparing it to the recorded best value. By default, this will be the validation loss.

history_2=model.fit(x_train, y_train, validation_data=(x_test,y_test),batch_size=batch_size, epochs=epochs,callbacks=[callback], 
          validation_split=0.1)

Now your code saves the last model that achieved the best result on dev set before the training was stopped by the early stopping callback.

At the end of the training, when your waiting period has overshot the patience parameter, the model’s weights are returned to be the best weights (weights of the model at the time of lowest validation loss).

#In EarlyStopping 
def on_epoch_end(self, epoch, logs=None):
.....
      self.wait += 1
      if self.wait >= self.patience:
        self.stopped_epoch = epoch
        self.model.stop_training = True
        if self.restore_best_weights:
          if self.verbose > 0:
            print('Restoring model weights from the end of the best epoch.')
          self.model.set_weights(self.best_weights)

Related Post

Run this code in Google colab.