Multi-Label Text Classification in Keras

In this tutorial, we create a multi-label text classification model for predicts the probability of each type of toxicity for each comment. This model is capable of detecting different types of toxicity like threats, obscenity, insults, and identity-based hate. We need to create a model which predicts the probability of each type of toxicity for each comment.

Different between multi-class and multi-label Classification

In Multi-Class classification there are more than two classes; e.g., classify a set of images of fruits which may be oranges, apples, or pears. Each sample is assigned one and only one label: a fruit can be either an apple or an orange.

In Multi-Label classification, each sample has a set of target labels. A comment might be threats, obscenity, insults, and identity-based hate at the same time or none of these.

Multi-class classification uses softmax activation function in the output layer. The probability of each class is dependent on the other classes. As the probability of one class increases, the probability of the other class decreases.

The softmax activation function is not appropriate in Multi-label classification because it has more than one label for a single text. The probabilities are independent of each other. Here we use the sigmoid activation function. This will predict the probability for each class independently.

Download Dataset

Here, we use Toxic Comment Classification Challenge dataset from Kaggle. It provided Wikipedia comments which have been labeled by human raters for toxic behavior. The types of toxicity are:

toxic
severe_toxic
obscene
threat
insult
Identity_hate

Prepare Dataset

First, we format our text and labels into tensors that can be fed into a neural network. To do this, we use Keras utilities keras.preprocessing.text.Tokenizer and keras.preprocessing.sequence.pad_sequences.

import tensorflow as tf
import numpy as np
import pandas as pd

import os

TRAIN_DATA = "data/train.csv"
GLOVE_EMBEDDING = "embedding/glove.6B.100d.txt"

train = pd.read_csv(TRAIN_DATA)

train["comment_text"].fillna("fillna")

x_train = train["comment_text"].str.lower()
y_train = train[["toxic", "severe_toxic", "obscene", "threat", "insult", "identity_hate"]].values

max_words = 100000
max_len = 150

embed_size = 100

tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=max_words, lower=True)

tokenizer.fit_on_texts(x_train)

x_train = tokenizer.texts_to_sequences(x_train)

x_train = tf.keras.preprocessing.sequence.pad_sequences(x_train, maxlen=max_len)

Use pre-train embeddings

In this tutorial, We use pre-trained word embedding for Text classification. We will use the 100-dimensional GloVe embeddings of 400k words computed on a 2014 dump of English Wikipedia. You can download them here.

we compute an index mapping words to known embeddings, by parsing the data dump of pre-trained embeddings:

embeddings_index = {}

with open(GLOVE_EMBEDDING, encoding='utf8') as f:
    for line in f:
        values = line.rstrip().rsplit(' ')
        word = values[0]
        embed = np.asarray(values[1:], dtype='float32')
        embeddings_index[word] = embed

word_index = tokenizer.word_index

num_words = min(max_words, len(word_index) + 1)

embedding_matrix = np.zeros((num_words, embed_size), dtype='float32')

for word, i in word_index.items():

    if i >= max_words:
        continue

    embedding_vector = embeddings_index.get(word)

    if embedding_vector is not None:
        embedding_matrix[i] = embedding_vector

Now, We load this embedding matrix into an Embedding layer. Note that we set trainable=False to prevent the weights from being updated during training.

input = tf.keras.layers.Input(shape=(max_len,))

x = tf.keras.layers.Embedding(max_words, embed_size, weights=[embedding_matrix], trainable=False)(input)

Bidirectional Layer

It propagates the input forward and backward through the RNN layer and then concatenates the output. This helps the RNN to learn long-range dependencies.

x = tf.keras.layers.Bidirectional(tf.keras.layers.GRU(128, return_sequences=True, dropout=0.1,
                                                      recurrent_dropout=0.1))(x)

x = tf.keras.layers.Conv1D(64, kernel_size=3, padding="valid", kernel_initializer="glorot_uniform")(x)

avg_pool = tf.keras.layers.GlobalAveragePooling1D()(x)
max_pool = tf.keras.layers.GlobalMaxPooling1D()(x)

x = tf.keras.layers.concatenate([avg_pool, max_pool])

preds = tf.keras.layers.Dense(6, activation="sigmoid")(x)

model = tf.keras.Model(input, preds)

model.summary()

model.compile(loss='binary_crossentropy', optimizer=tf.keras.optimizers.Adam(lr=1e-3), metrics=['accuracy'])

Here we use sigmoid activation instead of softmax for the output layer and binary_crossentropy for the loss function.

Train Model

batch_size = 128

checkpoint_path = "training_1/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

cp_callback = tf.keras.callbacks.ModelCheckpoint(checkpoint_path,
                                                 save_weights_only=True,
                                                 verbose=1)

callbacks = [
    tf.keras.callbacks.EarlyStopping(patience=5, monitor='val_loss'),
    tf.keras.callbacks.TensorBoard(log_dir='./logs'),
    cp_callback
]

model.fit(x_train, y_train, validation_split=0.2, batch_size=batch_size,
          epochs=1, callbacks=callbacks, verbose=1)

Predictions

latest = tf.train.latest_checkpoint(checkpoint_dir)

model.load_weights(latest)

predictions = model.predict(np.expand_dims(x_train[43], 0))

print(tokenizer.sequences_to_texts([x_train[43]]))
print(y_train[43])
print(predictions)