In this tutorial, we create a multi-label text classification model for predicts the probability of each type of toxicity for each comment. This model is capable of detecting different types of toxicity like threats, obscenity, insults, and identity-based hate. We need to create a model which predicts the probability of each type of toxicity for each comment.
Different between multi-class and multi-label Classification
In Multi-Class classification there are more than two classes; e.g., classify a set of images of fruits which may be oranges, apples, or pears. Each sample is assigned one and only one label: a fruit can be either an apple or an orange.
In Multi-Label classification, each sample has a set of target labels. A comment might be threats, obscenity, insults, and identity-based hate at the same time or none of these.
Multi-class classification uses softmax activation function in the output layer. The probability of each class is dependent on the other classes. As the probability of one class increases, the probability of the other class decreases.
The softmax activation function is not appropriate in Multi-label classification because it has more than one label for a single text. The probabilities are independent of each other. Here we use the sigmoid activation function. This will predict the probability for each class independently.
Download Dataset
Here, we use Toxic Comment Classification Challenge dataset from Kaggle. It provided Wikipedia comments which have been labeled by human raters for toxic behavior. The types of toxicity are:
- toxic
- severe_toxic
- obscene
- threat
- insult
- Identity_hate
Prepare Dataset
First, we format our text and labels into tensors that can be fed into a neural network. To do this, we use Keras utilities keras.preprocessing.text.Tokenizer
and keras.preprocessing.sequence.pad_sequences
.
import tensorflow as tf
import numpy as np
import pandas as pd
import os
TRAIN_DATA = "data/train.csv"
GLOVE_EMBEDDING = "embedding/glove.6B.100d.txt"
train = pd.read_csv(TRAIN_DATA)
train["comment_text"].fillna("fillna")
x_train = train["comment_text"].str.lower()
y_train = train[["toxic", "severe_toxic", "obscene", "threat", "insult", "identity_hate"]].values
max_words = 100000
max_len = 150
embed_size = 100
tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=max_words, lower=True)
tokenizer.fit_on_texts(x_train)
x_train = tokenizer.texts_to_sequences(x_train)
x_train = tf.keras.preprocessing.sequence.pad_sequences(x_train, maxlen=max_len)
Use pre-train embeddings
In this tutorial, We use pre-trained word embedding for Text classification. We will use the 100-dimensional GloVe embeddings of 400k words computed on a 2014 dump of English Wikipedia. You can download them here.
we compute an index mapping words to known embeddings, by parsing the data dump of pre-trained embeddings:
embeddings_index = {}
with open(GLOVE_EMBEDDING, encoding='utf8') as f:
for line in f:
values = line.rstrip().rsplit(' ')
word = values[0]
embed = np.asarray(values[1:], dtype='float32')
embeddings_index[word] = embed
word_index = tokenizer.word_index
num_words = min(max_words, len(word_index) + 1)
embedding_matrix = np.zeros((num_words, embed_size), dtype='float32')
for word, i in word_index.items():
if i >= max_words:
continue
embedding_vector = embeddings_index.get(word)
if embedding_vector is not None:
embedding_matrix[i] = embedding_vector
Now, We load this embedding matrix into an Embedding
layer. Note that we set trainable=False
to prevent the weights from being updated during training.
input = tf.keras.layers.Input(shape=(max_len,))
x = tf.keras.layers.Embedding(max_words, embed_size, weights=[embedding_matrix], trainable=False)(input)
Bidirectional Layer
It propagates the input forward and backward through the RNN layer and then concatenates the output. This helps the RNN to learn long-range dependencies.
x = tf.keras.layers.Bidirectional(tf.keras.layers.GRU(128, return_sequences=True, dropout=0.1,
recurrent_dropout=0.1))(x)
x = tf.keras.layers.Conv1D(64, kernel_size=3, padding="valid", kernel_initializer="glorot_uniform")(x)
avg_pool = tf.keras.layers.GlobalAveragePooling1D()(x)
max_pool = tf.keras.layers.GlobalMaxPooling1D()(x)
x = tf.keras.layers.concatenate([avg_pool, max_pool])
preds = tf.keras.layers.Dense(6, activation="sigmoid")(x)
model = tf.keras.Model(input, preds)
model.summary()
model.compile(loss='binary_crossentropy', optimizer=tf.keras.optimizers.Adam(lr=1e-3), metrics=['accuracy'])
Here we use sigmoid activation instead of softmax for the output layer and binary_crossentropy for the loss function.
Train Model
batch_size = 128
checkpoint_path = "training_1/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)
cp_callback = tf.keras.callbacks.ModelCheckpoint(checkpoint_path,
save_weights_only=True,
verbose=1)
callbacks = [
tf.keras.callbacks.EarlyStopping(patience=5, monitor='val_loss'),
tf.keras.callbacks.TensorBoard(log_dir='./logs'),
cp_callback
]
model.fit(x_train, y_train, validation_split=0.2, batch_size=batch_size,
epochs=1, callbacks=callbacks, verbose=1)
Predictions
latest = tf.train.latest_checkpoint(checkpoint_dir)
model.load_weights(latest)
predictions = model.predict(np.expand_dims(x_train[43], 0))
print(tokenizer.sequences_to_texts([x_train[43]]))
print(y_train[43])
print(predictions)

Related Post
How to use a saved Keras model to Predict Text from scratch
Loss function for multi-class and multi-label classification in Keras and PyTorch