In this post, you will learn how to implement a skip gram model in TensorFlow to generate word vectors and then use TensorBoard to visualize them.

TensorBoard visualize

Jupyter notebook that can run locally, or on Colaboratory.

One of the key ideas of word embeddings is a way of representing words that model automatically understand analogies like that “Man is the Woman as King is the Queen”.You’ll be able to build NLP application even relatively small training sets.

Word2Vec


Word2Vec use the theory of meaning to predict between every word and its context words. There are two methods to producing word vectors 1. Continuous Bag-of-Words model (CBOW) and 2.Skip-Gram model. For this post we implement is a Skip-Gram method.

Skip-Gram

TensorFlow Skip-gram model

The idea of the skip-gram model is, for each estimation step, you’re taking one word as the center word. In above example “for” is the center word and you’re going to try and predict words in its context out to some window size. The model is going to define a probability distribution that is the probability if a word appearing in the context given in this center word.

1.Setup


This code requires Python 3 and TensorFlow v1.4 +.

2. Download Dataset


The “IMDB dataset” is a set of around 50,000 positive or negative reviews for movies from the Internet Movie Database. Each review consists of a series of word indexes that go from 4(the most frequent word in the dataset, the) to 4999.Index 1 for the beginning of the sentence and 2 for unknown.TensorFlow provides Keras API for download the dataset.

3. Pad Sequences


After Loaded the data in memory pad each of the sentences with 0 to a fixed size (200). You have two 2-dimensional (25000X200)arrays for training and testing respectively.

4. Input Function


You needs to convert the data from numpy arrays into Tensors. The tf.data provide classes that allow you to easily load data, manipulate it, and pipe it into your model.

To construct a Dataset from some tensors in memory, you can use tf.data.Dataset.from_tensor_slices().
You might create one function to import the training set and another function to import the test set.

This input function builds an input pipeline that yields batches of (features, labels) pairs, where features is a dictionary feature.

5. Create Feature Columns


When you build an Estimator model, you pass a list of feature columns that describe each of the features you want the model to use.

categorical_column_with_identity is the right choice for text input. You need to convert your existing feature column into an embedding_column. The representation seen by the model is the mean of the embeddings for each token. We can plug in the embedded features into a pre-canned DNNClassifier.

6. Estimator


You have to not worry about creating the computational graph or sessions since Pre-made Estimators handle all the “plumbing” for you. DNNClassifier is a pre-made Estimator class that trains classification models through dense, feed-forward neural networks.

7.Train Model


Estimators expect an input_fn to take no arguments.

Evaluate Model


The following code block evaluates the accuracy of the trained model on the test data

Running this code yields the following output (or something similar):

Test set accuracy: 0.821

8. Visualizing Embeddings


TensorBoard includes the Embedding Projector, a tool for interactively visualize embeddings.It can read embeddings from your model and render them in two or three dimensions.

Metadata

You need to attach labels to the data points. You can do this by generating a metadata file containing the labels for each point and clicking “Load data” in the data panel of the Embedding Projector.

 

Leave a Reply