In this tutorial, we will build a TensorFlow RNN model for Time Series Prediction. This model is used to predict future values based on previously observed values. This model will try to predict the next value in a short sequence based on historical data.
Time series are widely used for non-stationary data, like predict stock markets, temperatures, traffic or sales data based on past patterns. They can also be adapted to generate text.
Simulate Time Series Data
It’s easier to learn with a small toy dataset that you can generate as much as you want. Let’s generate a bunch of time-series data.
from sklearn.cross_validation import train_test_split
import numpy as np
from matplotlib import pyplot as plt
import tensorflow as tf
from tensorflow.python.platform import tf_logging as logging
logging.set_verbosity(logging.INFO)
logging.log(logging.INFO, "Tensorflow version " + tf.__version__)
def generate_time_series(datalen):
freq1 = 0.2
freq2 = 0.15
noise = [np.random.random() * 0.1 for i in range(datalen)]
x1 = np.sin(np.arange(0, datalen) * freq1) + noise
x2 = np.sin(np.arange(0, datalen) * freq2) + noise
x = x1 + x2
return x.astype(np.float32)
DATA_SEQ_LEN = 24000
data = generate_time_series(DATA_SEQ_LEN)
Dataset Input Function
We will give it a sequence of number and ask it to predict the next number in the sequence using GRU cells. We unroll it a certain number of times. We put our sequence number on the inputs. It will produce some kind of number on the output. To teach it we force a sequence on the outputs which is the same sequence shifted by one number. This is what we will be teaching. Let’s write this in TensoFlow.
In this tutorial, we will use Dataset API to input data in model and Estimator API to train model. You need to provide an input_fn to read your data. You provide a function that returns inputs and labels. The inputs are a dictionary of all your inputs and the labels is a tensor.
SEQLEN = 16 # unrolled sequence length
BATCHSIZE = 32
X = data
Y = np.roll(data, -1)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y,
test_size=.2,
random_state=0)
X_train = np.reshape(X_train, [-1, SEQLEN])
Y_train = np.reshape(Y_train, [-1, SEQLEN])
X_test = np.reshape(X_test, [-1, SEQLEN])
Y_test = np.reshape(Y_test, [-1, SEQLEN])
def train_dataset():
dataset = tf.data.Dataset.from_tensor_slices((X_train, Y_train))
dataset = dataset.repeat()
dataset = dataset.shuffle(DATA_SEQ_LEN * 4 // SEQLEN)
dataset = dataset.batch(BATCHSIZE)
samples, labels = dataset.make_one_shot_iterator().get_next()
return samples, labels
def eval_dataset():
evaldataset = tf.data.Dataset.from_tensor_slices((X_test, Y_test))
evaldataset = evaldataset.repeat(1)
evaldataset = evaldataset.batch(BATCHSIZE)
samples, labels = evaldataset.make_one_shot_iterator().get_next()
return samples, labels
Create RNN Model
There is a full API for working with RNN in TensorFlow. It starts with a GRU cell. This defines all the weights and biases. The only parameter you need to specify is this internal size of the vectors in this cell.
RNN_CELLSIZE = 80
N_LAYERS = 2
DROPOUT_PKEEP = 0.7
def model_rnn_fn(features, labels, mode):
X = tf.expand_dims(features, axis=2)
batchsize = tf.shape(X)[0]
seqlen = tf.shape(X)[1]
cells = [tf.nn.rnn_cell.GRUCell(RNN_CELLSIZE) for _ in range(N_LAYERS)]
cells[:-1] = [tf.nn.rnn_cell.DropoutWrapper(cell, output_keep_prob=DROPOUT_PKEEP) for cell in cells[:-1]]
...
Deep NN
If you want to deep neural network. Let’s stack it, there is a function for that it’s called MultiRNNCell you pass it a cell and how many times you want this cell and it creates a new cell which is a stacked cell.
The last thing we need to do is to unroll it. In TensorFlow you do this using the dynamic RNN function you give it a stacked cell that you just produced. How many times do you unroll? That will depend on the shape of X. X here my input is a sequence of the number. If I feed it with sequences of 16 numbers my network will be unrolled 16 times. what I get as an output are results? It’s all the ages on the bottom.
.....
# a stacked RNN cell still works like an RNN cell
cell = tf.nn.rnn_cell.MultiRNNCell(cells, state_is_tuple=False)
# X[BATCHSIZE, SEQLEN, 1], Hin[BATCHSIZE, RNN_CELLSIZE*N_LAYERS]
Yn, H = tf.nn.dynamic_rnn(cell, X, dtype=tf.float32)
Yn = tf.reshape(Yn, [batchsize * seqlen, RNN_CELLSIZE])
Yr = tf.layers.dense(Yn, 1) # Yr [BATCHSIZE*SEQLEN, 1]
Yr = tf.reshape(Yr, [batchsize, seqlen, 1]) # Yr [BATCHSIZE, SEQLEN, 1]
Yout = Yr[:, -1, :] # Last output Yout [BATCHSIZE, 1]
loss = train_op = None
if mode != tf.estimator.ModeKeys.PREDICT:
labels = tf.expand_dims(labels, axis=2)
loss = tf.losses.mean_squared_error(Yr, labels) # la bels[BATCHSIZE, SEQLEN, 1]
lr = 0.001
optimizer = tf.train.AdamOptimizer(learning_rate=lr)
train_op = tf.contrib.training.create_train_op(loss, optimizer)
return tf.estimator.EstimatorSpec(
mode=mode,
predictions={"Yout": Yout},
loss=loss,
train_op=train_op
)
I will need to apply an activation layer on those to obtain my result sequence and I get also the last state H which is the state that will need to pass back in the beginning as I continue my training.
Create the Estimator
Estimator is a TensorFlow class for performing high-level model training, evaluation, and inference for our model.
training_config = tf.estimator.RunConfig(model_dir="./outputdir") estimator = tf.estimator.Estimator(model_fn=model_rnn_fn, config=training_config)
The model_fn argument specifies the model function to use for training, evaluation, and prediction; we pass it the model_rnn_fn. The model_dir argument specifies the directory where model data and checkpoints will be saved.
Train RNN Model
Now we’re ready to train our model, which we can do by calling train() on estimator.
estimator.train(input_fn=train_dataset,steps=2000)
Predictions (inferring) from Trained Model
We now have a trained model. We can now use the trained model to predict time series data based on some unlabeled data. As with training, we make predictions using a single function call.
results = estimator.predict(eval_dataset) Yout_ = [result["Yout"] for result in results] actual = Y_test[:, -1] colors = plt.rcParams['axes.prop_cycle'].by_key()['color'] plt.plot(actual, label="Actual Values", color='green') plt.plot(Yout_, label="Predicted Values", color='red', ) plt.show()
