LSTM Model in Torch not Learning - neural-network

Question

I have a toy dataset with one input and one output, the inputs are random 0s or 1s and the output equals the input from 3 samples previous
Input, Output
0,1
1,1
0,0
1,0
0,1
0,0
Model 1:
lstm = nn.Sequential()
lstm:add(nn.LSTM(1,20,8))
lstm:add(nn.Linear(20,100))
lstm:add(nn.Sigmoid())
lstm:add(nn.Linear(100,1))
lstm:add(nn.Sigmoid())
Model 2:
lstm = nn.Sequencer(
nn.Sequential()
:add(nn.LSTM(1,100))
:add(nn.Linear(100,1))
:add(nn.Sigmoid())
)
Criterion:
criterion = nn.BCECriterion()
trainer = nn.StochasticGradient(lstm, criterion)
trainer.learningRate = .01
trainer.maxIteration = 25
The first model doesn't learn and the second doesn't work (possible because my data is the wrong kind for a Sequencer?)
edit:
Training:
lstm:training()
print("Training")
for epoch=1, 5 do
err = 0
for i = 1,data:size() do
input = {}
output = {}
table.insert(input, data[i][1])
table.insert(output, data[i][2])
--input = data[i][1]
--output = data[i][2]
err = err + criterion:forward(lstm:updateOutput(input)[1], output)
lstm:zeroGradParameters()
lstm:backward(input, {criterion:backward(lstm.output[1], output)})
lstm.modules[1]:backwardThroughTime()
lstm:updateParameters(0.01)
lstm.modules[1]:forget()
end
print (err / data:size())
end
I now get an error when running Model 2 through training. The error
AbstractRecurrent.lua:61: Missing gradInput
occurs on line
lstm.modules[1]:backwardThroughTime()

Answer 1

look at the toy example for using LSTM with Sequencer. The model doesn't do anything except try to predict random numbers.
require 'nn'
require 'torch'
require 'rnn'
lstm = nn.Sequencer(
nn.Sequential()
:add(nn.LSTM(1,100))
:add(nn.Linear(100,1))
:add(nn.Sigmoid())
)
-- Encapsulate thr criterion using a Sequencer
-- Just provide the entire sequence as input and the corresponding
-- target sequence as expected output
criterion = nn.SequencerCriterion(nn.BCECriterion())
data = torch.zeros(10,2)
for i=1,data:size(1) do
data[i][1] = torch.uniform()
data[i][2] = torch.uniform()
end
local inputs, targets = {},{}
for epoch=1,5 do
lstm:training()
for i=1,data:size(1) do
--useful for minibatch
inputs[1] = torch.zeros(1)
inputs[1][1] = data[i][1]
targets[1] = torch.zeros(1)
targets[1][1] = data[i][2]
local output = lstm:forward(inputs)
local err = criterion:forward(output, targets)
local gradOutputs = criterion:backward(output, targets)
-- Sequencer handles the backwardThroughTime internally
lstm:backward(inputs, gradOutputs)
lstm:updateParameters(0.01)
lstm:zeroGradParameters()
inputs = {}
targets = {}
end
end