Torch: NN handling text and numeric input - neural-network

Question

I have the following NN architecture:
Part 1:
nn.Sequential {
[input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> output]
(1): nn.TemporalConvolution
(2): nn.TemporalMaxPooling
(3): nn.TemporalConvolution
(4): nn.TemporalMaxPooling
(5): nn.Reshape(14336)
(6): nn.Dropout(0.500000)
(7): nn.Linear(14336 -> 128)
}
Part 2:
nn.Sequential {
[input -> (1) -> (2) -> (3) -> output]
(1): nn.Linear(4 -> 8)
(2): nn.ReLU
(3): nn.Linear(8 -> 4)
}
What i would like to do is to use the output of these two parts as input to another part:
nn.Sequential {
[input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> output]
(1): nn.Linear(132 -> 32)
(2): nn.ReLU
(3): nn.Linear(32 -> 32)
(4): nn.ReLU
(5): nn.Linear(32 -> 2)
(6): nn.LogSoftMax
}
Notice how part 1 has 128 outputs and part 2 has 4 and finally part 3 has 132 inputs. So basically what i want is a network that takes two types of input (part 1 is for text, part 2 for a number vector) and to use both these information in the third layer for a 2 class classification.
I have looked at the various containers but nothing seems like it is what i need. Specifically i have looked at nn.Parallel but from the docs it looks like it does something completely different (same input for two different modules). The first problem is what should the input look like for the network (since each part takes a different type of Tensor, i thought that a simple table (array) would be fine, with its first element being a 2D tensor and second a 1D tensor) and how to plug its outputs into another network so i can use forward/backward calls as usual.
Is there any way how to do this?
Thanks!

Answer 1

What you need is nn.ParallelTable and nn.JoinTable.
local parallel = nn.ParallelTable()
parallel:add(part1)
parallel:add(part2)
local net = nn.Sequential()
net:add(parallel) -- (A)
net:add(nn.JoinTable(1)) -- (B)
net:add(part3) -- (C)
(A):
parallel will take a table of 2 tensors (in your case, text and numbers), forward first tensor to part1, second tensor to part2 and output both results in another table of 2 tensors.
(B):
The following nn.JoinTable takes this table as input and concatenates the 2 tensors in a single one. You may have to play with the parameter handling the concatenation dimension (1 in my example) depending on the shape of your tensors.
(C):
Finally you can add the third part of your network taking the concatenated tensor as input.