I running the python code in the Google Colab. The code is related to execution a Spiking Neural Networks.
When I plotting neuron's activations, there is a runtime error: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
notice: This function is derived from:
https:https://github.com/guillaume-chevalier/Spiking-Neural-Network-SNN-with-PyTorch-where-Backpropagation-engenders-STDP/blob/master/Spiking%20Neural%20Networks%20with%20PyTorch.ipynb
plotting neuron's activations:
spiking_model.visualize_all_neurons(x)>
def visualize_all_neurons(self, x):
assert x.shape[0] == 1 and len(x.shape) == 4,("Pass only 1 example to SpikingNet.visualize(x) with outer dimension shape of 1.")
_, layers_state = self.forward_through_time(x)
for i, (all_layer_states, all_layer_outputs) in enumerate(layers_state):
layer_state = torch.stack(all_layer_states).data.cpu().numpy().squeeze().transpose()
layer_output = torch.stack(all_layer_outputs).data.cpu().numpy().squeeze().transpose()
self.plot_layer(layer_state, title="Inner state values of neurons for layer {}".format(i))
self.plot_layer(layer_output, title="Output spikes (activation) values of neurons for layer {}".format(i))
enter image description here
Related
How can I change this train argument(older version code) and use this in trainer extensions. What are the necessary changes to be made to use this code in Chainer: 5.4.0.
ValueError: train argument is not supported anymore. Use
chainer.using_config
[AutoEncoder/StackedAutoEncoder/Regression.py](https://github.com/quolc/chainer-ML-examples/blob/master/mnist-stacked-autoencoder/net.py)
[Train.py](https://github.com/quolc/chainer-ML-examples/blob/master/mnist-stacked-autoencoder/train_mnist_sae.py)
for epoch in range(0, n_epoch):
print(' epoch {}'.format(epoch+1))
perm = np.random.permutation(N)
permed_data = np.array(input_data[perm])
sum_loss = 0
start = time.time()
for i in range(0, N, batchsize):
x = chainer.Variable(permed_data[i:i+batchsize])
y = chainer.Variable(permed_data[i:i+batchsize])
optimizer.update(model, x, y)
sum_loss += float(model.loss.data) * len(y.data)
end = time.time()
throughput = N / (end - start)
print(' train mean loss={}, throughput={} data/sec'.format(sum_loss
/ N, throughput))
sys.stdout.flush()
# prepare train data for next layer
x = chainer.Variable(np.array(train_data))
train_data_for_next_layer = cuda.to_cpu(ae.encode(x, train=False).data)
In errors it points out to two different sections:
1. optimizer.update(model, x, y)
2. prepare train data for next layer second line where they mismatch the number of nodes in each layer. The error code is given below.
InvalidType:
Invalid operation is performed in: LinearFunction (Forward)
Expect: prod(in_types[0].shape[1:]) == in_types[1].shape[1]
Actual: 784 != 250
As to train argument, the details are written here: https://docs.chainer.org/en/stable/upgrade_v2.html
train argument is used by dropout in v1, but now Chainer uses config to manage its phase: in training or not.
So, there are two things to do.
First, remove train arguments from scripts.
Second, move inference code in the context.
with chainer.using_config(‘train’, False):
# define the inference process
prepare train data for next layer second line where they mismatch the number of nodes in each layer.
Could you share the error messages?
A Triplet network (inspired by "Siamese network") is comprised of 3 instances of the same feed-forward network (with shared parameters). When fed with 3 samples, the network outputs 2 intermediate values - the L2 (Euclidean) distances between the embedded representation of two of its inputs from
the representation of the third.
I'm using pairs of three images for feeding the network (x = anchor image, a standard image, x+ = positive image, an image containing the same object as x - actually, x+ is same class as x, and x- = negative image, an image with different class than x.
I'm using the triplet loss cost function described here.
How do I determine the network's accuracy?
I am assuming that your are doing work for image retrieval or similar tasks.
You should first generate some triplet, either randomly or using some hard (semi-hard) negative mining method. Then you split your triplet into train and validation set.
If you do it this way, then you can define your validation accuracy as proportion of the number of triplet in which feature distance between anchor and positive is less than that between anchor and negative in your validation triplet. You can see an example here which is written in PyTorch.
As another way, you can directly measure in term of your final testing metric. For example, for image retrieval, typically, we measure the performance of model on test set using mean average precision. If you use this metric, you should first define some queries on your validation set and their corresponding ground truth image.
Either of the above two metric is fine. Choose whatever you think fit your case.
So I am performing a similar task of using Triplet loss for classification. Here is how I used the novel loss method with a classifier.
First, train your model using the standard triplet loss function for N epochs. Once you are sure that the model ( we shall refer to this as the embedding generator) is trained, save the weights as we shall be using these weights ahead.
Let's say that your embedding generator is defined as:
class EmbeddingNetwork(nn.Module):
def __init__(self):
super(EmbeddingNetwork, self).__init__()
self.conv1 = nn.Sequential(
nn.Conv2d(1, 64, (7,7), stride=(2,2), padding=(3,3)),
nn.BatchNorm2d(64),
nn.LeakyReLU(0.001),
nn.MaxPool2d((3, 3), 2, padding=(1,1))
)
self.conv2 = nn.Sequential(
nn.Conv2d(64,64,(1,1), stride=(1,1)),
nn.BatchNorm2d(64),
nn.LeakyReLU(0.001),
nn.Conv2d(64,192, (3,3), stride=(1,1), padding=(1,1)),
nn.BatchNorm2d(192),
nn.LeakyReLU(0.001),
nn.MaxPool2d((3,3),2, padding=(1,1))
)
self.fullyConnected = nn.Sequential(
nn.Linear(7*7*256,32*128),
nn.BatchNorm1d(32*128),
nn.LeakyReLU(0.001),
nn.Linear(32*128,128)
)
def forward(self,x):
x = self.conv1(x)
x = self.conv2(x)
x = self.fullyConnected(x)
return torch.nn.functional.normalize(x, p=2, dim=-1)
Now we shall using this embedding generator to create another classifier, fit the weights we saved before to this part of the network and then freeze this part so our classifier trainer does not interfere with the triplet model. This can be done as:
class classifierNet(nn.Module):
def __init__(self, EmbeddingNet):
super(classifierNet, self).__init__()
self.embeddingLayer = EmbeddingNet
self.classifierLayer = nn.Linear(128,62)
self.dropout = nn.Dropout(0.5)
def forward(self, x):
x = self.dropout(self.embeddingLayer(x))
x = self.classifierLayer(x)
return F.log_softmax(x, dim=1)
Now we shall load the weights we saved before and freeze them using:
embeddingNetwork = EmbeddingNetwork().to(device)
embeddingNetwork.load_state_dict(torch.load('embeddingNetwork.pt'))
classifierNetwork = classifierNet(embeddingNetwork)
Now train this classifier network using the standard classification losses like BinaryCrossEntropy or CrossEntropy.
I'm trying to write an XOR solution using neural networks and the sigmoid activation function. (With True=0.9 and False=0.1)
I'm at the backpropagation part now.
The formula I was given for computing weight adjustments is:
delta_weight(l,i,j) = gamma*output(l,i)*error_signal(l,j)
i.e - the weight adjustment for the link between layer 1 (hidden), node 2 and layer 2(output), node 0 is:
delta_weight(1,2,0)
I chose gamma=0.5
Since bias weights are associated with a single node I guessed the weight adjustment formula was:
delta_weight(l,i) = gamma*output(l,i)
My program is not working, clearly my guess was incorrect. Could someone help me along?
Thanks a bunch!
EDIT: CODE
def applyInputs(self, inps):
for i in range(len(self.layers)-1):
for n, node in enumerate(self.layers[i+1].nodes):
ans = 0
for m, mode in enumerate(self.layers[i].nodes):
ans += self.links[stringify(i,m,i+1,n)].weight * mode.output
if node.bias == True:
ans+= self.links[stringify(-1,-1,i+1,n)].weight
node.set_output(response(ans))
return self.layers[len(self.layers)-1].nodes[0].output
def computeErrorSignals(self, out): # 'out' is the output of the entire network (only 1 output node)
# output node error signal
output_node = self.layers[len(self.layers)-1].nodes[0]
fin_err = (out - output_node.output)*output_node.output*(1-output_node.output)
output_node.set_error(fin_err)
# hidden node error signals
for j in range(len(self.layers[1].nodes)):
hid_node = self.layers[1].nodes[j]
err = (hid_node.output)*(1-hid_node.output)*self.layers[2].nodes[0].error_signal*self.links[stringify(1,j,2,0)].weight
hid_node.set_error(err)
def computeWeightAdjustments(self):
for i in range(len(self.layers)-1):
for n, node in enumerate(self.layers[i+1].nodes):
for m, mode in enumerate(self.layers[i].nodes):
self.links[stringify(i,m,i+1,n)].weight += ((0.5)*self.layers[i+1].nodes[n].error_signal*self.layers[i].nodes[m].output)
if node.bias == True:
self.links[stringify(-1,-1,i+1,n)].weight += ((0.5)*self.layers[i].nodes[m].output)
Keeping all parameters constant, I get different Mean Average Percentage Errors on my test data on retraining the neural network. Why is this so? Aren't all components of the neural network training process deterministic? Sometimes, I see a difference of up to 1% on successive trainings.
The training code is below
netFeb = newfit(trainX', trainY', networkConfigFeb);
netFeb.performFcn = 'mae';
netFeb = trainlm(netFeb, trainX', trainY');
% Index for the testing Data
startingInd = find(trainInd == 0, 1, 'first');
endingInd = startingInd + daysInMonth('Feb') - 1 ;
% Testing Data
testX = X(startingInd:endingInd,:);
testY = dailyPeakLoad(startingInd:endingInd,:);
actualLoadFeb = testY;
% Calculate the Forcast Load and the Mean Absolute Percentage Error
forecastLoadFeb = sim(netFeb, testX'';
errFeb = testY - forecastLoadFeb;
errpct = abs(errFeb)./testY*100;
MAPEFeb = mean(errpct(~isinf(errpct)));
As A. Donda hinted, since neural networks initialize their weights randomly, they will generate different networks after training. Thus it will give you different performance. While the training process is deterministic, the initial values are not! You may end up in different local minimums as a result or stop in different places.
If you wish to see why, take a look at Why should weights of Neural Networks be initialized to random numbers?
Edit 1:
Notes
Since the user is defining the testing/training data manually, there is no randomization of the training data sets selected
I create and then train a network with following lines:
% here input is 9x543, target is 2x543 and hidden is 6 and output has 2 neurons
net = newff(input,target,{hidden},{'logsig','logsig'},'trainlm');
[net,tr] = train(net,input,target);
After training phase, I simulate my network with following line:
out1 = sim(net,input);
Then I get weights of network with following lines:
iwNet = net.IW{1,1};
lwNet = net.LW{2,1};
b1Net = net.b{1,1};
b2Net = net.b{2,1};
I implement my own network to get predicted output of input values:
% here input is 543x9 and target is 543x2
out2=logsig(logsig(input*iw'+repmat(b1',size(input,1),1))*lw'+repmat(b2',size(input,1),1));
I was expecting to be out1==out2, since I am using same activation functions, weights, and bias. But out2 has very interesting results. (still 1st column is 1 and second column of output is zero)
Did I miss something?
The problem might be pre-processing functions. Check if you included any:
net.inputs{1}.processFcns