I am new in cnn, and I wanted to know how may I improve my model? Augmentation is already done. Thanks in advance.
model = Sequential()
model.add(Conv2D(16, (3,3), activation='relu', strides=(1,1),
padding='same', input_shape=input_shape))
model.add(Conv2D(32, (3,3), activation='relu', strides=(1,1),
padding='same'))
model.add(Conv2D(64, (3,3), activation='relu', strides=(1,1),
padding='same'))
#model.add(Conv2D(128, (3,3), activation='relu', strides=(1,1),
# padding='same'))
#model.add(MaxPool2D(2,2))#AveragePooling2D
model.add(AveragePooling2D(2,2))#AveragePooling2D
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(10, activation='softmax'))
model.summary()
#opt = keras.optimizers.Adam(learning_rate=0.001)
model.compile(loss='categorical_crossentropy',
optimizer= "Adam",
metrics=\['acc'\] )][1]][1]
history = model.fit(X, y, epochs=150, batch_size=32,
shuffle=True, validation_split=0.1
callbacks = [checkpoint])
Epoch 00140: val_acc did not improve from 0.93082
Epoch 141/150
28620/28620 [==============================] - 37s 1ms/step - loss: 0.1654 - acc: 0.9401 - val_loss: 0.2388 - val_acc: 0.9267
Epoch 00141: val_acc did not improve from 0.93082
Epoch 142/150
28620/28620 [==============================] - 38s 1ms/step - loss: 0.1314 - acc: 0.9516 - val_loss: 0.2728 - val_acc: 0.9091
Epoch 00142: val_acc did not improve from 0.93082
Epoch 143/150
28620/28620 [==============================] - 37s 1ms/step - loss: 0.1425 - acc: 0.9476 - val_loss: 0.2439 - val_acc: 0.9242
Epoch 00143: val_acc did not improve from 0.93082
Epoch 144/150
28620/28620 [==============================] - 37s 1ms/step - loss: 0.1434 - acc: 0.9473 - val_loss: 0.3709 - val_acc: 0.8824
Epoch 00144: val_acc did not improve from 0.93082
Epoch 145/150
28620/28620 [==============================] - 37s 1ms/step - loss: 0.1483 - acc: 0.9468 - val_loss: 0.2544 - val_acc: 0.9208
Epoch 00145: val_acc did not improve from 0.93082
Epoch 146/150
28620/28620 [==============================] - 35s 1ms/step - loss: 0.1366 - acc: 0.9501 - val_loss: 0.2872 - val_acc: 0.9110
Epoch 00146: val_acc did not improve from 0.93082
Epoch 147/150
28620/28620 [==============================] - 36s 1ms/step - loss: 0.1476 - acc: 0.9465 - val_loss: 0.3147 - val_acc: 0.9013
Epoch 00147: val_acc did not improve from 0.93082
Epoch 148/150
28620/28620 [==============================] - 36s 1ms/step - loss: 0.1391 - acc: 0.9486 - val_loss: 0.2838 - val_acc: 0.9069
Epoch 00148: val_acc did not improve from 0.93082
Epoch 149/150
28620/28620 [==============================] - 35s 1ms/step - loss: 0.1392 - acc: 0.9486 - val_loss: 0.2541 - val_acc: 0.9211
Epoch 00149: val_acc did not improve from 0.93082
Epoch 150/150
28620/28620 [==============================] - 37s 1ms/step - loss: 0.1401 - acc: 0.9489 - val_loss: 0.2213 - val_acc: 0.9308
Epoch 00150: val_acc did not improve from 0.93082
I have built a model with LSTM - Linear modules in Pytorch for a classification problem (10 classes). I am training the model and for each epoch I output the loss and accuracy in the training set. The ouput is as follows:
epoch: 0 start!
Loss: 2.301875352859497
Acc: 0.11388888888888889
epoch: 1 start!
Loss: 2.2759320735931396
Acc: 0.29
epoch: 2 start!
Loss: 2.2510263919830322
Acc: 0.4872222222222222
epoch: 3 start!
Loss: 2.225804567337036
Acc: 0.6066666666666667
epoch: 4 start!
Loss: 2.199286699295044
Acc: 0.6511111111111111
epoch: 5 start!
Loss: 2.1704766750335693
Acc: 0.6855555555555556
epoch: 6 start!
Loss: 2.1381614208221436
Acc: 0.7038888888888889
epoch: 7 start!
Loss: 2.1007182598114014
Acc: 0.7194444444444444
epoch: 8 start!
Loss: 2.0557992458343506
Acc: 0.7283333333333334
epoch: 9 start!
Loss: 1.9998993873596191
Acc: 0.7427777777777778
epoch: 10 start!
Loss: 1.9277743101119995
Acc: 0.7527777777777778
epoch: 11 start!
Loss: 1.8325848579406738
Acc: 0.7483333333333333
epoch: 12 start!
Loss: 1.712520718574524
Acc: 0.7077777777777777
epoch: 13 start!
Loss: 1.6056485176086426
Acc: 0.6305555555555555
epoch: 14 start!
Loss: 1.5910680294036865
Acc: 0.4938888888888889
epoch: 15 start!
Loss: 1.6259561777114868
Acc: 0.41555555555555557
epoch: 16 start!
Loss: 1.892195224761963
Acc: 0.3655555555555556
epoch: 17 start!
Loss: 1.4949012994766235
Acc: 0.47944444444444445
epoch: 18 start!
Loss: 1.4332982301712036
Acc: 0.48833333333333334
For loss function I have used nn.CrossEntropyLoss and Adam Optimizer.
Although the loss is constantly decreasing, the accuracy increases until epoch 10 and then begins for some reason to decrease.
Why is this happening ?
Even if my model is overfitting, doesn't that mean that the accuracy should be high ?? (always speaking for accuracy and loss measured on the training set, not the validation set)
Decreasing loss does not mean improving accuracy always.
I will try to address this for the cross-entropy loss.
CE-loss= sum (-log p(y=i))
Note that loss will decrease if the probability of correct class increases and loss increases if the probability of correct class decreases. Now, when you compute average loss, you are averaging over all the samples, some of the probabilities may increase and some of them can decrease, making overall loss smaller but also accuracy drops.
I am new to PyTorch, currently working on a Transfer Learning simple code. When I am training my model, I am getting a big variance between increase and decrease of the accuracy and loss. I trained the network for 50 epochs, and below is the result:
Epoch [1/50], Loss: 0.5477, Train Accuracy: 63%
Epoch [2/50], Loss: 2.1935, Train Accuracy: 75%
Epoch [3/50], Loss: 1.8811, Train Accuracy: 79%
Epoch [4/50], Loss: 0.0671, Train Accuracy: 77%
Epoch [5/50], Loss: 0.2522, Train Accuracy: 80%
Epoch [6/50], Loss: 0.0962, Train Accuracy: 88%
Epoch [7/50], Loss: 1.8883, Train Accuracy: 74%
Epoch [8/50], Loss: 0.3565, Train Accuracy: 83%
Epoch [9/50], Loss: 0.0228, Train Accuracy: 81%
Epoch [10/50], Loss: 0.0124, Train Accuracy: 81%
Epoch [11/50], Loss: 0.0252, Train Accuracy: 84%
Epoch [12/50], Loss: 0.5184, Train Accuracy: 81%
Epoch [13/50], Loss: 0.1233, Train Accuracy: 86%
Epoch [14/50], Loss: 0.1704, Train Accuracy: 82%
Epoch [15/50], Loss: 2.3164, Train Accuracy: 79%
Epoch [16/50], Loss: 0.0294, Train Accuracy: 85%
Epoch [17/50], Loss: 0.2860, Train Accuracy: 85%
Epoch [18/50], Loss: 1.5114, Train Accuracy: 81%
Epoch [19/50], Loss: 0.1136, Train Accuracy: 86%
Epoch [20/50], Loss: 0.0062, Train Accuracy: 80%
Epoch [21/50], Loss: 0.0748, Train Accuracy: 84%
Epoch [22/50], Loss: 0.1848, Train Accuracy: 84%
Epoch [23/50], Loss: 0.1693, Train Accuracy: 81%
Epoch [24/50], Loss: 0.1297, Train Accuracy: 77%
Epoch [25/50], Loss: 0.1358, Train Accuracy: 78%
Epoch [26/50], Loss: 2.3172, Train Accuracy: 75%
Epoch [27/50], Loss: 0.1772, Train Accuracy: 79%
Epoch [28/50], Loss: 0.0201, Train Accuracy: 80%
Epoch [29/50], Loss: 0.3810, Train Accuracy: 84%
Epoch [30/50], Loss: 0.7281, Train Accuracy: 79%
Epoch [31/50], Loss: 0.1918, Train Accuracy: 81%
Epoch [32/50], Loss: 0.3289, Train Accuracy: 88%
Epoch [33/50], Loss: 1.2363, Train Accuracy: 81%
Epoch [34/50], Loss: 0.0362, Train Accuracy: 89%
Epoch [35/50], Loss: 0.0303, Train Accuracy: 90%
Epoch [36/50], Loss: 1.1700, Train Accuracy: 81%
Epoch [37/50], Loss: 0.0031, Train Accuracy: 81%
Epoch [38/50], Loss: 0.1496, Train Accuracy: 81%
Epoch [39/50], Loss: 0.5070, Train Accuracy: 76%
Epoch [40/50], Loss: 0.1984, Train Accuracy: 77%
Epoch [41/50], Loss: 0.1152, Train Accuracy: 79%
Epoch [42/50], Loss: 0.0603, Train Accuracy: 82%
Epoch [43/50], Loss: 0.2293, Train Accuracy: 84%
Epoch [44/50], Loss: 0.1304, Train Accuracy: 80%
Epoch [45/50], Loss: 0.0381, Train Accuracy: 82%
Epoch [46/50], Loss: 0.1833, Train Accuracy: 84%
Epoch [47/50], Loss: 0.0222, Train Accuracy: 84%
Epoch [48/50], Loss: 0.0010, Train Accuracy: 81%
Epoch [49/50], Loss: 1.0852, Train Accuracy: 79%
Epoch [50/50], Loss: 0.0167, Train Accuracy: 83%
There are some epochs that have a much better accuracy and loss than others. However, the model loses them in later epochs. As I know, the accuracy should improve every epoch. Did I write the training code wrongly? If not, then is that normal? Any way to solve it? Shall the previous accuracy be saved and only if the accuracy of the next epoch is greater than the previous one then train one more epoch? I have been working on Keras previously, and I haven't experienced that problem. I am fine tuning the resent by freezing previous weights and adding only 2 classes for the final layer. Below is my code:
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model_conv.fc.parameters(), lr=0.001, momentum=0.9)
num_epochs = 50
for epoch in range (num_epochs):
#Reset the correct to 0 after passing through all the dataset
correct = 0
for images,labels in dataloaders['train']:
images = Variable(images)
labels = Variable(labels)
if torch.cuda.is_available():
images = images.cuda()
labels = labels.cuda()
optimizer.zero_grad()
outputs = model_conv(images)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
_, predicted = torch.max(outputs, 1)
correct += (predicted == labels).sum()
train_acc = 100 * correct / dataset_sizes['train']
print ('Epoch [{}/{}], Loss: {:.4f}, Train Accuracy: {}%'
.format(epoch+1, num_epochs, loss.item(), train_acc))
I would say it depends on dataset and architecture. Hence, fluctuations are normal, but in general loss should improve.It could be a result of noise in the test dataset, i.e. wrongly labeled examples.
If the test accuracy starts to decrease it might be that your network is overfitting.
You might want to stop the learning just before you reach that point or take other steps to counter the overfitting problem.
Is it normal in PyTorch for accuracy to increase and decrease repeatedly
It should always go down compared on the one epoch level.
Compared to the one batch level it may fluctuate, but generally it should get smaller over time since this is the whole point when we minimize the loss we are improving accuracy.
I am training a two layer 1D CNN model using Keras with input and output function as softmax, SGD/adam optimizer but I see that the training accuracy increases while the validation accuracy remains or revolves around the same number.
Any reason why this could be happening ?
74336/74336 [==============================] - 64s 863us/step - loss: 0.3306 - acc: 0.9096 - val_loss: 0.6093 - val_acc: 0.8473
Epoch 2/30
74336/74336 [==============================] - 62s 840us/step - loss: 0.1980 - acc: 0.9464 - val_loss: 0.6631 - val_acc: 0.8235
Epoch 3/30
74336/74336 [==============================] - 63s 849us/step - loss: 0.1678 - acc: 0.9537 - val_loss: 0.5885 - val_acc: 0.8493
Epoch 4/30
74336/74336 [==============================] - 64s 859us/step - loss: 0.1497 - acc: 0.9591 - val_loss: 0.6043 - val_acc: 0.8482
Epoch 5/30
74336/74336 [==============================] - 64s 865us/step - loss: 0.1407 - acc: 0.9611 - val_loss: 0.5922 - val_acc: 0.8454
Epoch 6/30
74336/74336 [==============================] - 67s 900us/step - loss: 0.1292 - acc: 0.9644 - val_loss: 0.6744 - val_acc: 0.8275
Epoch 7/30
74336/74336 [==============================] - 67s 906us/step - loss: 0.1215 - acc: 0.9669 - val_loss: 0.6556 - val_acc: 0.8381
Epoch 8/30
74336/74336 [==============================] - 69s 929us/step - loss: 0.1170 - acc: 0.9676 - val_loss: 0.6184 - val_acc: 0.8510
Epoch 9/30
74336/74336 [==============================] - 70s 943us/step - loss: 0.1120 - acc: 0.9692 - val_loss: 0.6888 - val_acc: 0.8372
Epoch 10/30
74336/74336 [==============================] - 75s 1ms/step - loss: 0.1097 - acc: 0.9696 - val_loss: 0.7360 - val_acc: 0.8438
Epoch 11/30
74336/74336 [==============================] - 96s 1ms/step - loss: 0.1045 - acc: 0.9714 - val_loss: 0.6840 - val_acc: 0.8277
Epoch 12/30
74336/74336 [==============================] - 87s 1ms/step - loss: 0.1024 - acc: 0.9720 - val_loss: 0.6362 - val_acc: 0.8450
Epoch 13/30
74336/74336 [==============================] - 89s 1ms/step - loss: 0.1003 - acc: 0.9722 - val_loss: 0.7576 - val_acc: 0.8129
Epoch 14/30
74336/74336 [==============================] - 94s 1ms/step - loss: 0.0964 - acc: 0.9735 - val_loss: 0.7404 - val_acc: 0.8366
Epoch 15/30
74336/74336 [==============================] - 73s 985us/step - loss: 0.0949 - acc: 0.9739 - val_loss: 0.6569 - val_acc: 0.8369
Epoch 16/30
74336/74336 [==============================] - 74s 998us/step - loss: 0.0918 - acc: 0.9749 - val_loss: 0.7587 - val_acc: 0.8225
Epoch 17/30
74336/74336 [==============================] - 72s 971us/step - loss: 0.0885 - acc: 0.9754 - val_loss: 0.7721 - val_acc: 0.8438
Epoch 18/30
74336/74336 [==============================] - 74s 990us/step - loss: 0.0866 - acc: 0.9761 - val_loss: 0.6615 - val_acc: 0.8338
Epoch 19/30
74336/74336 [==============================] - 76s 1ms/step - loss: 0.0858 - acc: 0.9767 - val_loss: 0.6973 - val_acc: 0.8348
Epoch 20/30
74336/74336 [==============================] - 75s 1ms/step - loss: 0.0835 - acc: 0.9771 - val_loss: 0.6497 - val_acc: 0.8274
Epoch 21/30
74336/74336 [==============================] - 72s 972us/step - loss: 0.0825 - acc: 0.9769 - val_loss: 0.6922 - val_acc: 0.8339
Epoch 22/30
74336/74336 [==============================] - 73s 977us/step - loss: 0.0814 - acc: 0.9776 - val_loss: 0.7579 - val_acc: 0.7976
Epoch 23/30
74336/74336 [==============================] - 73s 978us/step - loss: 0.0801 - acc: 0.9775 - val_loss: 0.7615 - val_acc: 0.8093
Epoch 24/30
74336/74336 [==============================] - 73s 977us/step - loss: 0.0790 - acc: 0.9784 - val_loss: 0.7847 - val_acc: 0.8308
Epoch 25/30
74336/74336 [==============================] - 63s 853us/step - loss: 0.0780 - acc: 0.9780 - val_loss: 0.6529 - val_acc: 0.8472
Epoch 26/30
74336/74336 [==============================] - 61s 818us/step - loss: 0.0750 - acc: 0.9795 - val_loss: 0.6955 - val_acc: 0.8195
Epoch 27/30
74336/74336 [==============================] - 64s 858us/step - loss: 0.0749 - acc: 0.9788 - val_loss: 0.8086 - val_acc: 0.8407
Epoch 28/30
74336/74336 [==============================] - 58s 780us/step - loss: 0.0722 - acc: 0.9800 - val_loss: 0.7252 - val_acc: 0.8318
Epoch 29/30
74336/74336 [==============================] - 55s 742us/step - loss: 0.0720 - acc: 0.9799 - val_loss: 0.7773 - val_acc: 0.8251
Epoch 30/30
74336/74336 [==============================] - 63s 845us/step - loss: 0.0733 - acc: 0.9797 - val_loss: 0.6815 - val_acc: 0.8275
It might be that your model actually converged. Try to decrease your learning rate with reduceLROnplateau
The validation loss decrease while training loss is decreasing. This suggests possible overfitting. A few things to try to reduce overfitting:
add dropout
data augmentation - random shift, crop and rotation should be enough
use smaller model
use L2 weight regularization and weight decay
Im using a neural network implemented with the Keras library and below is the results during training. At the end it prints a test score and a test accuracy. I can't figure out exactly what the score represents, but the accuracy I assume to be the number of predictions that was correct when running the test.
Epoch 1/15 1200/1200 [==============================] - 4s - loss:
0.6815 - acc: 0.5550 - val_loss: 0.6120 - val_acc: 0.7525
Epoch 2/15 1200/1200 [==============================] - 3s - loss:
0.5481 - acc: 0.7250 - val_loss: 0.4645 - val_acc: 0.8025
Epoch 3/15 1200/1200 [==============================] - 3s - loss:
0.5078 - acc: 0.7558 - val_loss: 0.4354 - val_acc: 0.7975
Epoch 4/15 1200/1200 [==============================] - 3s - loss:
0.4603 - acc: 0.7875 - val_loss: 0.3978 - val_acc: 0.8350
Epoch 5/15 1200/1200 [==============================] - 3s - loss:
0.4367 - acc: 0.7992 - val_loss: 0.3809 - val_acc: 0.8300
Epoch 6/15 1200/1200 [==============================] - 3s - loss:
0.4276 - acc: 0.8017 - val_loss: 0.3884 - val_acc: 0.8350
Epoch 7/15 1200/1200 [==============================] - 3s - loss:
0.3975 - acc: 0.8167 - val_loss: 0.3666 - val_acc: 0.8400
Epoch 8/15 1200/1200 [==============================] - 3s - loss:
0.3916 - acc: 0.8183 - val_loss: 0.3753 - val_acc: 0.8450
Epoch 9/15 1200/1200 [==============================] - 3s - loss:
0.3814 - acc: 0.8233 - val_loss: 0.3505 - val_acc: 0.8475
Epoch 10/15 1200/1200 [==============================] - 3s - loss:
0.3842 - acc: 0.8342 - val_loss: 0.3672 - val_acc: 0.8450
Epoch 11/15 1200/1200 [==============================] - 3s - loss:
0.3674 - acc: 0.8375 - val_loss: 0.3383 - val_acc: 0.8525
Epoch 12/15 1200/1200 [==============================] - 3s - loss:
0.3624 - acc: 0.8367 - val_loss: 0.3423 - val_acc: 0.8650
Epoch 13/15 1200/1200 [==============================] - 3s - loss:
0.3497 - acc: 0.8475 - val_loss: 0.3069 - val_acc: 0.8825
Epoch 14/15 1200/1200 [==============================] - 3s - loss:
0.3406 - acc: 0.8500 - val_loss: 0.2993 - val_acc: 0.8775
Epoch 15/15 1200/1200 [==============================] - 3s - loss:
0.3252 - acc: 0.8600 - val_loss: 0.2960 - val_acc: 0.8775
400/400 [==============================] - 0s
Test score: 0.299598811865
Test accuracy: 0.88
Looking at the Keras documentation, I still don't understand what score is. For the evaluate function, it says:
Returns the loss value & metrics values for the model in test mode.
One thing I noticed is that when the test accuracy is lower, the score is higher, and when accuracy is higher, the score is lower.
For reference, the two relevant parts of the code:
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
score, acc = model.evaluate(x_test, y_test,
batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)
Score is the evaluation of the loss function for a given input.
Training a network is finding parameters that minimize a loss function (or cost function).
The cost function here is the binary_crossentropy.
For a target T and a network output O, the binary crossentropy can defined as
f(T,O) = -(T*log(O) + (1-T)*log(1-O) )
So the score you see is the evaluation of that.
If you feed it a batch of inputs it will most likely return the mean loss.
So yeah, if your model has lower loss (at test time), it should often have lower prediction error.
Loss is often used in the training process to find the "best" parameter values for your model (e.g. weights in neural network). It is what you try to optimize in the training by updating weights.
Accuracy is more from an applied perspective. Once you find the optimized parameters above, you use this metrics to evaluate how accurate your model's prediction is compared to the true data.
This answer provides a detailed info:
How to interpret "loss" and "accuracy" for a machine learning model