Keras Training Why Training accuracy Increases while the validation accuracy remains same? - neural-network

I am training a two layer 1D CNN model using Keras with input and output function as softmax, SGD/adam optimizer but I see that the training accuracy increases while the validation accuracy remains or revolves around the same number.
Any reason why this could be happening ?
74336/74336 [==============================] - 64s 863us/step - loss: 0.3306 - acc: 0.9096 - val_loss: 0.6093 - val_acc: 0.8473
Epoch 2/30
74336/74336 [==============================] - 62s 840us/step - loss: 0.1980 - acc: 0.9464 - val_loss: 0.6631 - val_acc: 0.8235
Epoch 3/30
74336/74336 [==============================] - 63s 849us/step - loss: 0.1678 - acc: 0.9537 - val_loss: 0.5885 - val_acc: 0.8493
Epoch 4/30
74336/74336 [==============================] - 64s 859us/step - loss: 0.1497 - acc: 0.9591 - val_loss: 0.6043 - val_acc: 0.8482
Epoch 5/30
74336/74336 [==============================] - 64s 865us/step - loss: 0.1407 - acc: 0.9611 - val_loss: 0.5922 - val_acc: 0.8454
Epoch 6/30
74336/74336 [==============================] - 67s 900us/step - loss: 0.1292 - acc: 0.9644 - val_loss: 0.6744 - val_acc: 0.8275
Epoch 7/30
74336/74336 [==============================] - 67s 906us/step - loss: 0.1215 - acc: 0.9669 - val_loss: 0.6556 - val_acc: 0.8381
Epoch 8/30
74336/74336 [==============================] - 69s 929us/step - loss: 0.1170 - acc: 0.9676 - val_loss: 0.6184 - val_acc: 0.8510
Epoch 9/30
74336/74336 [==============================] - 70s 943us/step - loss: 0.1120 - acc: 0.9692 - val_loss: 0.6888 - val_acc: 0.8372
Epoch 10/30
74336/74336 [==============================] - 75s 1ms/step - loss: 0.1097 - acc: 0.9696 - val_loss: 0.7360 - val_acc: 0.8438
Epoch 11/30
74336/74336 [==============================] - 96s 1ms/step - loss: 0.1045 - acc: 0.9714 - val_loss: 0.6840 - val_acc: 0.8277
Epoch 12/30
74336/74336 [==============================] - 87s 1ms/step - loss: 0.1024 - acc: 0.9720 - val_loss: 0.6362 - val_acc: 0.8450
Epoch 13/30
74336/74336 [==============================] - 89s 1ms/step - loss: 0.1003 - acc: 0.9722 - val_loss: 0.7576 - val_acc: 0.8129
Epoch 14/30
74336/74336 [==============================] - 94s 1ms/step - loss: 0.0964 - acc: 0.9735 - val_loss: 0.7404 - val_acc: 0.8366
Epoch 15/30
74336/74336 [==============================] - 73s 985us/step - loss: 0.0949 - acc: 0.9739 - val_loss: 0.6569 - val_acc: 0.8369
Epoch 16/30
74336/74336 [==============================] - 74s 998us/step - loss: 0.0918 - acc: 0.9749 - val_loss: 0.7587 - val_acc: 0.8225
Epoch 17/30
74336/74336 [==============================] - 72s 971us/step - loss: 0.0885 - acc: 0.9754 - val_loss: 0.7721 - val_acc: 0.8438
Epoch 18/30
74336/74336 [==============================] - 74s 990us/step - loss: 0.0866 - acc: 0.9761 - val_loss: 0.6615 - val_acc: 0.8338
Epoch 19/30
74336/74336 [==============================] - 76s 1ms/step - loss: 0.0858 - acc: 0.9767 - val_loss: 0.6973 - val_acc: 0.8348
Epoch 20/30
74336/74336 [==============================] - 75s 1ms/step - loss: 0.0835 - acc: 0.9771 - val_loss: 0.6497 - val_acc: 0.8274
Epoch 21/30
74336/74336 [==============================] - 72s 972us/step - loss: 0.0825 - acc: 0.9769 - val_loss: 0.6922 - val_acc: 0.8339
Epoch 22/30
74336/74336 [==============================] - 73s 977us/step - loss: 0.0814 - acc: 0.9776 - val_loss: 0.7579 - val_acc: 0.7976
Epoch 23/30
74336/74336 [==============================] - 73s 978us/step - loss: 0.0801 - acc: 0.9775 - val_loss: 0.7615 - val_acc: 0.8093
Epoch 24/30
74336/74336 [==============================] - 73s 977us/step - loss: 0.0790 - acc: 0.9784 - val_loss: 0.7847 - val_acc: 0.8308
Epoch 25/30
74336/74336 [==============================] - 63s 853us/step - loss: 0.0780 - acc: 0.9780 - val_loss: 0.6529 - val_acc: 0.8472
Epoch 26/30
74336/74336 [==============================] - 61s 818us/step - loss: 0.0750 - acc: 0.9795 - val_loss: 0.6955 - val_acc: 0.8195
Epoch 27/30
74336/74336 [==============================] - 64s 858us/step - loss: 0.0749 - acc: 0.9788 - val_loss: 0.8086 - val_acc: 0.8407
Epoch 28/30
74336/74336 [==============================] - 58s 780us/step - loss: 0.0722 - acc: 0.9800 - val_loss: 0.7252 - val_acc: 0.8318
Epoch 29/30
74336/74336 [==============================] - 55s 742us/step - loss: 0.0720 - acc: 0.9799 - val_loss: 0.7773 - val_acc: 0.8251
Epoch 30/30
74336/74336 [==============================] - 63s 845us/step - loss: 0.0733 - acc: 0.9797 - val_loss: 0.6815 - val_acc: 0.8275


It might be that your model actually converged. Try to decrease your learning rate with reduceLROnplateau


The validation loss decrease while training loss is decreasing. This suggests possible overfitting. A few things to try to reduce overfitting:
add dropout
data augmentation - random shift, crop and rotation should be enough
use smaller model
use L2 weight regularization and weight decay

Related

Trouble with setting the objective function in BayesianOptimization tuner in keras_tuner

I'm trying to tune hyperparameters for an LSTM model in Keras using Keras tuner's BayesianOptimization tuner. I keep getting error messages that seem to object to what I put in the objective argument when I instantiate the tuner. Here's my code:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dropout, Dense
from kerastuner.tuners import BayesianOptimization
from kerastuner.engine.hyperparameters import HyperParameters
from kerastuner import Objective
def build_model(hp):
model = Sequential()
model.add(LSTM(hp.Int("input_unit", min_value=20, max_value=100, step=20),
batch_input_shape=(149,1,1), stateful=True, return_sequences=True))
for i in range(hp.Int("n_layers", 1, 2)):
model.add(LSTM(hp.Int(f"lstm_{i}_units", min_value=20, max_value=100, step=20),
stateful=True, return_sequences=True))
model.add(Dropout(hp.Float("Dropout_rate", min_value=0, max_value=0.5, step=0.1)))
model.add(Dense(1))
model.compile(loss="mean_squared_error",
optimizer=hp.Choice("optimizer", values=["rmsprop", "adam"]),
metrics=[keras.metrics.RootMeanSquaredError(name="rmse")])
return model
tuner = BayesianOptimization(build_model,
objective=Objective("root_mean_squared_error", direction="min"),
max_trials=5,
executions_per_trial=1)
tuner.search(X_train, y_train, epochs=30, batch_size=149, shuffle=False, validation_data=(X_test, y_test))
Below is the message I get back. As you can see, it appears that the code executes at least partially, since I'm getting a message back about what hyperparameters perform best so far, and I get the summary of the epochs. After that, a long line of error messages with the last one seeming to be related to the objective argument. Is the issue with how I am specifying the objective function when I instantiate the tuner, or is there some other issue?
INFO:tensorflow:Reloading Oracle from existing project ./untitled_project/oracle.json
INFO:tensorflow:Reloading Tuner from ./untitled_project/tuner0.json
Search: Running Trial #3
Value |Best Value So Far |Hyperparameter
20 |40 |input_layer
1 |0 |middle_layers
0.1 |0.2 |dropout_rate
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-1.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-1.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.momentum
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.rho
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'rms' for (root).layer_with_weights-1.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'rms' for (root).layer_with_weights-1.bias
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
Epoch 1/30
29/29 [==============================] - 3s 25ms/step - loss: 0.2129 - rmse: 0.4614 - val_loss: 0.2266 - val_rmse: 0.4761
Epoch 2/30
29/29 [==============================] - 0s 8ms/step - loss: 0.1665 - rmse: 0.4080 - val_loss: 0.2074 - val_rmse: 0.4554
Epoch 3/30
29/29 [==============================] - 0s 7ms/step - loss: 0.1618 - rmse: 0.4022 - val_loss: 0.2002 - val_rmse: 0.4474
Epoch 4/30
29/29 [==============================] - 0s 7ms/step - loss: 0.1575 - rmse: 0.3969 - val_loss: 0.1906 - val_rmse: 0.4366
Epoch 5/30
29/29 [==============================] - 0s 6ms/step - loss: 0.1513 - rmse: 0.3890 - val_loss: 0.1788 - val_rmse: 0.4228
Epoch 6/30
29/29 [==============================] - 0s 6ms/step - loss: 0.1421 - rmse: 0.3770 - val_loss: 0.1638 - val_rmse: 0.4047
Epoch 7/30
29/29 [==============================] - 0s 6ms/step - loss: 0.1283 - rmse: 0.3583 - val_loss: 0.1440 - val_rmse: 0.3794
Epoch 8/30
29/29 [==============================] - 0s 6ms/step - loss: 0.1073 - rmse: 0.3275 - val_loss: 0.1161 - val_rmse: 0.3408
Epoch 9/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0773 - rmse: 0.2781 - val_loss: 0.0807 - val_rmse: 0.2841
Epoch 10/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0509 - rmse: 0.2256 - val_loss: 0.0613 - val_rmse: 0.2475
Epoch 11/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0449 - rmse: 0.2119 - val_loss: 0.0594 - val_rmse: 0.2438
Epoch 12/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0440 - rmse: 0.2098 - val_loss: 0.0587 - val_rmse: 0.2423
Epoch 13/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0434 - rmse: 0.2084 - val_loss: 0.0583 - val_rmse: 0.2414
Epoch 14/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0431 - rmse: 0.2076 - val_loss: 0.0580 - val_rmse: 0.2408
Epoch 15/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0429 - rmse: 0.2070 - val_loss: 0.0578 - val_rmse: 0.2405
Epoch 16/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0427 - rmse: 0.2066 - val_loss: 0.0577 - val_rmse: 0.2402
Epoch 17/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0426 - rmse: 0.2064 - val_loss: 0.0576 - val_rmse: 0.2400
Epoch 18/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0425 - rmse: 0.2062 - val_loss: 0.0575 - val_rmse: 0.2398
Epoch 19/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0424 - rmse: 0.2060 - val_loss: 0.0575 - val_rmse: 0.2397
Epoch 20/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0424 - rmse: 0.2059 - val_loss: 0.0574 - val_rmse: 0.2396
Epoch 21/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0423 - rmse: 0.2058 - val_loss: 0.0573 - val_rmse: 0.2395
Epoch 22/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0423 - rmse: 0.2057 - val_loss: 0.0573 - val_rmse: 0.2394
Epoch 23/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0423 - rmse: 0.2056 - val_loss: 0.0572 - val_rmse: 0.2393
Epoch 24/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0422 - rmse: 0.2055 - val_loss: 0.0572 - val_rmse: 0.2392
Epoch 25/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0422 - rmse: 0.2055 - val_loss: 0.0572 - val_rmse: 0.2391
Epoch 26/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0422 - rmse: 0.2054 - val_loss: 0.0571 - val_rmse: 0.2390
Epoch 27/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0422 - rmse: 0.2054 - val_loss: 0.0571 - val_rmse: 0.2390
Epoch 28/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0422 - rmse: 0.2053 - val_loss: 0.0571 - val_rmse: 0.2389
Epoch 29/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0421 - rmse: 0.2053 - val_loss: 0.0570 - val_rmse: 0.2388
Epoch 30/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0421 - rmse: 0.2052 - val_loss: 0.0570 - val_rmse: 0.2388
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/tmp/ipykernel_371/3389478107.py in <module>
26 executions_per_trial=1)
27
---> 28 tuner.search(X_train, y_train, epochs=30, batch_size=149, shuffle=False, validation_data=(X_test, y_test))
~/comp_4447/dst2_env/lib/python3.9/site-packages/keras_tuner/engine/base_tuner.py in search(self, *fit_args, **fit_kwargs)
177
178 self.on_trial_begin(trial)
--> 179 results = self.run_trial(trial, *fit_args, **fit_kwargs)
180 # `results` is None indicates user updated oracle in `run_trial()`.
181 if results is None:
~/comp_4447/dst2_env/lib/python3.9/site-packages/keras_tuner/engine/tuner.py in run_trial(self, trial, *args, **kwargs)
292 callbacks.append(model_checkpoint)
293 copied_kwargs["callbacks"] = callbacks
--> 294 obj_value = self._build_and_fit_model(trial, *args, **copied_kwargs)
295
296 histories.append(obj_value)
~/comp_4447/dst2_env/lib/python3.9/site-packages/keras_tuner/engine/tuner.py in _build_and_fit_model(self, trial, *args, **kwargs)
221 model = self._try_build(hp)
222 results = self.hypermodel.fit(hp, model, *args, **kwargs)
--> 223 return tuner_utils.convert_to_metrics_dict(
224 results, self.oracle.objective, "HyperModel.fit()"
225 )
~/comp_4447/dst2_env/lib/python3.9/site-packages/keras_tuner/engine/tuner_utils.py in convert_to_metrics_dict(results, objective, func_name)
266 best_epoch = 0
267 for epoch, metrics in epoch_metrics.items():
--> 268 objective_value = objective.get_value(metrics)
269 # Support multi-objective.
270 if objective.name not in metrics:
~/comp_4447/dst2_env/lib/python3.9/site-packages/keras_tuner/engine/objective.py in get_value(self, logs)
53 The objective value.
54 """
---> 55 return logs[self.name]
56
57 def better_than(self, a, b):
KeyError: 'root_mean_squared_error'

What if :
objective=Objective("rmse", direction="min")
I don't know why it's not working. I tried it just now, modifying a notebook of mine to test, and 'rmse' works.
(p.s. I prefer Optuna though, for huge datasets, much faster)
Here are some snippets:
self.compile_metrics = ['AUC', 'binary_accuracy', 'Precision', 'Recall',
tf.keras.metrics.RootMeanSquaredError(
name='rmse', dtype=None
)]
self.model.compile(
optimizer = opt,
#loss = 'binary_crossentropy',
loss = 'mean_squared_error',
metrics = self.compile_metrics,
steps_per_execution = self.steps_per_execution,
)
# above are snippets from my_keras_tuner_class
dl = my_keras_tuner_class()
tuner = kt.BayesianOptimization(
dl.model_builder, # the hypermodel, found in custom class keras_tuning
#objective = 'val_binary_accuracy', # objective to optimize
objective = kt.Objective("rmse", direction="min"),
max_trials = nn_trials, # number of trials
distribution_strategy = tpu_strategy,
# this project name (a dir) is created in 'directory', and is used to
# resume tuning if the search is run again
directory = f'{workdir}',
project_name = 'kbayesian',
# if True, overwrite above directory if search is run again - i.e. don't resume
overwrite = True
Trial 2 Complete [00h 00m 16s]
rmse: 0.42514675855636597
Best rmse So Far: 0.42514675855636597
display(model.metrics_names)
-output:
['loss', 'auc', 'binary_accuracy', 'precision', 'recall', 'rmse']
# example trial
9s 453ms/step - loss: 2.5937 - auc: 0.6732 - binary_accuracy: 0.6353 - precision: 0.6174 - recall: 0.6237 - rmse: 0.5435 - val_loss: 1.1624 - val_auc: 0.7197 - val_binary_accuracy: 0.6970 - val_precision: 0.7083 - val_recall: 0.8500 - val_rmse: 0.5045

changing the loss function leads to neural network returns nans

I'm using the Deep SVDD on CIFAR10 for one-class classification. When I change the L2 norm to Lp for p<1 I got nans after some epochs.
It is working for loss= torch.mean((outputs - inputs)2)
But I got nan for loss= torch.mean((abs(outputs - inputs))(0.9))
The loss for each epoch is shown here:
INFO:root: Epoch 1/50 Time: 1.514 Loss: 84.51767029
INFO:root: Epoch 2/50 Time: 1.617 Loss: 82.70055634
INFO:root: Epoch 3/50 Time: 1.528 Loss: 80.92372467
INFO:root: Epoch 4/50 Time: 1.612 Loss: 79.23560699
INFO:root: Epoch 5/50 Time: 1.495 Loss: 77.56893951
INFO:root: Epoch 6/50 Time: 1.596 Loss: 75.95311737
INFO:root: Epoch 7/50 Time: 1.504 Loss: 74.40722260
INFO:root: Epoch 8/50 Time: 1.593 Loss: 72.84329010
INFO:root: Epoch 9/50 Time: 1.639 Loss: 71.34644287
INFO:root: Epoch 10/50 Time: 1.578 Loss: 69.86484253
INFO:root: Epoch 11/50 Time: 1.553 Loss: 68.41005692
INFO:root: Epoch 12/50 Time: 1.670 Loss: 66.96582977
INFO:root: Epoch 13/50 Time: 1.607 Loss: 65.56927887
INFO:root: Epoch 14/50 Time: 1.573 Loss: 64.20584961
INFO:root: Epoch 15/50 Time: 1.605 Loss: 62.85230591
INFO:root: Epoch 16/50 Time: 1.483 Loss: 61.53305466
INFO:root: Epoch 17/50 Time: 1.616 Loss: 60.22836166
INFO:root: Epoch 18/50 Time: 1.499 Loss: 58.94760498
INFO:root: Epoch 19/50 Time: 1.611 Loss: 57.73990845
INFO:root: Epoch 20/50 Time: 1.507 Loss: 56.51732086
INFO:root: Epoch 21/50 Time: 1.624 Loss: 55.30994400
INFO:root: Epoch 22/50 Time: 1.482 Loss: 54.13251587
INFO:root: Epoch 23/50 Time: 1.606 Loss: 52.98952118
INFO:root: Epoch 24/50 Time: 1.508 Loss: 51.86713654
INFO:root: Epoch 25/50 Time: 1.587 Loss: 50.76639069
INFO:root: Epoch 26/50 Time: 1.523 Loss: 49.68750381
INFO:root: Epoch 27/50 Time: 1.574 Loss: 48.62197098
INFO:root: Epoch 28/50 Time: 1.537 Loss: 47.59307220
INFO:root: Epoch 29/50 Time: 1.560 Loss: 46.58890167
INFO:root: Epoch 30/50 Time: 1.607 Loss: 45.59774643
INFO:root: Epoch 31/50 Time: 1.504 Loss: 44.61755203
INFO:root: Epoch 32/50 Time: 1.592 Loss: 43.67579239
INFO:root: Epoch 33/50 Time: 1.480 Loss: 42.76135941
INFO:root: Epoch 34/50 Time: 1.577 Loss: 41.84933487
INFO:root: Epoch 35/50 Time: 1.488 Loss: 40.96647171
INFO:root: Epoch 36/50 Time: 1.596 Loss: 40.10220779
INFO:root: Epoch 37/50 Time: 1.534 Loss: 39.26658310
INFO:root: Epoch 38/50 Time: 1.615 Loss: 38.44916168
INFO:root: Epoch 39/50 Time: 1.518 Loss: nan
INFO:root: Epoch 40/50 Time: 1.574 Loss: nan
INFO:root: Epoch 41/50 Time: 1.511 Loss: nan
INFO:root: Epoch 42/50 Time: 1.556 Loss: nan
INFO:root: Epoch 43/50 Time: 1.565 Loss: nan
INFO:root: Epoch 44/50 Time: 1.561 Loss: nan
INFO:root: Epoch 45/50 Time: 1.600 Loss: nan
INFO:root: Epoch 46/50 Time: 1.518 Loss: nan
INFO:root: Epoch 47/50 Time: 1.618 Loss: nan
INFO:root: Epoch 48/50 Time: 1.540 Loss: nan
INFO:root: Epoch 49/50 Time: 1.591 Loss: nan
INFO:root: Epoch 50/50 Time: 1.504 Loss: nan
For different learning rates and output dimensions still, the network returns nan after some epochs

What should I consider in my situation to decrease the val_loss?

I am new in cnn, and I wanted to know how may I improve my model? Augmentation is already done. Thanks in advance.
model = Sequential()
model.add(Conv2D(16, (3,3), activation='relu', strides=(1,1),
padding='same', input_shape=input_shape))
model.add(Conv2D(32, (3,3), activation='relu', strides=(1,1),
padding='same'))
model.add(Conv2D(64, (3,3), activation='relu', strides=(1,1),
padding='same'))
#model.add(Conv2D(128, (3,3), activation='relu', strides=(1,1),
# padding='same'))
#model.add(MaxPool2D(2,2))#AveragePooling2D
model.add(AveragePooling2D(2,2))#AveragePooling2D
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(10, activation='softmax'))
model.summary()
#opt = keras.optimizers.Adam(learning_rate=0.001)
model.compile(loss='categorical_crossentropy',
optimizer= "Adam",
metrics=\['acc'\] )][1]][1]
history = model.fit(X, y, epochs=150, batch_size=32,
shuffle=True, validation_split=0.1
callbacks = [checkpoint])
Epoch 00140: val_acc did not improve from 0.93082
Epoch 141/150
28620/28620 [==============================] - 37s 1ms/step - loss: 0.1654 - acc: 0.9401 - val_loss: 0.2388 - val_acc: 0.9267
Epoch 00141: val_acc did not improve from 0.93082
Epoch 142/150
28620/28620 [==============================] - 38s 1ms/step - loss: 0.1314 - acc: 0.9516 - val_loss: 0.2728 - val_acc: 0.9091
Epoch 00142: val_acc did not improve from 0.93082
Epoch 143/150
28620/28620 [==============================] - 37s 1ms/step - loss: 0.1425 - acc: 0.9476 - val_loss: 0.2439 - val_acc: 0.9242
Epoch 00143: val_acc did not improve from 0.93082
Epoch 144/150
28620/28620 [==============================] - 37s 1ms/step - loss: 0.1434 - acc: 0.9473 - val_loss: 0.3709 - val_acc: 0.8824
Epoch 00144: val_acc did not improve from 0.93082
Epoch 145/150
28620/28620 [==============================] - 37s 1ms/step - loss: 0.1483 - acc: 0.9468 - val_loss: 0.2544 - val_acc: 0.9208
Epoch 00145: val_acc did not improve from 0.93082
Epoch 146/150
28620/28620 [==============================] - 35s 1ms/step - loss: 0.1366 - acc: 0.9501 - val_loss: 0.2872 - val_acc: 0.9110
Epoch 00146: val_acc did not improve from 0.93082
Epoch 147/150
28620/28620 [==============================] - 36s 1ms/step - loss: 0.1476 - acc: 0.9465 - val_loss: 0.3147 - val_acc: 0.9013
Epoch 00147: val_acc did not improve from 0.93082
Epoch 148/150
28620/28620 [==============================] - 36s 1ms/step - loss: 0.1391 - acc: 0.9486 - val_loss: 0.2838 - val_acc: 0.9069
Epoch 00148: val_acc did not improve from 0.93082
Epoch 149/150
28620/28620 [==============================] - 35s 1ms/step - loss: 0.1392 - acc: 0.9486 - val_loss: 0.2541 - val_acc: 0.9211
Epoch 00149: val_acc did not improve from 0.93082
Epoch 150/150
28620/28620 [==============================] - 37s 1ms/step - loss: 0.1401 - acc: 0.9489 - val_loss: 0.2213 - val_acc: 0.9308
Epoch 00150: val_acc did not improve from 0.93082

Why does loss decrease but accuracy decreases too (Pytorch, LSTM)?

I have built a model with LSTM - Linear modules in Pytorch for a classification problem (10 classes). I am training the model and for each epoch I output the loss and accuracy in the training set. The ouput is as follows:
epoch: 0 start!
Loss: 2.301875352859497
Acc: 0.11388888888888889
epoch: 1 start!
Loss: 2.2759320735931396
Acc: 0.29
epoch: 2 start!
Loss: 2.2510263919830322
Acc: 0.4872222222222222
epoch: 3 start!
Loss: 2.225804567337036
Acc: 0.6066666666666667
epoch: 4 start!
Loss: 2.199286699295044
Acc: 0.6511111111111111
epoch: 5 start!
Loss: 2.1704766750335693
Acc: 0.6855555555555556
epoch: 6 start!
Loss: 2.1381614208221436
Acc: 0.7038888888888889
epoch: 7 start!
Loss: 2.1007182598114014
Acc: 0.7194444444444444
epoch: 8 start!
Loss: 2.0557992458343506
Acc: 0.7283333333333334
epoch: 9 start!
Loss: 1.9998993873596191
Acc: 0.7427777777777778
epoch: 10 start!
Loss: 1.9277743101119995
Acc: 0.7527777777777778
epoch: 11 start!
Loss: 1.8325848579406738
Acc: 0.7483333333333333
epoch: 12 start!
Loss: 1.712520718574524
Acc: 0.7077777777777777
epoch: 13 start!
Loss: 1.6056485176086426
Acc: 0.6305555555555555
epoch: 14 start!
Loss: 1.5910680294036865
Acc: 0.4938888888888889
epoch: 15 start!
Loss: 1.6259561777114868
Acc: 0.41555555555555557
epoch: 16 start!
Loss: 1.892195224761963
Acc: 0.3655555555555556
epoch: 17 start!
Loss: 1.4949012994766235
Acc: 0.47944444444444445
epoch: 18 start!
Loss: 1.4332982301712036
Acc: 0.48833333333333334
For loss function I have used nn.CrossEntropyLoss and Adam Optimizer.
Although the loss is constantly decreasing, the accuracy increases until epoch 10 and then begins for some reason to decrease.
Why is this happening ?
Even if my model is overfitting, doesn't that mean that the accuracy should be high ?? (always speaking for accuracy and loss measured on the training set, not the validation set)

Decreasing loss does not mean improving accuracy always.
I will try to address this for the cross-entropy loss.
CE-loss= sum (-log p(y=i))
Note that loss will decrease if the probability of correct class increases and loss increases if the probability of correct class decreases. Now, when you compute average loss, you are averaging over all the samples, some of the probabilities may increase and some of them can decrease, making overall loss smaller but also accuracy drops.

Test score vs test accuracy when evaluating model using Keras

Im using a neural network implemented with the Keras library and below is the results during training. At the end it prints a test score and a test accuracy. I can't figure out exactly what the score represents, but the accuracy I assume to be the number of predictions that was correct when running the test.
Epoch 1/15 1200/1200 [==============================] - 4s - loss:
0.6815 - acc: 0.5550 - val_loss: 0.6120 - val_acc: 0.7525
Epoch 2/15 1200/1200 [==============================] - 3s - loss:
0.5481 - acc: 0.7250 - val_loss: 0.4645 - val_acc: 0.8025
Epoch 3/15 1200/1200 [==============================] - 3s - loss:
0.5078 - acc: 0.7558 - val_loss: 0.4354 - val_acc: 0.7975
Epoch 4/15 1200/1200 [==============================] - 3s - loss:
0.4603 - acc: 0.7875 - val_loss: 0.3978 - val_acc: 0.8350
Epoch 5/15 1200/1200 [==============================] - 3s - loss:
0.4367 - acc: 0.7992 - val_loss: 0.3809 - val_acc: 0.8300
Epoch 6/15 1200/1200 [==============================] - 3s - loss:
0.4276 - acc: 0.8017 - val_loss: 0.3884 - val_acc: 0.8350
Epoch 7/15 1200/1200 [==============================] - 3s - loss:
0.3975 - acc: 0.8167 - val_loss: 0.3666 - val_acc: 0.8400
Epoch 8/15 1200/1200 [==============================] - 3s - loss:
0.3916 - acc: 0.8183 - val_loss: 0.3753 - val_acc: 0.8450
Epoch 9/15 1200/1200 [==============================] - 3s - loss:
0.3814 - acc: 0.8233 - val_loss: 0.3505 - val_acc: 0.8475
Epoch 10/15 1200/1200 [==============================] - 3s - loss:
0.3842 - acc: 0.8342 - val_loss: 0.3672 - val_acc: 0.8450
Epoch 11/15 1200/1200 [==============================] - 3s - loss:
0.3674 - acc: 0.8375 - val_loss: 0.3383 - val_acc: 0.8525
Epoch 12/15 1200/1200 [==============================] - 3s - loss:
0.3624 - acc: 0.8367 - val_loss: 0.3423 - val_acc: 0.8650
Epoch 13/15 1200/1200 [==============================] - 3s - loss:
0.3497 - acc: 0.8475 - val_loss: 0.3069 - val_acc: 0.8825
Epoch 14/15 1200/1200 [==============================] - 3s - loss:
0.3406 - acc: 0.8500 - val_loss: 0.2993 - val_acc: 0.8775
Epoch 15/15 1200/1200 [==============================] - 3s - loss:
0.3252 - acc: 0.8600 - val_loss: 0.2960 - val_acc: 0.8775
400/400 [==============================] - 0s
Test score: 0.299598811865
Test accuracy: 0.88
Looking at the Keras documentation, I still don't understand what score is. For the evaluate function, it says:
Returns the loss value & metrics values for the model in test mode.
One thing I noticed is that when the test accuracy is lower, the score is higher, and when accuracy is higher, the score is lower.

For reference, the two relevant parts of the code:
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
score, acc = model.evaluate(x_test, y_test,
batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)
Score is the evaluation of the loss function for a given input.
Training a network is finding parameters that minimize a loss function (or cost function).
The cost function here is the binary_crossentropy.
For a target T and a network output O, the binary crossentropy can defined as
f(T,O) = -(T*log(O) + (1-T)*log(1-O) )
So the score you see is the evaluation of that.
If you feed it a batch of inputs it will most likely return the mean loss.
So yeah, if your model has lower loss (at test time), it should often have lower prediction error.

Loss is often used in the training process to find the "best" parameter values for your model (e.g. weights in neural network). It is what you try to optimize in the training by updating weights.
Accuracy is more from an applied perspective. Once you find the optimized parameters above, you use this metrics to evaluate how accurate your model's prediction is compared to the true data.
This answer provides a detailed info:
How to interpret "loss" and "accuracy" for a machine learning model