Training of FCN matlab not decreasing loss - matlab

Question

I have a Neural Network based on Fully Convolutional Neural Network for semantic segmentation / pixelwise object detection ( FCN) and try to train it using Pascal-Context data. However, instead of 450 classes provided by the dataset, I only want to capture 12 + background.
I prepared a dataset with 7000+ images and label maps, and followed the tutorial on Semantic Segmentation ( Semantic Segmentation), and followed every step similarly to the one presented.
However, when I train my neural network my Mini-batch loss remains constant (checked until end of first 3 epochs, then tried to reduce the amount of images, but still the same value) and doesn't change, even if I change learning rate or other parameters. Is there something wrong with the training process?
So I tried to reduce the number of images to 400, and what I am getting is in picture below. It doesn't improve at all.
( First column represents entry number, second Iteration Number, Third Time Elapsed, Fourth Mini-batch loss, Fifth Accuracy on batch, and finally learning rate).
Below is my code:
% MAIN FUNCTION %
outputFolder= fullfile('imade');
imgDir = fullfile('imade','images');
labelDir = fullfile('imade','labels');
imds = imageDatastore(imgDir);
I = readimage(imds, 1);
% I = histeq(I);
% figure
% imshow(I)
classes = [
"bottle"
"chair"
"diningtable"
"person"
"pottedplant"
"sofa"
"tvmonitor"
"ground"
"wall"
"floor"
"keyboard"
"ceiling"
"background"
];
valueSet = {
[230 25 75]
[60 180 75]
[255 225 25]
[0 130 200]
[245 130 48]
[145 30 180]
[128 128 0]
[210 245 60]
[250 190 190]
[0 128 128]
[170 110 40]
[128 0 0]
[0 0 0]
};
pxds = pixelLabelDatastore(labelDir,classes,valueSet);
% show sample image with overlay
C = readimage(pxds, 1);
cmap = camvidColorMap;
B = labeloverlay(I,C,'ColorMap',cmap);
figure
imshow(B)
pixelLabelColorbar(cmap,classes)
% calculate frequency of class pixels
tbl = countEachLabel(pxds);
% resize images and labels to desired format
imageFolder = fullfile(outputFolder,'imagesReszed',filesep);
imds = resizeCamVidImages(imds,imageFolder);
labelFolder = fullfile(outputFolder,'labelsResized',filesep);
pxds = resizeCamVidPixelLabels(pxds,labelFolder);
% partition dataset into test and train
[imdsTrain, imdsTest, pxdsTrain, pxdsTest] = partitionCamVidData(imds,pxds);
% create FCN net
imageSize = [225 300];
numClasses = numel(classes);
lgraph = fcnLayers(imageSize,numClasses,'type','16s');
st = fullfile('imade','checkPoint');
% adjust based on occurrence of each class
imageFreq = tbl.PixelCount ./ tbl.ImagePixelCount;
classWeights = median(imageFreq) ./ imageFreq;
pxLayer = pixelClassificationLayer('Name','labels','ClassNames', tbl.Name, 'ClassWeights', classWeights);
lgraph = removeLayers(lgraph, 'pixelLabels');
lgraph = addLayers(lgraph, pxLayer);
lgraph = connectLayers(lgraph, 'softmax' ,'labels');
% set training parameters
options = trainingOptions('sgdm', ...
'Momentum', 0.9, ...
'InitialLearnRate', 1e-3, ...
'L2Regularization', 0.0005, ...
'MaxEpochs', 100, ...
'MiniBatchSize', 1, ...
'Shuffle', 'every-epoch', ...
'VerboseFrequency', 2);
datasource = pixelLabelImageSource(imdsTrain,pxdsTrain);
doTraining = true;
if doTraining
[net, info] = trainNetwork(datasource,lgraph,options);
else
data = load (strcat(st,filesep,'convnet_checkpoint__4607__2018_01_27__21_25_03.mat'));
war = data.net;
[sp, info] = trainNetwork(datasource,war.Layers,options);
end