Why YoloV3 has very poor detection rate in testing? - neural-network

Question

I trained YoloV3 for object detection.
I trained to reach Avg Recall: almost 1.0 for all images in the batch and Avg IOU is above 0.9.
But when I go for testing with images (but they are very similar to those trained images) not included in training, the detection threshold needs to be set to 0.1.
But testing with those images used in the training, their detection rate threshold can be set high like 0.7.
What could be wrong or what do I need to change in training?
My cfg for training is
[net]
# Testing
#batch=1
#subdivisions=1
# Training
batch=16
subdivisions=8
height=416
width=416
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1
learning_rate=0.001
burn_in=1000
max_batches = 20000
policy=steps
steps=5000,10000
scales=.1,.1
[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky
[maxpool]
size=2
stride=2
[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky
[maxpool]
size=2
stride=2
[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky
[maxpool]
size=2
stride=2
[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky
[maxpool]
size=2
stride=2
[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky
[maxpool]
size=2
stride=2
[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky
#######
[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=1024
activation=leaky
[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=1024
activation=leaky
[route]
layers=-9
[convolutional]
batch_normalize=1
size=1
stride=1
pad=1
filters=64
activation=leaky
[reorg]
stride=2
[route]
layers=-1,-4
[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=1024
activation=leaky
[convolutional]
size=1
stride=1
pad=1
filters=30
activation=linear
[region]
anchors = 2.00,1.24, 2.75,0.93, 2.84,1.75, 3.72,1.21, 4.63,1.59
bias_match=1
classes=1
coords=4
num=5
softmax=1
jitter=.3
rescore=1
object_scale=5
noobject_scale=1
class_scale=1
coord_scale=1
absolute=1
thresh = .6
random=1

Answer 1

I've encountered such a problem when used Yolo for non square images. Is it you case?
In train phase, Yolo resizes imaged to requires size (namely 416x416 in your net): https://github.com/pjreddie/darknet/blob/master/src/data.c#L971
But in test phase it resizes image preserving aspect ratio and add required borderd to image: https://github.com/pjreddie/darknet/blob/master/examples/detector.c#L587 (alternative code lines with dumb resize are commented below). Naturally, if input images are not square, they appeares to be quite different then in trainig :)
I use https://github.com/AlexeyAB/darknet fork and this issue was fixed there quite recently (https://github.com/AlexeyAB/darknet/commit/efaf684cb20b996db0cd6d99d20fabb4bc26859a). But it seems, original Yolo still has it.
PS. While Yolo is a great tool, its code has a lot of thing like this and it is a good idea to look at source code and debugger when using it :)

Answer 2

uncomment batch = 1 and subdivisions=1 and comment two last line:
# Testing
#batch=1
#subdivisions=1
# Training
batch=16
subdivisions=8