load on GPU gradually decreases using Tensorflow's queue - queue

Question

I've a large number of pickled data stored on my hard disk. I've a generator function that reads these pickled files in batch (batch_size = 512) and I'm using tensorflow's queue to speed-up the process. currently, my queue_size is 4096 and I'm using 6 threads as I've 6 physical cores. When I run the code and monitor my GPU load (I'm using TitanX), at the beginning, it looks OK:
But over time, I see less load on my GPU:
I also see increase in execution time per epoch:
Epoch 1 | Exec. time: 1646.523872
Epoch 2 | Exec. time: 1760.770192
Epoch 3 | Exec. time: 1861.450039
Epoch 4 | Exec. time: 1952.52812
Epoch 5 | Exec. time: 2167.598431
Epoch 6 | Exec. time: 2278.203603
Epoch 7 | Exec. time: 2320.280606
Epoch 8 | Exec. time: 2467.036160
Epoch 9 | Exec. time: 2584.932837
Epoch 10 | Exec. time: 2736.121618
...
Epoch 20 | Exec. time: 3841.635191
which the GPU load that I observe kind of explains it.
Now, the question is why is this happening? Is this a bug in tensorflow's queue? Have I done something wrong?! I'm using tensorflow ver. 1.4 and if it helps this is the way I defined my queue, enqueue and dequeue:
def get_train_queue(batch_size, data_generator, queue_size, num_threads):
# get train queue to parallelize loading data
q = tf.FIFOQueue(capacity = queue_size, dtypes = [tf.float32, tf.float32, tf.float32, tf.float32],
shapes = [[batch_size, x_height, x_width, num_channels],
[batch_size, num_classes],
[batch_size, latent_size],
[batch_size]])
batch = next(data_generator)
batch_z = np.random.uniform(-1.0, 1.0, size = (batch_size, latent_size))
mask = get_labled_mask(labeled_rate, batch_size)
enqueue_op = q.enqueue((batch[0], batch[1], batch_z, mask))
qr = tf.train.QueueRunner(q, [enqueue_op] * num_threads)
tf.train.add_queue_runner(qr)
return q
and
def train_per_batch(sess, q, train_samples_count, batch_size, parameters, epoch):
# train_per_batch and get train loss and accuracy
t_total = 0
for iteration in range(int(train_samples_count / batch_size)):
t_start = time.time()
data = q.dequeue()
feed_dictionary = {parameters['x']: sess.run(data[0]),
parameters['z']: sess.run(data[2]),
parameters['label']: sess.run(data[1]),
parameters['labeled_mask']: sess.run(data[3]),
parameters['dropout_rate']: dropout,
parameters['d_init_learning_rate']: D_init_learning_rate,
parameters['g_init_learning_rate']: G_init_learning_rate,
parameters['is_training']: True}
sess.run(parameters['D_optimizer'], feed_dict = feed_dictionary)
sess.run(parameters['G_optimizer'], feed_dict = feed_dictionary)
train_D_loss = sess.run(parameters['D_L'], feed_dict = feed_dictionary)
train_G_loss = sess.run(parameters['G_L'], feed_dict = feed_dictionary)
t_total += (time.time() - t_start)
I also tried tf.data.Dataset.from_generator() recommended by tensorflow 1.4 as:
train_dataset = tf.data.Dataset.from_generator(data_generator_training_from_pickles,
(tf.float32, tf.float32, tf.float32, tf.float32, tf.float32, tf.float32),
([batch_size, x_height, x_width, num_channels],
[batch_size, num_classes],
[batch_size, latent_size],
[batch_size])))
and then using:
def train_per_batch(sess, train_dataset, train_samples_count, batch_size, parameters, epoch):
# train_per_batch and get train loss and accuracy
t_total = 0
for iteration in range(int(train_samples_count / batch_size)):
t_start = time.time()
data = train_dataset.make_one_shot_iterator().get_next()
feed_dictionary = {parameters['x']: sess.run(data[0]),
parameters['z']: sess.run(data[2]),
parameters['label']: sess.run(data[1]),
parameters['labeled_mask']: sess.run(data[3]),
parameters['dropout_rate']: dropout,
parameters['d_init_learning_rate']: D_init_learning_rate,
parameters['g_init_learning_rate']: G_init_learning_rate,
parameters['is_training']: True}
sess.run(parameters['D_optimizer'], feed_dict = feed_dictionary)
sess.run(parameters['G_optimizer'], feed_dict = feed_dictionary)
train_D_loss = sess.run(parameters['D_L'], feed_dict = feed_dictionary)
train_G_loss = sess.run(parameters['G_L'], feed_dict = feed_dictionary)
t_total += (time.time() - t_start)
It's the worst. There is no queueing obviously: