How the softmax function and loss function work in multiple input keras model - tf.keras

Question

My model has two inputs branches, input1 2D grayscale images, and input2 color images. The two inputs branches are merged using the concatenate method and classified using a softmax function. The model is working fine but the problem is in understanding the operation of softmax in multiple inputs model and also how weights are updated in both the branches.

Answer 1

The softmax function and loss function perform similar to the single input/output model even in the case of multiple input/output model. If we only passed a single loss function to the model, the same loss function would be applied to every output, unless you specify different loss function and different activation function for multiple outputs.
Consider the following model, which has an image input of shape (32, 32, 3) (that's (height, width, channels)) and a timeseries input of shape (None, 10) (that's (timesteps, features)). Our model will have two outputs computed from the combination of these inputs: a "score" (of shape (1,)) and a probability distribution over five classes (of shape (5,)).
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np
image_input = keras.Input(shape=(32, 32, 3), name='img_input')
timeseries_input = keras.Input(shape=(None, 10), name='ts_input')
x1 = layers.Conv2D(3, 3)(image_input)
x1 = layers.GlobalMaxPooling2D()(x1)
x2 = layers.Conv1D(3, 3)(timeseries_input)
x2 = layers.GlobalMaxPooling1D()(x2)
x = layers.concatenate([x1, x2])
score_output = layers.Dense(1, name='score_output')(x)
class_output = layers.Dense(5, name='class_output')(x)
model = keras.Model(inputs=[image_input, timeseries_input],
outputs=[score_output, class_output])
Let's plot this model, so you can clearly see what we're doing here (note that the shapes shown in the plot are batch shapes, rather than per-sample shapes).
keras.utils.plot_model(model, 'multi_input_and_output_model.png', show_shapes=True)
If we only passed a single loss function to the model, the same loss function would be applied to every output, which is not appropriate here.
Passing data to a multi-input or multi-output model in fit works in a similar way as specifying a loss function in compile: you can pass lists of Numpy arrays (with 1:1 mapping to the outputs that received a loss function) or dicts mapping output names to Numpy arrays of training data.
model.compile(
optimizer=keras.optimizers.RMSprop(1e-3),
loss=[keras.losses.MeanSquaredError(),
keras.losses.CategoricalCrossentropy(from_logits=True)])
# Generate dummy Numpy data
img_data = np.random.random_sample(size=(100, 32, 32, 3))
ts_data = np.random.random_sample(size=(100, 20, 10))
score_targets = np.random.random_sample(size=(100, 1))
class_targets = np.random.random_sample(size=(100, 5))
# Fit on lists
model.fit([img_data, ts_data], [score_targets, class_targets],
batch_size=32,
epochs=3)
# Alternatively, fit on dicts
model.fit({'img_input': img_data, 'ts_input': ts_data},
{'score_output': score_targets, 'class_output': class_targets},
batch_size=32,
epochs=3)
Output -
Train on 100 samples
Epoch 1/3
100/100 [==============================] - 2s 22ms/sample - loss: 5.2477 - score_output_loss: 0.1809 - class_output_loss: 5.3292
Epoch 2/3
100/100 [==============================] - 0s 191us/sample - loss: 4.8558 - score_output_loss: 0.1235 - class_output_loss: 4.5884
Epoch 3/3
100/100 [==============================] - 0s 202us/sample - loss: 4.7482 - score_output_loss: 0.1421 - class_output_loss: 4.5786
Train on 100 samples
Epoch 1/3
100/100 [==============================] - 0s 260us/sample - loss: 4.6704 - score_output_loss: 0.1377 - class_output_loss: 4.5686
Epoch 2/3
100/100 [==============================] - 0s 204us/sample - loss: 4.6210 - score_output_loss: 0.2038 - class_output_loss: 4.5260
Epoch 3/3
100/100 [==============================] - 0s 188us/sample - loss: 4.6014 - score_output_loss: 0.1678 - class_output_loss: 4.3346
<tensorflow.python.keras.callbacks.History at 0x7f8cc43a9ac8>