Can I use rectangle images with a convolution neural network in Keras? - neural-network

Question

Say I'd like to use Keras's Convolutional2D function to build a CNN, can the input image be of size [224, 320, 3] instead of something like [224, 224, 3]?
Should I keep my images in their rectangle format or scale them to be square? I've tried making them into squares but the quality is greatly diminished + there is important data around the edges.
If I build it with rectangle input images, will it end up breaking down the line?
I'd also like to attach a decoder onto the end of the CNN to output the images in the same shape (essentially a VAE with rectangle images not squares).

Answer 1

Can I use Conv2D on arbitrary rectangles?
The short answer is yes. One of the big reasons that squares are used is that the math for the maxpooling/strides/padding is easy if it is exactly the same for both height and width. It just makes it easy. In the case of 224, you could use conv2d with padding=same, followed by maxpool several times to decrease both the height and width from 224, to 112, then 56, 28, 14, then finally 7.
When you do that with an input image of 224x320, then the progressing of reductions is as follows: 224x320, 112x160, 56x80, 28x40, 14x20, 7x10. Not a big deal, and it worked out pretty well. If instead the image was 224x300, it wouldn't get far before second dimension didn't divide nicely.
Here is some code in tensorflow for the encoder side of an autoencoder
import tensorflow as tf
import numpy as np
encoder = tf.keras.models.Sequential([
tf.keras.layers.InputLayer([224,320,3]),
tf.keras.layers.Conv2D(filters=16, kernel_size=5, padding='same', activation='tanh'),
tf.keras.layers.MaxPool2D(2),
tf.keras.layers.Conv2D(filters=16, kernel_size=5, padding='same', activation='tanh'),
tf.keras.layers.MaxPool2D(2),
tf.keras.layers.Conv2D(filters=16, kernel_size=5, padding='same', activation='tanh'),
tf.keras.layers.MaxPool2D(2),
tf.keras.layers.Conv2D(filters=16, kernel_size=5, padding='same', activation='tanh'),
tf.keras.layers.MaxPool2D(2),
tf.keras.layers.Conv2D(filters=16, kernel_size=5, padding='same', activation='tanh'),
tf.keras.layers.MaxPool2D(2),
tf.keras.layers.Conv2D(filters=32, kernel_size=5, padding='same', activation='tanh'),
])
data = np.zeros([1,224,320,3], dtype=np.float32)
print( encoder.predict(data).shape )
The output is
(1, 7, 10, 32)
The reverse can be used to make a decoder