I was looking at the docs of tensorflow about tf.nn.conv2d here. But I can't understand what it does or what it is trying to achieve. It says on the docs,
#1 : Flattens the filter to a 2-D matrix with shape
[filter_height * filter_width * in_channels, output_channels].
Now what does that do? Is that element-wise multiplication or just plain matrix multiplication? I also could not understand the other two points mentioned in the docs. I have written them below :
# 2: Extracts image patches from the the input tensor to form a virtual tensor of shape
[batch, out_height, out_width, filter_height * filter_width * in_channels].
# 3: For each patch, right-multiplies the filter matrix and the image patch vector.
It would be really helpful if anyone could give an example, a piece of code (extremely helpful) maybe and explain what is going on there and why the operation is like this.
I've tried coding a small portion and printing out the shape of the operation. Still, I can't understand.
I tried something like this:
op = tf.shape(tf.nn.conv2d(tf.random_normal([1,10,10,10]),
tf.random_normal([2,10,10,10]),
strides=[1, 2, 2, 1], padding='SAME'))
with tf.Session() as sess:
result = sess.run(op)
print(result)
I understand bits and pieces of convolutional neural networks. I studied them here. But the implementation on tensorflow is not what I expected. So it raised the question.
EDIT:
So, I implemented a much simpler code. But I can't figure out what's going on. I mean how the results are like this. It would be extremely helpful if anyone could tell me what process yields this output.
input = tf.Variable(tf.random_normal([1,2,2,1]))
filter = tf.Variable(tf.random_normal([1,1,1,1]))
op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='SAME')
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
print("input")
print(input.eval())
print("filter")
print(filter.eval())
print("result")
result = sess.run(op)
print(result)
output
input
[[[[ 1.60314465]
[-0.55022103]]
[[ 0.00595062]
[-0.69889867]]]]
filter
[[[[-0.59594476]]]]
result
[[[[-0.95538563]
[ 0.32790133]]
[[-0.00354624]
[ 0.41650501]]]]
Ok I think this is about the simplest way to explain it all.
Your example is 1 image, size 2x2, with 1 channel. You have 1 filter, with size 1x1, and 1 channel (size is height x width x channels x number of filters).
For this simple case the resulting 2x2, 1 channel image (size 1x2x2x1, number of images x height x width x x channels) is the result of multiplying the filter value by each pixel of the image.
Now let's try more channels:
input = tf.Variable(tf.random_normal([1,3,3,5]))
filter = tf.Variable(tf.random_normal([1,1,5,1]))
op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='VALID')
Here the 3x3 image and the 1x1 filter each have 5 channels. The resulting image will be 3x3 with 1 channel (size 1x3x3x1), where the value of each pixel is the dot product across channels of the filter with the corresponding pixel in the input image.
Now with a 3x3 filter
input = tf.Variable(tf.random_normal([1,3,3,5]))
filter = tf.Variable(tf.random_normal([3,3,5,1]))
op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='VALID')
Here we get a 1x1 image, with 1 channel (size 1x1x1x1). The value is the sum of the 9, 5-element dot products. But you could just call this a 45-element dot product.
Now with a bigger image
input = tf.Variable(tf.random_normal([1,5,5,5]))
filter = tf.Variable(tf.random_normal([3,3,5,1]))
op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='VALID')
The output is a 3x3 1-channel image (size 1x3x3x1).
Each of these values is a sum of 9, 5-element dot products.
Each output is made by centering the filter on one of the 9 center pixels of the input image, so that none of the filter sticks out. The xs below represent the filter centers for each output pixel.
.....
.xxx.
.xxx.
.xxx.
.....
Now with "SAME" padding:
input = tf.Variable(tf.random_normal([1,5,5,5]))
filter = tf.Variable(tf.random_normal([3,3,5,1]))
op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='SAME')
This gives a 5x5 output image (size 1x5x5x1). This is done by centering the filter at each position on the image.
Any of the 5-element dot products where the filter sticks out past the edge of the image get a value of zero.
So the corners are only sums of 4, 5-element dot products.
Now with multiple filters.
input = tf.Variable(tf.random_normal([1,5,5,5]))
filter = tf.Variable(tf.random_normal([3,3,5,7]))
op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='SAME')
This still gives a 5x5 output image, but with 7 channels (size 1x5x5x7). Where each channel is produced by one of the filters in the set.
Now with strides 2,2:
input = tf.Variable(tf.random_normal([1,5,5,5]))
filter = tf.Variable(tf.random_normal([3,3,5,7]))
op = tf.nn.conv2d(input, filter, strides=[1, 2, 2, 1], padding='SAME')
Now the result still has 7 channels, but is only 3x3 (size 1x3x3x7).
This is because instead of centering the filters at every point on the image, the filters are centered at every other point on the image, taking steps (strides) of width 2. The x's below represent the filter center for each output pixel, on the input image.
x.x.x
.....
x.x.x
.....
x.x.x
And of course the first dimension of the input is the number of images so you can apply it over a batch of 10 images, for example:
input = tf.Variable(tf.random_normal([10,5,5,5]))
filter = tf.Variable(tf.random_normal([3,3,5,7]))
op = tf.nn.conv2d(input, filter, strides=[1, 2, 2, 1], padding='SAME')
This performs the same operation, for each image independently, giving a stack of 10 images as the result (size 10x3x3x7)
2D convolution is computed in a similar way one would calculate 1D convolution: you slide your kernel over the input, calculate the element-wise multiplications and sum them up. But instead of your kernel/input being an array, here they are matrices.
In the most basic example there is no padding and stride=1. Let's assume your input and kernel are:
When you use your kernel you will receive the following output: , which is calculated in the following way:
14 = 4 * 1 + 3 * 0 + 1 * 1 + 2 * 2 + 1 * 1 + 0 * 0 + 1 * 0 + 2 * 0 + 4 * 1
6 = 3 * 1 + 1 * 0 + 0 * 1 + 1 * 2 + 0 * 1 + 1 * 0 + 2 * 0 + 4 * 0 + 1 * 1
6 = 2 * 1 + 1 * 0 + 0 * 1 + 1 * 2 + 2 * 1 + 4 * 0 + 3 * 0 + 1 * 0 + 0 * 1
12 = 1 * 1 + 0 * 0 + 1 * 1 + 2 * 2 + 4 * 1 + 1 * 0 + 1 * 0 + 0 * 0 + 2 * 1
TF's conv2d function calculates convolutions in batches and uses a slightly different format. For an input it is [batch, in_height, in_width, in_channels] for the kernel it is [filter_height, filter_width, in_channels, out_channels]. So we need to provide the data in the correct format:
import tensorflow as tf
k = tf.constant([
[1, 0, 1],
[2, 1, 0],
[0, 0, 1]
], dtype=tf.float32, name='k')
i = tf.constant([
[4, 3, 1, 0],
[2, 1, 0, 1],
[1, 2, 4, 1],
[3, 1, 0, 2]
], dtype=tf.float32, name='i')
kernel = tf.reshape(k, [3, 3, 1, 1], name='kernel')
image = tf.reshape(i, [1, 4, 4, 1], name='image')
Afterwards the convolution is computed with:
res = tf.squeeze(tf.nn.conv2d(image, kernel, [1, 1, 1, 1], "VALID"))
# VALID means no padding
with tf.Session() as sess:
print sess.run(res)
And will be equivalent to the one we calculated by hand.
For examples with padding/strides, take a look here.
Just to add to the other answers, you should think of the parameters in
filter = tf.Variable(tf.random_normal([3,3,5,7]))
as '5' corresponding to the number of channels in each filter. Each filter is a 3d cube, with a depth of 5. Your filter depth must correspond to your input image's depth. The last parameter, 7, should be thought of as the number of filters in the batch. Just forget about this being 4D, and instead imagine that you have a set or a batch of 7 filters. What you do is create 7 filter cubes with dimensions (3,3,5).
It is a lot easier to visualize in the Fourier domain since convolution becomes point-wise multiplication. For an input image of dimensions (100,100,3) you can rewrite the filter dimensions as
filter = tf.Variable(tf.random_normal([100,100,3,7]))
In order to obtain one of the 7 output feature maps, we simply perform the point-wise multiplication of the filter cube with the image cube, then we sum the results across the channels/depth dimension (here it's 3), collapsing to a 2d (100,100) feature map. Do this with each filter cube, and you get 7 2D feature maps.
I tried to implement conv2d (for my studying). Well, I wrote that:
def conv(ix, w):
# filter shape: [filter_height, filter_width, in_channels, out_channels]
# flatten filters
filter_height = int(w.shape[0])
filter_width = int(w.shape[1])
in_channels = int(w.shape[2])
out_channels = int(w.shape[3])
ix_height = int(ix.shape[1])
ix_width = int(ix.shape[2])
ix_channels = int(ix.shape[3])
filter_shape = [filter_height, filter_width, in_channels, out_channels]
flat_w = tf.reshape(w, [filter_height * filter_width * in_channels, out_channels])
patches = tf.extract_image_patches(
ix,
ksizes=[1, filter_height, filter_width, 1],
strides=[1, 1, 1, 1],
rates=[1, 1, 1, 1],
padding='SAME'
)
patches_reshaped = tf.reshape(patches, [-1, ix_height, ix_width, filter_height * filter_width * ix_channels])
feature_maps = []
for i in range(out_channels):
feature_map = tf.reduce_sum(tf.multiply(flat_w[:, i], patches_reshaped), axis=3, keep_dims=True)
feature_maps.append(feature_map)
features = tf.concat(feature_maps, axis=3)
return features
Hope I did it properly. Checked on MNIST, had very close results (but this implementation is slower). I hope this helps you.
In addition to other answers, conv2d operation is operating in c++ (cpu) or cuda for gpu machines that requires to flatten and reshape data in certain way and use gemmBLAS or cuBLAS(cuda) matrix multiplication.
It's performing convulition throught the picture when you are trying for example image classifation thuis function has all the parameters need to do that.
When you are basically can chose the filter dimension. Strides. Padding. Before to used its need to undestant the concepts of convolution
this explanation complements:
Keras Conv2d own filters
I had some doubts about the filter parameters in keras.conv2d because when I learned I was supposed to set my own filter design. But this parameters tells how many filters to test and keras itself will try to find the best filters weights.
Related
I have a vector A, say
A = [1, 0, 0, 0]
I want to perform an operation on this vector to get the next element. For example, say
A(i) = A(i - 1) * 5 [for i >= 2]
This can be easily achieved via a loop. But I want to achieve it by using vector operation. So far I have tried
A = [1, 0, 0, 0]
A(2:4) = A(1:3) * 5
But the content in A after this operation is coming as
A = [1 5 0 0]
The targeted answer should be
A = [1 5 25 125]
Please, mention the necessary changes to be made to achieve the target.
[Note: Please do not simply consider the above example as the elements which are power of 5 but consider A(i) = A(i - 1) * 5.]
how about that:
A(1)*5.^[0:numel(A)-1]
I have a feature matrix of size ~1M x 3 where the columns are doc#,wordID#,wordcount
What's a fast way in Matlab to rearrange this feature matrix so it is instead of size #docs x # unique words i.e.
(length(unique(featurematrix(:,1))) x length(unique(featurematrix(:,2)))
so that each row instead represents an entire document, each column represents a different word, and the values are the wordcounts from the 3rd column of the original matrix?
I started writing a bunch of loops, but had the feeling there's probably some short idiomatic way to do this already built-in to Matlab.
You can actually use accumarray to accomplish this
data = [1, 1, 1;
1, 2, 2;
1, 5, 3;
2, 1, 4;
2, 3, 5];
result = accumarray(data(:,1:2), data(:,3))
% 1 2 0 0 3
% 4 0 5 0 0
Alternately you could use sparse
result = full(sparse(data(:,1), data(:,2), data(:,3)))
Basically I have an original 128 x 128 x 3 matrix which describes a RGB image (128 * 128 points with each points being a 1x3 vector containing Red, Green, and Blue intensity respectively) , and I also have 16 points chosen from them (this is the easy part), and now I want to do pairwise distance calculations between the 128 x 128 x 3 matrix and the 16 points.
But the problem is the Matlab function for that, "pdist2" only takes 2 matrices of size M1 x N and M2 x N and not anything else, so I plan to convert the 128 x 128 x 3 matrix to a (128 * 128) x 3 one. Later on, after some calculations with the newly transformed matrix, I need to convert it back to the original size in order to display the image and check the results. But I'm not sure if the elements will remain in their place or will they be shuffled around ? Please help me thank you very much!
From the documentation
The data type and number of elements in B are the same as the data type and number of elements in A. The elements in B preserve their column-wise ordering from A.
If you need the result to be the same size as the original, just store the size before the initial transformation and use this as the input to reshape after performing your manipulation of the data.
% Store the original size
originalSize = size(data);
% Reshape it to your new 2D array
data = reshape(data, [], 3);
% Do stuff
% Reshape it back to it's original size
data = reshape(data, originalSize);
In the 2D version, the elements won't technically be in the same position as they were in the 3D matrix because...well..it's 2D not 3D. But, if you reshape it back to the 3D (without moving elements around), the element ordering would be the same as the original 3D matrix.
Update
You can easily check this for yourself.
R = rand([10, 20, 3]);
isequal(R, reshape(reshape(R, [], 3), size(R)))
The reason for why this is is because reshape does not actually change the underlying data but rather the way in which it is accessed. We can easily check this by using format debug to see where the data is stored.
We can also use a little anonymous function I wrote to see where in memory a given variable is stored.
format debug;
memoryLocation = #(x)regexp(evalc('disp(x)'), '(?<=pr\s*=\s*)[a-z0-9]*', 'match')
Ok so let's create a matrix and check where MATLAB stored it in memory
A = rand(10);
memoryLocation(A)
% 7fa58f2ed9c0
Now let's reshape it and check the memory location again to see if it is in a different place (i.e. the order or values were modified)
B = reshape(A, [], 10);
memoryLocation(B)
% 7fa58f2ed9c0
As you can see, the memory location hasn't changed meaning that the ordering of elements has to be the same otherwise MATLAB would have needed to make a copy in memory.
Underlying data representation and why Suever's answer is correct:
The underlying array data in MATLAB is essentially an array of double precision floating point. In c++ it would be a:
double *array_data;
MATLAB stores data in column major format. If an arrow had n_rows rows, the element A_{i,j} (i.e. ith row, jth column (zero indexed) would be given by:
array_data[i + j * n_rows]
When you call reshape function, what changes are the variables n_rows, n_cols, etc.... It doesn't change array_data.
Example (no need to touch array data to resize array):
array_data = [1, 2, 3, 4, 5, 6];
With n_rows = 2 and column-major format, this would be:
A = [1, 3, 5
2, 4, 6]
A11 = array_data[0 + 0 * 2] = array_data[0] = 1
A21 = array_data[1 + 0 * 2] = array_data[1] = 2
A12 = array_data[0 + 1 * 2] = array_data[2] = 3
A22 = array_data[1 + 1 * 2] = array_data[3] = 4
A13 = array_data[0 + 2 * 2] = array_data[4] = 5
A23 = array_data[1 + 2 * 2] = array_data[5] = 6
With n_rows = 3 and the same underlying array_data, you would have:
A = [1, 4
2, 5
3, 6]
A11 = array_data[0 + 0 * 3] = array_data[0] = 1
A21 = array_data[1 + 0 * 3] = array_data[1] = 2
A31 = array_data[2 + 0 * 3] = array_data[2] = 3
A12 = array_data[0 + 1 * 3] = array_data[3] = 4
A22 = array_data[1 + 1 * 3] = array_data[4] = 5
A32 = array_data[2 + 1 * 3] = array_data[5] = 6
resize just changes n_rows, n_cols etc...
According to this paper, the output shape is N + H - 1, N is input height or width, H is kernel height or width. This is obvious inverse process of convolution. This tutorial gives a formula to calculate the output shape of convolution which is (W−F+2P)/S+1, W - input size, F - filter size, P - padding size, S - stride. But in Tensorflow, there are test cases like:
strides = [1, 2, 2, 1]
# Input, output: [batch, height, width, depth]
x_shape = [2, 6, 4, 3]
y_shape = [2, 12, 8, 2]
# Filter: [kernel_height, kernel_width, output_depth, input_depth]
f_shape = [3, 3, 2, 3]
So we use y_shape, f_shape and x_shape, according to formula (W−F+2P)/S+1 to calculate padding size P. From (12 - 3 + 2P) / 2 + 1 = 6, we get P = 0.5, which is not an integer. How does deconvolution works in Tensorflow?
for deconvolution,
output_size = strides * (input_size-1) + kernel_size - 2*padding
strides, input_size, kernel_size, padding are integer
padding is zero for 'valid'
The formula for the output size from the tutorial assumes that the padding P is the same before and after the image (left & right or top & bottom).
Then, the number of places in which you put the kernel is:
W (size of the image) - F (size of the kernel) + P (additional padding before) + P (additional padding after).
But tensorflow also handles the situation where you need to pad more pixels to one of the sides than to the other, so that the kernels would fit correctly. You can read more about the strategies to choose the padding ("SAME" and "VALID") in the docs. The test you're talking about uses method "VALID".
This discussion is really helpful. Just add some additional information.
padding='SAME' can also let the bottom and right side get the one additional padded pixel. According to TensorFlow document, and the test case below
strides = [1, 2, 2, 1]
# Input, output: [batch, height, width, depth]
x_shape = [2, 6, 4, 3]
y_shape = [2, 12, 8, 2]
# Filter: [kernel_height, kernel_width, output_depth, input_depth]
f_shape = [3, 3, 2, 3]
is using padding='SAME'. We can interpret padding='SAME' as:
(W−F+pad_along_height)/S+1 = out_height,
(W−F+pad_along_width)/S+1 = out_width.
So (12 - 3 + pad_along_height) / 2 + 1 = 6, and we get pad_along_height=1. And pad_top=pad_along_height/2 = 1/2 = 0(integer division), pad_bottom=pad_along_height-pad_top=1.
As for padding='VALID', as the name suggested, we use padding when it is proper time to use it. At first, we assume that the padded pixel = 0, if this doesn't work well, then we add 0 padding where any value outside the original input image region. For example, the test case below,
strides = [1, 2, 2, 1]
# Input, output: [batch, height, width, depth]
x_shape = [2, 6, 4, 3]
y_shape = [2, 13, 9, 2]
# Filter: [kernel_height, kernel_width, output_depth, input_depth]
f_shape = [3, 3, 2, 3]
The output shape of conv2d is
out_height = ceil(float(in_height - filter_height + 1) / float(strides[1]))
= ceil(float(13 - 3 + 1) / float(3)) = ceil(11/3) = 6
= (W−F)/S + 1.
Cause (W−F)/S+1 = (13-3)/2+1 = 6, the result is an integer, we don't need to add 0 pixels around the border of the image, and pad_top=1/2, pad_left=1/2 in the TensorFlow document padding='VALID' section are all 0.
I have an image U, and when I want to convolve it with a box filter:
0 1 0
1 -4 1
0 1 0
I use imfilter function with a constant 2D array and there is no problem. But, when I have the following operation:
u(i,j) = v(i-1,j)^2 * u(i-1,j) + v(i+1,j)^2 * u(i+1,j) + v(i, j+1)^2 * u(i,j+1) + v(i,j-1)^2 * u(i,j-1)
(A simplified version of my filter). In other words, my filter to be used over image U is related to the pixel values of image V, but in the same location which the filter is applied. Is there a way to implement such an operation in MATLAB, WITHOUT using nested for loops for each pixel?
You can solve it using im2col and col2im as following:
% Filtering A using B
mA = randn(10, 10); mB = randn(10, 10);
mACol = im2col(mA, [3, 3], 'sliding'); mBCol = im2col(mB, [3, 3],
'sliding');
mAColFilt = sum(mACol .* (mBCol .^ 2));
mAFilt = col2im(mAColFilt, [3, 3], [10, 10]);
I skipped the to get the correct coefficients (In your case, zero few of them and raise to the power of 2 the rest, I only raised all to the power of 2).
Pay attention that the filtered image is smaller (Bounadries of the filter).
You should pad it as required.