I'm trying to convert a convolution layer to a fully-connected layer.
For example, there is an example of 3×3 input and 2x2 kernel:
which is equivalent to a vector-matrix multiplication，
Is there a function in PyTorch to get the matrix B?
I can only partially answer your question:
In your example above, you write the kernel as matrix and the input as a vector. If you are fine with writing the input as a matrix, you can use torch.nn.Unfold which explicitly calculates a convolution in the documentation:
# Convolution is equivalent with Unfold + Matrix Multiplication + Fold (or view to output shape)
inp = torch.randn(1, 3, 10, 12)
w = torch.randn(2, 3, 4, 5)
inp_unf = torch.nn.functional.unfold(inp, (4, 5))
out_unf = inp_unf.transpose(1, 2).matmul(w.view(w.size(0), -1).t()).transpose(1, 2)
out = out_unf.view(1, 2, 7, 8)
(torch.nn.functional.conv2d(inp, w) - out).abs().max()
# tensor(1.9073e-06)
If, however, you need to calculate the matrix for the kernel (the smaller matrix) you can use this function, which is based on Warren Weckessers answer:
def toeplitz_1_ch(kernel, input_size):
# shapes
k_h, k_w = kernel.shape
i_h, i_w = input_size
o_h, o_w = i_h-k_h+1, i_w-k_w+1
# construct 1d conv toeplitz matrices for each row of the kernel
toeplitz = []
for r in range(k_h):
toeplitz.append(linalg.toeplitz(c=(kernel[r,0], *np.zeros(i_w-k_w)), r=(*kernel[r], *np.zeros(i_w-k_w))) )
# construct toeplitz matrix of toeplitz matrices (just for padding=0)
h_blocks, w_blocks = o_h, i_h
h_block, w_block = toeplitz[0].shape
W_conv = np.zeros((h_blocks, h_block, w_blocks, w_block))
for i, B in enumerate(toeplitz):
for j in range(o_h):
W_conv[j, :, i+j, :] = B
W_conv.shape = (h_blocks*h_block, w_blocks*w_block)
return W_conv
which is not in pytorch but in numpy. This is for padding = 0 but can easily be adjusted by changing h_blocks and w_blocks and W_conv[i+j, :, j, :].
Update: Multiple output channels are just multiple of these matrices, as each output has its own kernel. Multiple input channels also have their own kernels - and their own matrices - over which you average after the convolution. This can be implemented as follows:
def conv2d_toeplitz(kernel, input):
"""Compute 2d convolution over multiple channels via toeplitz matrix
Args:
kernel: shape=(n_out, n_in, H_k, W_k)
input: shape=(n_in, H_i, W_i)"""
kernel_size = kernel.shape
input_size = input.shape
output_size = (kernel_size[0], input_size[1] - (kernel_size[1]-1), input_size[2] - (kernel_size[2]-1))
output = np.zeros(output_size)
for i,ks in enumerate(kernel): # loop over output channel
for j,k in enumerate(ks): # loop over input channel
T_k = toeplitz_1_ch(k, input_size[1:])
output[i] += T_k.dot(input[j].flatten()).reshape(output_size[1:]) # sum over input channels
return output
To check the correctness:
k = np.random.randn(4*3*3*3).reshape((4,3,3,3))
i = np.random.randn(3,7,9)
out = conv2d_toeplitz(k, i)
# check correctness of convolution via toeplitz matrix
print(np.sum((out - F.conv2d(torch.tensor(i).view(1,3,7,9), torch.tensor(k)).numpy())**2))
>>> 1.0063523219807736e-28
Update 2:
It is also possible to do this without looping in one matrix:
def toeplitz_mult_ch(kernel, input_size):
"""Compute toeplitz matrix for 2d conv with multiple in and out channels.
Args:
kernel: shape=(n_out, n_in, H_k, W_k)
input_size: (n_in, H_i, W_i)"""
kernel_size = kernel.shape
output_size = (kernel_size[0], input_size[1] - (kernel_size[1]-1), input_size[2] - (kernel_size[2]-1))
T = np.zeros((output_size[0], int(np.prod(output_size[1:])), input_size[0], int(np.prod(input_size[1:]))))
for i,ks in enumerate(kernel): # loop over output channel
for j,k in enumerate(ks): # loop over input channel
T_k = toeplitz_1_ch(k, input_size[1:])
T[i, :, j, :] = T_k
T.shape = (np.prod(output_size), np.prod(input_size))
return T
The input has to be flattened and the output reshaped after multiplication.
Checking for correctness (using the same i and k as above):
T = toeplitz_mult_ch(k, i.shape)
out = T.dot(i.flatten()).reshape((1,4,5,7))
# check correctness of convolution via toeplitz matrix
print(np.sum((out - F.conv2d(torch.tensor(i).view(1,3,7,9), torch.tensor(k)).numpy())**2))
>>> 1.5486060830252635e-28
You can use my code for convolution with circular padding:
import numpy as np
import scipy.linalg as linalg
def toeplitz_1d(k, x_size):
k_size = k.size
r = *k[(k_size // 2):], *np.zeros(x_size - k_size), *k[:(k_size // 2)]
c = *np.flip(k)[(k_size // 2):], *np.zeros(x_size - k_size), *np.flip(k)[:(k_size // 2)]
t = linalg.toeplitz(c=c, r=r)
return t
def toeplitz_2d(k, x_size):
k_h, k_w = k.shape
i_h, i_w = x_size
ks = np.zeros((i_w, i_h * i_w))
for i in range(k_h):
ks[:, i*i_w:(i+1)*i_w] = toeplitz_1d(k[i], i_w)
ks = np.roll(ks, -i_w, 1)
t = np.zeros((i_h * i_w, i_h * i_w))
for i in range(i_h):
t[i*i_h:(i+1)*i_h,:] = ks
ks = np.roll(ks, i_w, 1)
return t
def toeplitz_3d(k, x_size):
k_oc, k_ic, k_h, k_w = k.shape
i_c, i_h, i_w = x_size
t = np.zeros((k_oc * i_h * i_w, i_c * i_h * i_w))
for o in range(k_oc):
for i in range(k_ic):
t[(o * (i_h * i_w)):((o+1) * (i_h * i_w)), (i * (i_h * i_w)):((i+1) * (i_h * i_w))] = toeplitz_2d(k[o, i], (i_h, i_w))
return t
if __name__ == "__main__":
import torch
k = np.random.randint(50, size=(3, 2, 3, 3))
x = np.random.randint(50, size=(2, 5, 5))
t = toeplitz_3d(k, x.shape)
y = t.dot(x.flatten()).reshape(3, 5, 5)
xx = torch.nn.functional.pad(torch.from_numpy(x.reshape(1, 2, 5, 5)), pad=(1, 1, 1, 1), mode='circular')
yy = torch.conv2d(xx, torch.from_numpy(k))
err = ((y - yy.numpy()) ** 2).sum()
print(err)
While the other answers are correct, there is a faster way. In your example, you give an input of size 3x3 with a kernel of size 2x2. And your resulting circulant matrix multiplied by the input image is 9x9x4 operations, or 324 in total. Here is a method that does this with 4 x 4 x 4, or 64 operations in total. We will use Pytorch, but this could be done in Numpy, as well.
Assume an image input of shape (batch, channels, height, width):
import torch
def get_kernel_inputs(image, kernel):
out = torch.empty(image.size()[0], 0, 1, kernel.size()[-2] * kernel.size()[-1])
for k in range(image.size()[-2] - kernel.size()[-2] + 1):
for l in range(image.size()[-1] - kernel.size()[-1] + 1):
out = torch.cat([out,image[:, :, k:k+kernel.size()[-2],l:l + kernel.size()[-1]].reshape(image.size()[0], -1, 1, kernel.size()[-1] * kernel.size()[-2])], dim=1)
return out
Now let's test to see what size out this gives:
img = torch.rand(1, 1, 3, 3)
kernel = torch.rand(2, 2)
kernelized_img = get_kernel_inputs(img, kernel)
print(kernelized_img.size())
This yields a size of:
torch.Size([1, 4, 1, 4])
So there are 16 values stored in the above tensor. Now let's matrix multiply:
print(torch.matmul(kernelized_img, kernel.view(4)))
This is 16 x 4 multiplications.
Finally, let's test that this is, in fact, giving out the correct value by using the Torch Conv2d module:
import torch.nn as nn
mm = nn.Conv2d(1, 1, (2,2), bias=False)
with torch.no_grad():
kernel_test = mm.weight
print("Control ", mm(img))
print("Test", torch.matmul(kernelized_img, kernel_test.view(4)).view(1, 1, 2, 2))
Control tensor([[[[-0.0089, 0.0178],
[-0.1419, 0.2720]]]], grad_fn=<ThnnConv2DBackward>)
Test tensor([[[[-0.0089, 0.0178],
[-0.1419, 0.2720]]]], grad_fn=<ViewBackward>)
All we are doing differently in the above is reshaping the image instead of the kernel.
Setting the image height and width equal and the kernel height and width equal, where
i=image height/width
k=kernel height/width
Then the difference in the number of calculations in the Toeplitz method vs. the above method is:
Edit Addition:
The above implementation only worked on single-channel inputs. For this definition to work on multiple channel inputs and outputs, plus handle batches, can do the following:
def get_kernel_inputs(image, kernel):
out=torch.empty(image.size()[0], image.size()[1], 0, kernel.size()[-2]*kernel.size()[-1])
out_size=[image.size()[-2]-kernel.size()[-2]+1,(image.size()[-1]-kernel.size()[-1]+1)]
for k in range(out_size[0]):
for l in range(out_size[1]):
out=torch.cat([out,image[:,:,k:k+kernel.size()[-2],l:l+kernel.size()[-1]].reshape(image.size()[0],-1,1,kernel.size()[-1]*kernel.size()[-2])],dim=2)
preout=out.permute(0,2,1,3).reshape(image.size()[0],-1,image.size()[1]*kernel.size()[-2]*kernel.size()[-1])
kernel1 = kernel.view(kernel.size()[0], -1)
out = torch.matmul(preout, kernel1.T).permute(0, 2, 1).reshape(image.size()[0], kernel.size()[0],
out_size[0], out_size[1])
return out
images=torch.rand(5, 3, 32, 32)
mm=nn.Conv2d(3, 32, (3, 3), bias=False)
#Set the kernel to Conv2d init for testing
with torch.no_grad():
kernel=mm.weight
print(get_kernel_inputs(images, kernel))
print(mm(images))
My understanding was, that these three should be the same, however matlab gives completely different results. The first and third one are in sync with what I calculated by hand, the second one is different.
x_1 = [1, 2, 0, 5];
x_2 = [1/2, -1/4, 1, 0, 3/4];
y_2_1 = ifft(fft(x_1, 2) .* fft(x_2, 2))
y_2_2 = cconv(x_2, x_1, 2)
y_2_3 = cconv(x_2(1:2), x_1(1:2), 2)
From the documentation:
The modulo-2 circular convolution is equivalent to splitting the linear convolution into two-element arrays and summing the arrays.
So it is not the same to do
res = cconv(x_2, x_1, 2);
as to do
res2 = cconv(x_2, x_1);
res2 = res(1:2);
The former is equivalent to
res = cconv(x_2, x_1);
res = res(1:2) + res(3:4) + res(5:6) + ...;
(padding with zeros if res is odd in size).
On the other hand,
res3 = ifft(fft(x_1, 2) .* fft(x_2, 2));
is equivalent to
res3 = fft(x_1(1:2)) .* fft(x_2(1:2));
res3 = ifft(res3);
and different from either of the two cconv results.
I'm using Matlab 2015a.
Here is the function I want to optimise:
function result = fig_of_merit(x, a, b, c, d)
result = 1;
end
This is how I want to optimise the function:
x1 = [10*10^-6, 120];
x2 = [300*10^-6, 175];
fminbnd(#(x) fig_of_merit(x, 1, 2, 3, 4),x1,x2);
I keep getting the following error:
Error using *
Inner matrix dimensions must agree.
Error in fminbnd (line 291)
x = xf + si * max( abs(d), tol1 );
Error in test (line 5)
fminbnd(#(x) fig_of_merit(x, 1, 2, 3, 4),x1,x2);
What's wrong with my code? I did the same for optimisation without bounds using the fminsearch function and everything was fine.
It has something to do with the anonymous function I used - when I reduced the vectors x1 and x2 to scalars, it worked:
fminbnd(#(x) fig_of_merit(x, 1, 2, 3, 4),-4, 5);
It doesn't satisfy me, because I want to optimise 2 parameters at once.
The fminbnd function only works on scalar parameters. It can't optimize two parameters at once. However, the fmincon function can do this:
x1 = [10*10^-6, 120];
x2 = [300*10^-6, 175];
fmincon(#(x) fig_of_merit(x, 1, 2, 3, 4),ones(size(x1)),[],[],[],[],x1,x2);
I have the following homework question:
Apply linear least squares with the two models S1(A, B, C) = Ax^2 + Bx
+ C and S2(A, B, C, D) = Ax^3 + Bx^2 + Cx + D to the data set (0, 4), (1, −1), (2, 6), (3, 1), (4, −4), (5, −9). Solve in MATLAB using
lspoly. Report the values of the parameters A, B, C and D clearly, and
produce a plot showing the data and both fitting curves.
I am working with the following function in MATLAB:
Function 1
function y = horner(a,c)
n=length(a)-1;
y=a(n+1);
for k = n:-1:1
y = a(k)+ c*y;
end
Function 2
function C = lspoly(x,y,M)
n = length(x);
F = zeros(n,M+1);
for k = 1:M+1
F(:,k) = x'.^(k-1);
end
A = F'*F;
B = F'*y';
C = A\B;
And this is the code I wrote for the problem:
clc
clear all
close all
x = [0, 1, 2, 3, 4, 5];
y = [4, -1, 6, 1, -4, -9];
C = lspoly(x,y,2); % finds the parameters
xx = 0:0.01:5;
yy = zeros(1, length(xx));
for i=1:length(xx)
yy(i) = horner(C,xx(i));
end
CC = lspoly(x,y,3); % finds the parameters
xxx = 0:0.1:5;
yyy = zeros(1, length(xxx));
for i=1:length(xxx)
yyy(i) = horner(CC,xxx(i));
end
figure(1)
plot(x, y, 'o', xx, yy, 'r-', xxx, yyy, 'b-')
I am encountering some issues with this code. When I try to run the program, I get the following error:
In an assignment A(I) = B, the number of elements in B and I must be
the same.
Error in HW7_1 (line 14) yy(i) = horner(C,xx(i));
I can't really wrap my head around what exactly I need to do to fix this issue. I tried breaking down my program piece by piece to determine results at different spots in the code, but so far nothing of note has been found.
Can someone help me fix this error?
It works just fine for me in Octave 3.8. The only thing I can think of is there a built-in function called horner in MATLAB (part of the Symbolic Math Toolbox), so maybe your code is calling that function instead of yours. Maybe try renaming it to something different, like my_horner or similar.
A quick question, I have a sparse matrix A and a index list b = [2, 3, 4 ...], I want to set A(2, 2), A(3, 3), A(4, 4)... = 0. I tried A(b, b) = 0, the result is that A(2, 3), A(2, 4), A(3, 2), A(3, 4), A(4, 2), A(4, 3) and some non-diagonal elements of this matrix are also set zero, how to deal with that?
Use linear indexing:
A(sub2ind(size(A),b,b)) = 0;
This works for full or sparse matrices.