I am trying to write a function which performs Gaussian elimination with scaled row pivoting. I almost have it right, but my answer is not quite correct, so something must be wrong in my code. I have written:
function [B,h] = factorization(A)
n = length(A);
p = zeros(1,n);
s = zeros(1,n);
for i = 1:n
p(i) = i;
s(i) = max(abs(A(i,1:n)));
end
for k = 1:(n-1)
m = abs(A(k:n,k));
q = length(m);
v = zeros(1,q);
w = s(k:n);
for j = 1:q
v(j) = m(j)/(w(j));
end
[pivot,pivot] = max(abs(v(1:end)));
if pivot ~= 1
var = p(k);
p(k) = p(pivot);
p(pivot) = var;
end
for i = (k+1):n
z = A(p(i),k)/A(p(k),k);
A(p(i),k) = z;
for j = (k+1):n
A(p(i),j) = A(p(i),j) - z*A(p(k),j);
end
end
end
B = A;
h = p;
Say then that I use the matrix A = [2 3 -6; 1 -6 8; 3 -2 1] as input. My code gives me the output: B = [0.6667 -0.8125 -0.4375; 0.3333 -5.3333 7.6667; 3 -2 1], h = [3 2 1]. The correct answer, however, should be: B = [0.0007 4.3333 -6.6667; 0.3333 -1.2308 -0.5385; 3 -2 1], h = [3 1 2]
I can't see where in the code I'm doing wrong, so if anyone could help me out, I would be very grateful!
Related
I have some matlab code as follow, constructing KNN similarity weight matrix.
[D,I] = pdist2(X, X, 'squaredeuclidean', 'Smallest', k+1);
D = D < threshold;
W = zeros(n, n);
for i=1:size(I,2)
W(I(:,i), i) = D(:,i);
W(i, I(:,i)) = D(:,i)';
end
I want to vectorize the for loop. I have tried
W(I) = D;
but failed to get the correct value.
I add test case here:
n = 5;
D = [
1 1 1 1 1
0 1 1 1 1
0 0 0 0 0
];
I = [
1 2 3 4 5
5 4 5 2 3
3 1 1 1 1
];
There are some undefined variables that makes it hard to check what it is doing, but this should do the same as your for loop:
D,I] = pdist2(X, X, 'squaredeuclidean', 'Smallest', k+1);
D = D < threshold;
W = zeros(n);
% set the diagonal values
W(sub2ind(size(X), I(1, :), I(1, :))) = D(1,:);
% set the other values
W(sub2ind(size(W), I(2, :), 1:size(I, 2))) = D(2, :);
W(sub2ind(size(W), 1:size(I, 2), I(2, :))) = D(2, :).';
I splited the directions, it works now with your test case.
A possible solution:
idx1 = reshape(1:n*n,n,n).';
idx2 = bsxfun(#plus,I,0:n:n*size(I,2)-1);
W=zeros(n,n);
W(idx2) = D;
W(idx1(idx2)) = D;
Here assumed that you repeatedly want to compute D and I so compute idx only one time and use it repeatedly.
n = 5;
idx1 = reshape(1:n*n,n,n).';
%for k = 1 : 1000
%[D,I] = pdist2(X, X, 'squaredeuclidean', 'Smallest', k+1);
%D = D < threshold;
idx2 = bsxfun(#plus,I,0:n:n*size(I,2)-1);
W=zeros(n,n);
W(idx2) = D;
W(idx1(idx2)) = D;
%end
But if n isn't constant and it varies in each iteration it is better to change the way idx1 is computed:
n = 5;
%for k = 1 : 1000
%n = randi([2 10]);%n isn't constant
%[D,I] = pdist2(X, X, 'squaredeuclidean', 'Smallest', k+1);
%D = D < threshold;
idx1 = bsxfun(#plus,(0:n:n^2-1).',1:size(I,2));
idx2 = bsxfun(#plus,I,0:n:n*size(I,2)-1);
W=zeros(n,n);
W(idx2) = D;
W(idx1(idx2)) = D;
%end
You can cut some corners with linear indices but if your matrices are big then you should only take the nonzero components of D. Following copies all values of D
W = zeros(n);
W(reshape(sub2ind([n,n],I,[1;1;1]*[1:n]),1,[])) = reshape(D,1,[]);
How can I speed up the following MATLAB code, using vectorization? Right now the single line in the loop is taking hours to run for the case upper = 1e7.
Here is the commented code with sample output:
p = 8;
lower = 1;
upper = 1e1;
n = setdiff(lower:upper,primes(upper)); % contains composite numbers between lower + upper
x = ones(length(n),p); % Preallocated 2-D array of ones
% This loop stores the unique prime factors of each composite
% number from 1 to n, in each row of x. Since the rows will have
% varying lengths, the rows are padded with ones at the end.
for i = 1:length(n)
x(i,:) = [unique(factor(n(i))) ones(1,p-length(unique(factor(n(i)))))];
end
output:
x =
1 1 1 1 1 1 1 1
2 1 1 1 1 1 1 1
2 3 1 1 1 1 1 1
2 1 1 1 1 1 1 1
3 1 1 1 1 1 1 1
2 5 1 1 1 1 1 1
For example, the last row contains the prime factors of 10, if we ignore the ones. I have made the matrix 8 columns wide to account for the many prime factors of numbers up to 10 million.
Thanks for any help!
This is not vectorization, but this version of the loop will save about half of the time:
for k = 1:numel(n)
tmp = unique(factor(n(k)));
x(k,1:numel(tmp)) = tmp;
end
Here is a quick benchmark for this:
function t = getPrimeTime
lower = 1;
upper = 2.^(1:8);
t = zeros(numel(upper),2);
for k = 1:numel(upper)
n = setdiff(lower:upper(k),primes(upper(k))); % contains composite numbers between lower to upper
t(k,1) = timeit(#() getPrime1(n));
t(k,2) = timeit(#() getPrime2(n));
disp(k)
end
p = plot(log2(upper),log10(t));
p(1).Marker = 'o';
p(2).Marker = '*';
xlabel('log_2(range of numbers)')
ylabel('log(time (sec))')
legend({'getPrime1','getPrime2'})
end
function x = getPrime1(n) % the originel function
p = 8;
x = ones(length(n),p); % Preallocated 2-D array of ones
for k = 1:length(n)
x(k,:) = [unique(factor(n(k))) ones(1,p-length(unique(factor(n(k)))))];
end
end
function x = getPrime2(n)
p = 8;
x = ones(numel(n),p); % Preallocated 2-D array of ones
for k = 1:numel(n)
tmp = unique(factor(n(k)));
x(k,1:numel(tmp)) = tmp;
end
end
Here's another approach:
p = 8;
lower = 1;
upper = 1e1;
p = 8;
q = primes(upper);
n = setdiff(lower:upper, q);
x = bsxfun(#times, q, ~bsxfun(#mod, n(:), q));
x(~x) = inf;
x = sort(x,2);
x(isinf(x)) = 1;
x = [x ones(size(x,1), p-size(x,2))];
This seems to be faster than the other two options (but is uses more memory). Borrowing EBH's benchmarking code:
function t = getPrimeTime
lower = 1;
upper = 2.^(1:12);
t = zeros(numel(upper),3);
for k = 1:numel(upper)
n = setdiff(lower:upper(k),primes(upper(k)));
t(k,1) = timeit(#() getPrime1(n));
t(k,2) = timeit(#() getPrime2(n));
t(k,3) = timeit(#() getPrime3(n));
disp(k)
end
p = plot(log2(upper),log10(t));
p(1).Marker = 'o';
p(2).Marker = '*';
p(3).Marker = '^';
xlabel('log_2(range of numbers)')
ylabel('log(time (sec))')
legend({'getPrime1','getPrime2','getPrime3'})
grid on
end
function x = getPrime1(n) % the originel function
p = 8;
x = ones(length(n),p); % Preallocated 2-D array of ones
for k = 1:length(n)
x(k,:) = [unique(factor(n(k))) ones(1,p-length(unique(factor(n(k)))))];
end
end
function x = getPrime2(n)
p = 8;
x = ones(numel(n),p); % Preallocated 2-D array of ones
for k = 1:numel(n)
tmp = unique(factor(n(k)));
x(k,1:numel(tmp)) = tmp;
end
end
function x = getPrime3(n) % Approach in this answer
p = 8;
q = primes(max(n));
x = bsxfun(#times, q, ~bsxfun(#mod, n(:), q));
x(~x) = inf;
x = sort(x,2);
x(isinf(x)) = 1;
x = [x ones(size(x,1), p-size(x,2))];
end
I want to speed up my code. I always use vectorization. But in this code I have no idea how to avoid the for-loop. I would really appreciate a hint how to proceed.
thank u so much for your time.
close all
clear
clc
% generating sample data
x = linspace(10,130,33);
y = linspace(20,100,22);
[xx, yy] = ndgrid(x,y);
k = 2*pi/50;
s = [sin(k*xx+k*yy)];
% generating query points
xi = 10:5:130;
yi = 20:5:100;
[xxi, yyi] = ndgrid(xi,yi);
P = [xxi(:), yyi(:)];
% interpolation algorithm
dx = x(2) - x(1);
dy = y(2) - y(1);
x_ = [x(1)-dx x x(end)+dx x(end)+2*dx];
y_ = [y(1)-dy y y(end)+dy y(end)+2*dy];
s_ = [s(1) s(1,:) s(1,end) s(1,end)
s(:,1) s s(:,end) s(:,end)
s(end,1) s(end,:) s(end,end) s(end,end)
s(end,1) s(end,:) s(end,end) s(end,end)];
si = P(:,1)*0;
M = 1/6*[-1 3 -3 1
3 -6 3 0
-3 0 3 0
1 4 1 0];
tic
for nn = 1:numel(P(:,1))
u = mod(P(nn,1)- x_(1), dx)/dx;
jj = floor((P(nn,1) - x_(1))/dx) + 1;
v = mod(P(nn,2)- y_(1), dy)/dy;
ii = floor((P(nn,2) - y_(1))/dy) + 1;
D = [s_(jj-1,ii-1) s_(jj-1,ii) s_(jj-1,ii+1) s_(jj-1,ii+2)
s_(jj,ii-1) s_(jj,ii) s_(jj,ii+1) s_(jj,ii+2)
s_(jj+1,ii-1) s_(jj+1,ii) s_(jj+1,ii+1) s_(jj+1,ii+2)
s_(jj+2,ii-1) s_(jj+2,ii) s_(jj+2,ii+1) s_(jj+2,ii+2)];
U = [u.^3 u.^2 u 1];
V = [v.^3 v.^2 v 1];
si(nn) = U*M*D*M'*V';
end
toc
scatter3(P(:,1), P(:,2), si)
hold on
mesh(xx,yy,s)
This is the full example and is a cubic B-spline surface interpolation algorithm in 2D space.
I asked a question a few days before but I guess it was a little too complicated and I don't expect to get any answer.
My problem is that I need to use ANN for classification. I've read that much better cost function (or loss function as some books specify) is the cross-entropy, that is J(w) = -1/m * sum_i( yi*ln(hw(xi)) + (1-yi)*ln(1 - hw(xi)) ); i indicates the no. data from training matrix X. I tried to apply it in MATLAB but I find it really difficult. There are couple things I don't know:
should I sum each outputs given all training data (i = 1, ... N, where N is number of inputs for training)
is the gradient calculated correctly
is the numerical gradient (gradAapprox) calculated correctly.
I have following MATLAB codes. I realise I may ask for trivial thing but anyway I hope someone can give me some clues how to find the problem. I suspect the problem is to calculate gradients.
Many thanks.
Main script:
close all
clear all
L = #(x) (1 + exp(-x)).^(-1);
NN = #(x,theta) theta{2}*[ones(1,size(x,1));L(theta{1}*[ones(size(x,1),1) x]')];
% theta = [10 -30 -30];
x = [0 0; 0 1; 1 0; 1 1];
y = [0.9 0.1 0.1 0.1]';
theta0 = 2*rand(9,1)-1;
options = optimset('gradObj','on','Display','iter');
thetaVec = fminunc(#costFunction,theta0,options,x,y);
theta = cell(2,1);
theta{1} = reshape(thetaVec(1:6),[2 3]);
theta{2} = reshape(thetaVec(7:9),[1 3]);
NN(x,theta)'
Cost function:
function [jVal,gradVal,gradApprox] = costFunction(thetaVec,x,y)
persistent index;
% 1 x x
% 1 x x
% 1 x x
% x = 1 x x
% 1 x x
% 1 x x
% 1 x x
m = size(x,1);
if isempty(index) || index > size(x,1)
index = 1;
end
L = #(x) (1 + exp(-x)).^(-1);
NN = #(x,theta) theta{2}*[ones(1,size(x,1));L(theta{1}*[ones(size(x,1),1) x]')];
theta = cell(2,1);
theta{1} = reshape(thetaVec(1:6),[2 3]);
theta{2} = reshape(thetaVec(7:9),[1 3]);
Dew = cell(2,1);
DewApprox = cell(2,1);
% Forward propagation
a0 = x(index,:)';
z1 = theta{1}*[1;a0];
a1 = L(z1);
z2 = theta{2}*[1;a1];
a2 = L(z2);
% Back propagation
d2 = 1/m*(a2 - y(index))*L(z2)*(1-L(z2));
Dew{2} = [1;a1]*d2;
d1 = [1;a1].*(1 - [1;a1]).*theta{2}'*d2;
Dew{1} = [1;a0]*d1(2:end)';
% NNRes = NN(x,theta)';
% jVal = -1/m*sum(NNRes-y)*NNRes*(1-NNRes);
jVal = -1/m*(a2 - y(index))*a2*(1-a2);
gradVal = [Dew{1}(:);Dew{2}(:)];
gradApprox = CalcGradApprox(0.0001);
index = index + 1;
function output = CalcGradApprox(epsilon)
output = zeros(size(gradVal));
for n=1:length(thetaVec)
thetaVecMin = thetaVec;
thetaVecMax = thetaVec;
thetaVecMin(n) = thetaVec(n) - epsilon;
thetaVecMax(n) = thetaVec(n) + epsilon;
thetaMin = cell(2,1);
thetaMax = cell(2,1);
thetaMin{1} = reshape(thetaVecMin(1:6),[2 3]);
thetaMin{2} = reshape(thetaVecMin(7:9),[1 3]);
thetaMax{1} = reshape(thetaVecMax(1:6),[2 3]);
thetaMax{2} = reshape(thetaVecMax(7:9),[1 3]);
a2min = NN(x(index,:),thetaMin)';
a2max = NN(x(index,:),thetaMax)';
jValMin = -1/m*(a2min-y(index))*a2min*(1-a2min);
jValMax = -1/m*(a2max-y(index))*a2max*(1-a2max);
output(n) = (jValMax - jValMin)/2/epsilon;
end
end
end
EDIT:
Below I present the correct version of my costFunction for those who may be interested.
function [jVal,gradVal,gradApprox] = costFunction(thetaVec,x,y)
m = size(x,1);
L = #(x) (1 + exp(-x)).^(-1);
NN = #(x,theta) L(theta{2}*[ones(1,size(x,1));L(theta{1}*[ones(size(x,1),1) x]')]);
theta = cell(2,1);
theta{1} = reshape(thetaVec(1:6),[2 3]);
theta{2} = reshape(thetaVec(7:9),[1 3]);
Delta = cell(2,1);
Delta{1} = zeros(size(theta{1}));
Delta{2} = zeros(size(theta{2}));
D = cell(2,1);
D{1} = zeros(size(theta{1}));
D{2} = zeros(size(theta{2}));
jVal = 0;
for in = 1:size(x,1)
% Forward propagation
a1 = [1;x(in,:)']; % added bias to a0
z2 = theta{1}*a1;
a2 = [1;L(z2)]; % added bias to a1
z3 = theta{2}*a2;
a3 = L(z3);
% Back propagation
d3 = a3 - y(in);
d2 = theta{2}'*d3.*a2.*(1 - a2);
Delta{2} = Delta{2} + d3*a2';
Delta{1} = Delta{1} + d2(2:end)*a1';
jVal = jVal + sum( y(in)*log(a3) + (1-y(in))*log(1-a3) );
end
D{1} = 1/m*Delta{1};
D{2} = 1/m*Delta{2};
jVal = -1/m*jVal;
gradVal = [D{1}(:);D{2}(:)];
gradApprox = CalcGradApprox(x(in,:),0.0001);
% Nested function to calculate gradApprox
function output = CalcGradApprox(x,epsilon)
output = zeros(size(thetaVec));
for n=1:length(thetaVec)
thetaVecMin = thetaVec;
thetaVecMax = thetaVec;
thetaVecMin(n) = thetaVec(n) - epsilon;
thetaVecMax(n) = thetaVec(n) + epsilon;
thetaMin = cell(2,1);
thetaMax = cell(2,1);
thetaMin{1} = reshape(thetaVecMin(1:6),[2 3]);
thetaMin{2} = reshape(thetaVecMin(7:9),[1 3]);
thetaMax{1} = reshape(thetaVecMax(1:6),[2 3]);
thetaMax{2} = reshape(thetaVecMax(7:9),[1 3]);
a3min = NN(x,thetaMin)';
a3max = NN(x,thetaMax)';
jValMin = 0;
jValMax = 0;
for inn=1:size(x,1)
jValMin = jValMin + sum( y(inn)*log(a3min) + (1-y(inn))*log(1-a3min) );
jValMax = jValMax + sum( y(inn)*log(a3max) + (1-y(inn))*log(1-a3max) );
end
jValMin = 1/m*jValMin;
jValMax = 1/m*jValMax;
output(n) = (jValMax - jValMin)/2/epsilon;
end
end
end
I've only had a quick eyeball over your code. Here are some pointers.
Q1
should I sum each outputs given all training data (i = 1, ... N, where
N is number of inputs for training)
If you are talking in relation to the cost function, it is normal to sum and normalise by the number of training examples in order to provide comparison between.
I can't tell from the code whether you have a vectorised implementation which will change the answer. Note that the sum function will only sum up a single dimension at a time - meaning if you have a (M by N) array, sum will result in a 1 by N array.
The cost function should have a scalar output.
Q2
is the gradient calculated correctly
The gradient is not calculated correctly - specifically the deltas look wrong. Try following Andrew Ng's notes [PDF] they are very good.
Q3
is the numerical gradient (gradAapprox) calculated correctly.
This line looks a bit suspect. Does this make more sense?
output(n) = (jValMax - jValMin)/(2*epsilon);
EDIT: I actually can't make heads or tails of your gradient approximation. You should only use forward propagation and small tweaks in the parameters to compute the gradient. Good luck!
I need to evaluate following expression (in pseudo-math notation):
∑ipi⋅n
where p is a matrix of three-element vectors and n is a three-element vector. I can do this with for loops as follows but I can't figure out
how to vectorize this:
p = [1 1 1; 2 2 2];
n = [3 3 3];
s = 0;
for i = 1:size(p, 1)
s = s + dot(p(i, :), n)
end
Why complicate things? How about simple matrix multiplication:
s = sum(p * n(:))
where p is assumed to be an M-by-3 matrix.
I think you can do it with bsxfun:
sum(sum(bsxfun(#times,p,n)))
----------
% Is it the same for this case?
----------
n = 200; % depending on the computer it might be
m = 1000*n; % that n needs to be chosen differently
A = randn(n,m);
x = randn(n,1);
p = zeros(m,1);
q = zeros(1,m);
tic;
for i = 1:m
p(i) = sum(x.*A(:,i));
q(i) = sum(x.*A(:,i));
end
time = toc; disp(['time = ',num2str(time)]);