I am trying to solve the following optimization problem in octave
The first contraint is that A be positive semi-definite.
S is a set of data points such that if (xi,xj) is in S then xi is similar to xj and D is a set of data points such that if (xi,xj) is in D then xi and xj are dissimilar. Note that the above formula is 2 separate sums and the second sum is not nested. Also xi and xj are assumed to be column vectors of length N.
Because this is a nonlinear optimization I am trying to use octave's nonlinear program solver, sqp.
The problem is that if I just provide it with the function to optimize, on some small toy tests the, BFGS method to find the Hessian
fails. Because of this I tried to provide my own Hessian function but now this problem occurs
error: __qp__: operator *: nonconformant arguments (op1 is 2x2, op2 is 3x1)
error: called from:
error: /usr/share/octave/3.6.3/m/optimization/qp.m at line 393, column 26
error: /usr/share/octave/3.6.3/m/optimization/sqp.m at line 414, column 32
when I make the following call to sqp
[A, ~, Info] = sqp(initial_guess, {#toOpt, #CalculateGradient,#CalculateHessian},
[],[],0,[],maxiter);
I simplified the constraint that A be positive semi-definite and diagonal by only solving for the diagonal entries and constraining all the diagonal entries to be >=0. initial_guess is a vector of ones that is N long.
Here is my code to calculate what I believe to be the Hessian matrix
%Hessian = CalculateHessian(A)
%calculates the Hessian of the function we are optimizing as follows
%H(i,j) = (sumsq(D(:,i),1) * sumsq(D(:,j),1)) / (sum(A.*sumsq(D,1))^2)
%where D is a matrix of of differences between observations that are dissimilar, with one difference on each row
%and sumsq is the sum of the squares
%input A: the current guess for A
%output Hessian: The hessian of the function we are optimizing
function Hessian = CalculateHessian(A)
global HessianNumerator; %this is a matrix with the numerator of H(i,j)
global Dsum_of_squares; %the sum of the squares of the differences of each dimensions of the dissimilar observations
if(iscolumn(A)) %if A is a column vector
A = A'; %make it a row vector. necessary to prevent broadcasting
endif
if(~isempty(Dsum_of_squares)) %if disimilar constraints were provided
Hessian = HessianNumerator / (sum(A.*Dsum_of_squares)^2)
else
Hessian = HessianNumerator; %the hessian is a matrix of 0s
endif
endfunction
and Dsum_of_squares and HessianNumertor are
[dissimilarRow,dissimilarColumn] = find(D); %find which observations are dissimilar to each other
DissimilarDiffs = X(dissimilarRow,:) - X(dissimilarColumn,:); %take the difference between the dissimilar observations
Dsum_of_squares = sumsq(DissimilarDiffs,1);
HessianNumerator = Dsum_of_squares .* Dsum_of_squares'; %calculate the numerator of the Hessian. it is a constant value
X is a M x N matrix with one observation per row.
D is a M x M dissimilarity matrix. if D(i,j) is 1 then row i of X is dissimlar to row j. 0 otherwise.
I believe my error is in one of the following areas (from least likely to most likely)
The math I used to derive the Hessian function is wrong. The formula I am using is in my comments for the function.
My implementation of the math.
The Hessian Matrix that sqp wants is different from the one described on the Hessian Matrix Wikipedia page.
Any help would be greatly appreciated. If you need me to post more code I would be happy to do so. Right now the amount of code to try and solve the optimization is about 160 lines.
Here is the test case I am running that causes the code to fail. It works if I only pass it the gradient function.
X = [1 2 3;
4 5 6;
7 8 9;
10 11 12];
S = [0 1 1 0;
1 0 0 0;
1 0 0 0;
0 0 0 0]; %this means row 1 of X is similar to rows 2 and 3
D = [0 0 0 0;
0 0 0 0;
0 0 0 1;
0 0 1 0]; %this means row 3 of X is dissimilar to row 4
gml(X,S,D, 200); %200 is the maximum number of iterations for sqp to run
Related
As a part of project I have to construct a cubic spline with natural boundary conditions without using any built-in MATLAB functions such as spline or csape.
I tried programming the following function.
While I'm pretty sure it's correct up to the point where it calculates the coefficients q, I can't figure out how I will eventually get the actual cubic polynomials. What I am getting right now as an outpout when calling the function is 9 distinct values for S.
Any help or hints would be appreciated.
function S=cubic_s(x,y)
n=length(x);
%construction of the tri-diagonal matrix
for j=1:n
V(j,1)=1;
V(j,2)=4;
V(j,3)=1;
end
%the first row should be (1,0,...,0) and the last (0,0,...,0,1)
V(1,2)=1; V(n,2)=1; V(2,3)=0; V(n-1,1)=0;
d=[-1 0 1];
A=spdiags(V,d,n,n);
%construction of the vector b
b=zeros(n,1);
%the first and last elements of b must equal 0
b(1)=0; b(n)=0;
%distance between two consecutive points
h=(x(n)-x(1))/(n-1);
for j=2:n-1
b(j,1)=(6/h^2)*(y(j+1)-2*y(j)+y(j-1));
end
%solving for the coefficients q
q=A\b;
%finding the polynomials with the formula for the cubic spline
for j=1:n-1
for z=x(j):0.01:x(j+1)
S(j)=(q(j)/(6*h))*(x(j+1)-z)^3+(q(j+1)/(6*h))*(z-x(j))^3+(z-x(j))* (y(j+1)/h-(q(j+1)*h)/6)+(x(j+1)-z)*(y(j)/h-(q(j)*h)/6);
end
end
You should save S every z-time, see picture and code below
function plot_spline
x = (0:10);
y = [1 4 3 7 1 5 2 1 6 2 3];
xx = x(1):0.01:x(2);
[XX,YY]=cubic_s(x,y);
plot(x,y,'*r', XX,YY,'-k')
function [XX,YY]=cubic_s(x,y)
n=length(x);
%construction of the tri-diagonal matrix
for j=1:n
V(j,1)=1;
V(j,2)=4;
V(j,3)=1;
end
%the first row should be (1,0,...,0) and the last (0,0,...,0,1)
V(1,2)=1; V(n,2)=1; V(2,3)=0; V(n-1,1)=0;
d=[-1 0 1];
A=spdiags(V,d,n,n);
%construction of the vector b
b=zeros(n,1);
%the first and last elements of b must equal 0
b(1)=0; b(n)=0;
%distance between two consecutive points
h=(x(n)-x(1))/(n-1);
for j=2:n-1
b(j,1)=(6/h^2)*(y(j+1)-2*y(j)+y(j-1));
end
%solving for the coefficients q
q=A\b;
%finding the polynomials with the formula for the cubic spline
enum = 1;
for j=1:n-1
for z=x(j):0.01:x(j+1)
YY(enum)=(q(j)/(6*h))*(x(j+1)-z)^3+(q(j+1)/(6*h))*(z-x(j))^3+(z-x(j))* (y(j+1)/h-(q(j+1)*h)/6)+(x(j+1)-z)*(y(j)/h-(q(j)*h)/6);
XX(enum)=z;
enum = enum+1;
end
end
I am new to MATLAB's environment and no matter how much I have struggled it just seems that I cannot get the concept of how to construct a ML algorithm for a multivariate Bernoulli.
I have a dataset of N variables (x1,x2,...,xN) and each variable is a vector of D dimensions (Dx1), with a parameter vector in the form p=(p1,p2,...,pD) . So the Bernoulli distribution should have the form:
Pr(X|p)=Πp(d)^x(nd)*(1-p(d))^(1-x(nd))
The code that I created uses MATLAB's mle function:
for n=1:D
prob(n)=mle(dataset(:,n),'distribution', 'bernoulli');
end
which gives me a D vector of estimated probabilities from the dataset.
But, what I am really interested in is how to implement the ML on a step-by-step MATLAB process and not just use the mle.
Thank you very much.
phat for a Bernoulli distribution is proportion of successes to the number of trials. If you'd like to do it manually, you can just count the number of successes (either 1 or 0) in each of your vectors then divide it by the length of the vector. Here's a quick way assuming 1's are successes stored vertically in the matrix.
bern_mat = [0 0 1 0 1 1; 1 1 0 1 0 0 ; 1 0 1 0 1 1]; % 3x6 matrix of 1's and 0's
phat = sum(bern_mat,1)/size(bern_mat,1); % sum across the first dim then divide by size of first dim.
I was trying to get an vector optimized version of the linear rectifier. i.e. y = max(0,x). So what it should compute its element wise max of zero and x_i. I obviously implemented:
function [ y ] = rectSig( x )
%rectSig computes vector-wise rectified linear function
% computes y = [..., max(0,x_i), ...]
n=length(x);
y = zeros(1,n);
for i=1:1:length(x);
y(i) = max(0,x(i));
end
end
however, I know that looping like this in MATLAB is ill advised. So I was wondering if there was a better way to do this or if obviously matlab had its own implementation of a vectorized version of such a function? I always try to avoid loops if I can in matlab if there is a way to vectorize my code. It usually tends to speed things up.
Btw, I obviously tried googling it but didn't really get the result I expected...
The solution is as simple as
y = max(x,0);
This works for x being a column, row vector, matrix, higher dimensional matrix, etc. On the other hand
y = max(zeros(1,length(x)),x);
only works for x being a row vector. It fails when x is a column vector or matrix.
max accepts matrix inputs:
x = -5:5;
comparisonvector = zeros(size(x));
y = max(comparisonvector, x);
Returns:
y =
0 0 0 0 0 0 1 2 3 4 5
I'm trying to solve a system of DAE (mixture of ODE and Algebraic Eq). So, my system looks like this
M dy/dt = F(t,y)
The matrix M is basically a constant mass matrix 1 in the diagonal with all zeros on the few last lines and few last column.
so for an example mass matrix looks like this
M = [1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 0]
my vector Fx is in dimension (4x1) if we take the matrix example.
How do I proceed if I get this error?
Matrix is singular, close to singular or badly scaled. Results may be inaccurate. RCOND = NaN
my code for the system resolution :
Mass = Mass_Matrix( Nc,NT );
opt = odeset('RelTol', 10.0^(-3), 'AbsTol' , 10.0^(-3), 'Mass', Mass ,'MassSingular', 'yes', 'OutputFcn',#odeprint);
[T,Y] = ode15s(#(t,y)Function_Fx( t, y, Resolution, y01),[0 1],y0,opt);
and my vector Fx is created by the Function_Fx
function Fx = Function_Fx( t, y, Resolution, y01)
With Resolution being a struct variables, and y01 a stored values.
My question is where could be the problem, is it the F(t,y) or the M? Given M is constant mass matrix.
Thank you for your help
I've solved this problem. It was because my F(t,y) is wrong. I checked the jacobian only by then I managed to see that I have some lines of zeros.
I am using Matlab to find the spectral radius of the Jacobi iteration matrix where A=[4 2 1;1 3 1;1 1 4].
I can't seem to input the correct commands to get the size of the error after 5 iterations. Can someone help me?
Here are a list of commands that I put into Matlab so far:
A=[4 2 1;1 3 1;1 1 4]
A =
4 2 1
1 3 1
1 1 4
D=diagonal(diagonal(A));L=(A,-1);U=(A,1);
b=([3 -1 4])
x0j=zeros([0 0 0]);
x=D\(-(U+L)*x0j+b);r=b-A*x %Jacobi iteration.
------------------------------------------------------------------------------
Error using *
Inputs must be 2-D, o enter code here r at least one input must be scalar.
To compute element wise TIMES, use TIMES (.*) instead.
The spectral radius of a matrix is the maximum of the modulus of its eigenvalues. It can be simply computed using max(abs(eig(·))).
However, as others have noticed, your whole code seems pretty mixed up and not actually valid Matlab code, so your problem is not really to compute the spectral radius, is it? The algorithm is very straightforward and easy to implement:
% diagonal part of A and rest
D = diag(diag(A));
R = A - D;
% iteration matrix and offset
T = - inv(D) * R;
C = inv(D) * b;
% spectral radius condition
rho = max(abs(eig(T)));
if rho >= 1
error('no convergence')
end
% initial guess
x = randn(size(b));
% iteration
while norm(A * x - b) > 1e-15
x = T * x + C;
end
Note that I used inv(D) to directly follow the description in Wikipedia, but the inverse of a diagonal matrix can be easily computed using diag(1 ./ diag(D)).
I don't really see why one would need to separate R into an upper and lower part. I suppose it has to do with numerical efficiency, but then, Matlab is a very efficient high-level language for matrix computations already. So actually there is no need to implement the Jacobi algorithm in it explicitly when you can simply write A \ b – except for educational purposes I guess.