I am currently using fmincon for minimizing a log-Likelihood function in respect to a 18*18 matrix. While on smaller problems the algorithm is very fast, it takes about 2h to converge in the current setup - as I am iterating over this minimisation problem, running through the code may take up to 2 weeks.
Is there a matlab-based, free alternative to fmincon that improves speed on such specific problems? (Costly solutions are discussed here, non-matlab solutions here.) Or would I need to call e.g. a python script from matlab?
The function I want to minimize:
function [L] = logL(A, U, Sigma_e, T, lags)
% A - parameters to optimize w.r.t
logL = 0;
for t = 1 : T - lags
logL(t, 1) = 0.5*(log(det(A * diag(Sigma_e(t,:)) * A' ) ) + ...
U(t,:) * (A * diag(Sigma_e(t,:)) * A' )^(-1) * U(t,:)' );
end
L = sum(logL);
and calling it by:
Options = optimset('Algorithm', 'active-set', 'Display', 'off', 'Hessian','bfgs', ...
'DerivativeCheck','on','Diagnostics','off','GradObj','off','LargeScale','off');
A = fmincon( #(A0)logL(A0, U, Sigma_e, T, lags), A0 , [], [] , [] , [] , [] , [] , [], Options);
(I have tried the different fmincon algorithms without much improvement). Note, T is quite large ~3000.
A and A0 are a 18*18 matrices,
Sigma_e is T*18,
U is T*18
I'm not aware of any speedy alternative to fminconst but you can vectorize the logL function to speed up the algorithm. Here is a vectorized version:
function [L] = logL(A, U, Sigma_e, T, lags)
ia = inv(A);
iat = ia.';
N = T - lags;
UU = zeros(N,1);
for t = 1: N
UU (t) = U(t,:) * (iat .* 1./Sigma_e(t,:) * ia) * U(t,:)';
end
L = 0.5 *sum( log(det(A) ^ 2 .* prod(Sigma_e(1:N,:),2)) + UU);
end
In some tests in Octave it nearly is 10X faster than the your solution.
Note that if some elements of Sigma_e are equal to zero you need to compute UU as:
UU (t)=U(t,:) * (A * diag(Sigma_e(t,:)) * A' )^(-1) * U(t,:)';
These relations are used to convert the loop solution to the vectorized one:
det(a * b * c) == det(a) * det(b) * det(c)
det(a) == det(a.')
det(diag(a)) == prod(a)
(a * b * c)^-1 == c^-1 * b^-1 * a^-1
a * diag(b) == a .* b
inv(diag(a)) == diag(1./a)
Related
2nd ODE to solve in MATLAB:
( (a + f(t))·d²x/dt² + (b/2 + k(t))·dx/dt ) · dx/dt - g(t) = 0
Boundary condition:
dx/dt(0) = v0
where
t is the time,
x is the position
dx/dt is the velocity
d2x/dt2 is the acceleration
a, b, v0 are constants
f(t), k(t) and h(t) are KNOWN functions dependent on t
(I do not write them because they are quite big)
As an example, using symbolic variables:
syms t y
%% --- Initial conditions ---
phi = 12.5e-3;
v0 = 300;
e = 3e-3;
ro = 1580;
E = 43e9;
e_r = 0.01466;
B = 0.28e-3;
%% --- Intermediate calculations ---
v_T = sqrt(((1 + e_r) * 620e6) /E) - sqrt(E/ro) * e_r;
R_T = v_T * t;
m_acc = pi * e * ro *(R_T^2);
v_L = sqrt (E/ro);
R_L = v_L * t;
z = 2 * R_L;
E_4 = B * ((e_r^2)* B * (0.9^(z/B)-1)) /(log(0.9));
E_1 = E * e * pi * e_r^2 * (-phi* (phi - 2*v_T*t)) /16;
E_2 = pi * R_T^2 * 10e9;
E_3 = pi * R_T^2 * 1e6 * e;
%% Resolution of the problem
g_t = -diff(E_1 + E_2 + E_3, t);
f(t,y)=(g_t - (pi*v_T*e*ro/2 + E_4) * y^2 /(y* (8.33e-3 + m_acc))];
fun=matlabFunction(f);
[T,Y]=ode45(fun,[0 1], v0]);
How can I rewrite this to get x as y=dx/dt? I'm new to Matlab and any help is very welcome !
First, you chould use subs to evaluate a symbolic function. Another approach is to use matlabFunction to convert all symbolic expressions to anonymous functions, as suggested by Horchler.
Second, you're integrating the ODE as if it is 1st order in dx/dt. If you're interested in x(t) as well as dx/dt(t), then you'll have to modify the function like so:
fun = #(t,y) [y(2);
( subs(g) - (b/2 + subs(k))*y(2)*y(2) ) / ( y(2) * (a + subs(f))) ];
and of course, provide an initial value for x0 = x(0) as well as v0 = dx/dt(0).
Third, the absolute value of the parameters is hardly ever a real concern. IEEE754 double-precision floating point format can effortlessly represent numbers between 2.225073858507201e-308 and 1.797693134862316e+308 (realmin and realmax, respectively). So for the coefficients you gave (O(1014)), this is absolutely not a problem. You might lose a few digits of precision if you don't take precautions (rescale to [-1 +1], reformulate the problem in different units, ...), but the relative error due to this is more than likely to be tiny and insignificant compared to the algorithmic error made by ode45.
<RANDOM_OPINIONATED_RANT>
Fourth, WHY do you use symbolic math for this purpose?! You are doing a numerical integration, meaning, there is no analytic solution anyway. Why bother with symbolics then? Doing the integration with symbolics (through vpa even) is going to be dozens, hundreds, yes, often even thousands of times slower than keeping (or re-implementing) everything numerical (which some would argue is already slow in MATLAB compared to a bare-metal approach).
Yes, of course, for this specific, individual, isolated use case it may not matter much, but for the future I'd strongly advise you to learn to:
use symbolics for derivations, proving theorems, simplifying expressions, ...
use numerics to implement any algorithm or function from which actual numbers are expected.
In other words, symbolics for drafting, numerics for crunching. And exactly zero symbolics should appear in any good implementation of any algorithm.
Although it's possible to mix them to some extent, that does not mean it is a good idea to do so. In fact, that's almost never. And the few isolated cases where it is the only viable option are not a vindication of the approach.
They are rare, isolated cases after all, far from the abundant norm.
For me it bears resemblance with the evil eval, with similar reasons for why it Should. Be. Avoided.
</RANDOM_OPINIONATED_RANT>
With the full code, it's easy to come up with a complete solution:
% Initial conditions
phi = 12.5e-3;
v0 = 300;
x0 = 0; % (my assumption)
e = 3e-3;
ro = 1580;
E = 43e9;
e_r = 0.01466;
B = 0.28e-3;
% Intermediate calculations
v_T = sqrt(((1 + e_r) * 620e6) /E) - sqrt(E/ro) * e_r;
R_T = #(t) v_T * t;
m_acc = #(t) pi * e * ro *(R_T(t)^2);
v_L = sqrt (E/ro);
R_L = #(t) v_L * t;
z = #(t) 2 * R_L(t);
E_4 = #(t) B * ((e_r^2)* B * (0.9^(z(t)/B)-1)) /(log(0.9));
% UNUSED
%{
E_1 = #(t) -phi * E * e * pi * e_r^2 * (phi - 2*v_T*t) /16;
E_2 = #(t) pi * R_T(t)^2 * 10e9;
E_3 = #(t) pi * R_T(t)^2 * 1e6 * e;
%}
% Resolution of the problem
g_t = #(t) -( phi * E * e * pi * e_r^2 * v_T / 8 + ... % dE_1/dt
pi * 10e9 * 2 * R_T(t) * v_T + ... % dE_2/dt
pi * 1e6 * e * 2 * R_T(t) * v_T ); % dE_3/dt
% The derivative of Z = [x(t); x'(t)] equals Z' = [x'(t); x''(t)]
f = #(t,y)[y(2);
(g_t(t) - (0.5*pi*v_T*e*ro + E_4(t)) * y(2)^2) /(y(2) * (8.33e-3 + m_acc(t)))];
% Which is readily integrated
[T,Y] = ode45(f, [0 1], [x0 v0]);
% Plot solutions
figure(1)
plot(T, Y(:,1))
xlabel('t [s]'), ylabel('position [m]')
figure(2)
plot(T, Y(:,2))
xlabel('t [s]'), ylabel('velocity [m/s]')
Results:
Note that I've not used symbolics anywhere, except to double-check my hand-derived derivatives.
I'm trying to implement a RK implicit 2-order to convection-diffusion equation (1D) with fdm_2nd and gauss butcher coefficients: 'u_t = -uu_x + nu .u_xx' .
My goal is to compare the explit versus implcit scheme. The explicit rk which works well with a little number of viscosity. The curve of explicit schem show us a very nice shock wave.
I need your help to implement correctly the solver of the k(i) coefficient. I don't see how implement the newton method for all k(i).
do I need to implement it for all time-space steps ? or just in time ? The jacobian is maybe wrong but i don't see where. Or maybe i use the jacobian in wrong direction...
Actualy, my code works, but i think it's was wrong somewhere ... also the implicit curve does not move from the initial values.
here my function :
function [t,u] = burgers(t0,U,N,dx)
nu=0.01; %coefficient de viscosité
A=(diag(zeros(1,N))-diag(ones(1,N-1),1)+diag(ones(1,N-1),-1)) / (2*dx);
B=(-2*diag(ones(1,N))+diag(ones(1,N-1),1)+diag(ones(1,N-1),-1)) / (dx).^2;
t=t0;
u = - A * U.^2 + nu .* B * U;
the jacobian :
function Jb = burJK(U,dx,i)
%Opérateurs
a(1,1) = 1/4;
a(1,2) = 1/4 - (3).^(1/2) / 6;
a(2,1) = 1/4 + (3).^(1/2) / 6;
a(2,2) = 1/4;
Jb(1,1) = a(1,1) .* (U(i+1,1) - U(i-1,1))/ (2*dx) - 1;
Jb(1,2) = a(1,2) .* (U(i+1,1) - U(i-1,1))/ (2*dx);
Jb(2,1) = a(2,1) .* (U(i+1,2) - U(i-1,2))/ (2*dx);
Jb(2,2) = a(2,2) .* (U(i+1,2) - U(i-1,2))/ (2*dx) - 1;
Here my newton-code:
iter = 1;
iter_max = 100;
k=zeros(2,N);
k(:,1)=[0.4;0.6];
[w_1,f1] =burgers(n + c(1) * dt,uu + dt * (a(1,:) * k(:,iter)),iter,dx);
[w_2,f2] =burgers(n + c(2) * dt,uu + dt * (a(2,:) * k(:,iter)),iter,dx);
f1 = -k(1,iter) + f1;
f2 = -k(1,iter) + f2;
f(:,1)=f1;
f(:,2)=f2;
df = burJK(f,dx,iter+1);
while iter<iter_max-1 % K_newton
delta = df\f(iter,:)';
k(:,iter+1) = k(:,iter) - delta;
iter = iter+1;
[w_1,f1] =burgers(n + c(1) * dt,uu + dt * (a(1,:) * k(:,iter+1)),N,dx);
[w_2,f2] =burgers(n + c(2) * dt,uu + dt * (a(2,:) * k(:,iter+1)),N,dx);
f1 = -k(1,iter+1) + f1;
f2 = -k(1,iter+1) + f2;
f(:,1)=f1;
f(:,2)=f2;
df = burJK(f,dx,iter);
if iter>iter_max
disp('#');
else
disp('ok');
end
end
I'm a little rusty on exactly how to implement this in matlab, but I can walk your through the general steps and hopefully that will help. First we can consider the equation you are solving to fit the general class of problems that can be posed as
du/dt = F(u), where F is a linear or nonlinear function
For a Runge Kutta scheme you typically recast the problem something like this
k(i) = F(u+dt*a(i,i)*k(i)+ a(i,j)*k(j))
for a given stage. Now comes the tricky part, you you need to make 1-D vector constructed by stacking k(1) onto k(2). So the first half of the elements of the vector are k(1) and the second half are k(2). With this new combined vector you can then change F So that it operates on the two k's separately. This results in
K = FF(u+dt*a*K) where FF is F for the new double k vector, K
Ok, now we can implement the Newton's method. You will do this for each time step and until you have converged on the right answer and use it across all spatial points at the same time. What you do is you guess a K and compute the jacobian of G(K,U) = K-FF(FF(u+dt*a*K). G(K,U) should be only valued at zero when K is at the right solution. So in other words, do you Newton's method on K and when looking for convergence you need to see that it is converging at all spots. I would run the newton's method until max(abs(G(K,U)))< SolverTolerance.
Sorry I can't be more help on the matlab implementation, but hopefully I helped with explaining how to implement the newton's method.
I want to write my own 2 Dimensional DFT function with reduced loops.
What I try to implement is Discrete Fourier Transform:
Using the separability property of transform (actually exponential function), we can write this as multiplication of two 1 dimensional DFT. Then, we can calculate the exponential terms for rows (the matrix wM below) and columns (the matrix wN below) of transform. Then, for summation process we can multiply them as "F = wM * original_matrix * wN"
Here is the code I wrote:
f = imread('cameraman.tif');
[M, N, ~] = size(f);
wM = zeros(M, M);
wN = zeros(N, N);
for u = 0 : (M - 1)
for x = 0 : (M - 1)
wM(u+1, x+1) = exp(-2 * pi * 1i / M * x * u);
end
end
for v = 0 : (N - 1)
for y = 0 : (N - 1)
wN(y+1, v+1) = exp(-2 * pi * 1i / N * y * v);
end
end
F = wM * im2double(f) * wN;
The first thing is I dont want to use 2 loops which are MxM and NxN times running. If I used a huge matrix (or image), that would be a problem. Is there any chance to make this code faster (for example eliminating the loops)?
The second thing is displaying the Fourier Transform result. I use the codes below to display the transform:
% // "log" method
fl = log(1 + abs(F));
fm = max(fl(:));
imshow(im2uint8(fl / fm))
and
% // "abs" method
fa = abs(F);
fm = max(fa(:));
imshow(fa / fm)
When I use the "abs" method, I see only black figure, nothing else. What is wrong with "abs" method you think?
And the last thing is when I compare the transform result of my own function with MATLAB' s fft2() function', mine displays darker figure than MATLAB' s result. What am I missing here? Implementation misktake?
The transform result of my own function:
The transform result of MATLAB fft2() function:
I am happy you solved your problem but unfortunately you answer is not completely right. Indeed it does the job, but as I commented, im2double will normalize everything to 1, therefore showing the scaled result you have. What you want (if you are looking for performance) is not doing im2doubleand then multiply by 255, but directly casting to double().
You can eliminate loops by using meshgrid.
For example:
M = 1024;
tic
[ mX, mY ] = meshgrid( 0 : M - 1, 0 : M - 1 );
wM1 = exp( -2 * pi * 1i / M .* mX .* mY );
toc
tic
for u = 0 : (M - 1)
for x = 0 : (M - 1)
wM2( u + 1, x + 1 ) = exp( -2 * pi * 1i / M * x * u );
end
end
toc
all( wM1( : ) == wM2( : ) )
The timing on my system was:
Elapsed time is 0.130923 seconds.
Elapsed time is 0.493163 seconds.
I am trying to compute log(N(x | mu, sigma)) in MATLAB where
x is the data vector(Dimensions D x 1) , mu(Dimensions D x 1) is mean and sigma(Dimensions D x D) is covariance.
My present implementation is
function [loggaussian] = logmvnpdf(x,mu,Sigma)
[D,~] = size(x);
const = -0.5 * D * log(2*pi);
term1 = -0.5 * ((x - mu)' * (inv(Sigma) * (x - mu)));
term2 = - 0.5 * logdet(Sigma);
loggaussian = const + term1 + term2;
end
function y = logdet(A)
y = log(det(A));
end
For some cases I get an error
Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND =
NaN
I know you will point out that my data is not consistent, but I need to implement the function so that I can get the best approximate instead of throwing an warning. . How do I ensure that I always get a value.
I think the warning comes from using inv(Sigma). According to the documentation, you should avoid using inv where its use can be replaced by \ (mldivide). This will give you both better speed and accuracy.
For your code, instead of inv(Sigma) * (x - mu) use Sigma \ (x - mu).
The following approach should be (a little) less sensitive to ill-conditioning of the covariance matrix:
function logpdf = logmvnpdf (x, mu, K)
n = length (x);
R = chol (K);
const = 0.5 * n * log (2 * pi);
term1 = 0.5 * sum (((R') \ (x - mu)) .^ 2);
term2 = sum (log (diag (R)));
logpdf = - (const + term1 + term2);
end
If K is singular or near-singular, you can still have warnings (or errors) when calling chol.
I have written a simple matlab / octave function to create the sum of sinusoids with independent amplitude, frequency and phase for each component. Is there a cleaner way to write this?
## Create a sum of cosines with independent amplitude, frequency and
## phase for each component:
## samples(t) = SUM(A[i] * sin(2 * pi * F[i] * t + Phi[i])
## Return samples as a column vector.
##
function signal = sum_of_cosines(A = [1.0],
F = [440],
Phi = [0.0],
duration = 1.0,
sampling_rate = 44100)
t = (0:1/sampling_rate:(duration-1/sampling_rate));
n = length(t);
signal = sum(repmat(A, n, 1) .* cos(2*pi*t' * F + repmat(Phi, n, 1)), 2);
endfunction
In particular, the calls to repmat() seem a bit clunky -- is there some nifty vectorization technique waiting for me to learn?
Is this the same?
signal = cos(2*pi*t' * F + repmat(Phi, n, 1)), 2) * A';
And then maybe
signal = real(exp(j*2*pi*t'*F) * (A .* exp(j*Phi))');
If you are memory constrained, this should work nicely:
e_jtheta = exp(j * 2 * pi * F / sampling_rate);
phasor = A .* exp(j*Phi);
samples = zeros(duration,1);
for k = 1:duration
samples(k) = real((e_jtheta .^ k) * phasor');
end
For row vectors A, F, and Phi, so you can use bsxfun to get rid of the repmat, but it is arguably uglier looking:
signal = cos(bsxfun(#plus, 2*pi*t' * F, Phi)) * A';
Heh. When I call any of the above vectorized versions with length(A) = 10000, octave fills up VM and grinds to a halt (or at least, a slow crawl -- I haven't had the patience to wait for it to complete.
As a result, I've fallen back to the straightforward iterative version:
function signal = sum_of_cosines(A = [1.0],
F = [440],
Phi = [0.0],
duration = 1.0,
sampling_rate = 44100)
t = (0:1/sampling_rate:(duration-1/sampling_rate));
n = length(t);
signal = zeros(n, 1);
for i=1:length(A)
samples += A(i) * cos(2*pi*t'*F(i) + Phi(i));
endfor
endfunction
This version works plenty fast and teaches me a lesson about trying to be 'elegant'.
P.S.: This doesn't diminish my appreciation for the answers given by #BenVoigt and #chappjc -- I've learned something useful from both!