I want to compute below formula in Matlab (E-step of EM for Multinomial Mixture Model),
g and θ are matrix , θ and λ have below constrains:
but count of m is more than 1593 and when compute product of θ, number get very small and Matlab save it with zero.
Anyone can simplifying the g formula or use other tricks to solve this problem?
update:
data:
data.txt
(after downloads, change file extension to 'mat')
code:
function EM(data)
%% initialize
K=2;
[N M]=size(data);
g=zeros(N,K);
landa=ones(K,1) .* 0.5;
theta = rand(M, K);
theta = bsxfun(#rdivide, theta, sum(theta,1))';
%% EM
for i=1:10
%% E Step
for n=1:N
normalize=0;
for k=1:K
g(n,k)=landa(k) * prod(theta(k,:) .^ data(n,:));
normalize=normalize + landa(k) * prod(theta(k,:) .^ data(n,:));
end
g(n,:)=g(n,:) ./ normalize;
end
%% M Step
for k=1:K
landa(k)=sum(g(:,k)) / N ;
for m=1:M
theta(k,m)=(sum(g(:,k) .* data(:,m)) + 1) / (sum(g(:,k) .* sum(data,2)) + M);
end
end
end
end
You can use computations on logarithms instead of the actual values to avoid underflow problems.
To start things, we slightly reformat the E step code:
for n = 1 : N
for k = 1 : K
g(n, k) = lambda(k) * prod(theta(k, :) .^ data(n, :));
end
end
g = bsxfun(#rdivide, g, sum(g, 2));
So instead of accumulating the denominator in an extra variable normalize, we do the normalization in one step after both loops.
Now we introduce a variable lg with contains the logarithm of g:
for n = 1 : N
for k = 1 : K
lg(n, k) = log(lambda(k)) + sum(log(theta(k, :)) .* data(n, :));
end
end
g = exp(lg);
g = bsxfun(#rdivide, g, sum(g, 2));
So far, nothing is achieved. The underflow is just moved from within the loop to the conversion from lg to g via the exponential afterwards.
But, in the next line there is the normalization step, which means that the correct value of g is not really necessary: All that is important is that different values have the correct ratios between them. This means we can divide all values that jointly enter a normalization by an arbitrary constant, without changing the end result. On the logarithmic scale, this means subtracting something, and we choose this something to be the arithmetic mean of lg (corresponding to the harmonic mean of g):
lg = bsxfun(#minus, lg, mean(lg, 2));
g = exp(lg);
g = bsxfun(#rdivide, g, sum(g, 2));
Via the subtraction, logarithmic values are moved from something like -2000, which doesn't survive the exponential, to something like +50 or -30. Values of g now are sensible, and can be easily normalized to reach the correct end result.
Related
I'm trying to run a program in matlab to obtain the direct and inverse DFT for a grey scale image, but I'm not able to recover the original image after applying the inverse. I'm getting complex numbers as my inverse output. Is like i'm losing information. Any ideas on this? Here is my code:
%2D discrete Fourier transform
%Image Dimension
M=3;
N=3;
f=zeros(M,N);
f(2,1:3)=1;
f(3,1:3)=0.5;
f(1,2)=0.5;
f(3,2)=1;
f(2,2)=0;
figure;imshow(f,[0 1],'InitialMagnification','fit')
%Direct transform
for u=0:1:M-1
for v=0:1:N-1
for x=1:1:M
for y=1:1:N
F(u+1,v+1)=f(x,y)*exp(-2*pi*(1i)*((u*(x-1)/M)+(v*(y-1)/N)));
end
end
end
end
Fab=abs(F);
figure;imshow(Fab,[0 1],'InitialMagnification','fit')
%Inverse Transform
for x=0:1:M-1
for y=0:1:N-1
for u=1:1:M
for v=1:1:N
z(x+1,y+1)=(1/M*N)*F(u,v)*exp(2*pi*(1i)*(((u-1)*x/M)+((v-1)*y/N)));
end
end
end
end
figure;imshow(real(z),[0 1],'InitialMagnification','fit')
There are a couple of issues with your code:
You are not applying the definition of the DFT (or IDFT) correctly: you need to sum over the original variable(s) to obtain the transform. See the formula here; notice the sum.
In the IDFT the normalization constant should be 1/(M*N) (not 1/M*N).
Note also that the code could be made mucho more compact by vectorization, avoiding the loops; or just using the fft2 and ifft2 functions. I assume you want to compute it manually and "low-level" to verify the results.
The code, with the two corrections, is as follows. The modifications are marked with comments.
M=3;
N=3;
f=zeros(M,N);
f(2,1:3)=1;
f(3,1:3)=0.5;
f(1,2)=0.5;
f(3,2)=1;
f(2,2)=0;
figure;imshow(f,[0 1],'InitialMagnification','fit')
%Direct transform
F = zeros(M,N); % initiallize to 0
for u=0:1:M-1
for v=0:1:N-1
for x=1:1:M
for y=1:1:N
F(u+1,v+1) = F(u+1,v+1) + ...
f(x,y)*exp(-2*pi*(1i)*((u*(x-1)/M)+(v*(y-1)/N))); % add term
end
end
end
end
Fab=abs(F);
figure;imshow(Fab,[0 1],'InitialMagnification','fit')
%Inverse Transform
z = zeros(M,N);
for x=0:1:M-1
for y=0:1:N-1
for u=1:1:M
for v=1:1:N
z(x+1,y+1) = z(x+1,y+1) + (1/(M*N)) * ... % corrected scale factor
F(u,v)*exp(2*pi*(1i)*(((u-1)*x/M)+((v-1)*y/N))); % add term
end
end
end
end
figure;imshow(real(z),[0 1],'InitialMagnification','fit')
Now the original and recovered image differ only by very small values, of the order of eps, due to the usual floating-point inaccuacies:
>> f-z
ans =
1.0e-15 *
Columns 1 through 2
0.180411241501588 + 0.666133814775094i -0.111022302462516 - 0.027755575615629i
0.000000000000000 + 0.027755575615629i 0.277555756156289 + 0.212603775716506i
0.000000000000000 - 0.194289029309402i 0.000000000000000 + 0.027755575615629i
Column 3
-0.194289029309402 - 0.027755575615629i
-0.222044604925031 - 0.055511151231258i
0.111022302462516 - 0.111022302462516i
Firstly, the biggest error is that you are computing the Fourier transform incorrectly. When computing F, you need to be summing over x and y, which you are not doing. Here's how to rectify that:
F = zeros(M, N);
for u=0:1:M-1
for v=0:1:N-1
for x=1:1:M
for y=1:1:N
F(u+1,v+1)=F(u+1,v+1) + f(x,y)*exp(-2*pi*(1i)*((u*(x-1)/M)+(v*(y-1)/N)));
end
end
end
end
Secondly, in the inverse transform, your bracketing is incorrect. It should be 1/(M*N) not (1/M*N).
As an aside, at the cost of a bit more memory, you can speed up the computation by not nesting so many loops. Namely, when computing the FFT, do the following instead
x = (1:1:M)'; % x is a column vector
y = (1:1:N) ; % y is a row vector
for u = 0:1:M-1
for v = 0:1:N-1
F2(u+1,v+1) = sum(f .* exp(-2i * pi * (u*(x-1)/M + v*(y-1)/N)), 'all');
end
end
To take this method to the extreme, i.e. not using any loops at all, you would do the following (though this is not recommended, since you would lose code readability and the memory cost would increase exponentially)
x = (1:1:M)'; % x is in dimension 1
y = (1:1:N) ; % y is in dimension 2
u = permute(0:1:M-1, [1, 3, 2]); % x-freqs in dimension 3
v = permute(0:1:N-1, [1, 4, 3, 2]); % y-freqs in dimension 4
% sum the exponential terms in x and y, which are in dimensions 1 and 2.
% If you are using r2018a or older, the below summation should be
% sum(sum(..., 1), 2)
% instead of
% sum(..., [1,2])
F3 = sum(f .* exp(-2i * pi * (u.*(x-1)/M + v.*(y-1)/N)), [1, 2]);
% The resulting array F3 is 1 x 1 x M x N, to make it M x N, simply shiftdim or squeeze
F3 = squeeze(F3);
Before the question, thank you for reading this post.
I have defined a real function, f= #(x) sin(x) + x.^2 and I want to store the N translates of f from 0 to N-1, in a vector w. I mean, I want to store the vector w=( sin(x) + x.^2 , ...., sin(x- (N-1)) + (x-(N-1)).^2)
I have a vector, v, of N real numbers.
Question: How to get the inner product, u, of v and w? I want to store it as a function because I want to calculate the FFT coefficients of u.
I'm trying to do this with a loop but I'm making a mess with the loop/s.
Would you help me, please?
Many thanks.
What I did with the help of #m7913d:
%%%code
N=10;
v=rand(1,N);%%vector
tras=1:N;
tsi=t.*ones(1,N);
%%ff(x)=sin(x)+x.^2
ff=sin(tsi+tras) +(tsi+tras).^2 ;
total=sum(ff.*v,2);
vpa(subs(total,t,0.1));%%check
Fs=100;% Sampling frequency
tt=-pi:1/Fs:pi;% Time vector
L=length(tt); % Signal length
X=double(subs(total, t,tt));
n=2^nextpow2(L);
coef=fft(X,n);
You can generate a cell array of functions as follows:
N = 10;
f= #(x) sin(x) + x.^2;
u = cell(N, 1);
v = rand(N, 1);
g = 0;
for i=0:(N-1)
u{i+1} = #(x) v(i+1)*f(x + i);
g = #(x) g(x) + u{i+1}(x);
end
Note that I calculated the u vector at once, without using the intermediate variable w.
Note that the calculations will be easier (and faster) if you immediately insert your desired x vector. In that case, you do not have to hassle with function handles.
The question is solved. Here is the code
%%%code
N=10;
v=rand(1,N);%%vector
tras=1:N;
tsi=t.*ones(1,N);
%%ff(x)=sin(x)+x.^2
ff=sin(tsi+tras) +(tsi+tras).^2 ;
total=sum(ff.*v,2);
vpa(subs(total,t,0.1));%%check
Fs=100;% Sampling frequency
tt=-pi:1/Fs:pi;% Time vector
L=length(tt); % Signal length
X=double(subs(total, t,tt));
n=2^nextpow2(L);
coef=fft(X,n);
Thanks to everyone for your help.
I am aiming to minimize the below cost function over W
J = (E)^2
E = A - W .* B
Such that W(n+1) = W(n) - (u/2) * delJ
delJ = gradient of J = -2 * E .* B
u = step_size=0.2
where:
- A, B are STFT matrix of 2 audio signals (dimension is 257x4000 for a 16s audio with window size = 256 , 75% overlap, nfft=512)
- W is a matrix constructed with [257x1] vector repeated 4000 times (so that it become 257x4000] matrix
I have written my customize function as below. The problem is, the elements in A and B are so small (~e-20) that even after 1000 iteration that no change happens to g.
I am certainly missing something, if anybody can help or guide me to some link which explains the whole process for a new person.
[M,N] = size(A);
E =#(x) A - repmat(x,1,N) .* B; % Error Function
J = #(x) E(x) .^ 2; % Cost Function
G = #(x) -2 * E(x) .* B; % Gradiant Function
alpha = .2; % Step size
maxiter = 500; % Max iteration
dwmin = 1e-6; % Min change in gradiation
tolerence = 1e-6; % Max Gradiant norm
gnorm = inf;
w = rand(M,1);
dw = inf;
for i = 1:maxiter
g = G(w);
gnorm = norm(g);
wnew = w - (alpha/2)*g(:,1);
dw = norm(wnew-w)
if or(dw < dwmin, gnorm < tolerence)
break
end
end
w = wnew;
A & B are always positive real numbered vectors.
Your problem is actually a series of independent problems. If we index each row of A and B and each element of w with i, then minimizing the sum of squares across the error matrix
A - repmat(w, 1, N) .* B
is the same as minimizing the the sum of squares across the error vector
A(i, :) - w(i) * B(i, :)
for all rows separately. The latter problem can be solved using one of the least-squares operators of Matlab, in particular mrdivide or /:
for i = 1 : M
w(i) = A(i, :) / B(i, :);
end
As far as I can tell, there is no way to further vectorize this calculation.
In any case, there is no need to use a gradient descent or other form of optimization algorithm.
I'm pretty confused on how I would go about summing an infinite amount of matrices in MATLAB. Lets say I have this function (a gaussian):
%Set up grid/coordinate system
Ngrid=400;
w=Ngrid;
h=Ngrid;
%Create Gaussian Distribution
G = zeros ([w, h]);
Sig = 7.3; %I want the end/resultant G to be a summation of Sign from 7.3 to 10 with dx
for x = 1 : w
for y = 1 : h
G (x, y) = exp (-((Sig^-2)*((x-w/2+1)^2 + (y-h/2+1)^2)) / (2));
end
end
I essentially want the end/resultant function G to be a summation of Sign from 7.3 to 10 with dx (which is infinitesimally) small ie integration. How would I go about doing this? I am pretty confused. Can it even be done?
You don't appear to actually be summing G over a range of Sig values. You never change the value of Sig. In any case, assuming that dx isn't too small and that you have the memory this can be done without any loops, let alone two.
Ngrid = 400;
w = Ngrid;
h = Ngrid;
% Create range for Sig
dx = 0.1;
Sig = 7.3:dx:10;
% Build mesh of x and y points
x = 1:w;
y = 1:h;
[X,Y] = meshgrid(x,y);
% Evaluate columnized mesh points at each value of Sig, sum up, reshape to matrix
G = reshape(sum(exp(bsxfun(#rdivide,-((X(:)-w/2+1).^2+(Y(:)-h/2+1).^2),2*Sig.^2)),2),[h w]);
figure
imagesc(G)
axis equal
This results in a figure like this
The long complicated line above can be replaced by this (uses less memory, but may be slower):
G = exp(-((X-w/2+1).^2+(Y-h/2+1).^2)/(2*Sig(1)^2));
for i = 2:length(Sig)
G = G+exp(-((X-w/2+1).^2+(Y-h/2+1).^2)/(2*Sig(i)^2));
end
I'm trying to get Matlab to take this as a function of x_1 through x_n and y_1 through y_n, where k_i and r_i are all constants.
So far my idea was to take n from the user and make two 1×n vectors called x and y, and for the x_i just pull out x(i). But I don't know how to make an arbitrary sum in MATLAB.
I also need to get the gradient of this function, which I don't know how to do either. I was thinking maybe I could make a loop and add that to the function each time, but MATLAB doesn't like that.
I don't believe a loop is necessary for this calculation. MATLAB excels at vectorized operations, so would something like this work for you?
l = 10; % how large these vectors are
k = rand(l,1); % random junk values to work with
r = rand(l,1);
x = rand(l,1);
y = rand(l,1);
vals = k(1:end-1) .* (sqrt(diff(x).^2 + diff(y).^2) - r(1:end-1)).^2;
sum(vals)
EDIT: Thanks to #Amro for correcting the formula and simplifying it with diff.
You can solve for the gradient symbolically with:
n = 10;
k = sym('k',[1 n]); % Create n variables k1, k2, ..., kn
x = sym('x',[1 n]); % Create n variables x1, x2, ..., xn
y = sym('y',[1 n]); % Create n variables y1, y2, ..., yn
r = sym('r',[1 n]); % Create n variables r1, r2, ..., rn
% Symbolically sum equation
s = sum((k(1:end-1).*sqrt((x(2:end)-x(1:end-1)).^2+(y(2:end)-y(1:end-1)).^2)-r(1:end-1)).^2)
grad_x = gradient(s,x) % Gradient with respect to x vector
grad_y = gradient(s,y) % Gradient with respect to y vector
The symbolic sum and gradients can be evaluated and converted to floating point with:
% n random data values for k, x, y, and r
K = rand(1,n);
X = rand(1,n);
Y = rand(1,n);
R = rand(1,n);
% Substitute in data for symbolic variables
S = double(subs(s,{[k,x,y,r]},{[K,X,Y,R]}))
GRAD_X = double(subs(grad_x,{[k,x,y,r]},{[K,X,Y,R]}))
GRAD_Y = double(subs(grad_y,{[k,x,y,r]},{[K,X,Y,R]}))
The gradient function is the one overloaded for symbolic variables (type help sym/gradient) or see the more detailed documentation online).
Yes, you could indeed do this with a loop, considering that x, y, k, and r are already defined.
n = length(x);
s = 0;
for j = 2 : n
s = s + k(j-1) * (sqrt((x(j) - x(j-1)).^2 + (y(j) - y(j-1)).^2) - r(j-1)).^2
end
You should derive the gradient analytically and then plug in numbers. It should not be too hard to expand these terms and then find derivatives of the resulting polynomial.
Vectorized solution is something like (I wonder why do you use sqrt().^2):
is = 2:n;
result = sum( k(is - 1) .* abs((x(is) - x(is-1)).^2 + (y(is) - y(is-1)).^2 - r(is-1)));
You can either compute gradient symbolically or rewrite this code as a function and make a standard +-eps calculation. If you need a gradient to run optimization (you code looks like a fitness function) you could use algorithms that calculate them themselves, for example, fminsearch can do this