I have a set of data that is generated from an uniform distribution. Now I want to fit the corresponding histogram to an uniform distribution, such that there is a 'ㄇ' shape of line plotted on that histogram. I tried to fit it by using the MATLAB built-in function histfit, but there is no such an option of uniform distribution for histfit. How can I do it?
data = unifrnd(-100,100,1000,1);
%% MATLAB built-in function: 'histfit'
figure(1);
hh = histfit(data); % No options for 'histfit' to fit data to an uniform distribution
%% Manually fitting a histogram to an uniform distribution
figure(2);
numBars = length(hh(1).XData);
histogram(data, numBars);
% TODO: How to do next to plot a line that fits the data to an uniform distribution?
Since a uniform distribution is just 1/(b-a) for a region [a, b), you can define a function that calculates this
x = -200:200;
y = unifdist(x, -100, 100);
figure;
plot(x, y)
function ret = unifdist(x, a, b)
ret = ones(length(x), 1)/(b-a);
ret(x < a | x >= b) = 0;
end
There's probably a much simpler and faster way to do this, but it works.
min(data) and max(data) give an estimate of the two parameters of the uniform distribution, assuming data is uniformly distributed.
Note that this is a biased estimator, see here for a correction to remove the bias (one of the answers considers the case where the lower bound is not 0). [Link thanks to #flawr.]
As an alternative to #CrisLuengo's answer, you could use the Method of Moments to estimate the parameters a,b of the uniform distribution U[a,b]. Following equations are sufficient to solve for the parameters. The MoM just tells us to equate the (sample-) mean and variance to the distribution mean and variance:
mean(samples) = (a+b)/2, variance(samples) = (b-a)^2/12
This results in a, b = mean(samples) +- sqrt(3 * variance(samples)). In MATLAB you can compute this as follows:
m = mean(data);
v = var(data);
a = m - sqrt(3*v);
b = m + sqrt(3*v);
To plot this you can just define vector
x = linspace(-lower_limit, upper_limit, number_of_points);
y = (a < x) .* (x < b) ./ (b - a);
plot(x, y, '-r');
Related
I am trying to run (stochastic or batch) gradient descent when one uses the standard cross entropy (softmax loss):
when using as a model the Radial Basis Function (RBF) network (you can watch a lecture form caltech here if you want) when extended to multiclass classification (easily extended by simply feeding the output of the RBF network to a softmax layer. Note that P(y=l|x) is simply computed by passing the output of the RBF network through a softmax layer for each label l as follows:
where \theta_l indexes the parameters responsible for doing predictions for label l.
In this regard I wanted to optimize my model by computing the derivatives with respect to the parameters. Recall that the parameters to optimize in a radial basis function network are the weights c at the final layer and the centers t at the first layer. I have implemented and debugged how to compute the derivative with respect to the weights c. The code works as expected because the partial derivatives match the numerical derivatives. You can find the unit test code is here.
I have also tried writing the code that implements the derivative with respect to the centers but I just can't seem to make my implementation of the derivative match the numerical derivatives. The equation of the derivative of the loss J with respect to the centers t_k that I am trying implementing is as follow:
where h_{\theta_l} corresponds to the output of the RBF that is in charge of predicting label l. In fact, h_{\theta_l} is very simple to express:
My main issue is with computing the derivative of J with respect to t_k (the equation above). For that, I have implemented the following function that naively computes it without vectorizing it:
function [ dJ_dt ] = compute_dJ_dt(z,x,y,t,c)
%Computes dJ_dc
% Input:
% z = (K x 1)
% x = data point (D, 1)
% y = labels (1 x 1)
% t = centers (D x K)
% c = weights (K x L)
% Output:
% dJ_dc = (D x K)
[D,K] = size(t);
[~, L] = size(c);
dJ_dt = zeros(D, K);
for k=1:K
dJ_dt_k = zeros(D, 1);
for l=1:L
c_l = c(:,l);
dh_dt_l = compute_dh_dt(z,x,t,c_l); %(D x K)
delta = (y==l);
dJ_dt_k = dJ_dt_k + dh_dt_l(:,k) * delta;
end
dJ_dt(:,k) = -dJ_dt_k;
end
end
and it does not match the numerical derivatives code.
I have tried different things to check if it works and I will explain them all here. If anyone has additional ideas, feel free to share them, I sort of feel I ran out of good new ideas to try to debug this.
First a good natural question is, is my mathematical derivation of the derivative I am trying to implement correct? Even though I have not explicitly checked the mathematical derivation with someone, I have high confidence that its correct because the derivation for the partial derivative with respect to c and t in the model are identical and you only change the symbol \theta to whatever parameter you have in question. Since I have already implemented the derivative with respect to c and it passes all my derivative tests, I would assume the derivation of the derivative with respect to t or any parameter \theta should be correct. One can see my derivation of this equation at in math.stack exchange here.
One option could be that compute_dJ_dt doesn't actually implement the equation that I am expecting. This could indeed be the case and to check that I independently implemented a slightly more vectorized version of that code to see if I was actually implementing the equation that I had down on paper. Since the two version of the equation output the same derivative values, I have high confidence that they are computing, indeed the equation that I suspect (also if anyone has a way of further vectorizing this equation, that would be awesome! The vectorization I added is so trivial that it doesn't seem that interesting or that much of a performance gain, though, it does remove one for loop).
Since the equation I have on paper is (with high probability) correct and the implementation of the equation seems to be correct since two version of it output the same value, then it leads me to conclude that maybe, the numerical derivative code has a bug.
The numerical derivative code is so ridiculously simple that its hard to check what on earth could be wrong with it. The only thing that occurred to me that could be wrong is that it could be that my implementation of the softmax cost J is wrong, but I highly doubt it since...I already wrote a unit test for it! Plus, I use it to check the numerical derivatives with respect to c and the ones for c ALWAYS pass, so I can't imagine J to be wrong.
The last non-trivial thing to check is that compute_dh_dt is being computed correctly. I have written units tests for dh_dt and since they match their corresponding numerical derivatives on every run, I would suspect that code is correct.
At this point I am not 100% sure what else to try to I am hoping maybe someone has a good idea or maybe points out the stupid thing I am doing? I am not sure what to think right now. Thanks for the help and time community!
This is sort of a anti-climactic resolution, but I guess it was to be expected because this code seemed to be built from working components so it was bound to be a silly small mistake. The mistake was the in the code above that I pasted I should have been using the delta as the difference between indication of label and probability of that label but I forgot to subtract the probability. So the code above is:
delta = (y==l);
when it should have been:
prob_y_x_h_x = prob_y_x(h_x); % (L x 1)
ind_y_l = (y==l);
delta = ind_y_l - prob_y_x_h_x(l);
So the fixed code now passes the numerical tests and it looks as follow:
function [ dJ_dt ] = compute_dJ_dt(h_x,z,x,y,t,c)
%Computes dJ_dc
% Input:
% z = (K x 1)
% x = data point (D, 1)
% y = labels (1 x 1)
% t = centers (D x K)
% c = weights (K x L)
% Output:
% dJ_dc = (D x K)
[D,K] = size(t);
[~, L] = size(c);
dJ_dt = zeros(D, K);
for k=1:K
dJ_dt_k = zeros(D, 1);
for l=1:L
c_l = c(:,l);
dh_dt_l = compute_dh_dt(z,x,t,c_l); %(D x K)
prob_y_x_h_x = prob_y_x(h_x); % (L x 1)
ind_y_l = (y==l);
delta = ind_y_l - prob_y_x_h_x(l);
dJ_dt_k = dJ_dt_k + dh_dt_l(:,k) * delta;
end
dJ_dt(:,k) = -dJ_dt_k;
end
end
I still don't know how to further vectorize the code above, so I am still happy to receive feedback on that part of the question! Here is the vectorization I have so far:
function [ dJ_dt ] = compute_dJ_dt_vec(h_x,z,x,y,t,c)
%Computes dJ_dc
% Input:
% z = (K x 1)
% x = data point (D, 1)
% y = labels (1 x 1)
% t = centers (D x K)
% c = weights (K x L)
% Output:
% dJ_dc = (D x K)
[D,K] = size(t);
[~, L] = size(c);
dJ_dt = zeros(D, K);
for l=1:L
c_l = c(:,l);
dh_dt = compute_dh_dt(z,x,t,c_l); %(D x K)
ind_y_l = (y==l);
prob_y_x_h_x = prob_y_x(h_x); % (L x 1)
dJ_dh = repmat( ind_y_l - prob_y_x_h_x(l) , D, K); %(D x K)
dJ_dt = dJ_dt + dJ_dh.*dh_dt;
end
dJ_dt = -dJ_dt;
end
Given a M x M desired covariance, R, and a desired number of sample vectors, N calculate a N x M Gaussian random vector, X in vanilla MATLAB (i.e. can't use r = mvnrnd(MU,SIGMA,cases)).
Not really sure how to tackle this, usually you need a covariance AND mean to generate a Gaussian random variable. I think sqrtm and chol could be useful.
If you have access to the MATLAB statistics toolbox you can type edit mvnrnd in MATLAB to see their solution.
[T p] = chol(sigma);
if m1 == c
mu = mu';
end
mu = mu(ones(cases,1),:);
r = randn(cases,c) * T + mu;
It feels almost like cheating to point this out, but editing MATLAB's source is very useful to understand things in general. You can also search for mvnrnd.m on google if you don't have the toolbox.
Example:
% Gaussian mean and covariance
d = 2; % number of dimensions
mu = rand(1,d);
sigma = rand(d,d); sigma = sigma*sigma';
% generate 100 samples from above distribution
num = 100;
X = mvnrnd(mu, sigma, num);
% plot samples (only for 2D case)
scatter(X(:,1), X(:,2), 'filled'), hold on
ezcontour(#(x,y) mvnpdf([x y], mu, sigma), xlim(), ylim())
title('X~N(\mu,\sigma)')
xlabel('X_1'), ylabel('X_2')
The above code uses functions from the Statistics toolbox (mvnrnd and mvnpdf). If you don't have access to it, consider these replacements (using the same concepts mentioned by others):
mvnrnd = #(mu,S,num) bsxfun(#plus, randn(num,numel(mu))*cholcov(S), mu);
mvnpdf = #(x,mu,S) exp(-0.5*(x-mu)*(S\(x-mu)')) / sqrt((2*pi)^d*det(S));
I have an ellipse in 2 dimensions, defined by a positive definite matrix X as follows: a point x is in the ellipse if x'*X*x <= 1. How can I plot this ellipse in matlab? I've done a bit of searching while finding surprisingly little.
Figured out the answer actually: I'd post this as an answer, but it won't let me (new user):
Figured it out after a bit of tinkering. Basically, we express the points on the ellipse border (x'*X*x = 1) as a weighted combination of the eigenvectors of X, which makes some of the math to find the points easier. We can just write (au+bv)'X(au+bv)=1 and work out the relationship between a,b. Matlab code follows (sorry it's messy, just used the same notation that I was using with pen/paper):
function plot_ellipse(X, varargin)
% Plots an ellipse of the form x'*X*x <= 1
% plot vectors of the form a*u + b*v where u,v are eigenvectors of X
[V,D] = eig(X);
u = V(:,1);
v = V(:,2);
l1 = D(1,1);
l2 = D(2,2);
pts = [];
delta = .1;
for alpha = -1/sqrt(l1)-delta:delta:1/sqrt(l1)+delta
beta = sqrt((1 - alpha^2 * l1)/l2);
pts(:,end+1) = alpha*u + beta*v;
end
for alpha = 1/sqrt(l1)+delta:-delta:-1/sqrt(l1)-delta
beta = -sqrt((1 - alpha^2 * l1)/l2);
pts(:,end+1) = alpha*u + beta*v;
end
plot(pts(1,:), pts(2,:), varargin{:})
I stumbled across this post while searching for this topic, and even though it's settled, I thought I might provide another simpler solution, if the matrix is symmetric.
Another way of doing this is to use the Cholesky decomposition of the semi-definite positive matrix E implemented in Matlab as the chol function. It computes an upper triangular matrix R such that X = R' * R. Using this, x'*X*x = (R*x)'*(R*x) = z'*z, if we define z as R*x.
The curve to plot thus becomes such that z'*z=1, and that's a circle. A simple solution is thus z = (cos(t), sin(t)), for 0<=t<=2 pi. You then multiply by the inverse of R to get the ellipse.
This is pretty straightforward to translate into the following code:
function plot_ellipse(E)
% plots an ellipse of the form xEx = 1
R = chol(E);
t = linspace(0, 2*pi, 100); % or any high number to make curve smooth
z = [cos(t); sin(t)];
ellipse = inv(R) * z;
plot(ellipse(1,:), ellipse(2,:))
end
Hope this might help!
I want generate a number in Gaussian and Uniform distributions in matlab.
I know this function randi and rand() but all of them are in normal (Gaussian) distribution. How can a generate a random number in uniform distribution?
Use rand(dimensions) for a Uniform Distribution between 0 and 1.
Use randn(dimensions) * sqrt(sigma) + mu for a Gaussian Distribution with a mean of mu and standard deviation of sigma.
randn is the function to generate Gaussian distributed variables (randi and rand produce uniformly distributed ones).
You can generate any distribution from rand().
For example , lets say you want to generate 100000 samples for rayleigh dist.The way to do this is that you invert the cdf of that particular function.The basic idea is that since the cdf has to be between 0 and 1 , we can find the value of the random variable by inputting the value of cdf b/w 0 and 1. So for rayleigh, it would be
for i = 1:100000
data(i) = (2*sigma^2 *(-(log(1 - rand(1,1)))))^.5;
end
You can do something similar for gaussian distribution.
Congrulations, you already generating pseudo-random numbers with a gaussian distribution. Normal distribution is a synonym for it.
The only other possible interpretation I can get from your question is that you want something that has mean != 0 and/or variance != 1. To do that, simply perform mean + sqrt(var) * randn(X).
It is true you can generate just about anything from rand but that it isn't always convenient, especially for some complicated distributions.
MATLAB has introduced Probability Distribution Objects which make this a lot easier and allow you to seamlessly access mean, var, truncate, pdf, cdf, icdf (inverse transform), median, and other functions.
You can fit a distribution to data. In this case, we use makedist to define the probability distribution object. Then we can generate using random.
% Parameters
mu = 10;
sigma = 3;
a = 5; b = 15;
N = 5000;
% Older Approaches Still Work
rng(1775)
Z = randn(N,1); % Standard Normal Z~N(0,1)
X = mu + Z*sigma; % X ~ Normal(mu,sigma)
U = rand(N,1); % U ~ Uniform(0,1)
V = a + (b-a)*U; % V ~ Uniform(a,b)
% New Approaches Are Convenient
rng(1775)
pdX = makedist('Normal',mu,sigma);
X2 = random(pdX,N,1);
pdV = makedist('Uniform',a,b);
V2 = random(pdV,N,1);
A reproducible example:
Support = (0:0.01:20)';
figure
s(1) = subplot(2,2,1)
h(1) = histogram(X,'Normalization','pdf')
xlabel('Normal')
s(2) = subplot(2,2,2)
h(2) = histogram(V,'Normalization','pdf')
xlabel('Uniform')
s(3) = subplot(2,2,3), hold on, box on
h(3) = histogram(X2,'Normalization','pdf')
plot(Support,pdf(pdX,Support),'r-','LineWidth',1.2)
xlabel('Normal (new)')
s(4) = subplot(2,2,4), hold on, box on
h(4) = histogram(V2,'Normalization','pdf')
plot(Support,pdf(pdV,Support),'r-','LineWidth',1.2)
xlabel('Uniform (new)')
xlim(s,[0 20])
References:
Uniform Distribution
Normal (Gaussian) Distribution
Following raj's answer: by using the Box-Muller Transform you can generate independent standard Normal/Gaussian random numbers:
N = 1e6; z = sqrt(-2*log(rand(N, 1))) .* cos(2*pi * rand(N, 1)); figure; hist(z, 100)
N = 1e6; z = sqrt(-2*log(rand(N, 1))) .* sin(2*pi * rand(N, 1)); figure; hist(z, 100)
If you want to apply the Inverse Transformation Method, you can use the Inverse Complementary Error Function (erfcinv):
N = 1e6; z = -sqrt(2) * erfcinv(2 * rand(1e6, 1)); figure; hist(z, 100)
But I hope randn works better.
By using randn function I want to create a Gaussian random variable X such that X ~ N(2,4) and plot this simulated PDF together with theoretic curve.
Matlab randn generates realisations from a normal distribution with zero mean and a standard deviation of 1.
Samples from any other normal distribution can simply be generated via:
numSamples = 1000;
mu = 2;
sigma = 4;
samples = mu + sigma.*randn(numSamples, 1);
You can verify this by plotting the histogram:
figure;hist(samples(:));
See the matlab help.
N = 1000;
x = [-20:20];
samples = 2 + 4*randn(N, 1);
ySamples = histc(samples,x) / N;
yTheoretical = pdf('norm', x, 2, 4);
plot(x, yTheoretical, x, ySamples)
randn(N, 1) creates an N-by-1 vector.
histc is histogram count by bins given in x - you can use hist to plot the result immediately, but here we want to divide it by N.
pdf contains many useful PDFs, normal is just one example.
remember this: X ~ N(mean, variance)
randn in matlab produces normal distributed random variables W with zero mean and unit variance.
To change the mean and variance to be the random variable X (with custom mean and variance), follow this equation:
X = mean + standard_deviation*W
Please be aware of that standard_deviation is square root of variance.
N = 1000;
x = [-20:20];
samples = 2 + sqrt(4)*randn(N, 1);
ySamples = histc(samples,x) / N;
yTheoretical = pdf('norm', x, 2, sqrt(4)); %put std_deviation not variance
plot(x, yTheoretical, x, ySamples)
A quick and easy way to achieve this using one line of code is to use :
mu = 2;
sigma = 2;
samples = normrnd(mu,sigma,M,N);
This will generate an MxN matrix, sampled from N(μ,𝜎), (= N(2,2) in this particular case).
For additional information, see normrnd.