Related
How to normalize a histogram such that the area under the probability density function is equal to 1?
My answer to this is the same as in an answer to your earlier question. For a probability density function, the integral over the entire space is 1. Dividing by the sum will not give you the correct density. To get the right density, you must divide by the area. To illustrate my point, try the following example.
[f, x] = hist(randn(10000, 1), 50); % Create histogram from a normal distribution.
g = 1 / sqrt(2 * pi) * exp(-0.5 * x .^ 2); % pdf of the normal distribution
% METHOD 1: DIVIDE BY SUM
figure(1)
bar(x, f / sum(f)); hold on
plot(x, g, 'r'); hold off
% METHOD 2: DIVIDE BY AREA
figure(2)
bar(x, f / trapz(x, f)); hold on
plot(x, g, 'r'); hold off
You can see for yourself which method agrees with the correct answer (red curve).
Another method (more straightforward than method 2) to normalize the histogram is to divide by sum(f * dx) which expresses the integral of the probability density function, i.e.
% METHOD 3: DIVIDE BY AREA USING sum()
figure(3)
dx = diff(x(1:2))
bar(x, f / sum(f * dx)); hold on
plot(x, g, 'r'); hold off
Since 2014b, Matlab has these normalization routines embedded natively in the histogram function (see the help file for the 6 routines this function offers). Here is an example using the PDF normalization (the sum of all the bins is 1).
data = 2*randn(5000,1) + 5; % generate normal random (m=5, std=2)
h = histogram(data,'Normalization','pdf') % PDF normalization
The corresponding PDF is
Nbins = h.NumBins;
edges = h.BinEdges;
x = zeros(1,Nbins);
for counter=1:Nbins
midPointShift = abs(edges(counter)-edges(counter+1))/2;
x(counter) = edges(counter)+midPointShift;
end
mu = mean(data);
sigma = std(data);
f = exp(-(x-mu).^2./(2*sigma^2))./(sigma*sqrt(2*pi));
The two together gives
hold on;
plot(x,f,'LineWidth',1.5)
An improvement that might very well be due to the success of the actual question and accepted answer!
EDIT - The use of hist and histc is not recommended now, and histogram should be used instead. Beware that none of the 6 ways of creating bins with this new function will produce the bins hist and histc produce. There is a Matlab script to update former code to fit the way histogram is called (bin edges instead of bin centers - link). By doing so, one can compare the pdf normalization methods of #abcd (trapz and sum) and Matlab (pdf).
The 3 pdf normalization method give nearly identical results (within the range of eps).
TEST:
A = randn(10000,1);
centers = -6:0.5:6;
d = diff(centers)/2;
edges = [centers(1)-d(1), centers(1:end-1)+d, centers(end)+d(end)];
edges(2:end) = edges(2:end)+eps(edges(2:end));
figure;
subplot(2,2,1);
hist(A,centers);
title('HIST not normalized');
subplot(2,2,2);
h = histogram(A,edges);
title('HISTOGRAM not normalized');
subplot(2,2,3)
[counts, centers] = hist(A,centers); %get the count with hist
bar(centers,counts/trapz(centers,counts))
title('HIST with PDF normalization');
subplot(2,2,4)
h = histogram(A,edges,'Normalization','pdf')
title('HISTOGRAM with PDF normalization');
dx = diff(centers(1:2))
normalization_difference_trapz = abs(counts/trapz(centers,counts) - h.Values);
normalization_difference_sum = abs(counts/sum(counts*dx) - h.Values);
max(normalization_difference_trapz)
max(normalization_difference_sum)
The maximum difference between the new PDF normalization and the former one is 5.5511e-17.
hist can not only plot an histogram but also return you the count of elements in each bin, so you can get that count, normalize it by dividing each bin by the total and plotting the result using bar. Example:
Y = rand(10,1);
C = hist(Y);
C = C ./ sum(C);
bar(C)
or if you want a one-liner:
bar(hist(Y) ./ sum(hist(Y)))
Documentation:
hist
bar
Edit: This solution answers the question How to have the sum of all bins equal to 1. This approximation is valid only if your bin size is small relative to the variance of your data. The sum used here correspond to a simple quadrature formula, more complex ones can be used like trapz as proposed by R. M.
[f,x]=hist(data)
The area for each individual bar is height*width. Since MATLAB will choose equidistant points for the bars, so the width is:
delta_x = x(2) - x(1)
Now if we sum up all the individual bars the total area will come out as
A=sum(f)*delta_x
So the correctly scaled plot is obtained by
bar(x, f/sum(f)/(x(2)-x(1)))
The area of abcd`s PDF is not one, which is impossible like pointed out in many comments.
Assumptions done in many answers here
Assume constant distance between consecutive edges.
Probability under pdf should be 1. The normalization should be done as Normalization with probability, not as Normalization with pdf, in histogram() and hist().
Fig. 1 Output of hist() approach, Fig. 2 Output of histogram() approach
The max amplitude differs between two approaches which proposes that there are some mistake in hist()'s approach because histogram()'s approach uses the standard normalization.
I assume the mistake with hist()'s approach here is about the normalization as partially pdf, not completely as probability.
Code with hist() [deprecated]
Some remarks
First check: sum(f)/N gives 1 if Nbins manually set.
pdf requires the width of the bin (dx) in the graph g
Code
%http://stackoverflow.com/a/5321546/54964
N=10000;
Nbins=50;
[f,x]=hist(randn(N,1),Nbins); % create histogram from ND
%METHOD 4: Count Densities, not Sums!
figure(3)
dx=diff(x(1:2)); % width of bin
g=1/sqrt(2*pi)*exp(-0.5*x.^2) .* dx; % pdf of ND with dx
% 1.0000
bar(x, f/sum(f));hold on
plot(x,g,'r');hold off
Output is in Fig. 1.
Code with histogram()
Some remarks
First check: a) sum(f) is 1 if Nbins adjusted with histogram()'s Normalization as probability, b) sum(f)/N is 1 if Nbins is manually set without normalization.
pdf requires the width of the bin (dx) in the graph g
Code
%%METHOD 5: with histogram()
% http://stackoverflow.com/a/38809232/54964
N=10000;
figure(4);
h = histogram(randn(N,1), 'Normalization', 'probability') % hist() deprecated!
Nbins=h.NumBins;
edges=h.BinEdges;
x=zeros(1,Nbins);
f=h.Values;
for counter=1:Nbins
midPointShift=abs(edges(counter)-edges(counter+1))/2; % same constant for all
x(counter)=edges(counter)+midPointShift;
end
dx=diff(x(1:2)); % constast for all
g=1/sqrt(2*pi)*exp(-0.5*x.^2) .* dx; % pdf of ND
% Use if Nbins manually set
%new_area=sum(f)/N % diff of consecutive edges constant
% Use if histogarm() Normalization probability
new_area=sum(f)
% 1.0000
% No bar() needed here with histogram() Normalization probability
hold on;
plot(x,g,'r');hold off
Output in Fig. 2 and expected output is met: area 1.0000.
Matlab: 2016a
System: Linux Ubuntu 16.04 64 bit
Linux kernel 4.6
For some Distributions, Cauchy I think, I have found that trapz will overestimate the area, and so the pdf will change depending on the number of bins you select. In which case I do
[N,h]=hist(q_f./theta,30000); % there Is a large range but most of the bins will be empty
plot(h,N/(sum(N)*mean(diff(h))),'+r')
There is an excellent three part guide for Histogram Adjustments in MATLAB (broken original link, archive.org link),
the first part is on Histogram Stretching.
I am having trouble plotting a histogram of the x-values of my data points together with a line showing the relationship between x and y, mainly because the scale in the y direction of the histogram is not of the same magnitude as the scale in the line plot. For example:
% generate data
rng(1, 'twister')
x = randn(10000,1);
y = x.^2
% plot line, histogram, then histogram and line.
subplot(3,1,1)
scatter(x, y, 1, 'filled')
ax = gca;
maxlim = max(ax.XLim); % store maximum y-value to rescale histogram to this value
subplot(3,1,2)
h = histogram(x, 'FaceAlpha', 0.2)
subplot(3,1,3)
scatter(x, y, 1, 'filled')
hold on
h = histogram(x, 'FaceAlpha', 0.2)
Produces the following:
where the line chart is completely obscured by the histogram.
Now, one might naively try to rescale the histogram using:
h.Values = h.Values/max(h.Values) * maxlim;
which gives
You cannot set the read-only property 'Values' of Histogram.
Alternatively one can get the bin counts using histcounts, but as far as I can tell, the bar function does not allow one to set the face alpha or have other configurability as per the call to histogram.
As discussed in the comments there are several solutions that depend on the version of Matlab you are using. To restate the problem, the histogram function allows you to control many graphics properties like transparency, but only gives you a limited number of options to change the height of the bars. With histcounts you can get the bar heights and rescale them however you want, but you must plot the bars yourself.
First option: use histogram
As you cannot rescale the histogram heights, you must plot them on separate axis.
From release 2016a and onwards, you can use yyaxis left for the scatter plot and yyaxis right for the histogram, see Matlab documentation:
Prior to this one must manually create and set separate y-axis. Although I have not found a good simple example of this, this is perhaps the most relevant answer here: plot two histograms (using the same y-axis) and a line plot (using a different y-axis) on the same figure
Using histcounts and manually creating a bar chart
Using my example, we can get counts as follows:
[Values, Edges] = histcounts(x);
And rescaling:
Values = Values / max(Values) * maxlim;
and finding centres of bars:
bar_centres = 0.5*(Edges(1:end-1) + Edges(2:end));
Up to release 2014a, bar charts had a 'children' property for the patches that allows transparency to be controlled, e.g.:
% plot histogram
b1 = bar(bar_centres,Values);
% change transparency
set(get(b1,'Children'),'FaceAlpha',0.3)
After 2014a bar charts no longer have this property, and to get around it I plot the patches myself using the code from this mathworks q&a, replicated here:
function ptchs = createPatches(x,y,offset,c,FaceAlpha)
%createPatches.m
% This file will create a bar plot with the option for changing the
% FaceAlpha property. It is meant to be able to recreate the functionality
% of bar plots in versions prior to 2014b. It will create the rectangular
% patches with a base centered at the locations in x with a bar width of
% 2*offset and a height of y.
% Ensure x and y are numeric vectors
validateattributes(x,{'numeric'},{'vector'});
validateattributes(y,{'numeric'},{'vector'});
validateattributes(c,{'char'},{'scalar'});
%#TODO Allow use of vector c
% Check size(x) is same as size(y)
assert(all(size(x) == size(y)),'x and y must be same size');
% Default FaceAlpha = 1
if nargin < 5
FaceAlpha = 1;
end
if FaceAlpha > 1 || FaceAlpha <= 0
warning('FaceAlpha has been set to 1, valid range is (0,1]');
FaceAlpha = 1;
end
ptchs = cell(size(x)); % For storing the patch objects
for k = 1:length(x)
leftX = x(k) - offset; % Left Boundary of x
rightX = x(k) + offset; % Right Boundary of x
ptchs{k} = patch([leftX rightX rightX leftX],...
[0 0 y(k) y(k)],c,'FaceAlpha',FaceAlpha, ...
'EdgeColor', 'none');
end
end
I made one change: that is, imposed the no edge condition. Then, it is perfectly fine to use:
createPatches(bin_centres, Values, 1,'k', 0.2)
to create the bars.
I apologize for asking this, I believe this is a simple task, but I don't know how to do it.
Suppose I have a formula y = (exp(-x) + x^2)/sqrt(pi(x) and I want to plot it as y versus x^2.
How does one do this?
Like this:
X = 0:0.1:5; %// Get the x values
x = X.^2; %// Square them
%// Your formula had errors, I fixed them but I could have misinterpreted here, please check
y = (exp(-x) + x.^2)./sqrt(pi*x); %// Calculate y at intervals based on the squared x. This is still y = f(x), I'm just calculating it at the points at which I want to plot it.
plot(x,y) %//Plot against the square X.
At this point this is no different to having just plotted it normally. What you want is to make the tickmarks go up in values of X.^2. This does not change the y-values nor distort the function, it just changes what it looks like visually. Similar to plotting against a log scale:
set(gca, 'XTick', X.^2) %//Set the tickmarks to be squared
The second method gives you a plot like
edit:
Actually I think you were asking for this:
x = 0:0.1:5;
y = x.^2; %// Put your function in here, I'm using a simple quadratic for illustrative purposes.
plot(x.^2,y) %//Plot against the square X. Now your y values a f(x^2) which is wrong, but we'll fix that later
set(gca, 'XTick', (0:0.5:5).^2) %//Set the tickmarks to be a nonlinear intervals
set(gca, 'XTickLabel', 0:0.5:5) %//Cahnge the labels to be the original x values, now accroding to the plot y = f(x) again but has the shape of f(x^2)
So here I'm plotting a simple quadratic, but if I plot it against a squared x it should become linear. However I still want to read off the graph that y=x^2, not y=x, I just want it to look like y=x. So if I read the y value for the x value of 4 on that graph i will get 16 which is still the same correct original y value.
Here's my answer: it is similar to Dan's one, but fundamentally different. You can calculate the values of y as a function of x, but plot them as a function of x^2, which is what the OP was asking, if my understanding is correct:
x = 0:0.1:5; %// Get the x values
x_squared = x.^2; %// Square them
%// Your formula had errors, I fixed them but I could have misinterpreted here, please check
y = (exp(-x) + x.^2)./sqrt(pi*x); %// Calculate y based on x, not the square of x
plot(x_squared,y) %//Plot against the square of x
As Dan mentioned, you can always change the tickmarks:
x_ticks = (0:0.5:5).^2; % coarser vector to avoid excessive number of ticks
set(gca, 'XTick', x_ticks) %//Set the tickmarks to be squared
I have a matrix with x and y coordinates as well as the temperature values for each of my data points. When I plot this in a scatter plot, some of the data points will obscure others and therefore, the plot will not give a true representation of how the temperature varies in my data set.
To fix this, I would like to decrease the resolution of my graph and create pixels which represent the average temperature for all data points within the area of the pixel. Another way to think about the problem that I need to put a grid over the current plot and average the values within each segment of the grid.
I have found this thread - Generate a heatmap in MatPlotLib using a scatter data set - which shows how to use python to achieve the end result that I want. However, my current code is in MATLAB and even though I have tried different suggestions such as heatmap, contourf and imagesc, I can't get the result I want.
You can "reduce the resolution" of your data using accumarray, where you specify which output "bin" each point should go in and specify that you wish to take a mean over all points in that bin.
Some example data:
% make points that overlap a lot
n = 10000
% NOTE: your points do not need to be sorted.
% I only sorted so we can visually see if the code worked,
% see the below plot
Xs = sort(rand(n, 1));
Ys = rand(n, 1);
temps = sort(rand(n, 1));
% plot
colormap("hot")
scatter(Xs, Ys, 8, temps)
(I only sorted by Xs and temps in order to get the stripy pattern above so that we can visually verify if the "reduced resolution" worked)
Now, suppose I want to decrease the resolution of my data by getting just one point per 0.05 units in the X and Y direction, being the average of all points in that square (so since my X and Y go from 0 to 1, I'll get 20*20 points total).
% group into bins of 0.05
binsize = 0.05;
% create the bins
xbins = 0:binsize:1;
ybins = 0:binsize:1;
I use histc to work out which bin each X and Y is in (note - in this case since the bins are regular I could also do idxx = floor((Xs - xbins(1))/binsize) + 1)
% work out which bin each X and Y is in (idxx, idxy)
[nx, idxx] = histc(Xs, xbins);
[ny, idxy] = histc(Ys, ybins);
Then I use accumarray to do a mean of temps within each bin:
% calculate mean in each direction
out = accumarray([idxy idxx], temps', [], #mean);
(Note - this means that the point in temps(i) belongs to the "pixel" (of our output matrix) at row idxy(1) column idxx(1). I did [idxy idxx] as opposed to [idxx idxy] so that the resulting matrix has Y == rows and X == columns))
You can plot like this:
% PLOT
imagesc(xbins, ybins, out)
set(gca, 'YDir', 'normal') % flip Y axis back to normal
Or as a scatter plot like this (I plot each point in the midpoint of the 'pixel', and drew the original data points on too for comparison):
xx = xbins(1:(end - 1)) + binsize/2;
yy = ybins(1:(end - 1)) + binsize/2;
[xx, yy] = meshgrid(xx, yy);
scatter(Xs, Ys, 2, temps);
hold on;
scatter(xx(:), yy(:), 20, out(:));
How to normalize a histogram such that the area under the probability density function is equal to 1?
My answer to this is the same as in an answer to your earlier question. For a probability density function, the integral over the entire space is 1. Dividing by the sum will not give you the correct density. To get the right density, you must divide by the area. To illustrate my point, try the following example.
[f, x] = hist(randn(10000, 1), 50); % Create histogram from a normal distribution.
g = 1 / sqrt(2 * pi) * exp(-0.5 * x .^ 2); % pdf of the normal distribution
% METHOD 1: DIVIDE BY SUM
figure(1)
bar(x, f / sum(f)); hold on
plot(x, g, 'r'); hold off
% METHOD 2: DIVIDE BY AREA
figure(2)
bar(x, f / trapz(x, f)); hold on
plot(x, g, 'r'); hold off
You can see for yourself which method agrees with the correct answer (red curve).
Another method (more straightforward than method 2) to normalize the histogram is to divide by sum(f * dx) which expresses the integral of the probability density function, i.e.
% METHOD 3: DIVIDE BY AREA USING sum()
figure(3)
dx = diff(x(1:2))
bar(x, f / sum(f * dx)); hold on
plot(x, g, 'r'); hold off
Since 2014b, Matlab has these normalization routines embedded natively in the histogram function (see the help file for the 6 routines this function offers). Here is an example using the PDF normalization (the sum of all the bins is 1).
data = 2*randn(5000,1) + 5; % generate normal random (m=5, std=2)
h = histogram(data,'Normalization','pdf') % PDF normalization
The corresponding PDF is
Nbins = h.NumBins;
edges = h.BinEdges;
x = zeros(1,Nbins);
for counter=1:Nbins
midPointShift = abs(edges(counter)-edges(counter+1))/2;
x(counter) = edges(counter)+midPointShift;
end
mu = mean(data);
sigma = std(data);
f = exp(-(x-mu).^2./(2*sigma^2))./(sigma*sqrt(2*pi));
The two together gives
hold on;
plot(x,f,'LineWidth',1.5)
An improvement that might very well be due to the success of the actual question and accepted answer!
EDIT - The use of hist and histc is not recommended now, and histogram should be used instead. Beware that none of the 6 ways of creating bins with this new function will produce the bins hist and histc produce. There is a Matlab script to update former code to fit the way histogram is called (bin edges instead of bin centers - link). By doing so, one can compare the pdf normalization methods of #abcd (trapz and sum) and Matlab (pdf).
The 3 pdf normalization method give nearly identical results (within the range of eps).
TEST:
A = randn(10000,1);
centers = -6:0.5:6;
d = diff(centers)/2;
edges = [centers(1)-d(1), centers(1:end-1)+d, centers(end)+d(end)];
edges(2:end) = edges(2:end)+eps(edges(2:end));
figure;
subplot(2,2,1);
hist(A,centers);
title('HIST not normalized');
subplot(2,2,2);
h = histogram(A,edges);
title('HISTOGRAM not normalized');
subplot(2,2,3)
[counts, centers] = hist(A,centers); %get the count with hist
bar(centers,counts/trapz(centers,counts))
title('HIST with PDF normalization');
subplot(2,2,4)
h = histogram(A,edges,'Normalization','pdf')
title('HISTOGRAM with PDF normalization');
dx = diff(centers(1:2))
normalization_difference_trapz = abs(counts/trapz(centers,counts) - h.Values);
normalization_difference_sum = abs(counts/sum(counts*dx) - h.Values);
max(normalization_difference_trapz)
max(normalization_difference_sum)
The maximum difference between the new PDF normalization and the former one is 5.5511e-17.
hist can not only plot an histogram but also return you the count of elements in each bin, so you can get that count, normalize it by dividing each bin by the total and plotting the result using bar. Example:
Y = rand(10,1);
C = hist(Y);
C = C ./ sum(C);
bar(C)
or if you want a one-liner:
bar(hist(Y) ./ sum(hist(Y)))
Documentation:
hist
bar
Edit: This solution answers the question How to have the sum of all bins equal to 1. This approximation is valid only if your bin size is small relative to the variance of your data. The sum used here correspond to a simple quadrature formula, more complex ones can be used like trapz as proposed by R. M.
[f,x]=hist(data)
The area for each individual bar is height*width. Since MATLAB will choose equidistant points for the bars, so the width is:
delta_x = x(2) - x(1)
Now if we sum up all the individual bars the total area will come out as
A=sum(f)*delta_x
So the correctly scaled plot is obtained by
bar(x, f/sum(f)/(x(2)-x(1)))
The area of abcd`s PDF is not one, which is impossible like pointed out in many comments.
Assumptions done in many answers here
Assume constant distance between consecutive edges.
Probability under pdf should be 1. The normalization should be done as Normalization with probability, not as Normalization with pdf, in histogram() and hist().
Fig. 1 Output of hist() approach, Fig. 2 Output of histogram() approach
The max amplitude differs between two approaches which proposes that there are some mistake in hist()'s approach because histogram()'s approach uses the standard normalization.
I assume the mistake with hist()'s approach here is about the normalization as partially pdf, not completely as probability.
Code with hist() [deprecated]
Some remarks
First check: sum(f)/N gives 1 if Nbins manually set.
pdf requires the width of the bin (dx) in the graph g
Code
%http://stackoverflow.com/a/5321546/54964
N=10000;
Nbins=50;
[f,x]=hist(randn(N,1),Nbins); % create histogram from ND
%METHOD 4: Count Densities, not Sums!
figure(3)
dx=diff(x(1:2)); % width of bin
g=1/sqrt(2*pi)*exp(-0.5*x.^2) .* dx; % pdf of ND with dx
% 1.0000
bar(x, f/sum(f));hold on
plot(x,g,'r');hold off
Output is in Fig. 1.
Code with histogram()
Some remarks
First check: a) sum(f) is 1 if Nbins adjusted with histogram()'s Normalization as probability, b) sum(f)/N is 1 if Nbins is manually set without normalization.
pdf requires the width of the bin (dx) in the graph g
Code
%%METHOD 5: with histogram()
% http://stackoverflow.com/a/38809232/54964
N=10000;
figure(4);
h = histogram(randn(N,1), 'Normalization', 'probability') % hist() deprecated!
Nbins=h.NumBins;
edges=h.BinEdges;
x=zeros(1,Nbins);
f=h.Values;
for counter=1:Nbins
midPointShift=abs(edges(counter)-edges(counter+1))/2; % same constant for all
x(counter)=edges(counter)+midPointShift;
end
dx=diff(x(1:2)); % constast for all
g=1/sqrt(2*pi)*exp(-0.5*x.^2) .* dx; % pdf of ND
% Use if Nbins manually set
%new_area=sum(f)/N % diff of consecutive edges constant
% Use if histogarm() Normalization probability
new_area=sum(f)
% 1.0000
% No bar() needed here with histogram() Normalization probability
hold on;
plot(x,g,'r');hold off
Output in Fig. 2 and expected output is met: area 1.0000.
Matlab: 2016a
System: Linux Ubuntu 16.04 64 bit
Linux kernel 4.6
For some Distributions, Cauchy I think, I have found that trapz will overestimate the area, and so the pdf will change depending on the number of bins you select. In which case I do
[N,h]=hist(q_f./theta,30000); % there Is a large range but most of the bins will be empty
plot(h,N/(sum(N)*mean(diff(h))),'+r')
There is an excellent three part guide for Histogram Adjustments in MATLAB (broken original link, archive.org link),
the first part is on Histogram Stretching.