I have a vector d:
d = [
1.19011941712580e-06
6.39136179286748e-06
1.26442316296575e-05
1.81039120389278e-05
1.91304903300688e-05
2.19912290910362e-05
2.94113112667430e-05
3.42238417249065e-05
4.14201181268186e-05
5.76014376298924e-05
6.81337071520188e-05
0.000108396864465101
0.000130922201344182
0.000145712942644687
0.000174386494384153
0.000262758083529471
03050975943883
0.000373066486719321
0.000423949134658855
0.000489079623696380
0.000548432526451254
0.000694787830192734
0.000881370593483890
0.00125516689720339
0.00145237435686831
0.00815957230852142
0.0210146005799470
0.0507995676939279
0.0541594307796186
1
]
Plotting d:
plot(d, 'x:')
In this situation, [M, F] = mode(d) gives a result that I didn't want.
Is there any function that counts the most frequent values which takes a sort of tolerance into account?
Clustering can be considered. However, In the figure above, clustering may assign d(27:29) into the left side cluster.
Current approach is normilizing and thresholding:
d_norm = d / max(d);
v = d_norm(d_norm < 0.01); % 1 percent threshold
However, I think it is a sort of hard-coded and not a good approach.
Histcounts is your friend!
Your "threshold" can be easily translated to histogram bins if you know your range. In your case, the range is 0-1, if you choose a threshold of 0.01 then 100 is your bin count.
counts=histcounts(d,100)
Related
I'm new to Matlab and want to write a program that chooses the value of a parameter (P) to minimize the difference between two vectors, where each vector is a variable in a dataframe. The first vector (call it A) is a predetermined vector of 1s and 0s, and the second vector (call it B) has each of its entries determined as an indicator function that depends on the value of the parameter P and other variables in the dataframe. For instance, let C be a third variable in the dataset, so
A = [1, 0, 0, 1, 0]
B = [x, y, z, u, v]
where x = 1 if (C[1]+10)^0.5 - P > (C[1])^0.5 and otherwise x = 0, and similarly, y = 1 if (C[2]+10)^0.5 - P > (C[2])^0.5 and otherwise y = 0, and so on.
I'm not really sure where to start with the code, except that it might be useful to use the fminsearch command. Any suggestions?
Edit: I changed the above by raising to a power, which is closer to the actual example that I have. I'm also providing a complete example in response to a comment:
Let A be as above, and let C = [10, 1, 100, 1000, 1]. Then my goal with the Matlab code would be to choose a value of P to minimize the differences between the coordinates of the vectors A and B, where B[1] = 1 if (10+10)^0.5 - P > (10)^0.5 and otherwise B[1] = 0, and similarly B[2] = 1 if (1+10)^0.5 - P > (1)^0.5 and otherwise B[2] = 0, etc. So I want to choose P to maximize the likelihood that A[1] = B[1], A[2] = B[2], etc.
I have the following setup in Matlab, where ds is the name of my dataset:
ds.B = zeros(size(ds,1),1); % empty vector to fill
for i = 1:size(ds,1)
if ((ds.C(i) + 10)^(0.5) - P > (ds.C(i))^(0.5))
ds.B(i) = 1;
else
ds.B(i) = 0;
end
end
Now I want to choose the value of P to minimize the difference between A and B. How can I do this?
EDIT: I'm also wondering how to do this when the inequality is something like (C[i]+10)^0.5 - P*D[i] > (C[i])^0.5, where D is another variable in my dataset. Now P is a scalar being multiplied rather than just added. This seems more complicated since I can't solve for P exactly. How can I solve the problem in this case?
EDIT 1: It seems fminbnd() isn't optimal, likely due to the stairstep nature of the indicator function. I've updated to test the midpoints of all the regions between indicator function flips, plus endpoints.
EDIT 2: Updated to include dataset D as a coefficient of P.
If you can package your distance calculation up in a single function based on P, you can then search for its minimum.
arraySize = 1000;
ds.A = double(rand([arraySize,1]) > 0.5);
ds.C = rand(size(ds.A));
ds.D = rand(size(ds.A));
B = #(P)double((ds.C+10).^0.5 - P.*ds.D > ds.C.^0.5);
costFcn = #(P)sqrt(sum((ds.A-B(P)).^2));
% Solving the equation (C+10)^0.5 - P*D = C^0.5 for P, and sorting the results
BCrossingPoints = sort(((ds.C+10).^0.5-ds.C.^0.5)./ds.D);
% Taking the average of each crossing point with its neighbors
BMidpoints = (BCrossingPoints(1:end-1)+BCrossingPoints(2:end))/2;
% Appending endpoints onto the midpoints
PsToTest = [BCrossingPoints(1)-0.1; BMidpoints; BCrossingPoints(end)+0.1];
% Calculate the distance from A to B at each P to test
costResult = arrayfun(costFcn,PsToTest);
% Find the minimum cost
[~,lowestCostIndex] = min(costResult);
% Find the optimum P
optimumP = PsToTest(lowestCostIndex);
ds.B = B(optimumP);
semilogx(PsToTest,costResult)
xlabel('P')
ylabel('Distance from A to B')
1.- x is assumed positive real only, because with x<0 then complex values show up.
Since no comment is made in the question it seems reasonable to assume x real and x>0 only.
As requested, P 'the parameter' a scalar, P only has 2 significant states >0 or <0, let's see how is this:
2.- The following lines generate kind-of random A and C.
Then a sweep of p is carried out and distances d1 and d2 are calculated.
d1 is euclidean distance and d2 is the absolute of the difference between A and and B converting both from binary to decimal:
N=10
% A=[1 0 0 1 0]
A=randi([0 1],1,N);
% C=[10 1 1e2 1e3 1]
C=randi([0 1e3],1,N)
p=[-1e4:1:1e4]; % parameter to optimize
B=zeros(1,numel(A));
d1=zeros(1,numel(p)); % euclidean distance
d2=zeros(1,numel(p)); % difference distance
for k1=1:1:numel(p)
B=(C+10).^.5-p(k1)>C.^.5;
d1(k1)=(sum((B-A).^2))^.5;
d2(k1)=abs(sum(A.*2.^[numel(A)-1:-1:0])-sum(B.*2.^[numel(A)-1:-1:0]));
end
figure;
plot(p,d1)
grid on
xlabel('p');title('d1')
figure
plot(p,d2)
grid on
xlabel('p');title('d2')
The only degree of freedom to optimise seems to be the sign of P regardless of |P| value.
3.- f(p,x) has either no root, or just one root, depending upon p
The threshold funtion is
if f(x)>0 then B(k)==1 else B(k)==0
this is
f(p,x)=(x+10)^.5-p-x^.5
Now
(x+10).^.5-p>x.^.5 is same as (x+10).^.5-x.^.5>p
There's a range of p that keeps f(p,x)=0 without any (real) root.
For the particular case p=0 then (x+10).^.5 and x.^.5 do not intersect (until Inf reached = there's no intersection)
figure;plot(x,(x+10).^.5,x,x.^.5);grid on
[![enter image description here][3]][3]
y2=diff((x+10).^.5-x.^.5)
figure;plot(x(2:end),y2);
grid on;xlabel('x')
title('y2=diff((x+10).^.5-x.^.5)')
[![enter image description here][3]][3]
% 005
This means the condition f(x)>0 is always true holding all bits of B=1. With B=1 then d(A,B) turns into d(A,1), a constant.
However, for a certain value of p then there's one root and f(x)>0 is always false keeping all bits of B=0.
In this case d(A,B) the cost function turns into d(A,0) and this is A itself.
4.- P as a vector
The optimization gains in degrees of freedom if instead of P scalar, P is considered as vector.
For a given x there's a value of p that switches B(k) from 0 to 1.
Any value of p below such threshold keeps B(k)=0.
Equivalently, inverting f(x) :
g(p)=(10-p^2)^2/(4*p^2)>x
Values of x below this threshold bring B closer to A because for each element of B it's flipped to the element value of A.
Therefore, it's convenient to consider P as a vector, not a ascalar, and :
For all, or as many (as possible) elements of C to meet c(k)<(10-p^2)^2/(4*p^2) in order to get C=A or
minimize d(A,C)
5.- roots of f(p,x)
syms t positive
p=[-1000:.1:1000];
zp=NaN*ones(1,numel(p));
sol=zeros(1,numel(p));
for k1=1:1:numel(p)
p(k1)
eq1=(t+10)^.5-p(k1)-t^.5-p(k1)==0;
s1=solve(eq1,t);
if ~isempty(s1)
zp(k1)=s1;
end
end
nzp=~isnan(zp);
zp(nzp)
returns
=
620.0100 151.2900 64.5344 34.2225 20.2500 12.7211
8.2451 5.4056 3.5260 2.2500 1.3753 0.7803
0.3882 0.1488 0.0278
I want to construct a 3-dimensional Poisson distribution in Matlab with lambda parameters [0.4, 0.2, 0.6] and I want to truncate it to have support in [0;1;2;3;4;5]. The 3 components are independent.
This is what I do
clear
n=3; %number components of the distribution
supp_marginal=0:1:5;
suppsize_marginal=size(supp_marginal,2);
supp_temp=repmat(supp_marginal.',1,n);
supp_temp_cell=num2cell(supp_temp,1);
output_temp_cell=cell(1,n);
[output_temp_cell{:}] = ndgrid(supp_temp_cell{:});
supp=zeros(suppsize_marginal^n,n);
for h=1:n
temp=output_temp_cell{h};
supp(:,h)=temp(:);
end
suppsize=size(supp,1);
lambda_1=0.4;
lambda_2=0.2;
lambda_3=0.6;
pr_mass=zeros(suppsize,1);
for j=1:suppsize
pr_mass(j)=(poisspdf(supp(j,1),lambda_1).*...
poisspdf(supp(j,2),lambda_2).*...
poisspdf(supp(j,3),lambda_3))/...
sum(poisspdf(supp(:,1),lambda_1).*...
poisspdf(supp(:,2),lambda_2).*...
poisspdf(supp(j,3),lambda_3));
end
When I compute the mean of the obtained distribution, I get lambda_1 and lambda_2 but not lambda_3.
lambda_empirical=sum(supp.*repmat(pr_mass,1,3));
Question: why I do not get lambda_3?
tl;dr: Truncation changes the distribution so different means are expected.
This is expected as truncation itself has changed the distribution and certainly adjusts the mean. You can see this from the experiment below. Notice that for your chosen parameters, this just starts to become noticable around lambda = 0.6.
Similar to the wiki page, this illustrates the difference between E[X] (expectation of X without truncation; fancy word for mean) and E[ X | LB ≤ X ≤ UB] (expectation of X given it is on interval [LB,UB]). This conditional expectation implies a different distribution than the unconditional distribution of X (~Poisson(lambda)).
% MATLAB R2018b
% Setup
LB = 0; % lowerbound
UB = 5; % upperbound
% Simple test to compare theoretical means with and without truncation
TestLam = 0.2:0.01:1.5;
Gap = zeros(size(TestLam(:)));
for jj = 1:length(TestLam)
TrueMean = mean(makedist('Poisson','Lambda',TestLam(jj)));
TruncatedMean = mean(truncate(makedist('Poisson','Lambda',TestLam(jj)),LB,UB));
Gap(jj) = TrueMean-TruncatedMean;
end
plot(TestLam,Gap)
Notice the gap with these truncation bounds and a lambda of 0.6 is still small and is negligible as lambda approaches zero.
lam = 0.6; % <---- try different values (must be greater than 0)
pd = makedist('Poisson','Lambda',lam)
pdt = truncate(pd,LB,UB)
mean(pd) % 0.6
mean(pdt) % 0.5998
Other Resources:
1. Wiki for Truncated Distributions
2. What is a Truncated Distribution
3. MATLAB documentation for truncate(), makedist()
4. MATLAB: Working with Probability Distribution (Objects)
I have a vector in Matlab that looks something like:
vect = 0
100
300
500
700
1000
500
300
200
0
When normalised, each value should indicate the probability of a certain value, and my values are just 1 to 10 (i.e. 0% chance of 1, 100/sum(vect) chance of 2, etc).
How do I work out statistics on the value (in particular standard deviation)..? If I do mean(vect), I just end up with 360, and I get a similarly large value for standard deviation. The mean value should, of course, be around 5. I'm sure it wouldn't be too hard to code up manually at all, but there must be a way of doing this directly in Matlab, so I figured I'd ask!
I am not really sure if matlab have any built in function for this, but it is no big deal. Both are one liners anyway
vect = [0; 100; 300; 500; 700; 1000; 500; 300; 200; 0];
prob = vect./sum(vect);
val = [1:10].';
meanVal = sum(prob.*val);
stDev = sqrt( sum( prob.*val.^2 ) -sum(prob.*val)^2 );
EDIT:
There are two functions that does this. They are called mean and std as well. But they take a probability distribution object instead.
If you call stem(vect) you'll see that vect is the probability density function of a normally distributed variable, hence you can fit a normal distribution to vect without normalization
x = (1:length(vect))';
pdf = fitdist(x, 'normal', 'freq', vect);
The result has an average value of 5.63889 and a standard deviation of 1.66944.
I want to get the probability to get a value X higher than x_i, which means the cumulative distribution functions CDF. P(X>=x_i).
I've tried to do it in Matlab with this code.
Let's assume the data is in the column vector p1.
xp1 = linspace(min(p1), max(p1)); %range of bins
histp1 = histc(p1(:), xp1); %histogram od data
probp1 = histp1/sum(histp1); %PDF (probability distribution function)
`figure;plot(probp1, 'o') `
Now I want to calculate the CDF,
sorncount = flipud(histp1);
cumsump1 = cumsum(sorncount);
normcumsump1 = cumsump1/max(cumsump1);
cdf = flipud(normcumsump1);
figure;plot(xp1, cdf, 'ok');
I'm wondering whether anyone can help me to know if I'm ok or am I doing something wrong?
Your code works correctly, but is a bit more complicated than it could be. Since probp1 has been normalized to have sum equal to 1, the maximum of its cumulative sum is guaranteed to be 1, so there is no need to divide by this maximum. This shortens the code a bit:
xp1 = linspace(min(p1), max(p1)); %range of bins
histp1 = histc(p1(:), xp1); %count for each bin
probp1 = histp1/sum(histp1); %PDF (probability distribution function)
cdf = flipud(cumsum(flipud(histp1))); %CDF (unconventional, of P(X>=a) kind)
As Raab70 noted, most of the time CDF is understood as P(X<=a), in which case you don't need flipud: taking cumsum(histp1) is all that's needed.
Also, I would probably use histp1(end:-1:1) instead of flipud(histp1), so that the vector is flipped no matter if it's a row or column.
In order to find best fit (thru polyfit), i am getting negative p value but matlab is not accepting it (Subscript indices must either be real positive integers or logicals.). Is there any way that I can use it ? I can't think of alternative method. I'll always get negative values.
EDIT:
I am trying to flattening baseline of a curve, for that. I am running for loop to have fit from 1 to 3 order. And then I am using smallest normr s value to to find the best fit and then subtract it from the whole curve to get baseline straight. I tried with few curves it works well but not with all of the data because of above describes issue.
part of the code I am working on:
for i=1:3
[p,s]=polyfit(x,y,i);
a=s.normr;
b(i,1)=p(1);
normr(i,1)=a;
ind=find(b==min(b));
mn=b(ind,1);
Yflat=y-mn(1)*(x-mean(x));
ca{2,2}=Yflat;
clear a b normr p s rte ind ind2 Yflat
end
When I translate an image into negative coordinates,
I usually record an offset e.g.
offset = [ -5, -8.5 ]
and save the intensity values in matrix begin with (1, 1) as usual,
But when comes to calculation, let the coordinates array add up with the offset
e.g. [ actualX, actualY ] = [ x, y ] + offset ;
It may need extra efforts, but it works.
Good Luck!
The code below (your code from the comments + initialization of x, y) executes. What is the problem?
x = 1:50;
y = randn(size(x));
for i=1:3
[p,s]=polyfit(x,y,i);
a=s.normr;
b(i,1)=p(1);
normr(i,1)=a;
ind=find(b==min(b));
mn=b(ind,1);
Yflat=y-mn(1)*(x-mean(x));
ca{2,2}=Yflat;
end