Related
I found that calculating an index to specify outliers of a dataset according to how the boxplot works does not give the same results. Please find below an example where I create some data, extract the values from the boxplot (as seen in datatips in the figure window) and compare them to the values I calculated.
While the median and quartiles match up the upper and lower adjacent values do not. According to the Matlab help under 'Whisker', the adjacent values are calculated as q3 + w*(q3-q1) where q3 and q1 are the quantiles and w is the specified whisker length.
Am I calculating this wrong or is there any other mistake? I would like to be able to explain the error.
Screenshot of results table (please note the results vary due to random data)
%Boxplot test
% create random, normally distributed dataset
data = round(randn(1000,1)*10,2);
figure(10)
clf
boxplot(data,'Whisker',1.5)
clear stats tmp
% read data from boxplot, same values as can be seen in datatips in the figure window
h = findobj(gcf,'tag','Median');
tmp = get(h,'YData');
stats(1,1) = tmp(1);
h = findobj(gcf,'tag','Box');
tmp = get(h,'YData');
stats(1,2) = tmp(1);
stats(1,3) = tmp(2);
h = findobj(gcf,'tag','Upper Adjacent Value');
tmp = get(h,'YData');
stats(1,4) = tmp(1);
h = findobj(gcf,'tag','Lower Adjacent Value');
tmp = get(h,'YData');
stats(1,5) = tmp(1);
% calculated data
stats(2,1) = median(data);
stats(2,2) = quantile(data,0.25);
stats(2,3) = quantile(data,0.75);
range = stats(2,3) - stats(2,2);
stats(2,4) = stats(2,3) + 1.5*range;
stats(2,5) = stats(2,2) - 1.5*range;
% error calculation
for k=1:size(stats,2)
stats(3,k) = stats(2,k)-stats(1,k);
end %for k
% convert results to table with labels
T = array2table(stats,'VariableNames',{'Median','P25','P75','Upper','Lower'}, ...
'RowNames',{'Boxplot','Calculation','Error'});
While the calculation of the boundaries, e.g. q3 = q3 + w*(q3-q1), is correct, it is not displayed in the boxplot. What is actually displayed and marked as upper/lower adjacent value is the minimum and maximum of the values within the aforementioned boundaries.
Regarding the initial task leading to the question: For applying the same filtering of outliers as in the boxplot the calculated boundaries can be used.
I have the following data that predicts a curve in the middle of the two curves which has different equation and datas.I also need to spline and smothen the curve of the middle curve
I've tried searching other codes here in stackoverflow but this is the most close to the right solution. So far the plot for the two curve is right but the interpolated point gives me wrong plot.
Im trying to find the plot for val=30 given that (a25,vel25)=25 and (a50,vel50)=50. Please help me troubleshoot and get a table of data (x,y) for the generated interpolated curve. Thanks for your help
generated plot using this program
a50=[1.05
0.931818182
0.931818182
0.968181818
1.045454545
1.136363636
1.354545455
1.568181818
1.718181818
1.945454545
2.159090909
2.454545455
2.772727273
];
vel50=[0.85
0.705555556
0.605555556
0.533333333
0.472222222
0.45
0.427777778
0.45
0.477777778
0.533333333
0.611111111
0.711111111
0.827777778
];
a25=[0.5
0.613636364
0.686363636
0.795454545
0.918181818
0.963636364
1.090909091
1.236363636
1.304545455
1.431818182
1.545454545
1.659090909
1.818181818
];
vel25=[0.425555556
0.354444444
0.302222222
0.266666667
0.233333333
0.226666667
0.211111111
0.222222222
0.237777778
0.266666667
0.311111111
0.35
0.402222222
];
plot(a25,vel25,'b-');
hold on
plot(a50,vel50,'g-');
minX = min([a25 a50]);
maxX = max([a25,a50]);
xx = linspace(minX,maxX,100);
vel25_inter = interp1(a25,vel25,xx);
vel50_inter = interp1(a50,vel50,xx);
val = 30; % The interpolated point
interpVel = vel25_inter + ((val-25).*(vel50_inter-vel25_inter))./(50-25);
plot(xx,interpVel,'r-');
The question and answer linked in comment still apply and can be a solution.
In your case, it is not so direct because your data are not on the same grid and some are not monotonic, but once they are packaged properly, the easiest solution is still to use griddata.
By packaged properly, I mean finding the maximum common interval (on x, or what you call a), so the data can be interpolated between curve without producing NaNs.
This seems to work:
The red dashed line is the values interpolated at val=30, all the other lines are interpolations for values between 25 to 50.
The code to get there:
% back up original data, just for final plot
bkp_a50 = a50 ; bkp_vel50 = vel50 ;
% make second x vector monotonic
istart = find( diff(a50)>0 , 1 , 'first') ;
a50(1:istart-1) = [] ;
vel50(1:istart-1) = [] ;
% prepare a 3rd dimension vector (from 25 to 50)
T = [repmat(25,size(a25)) ; repmat(50,size(a50)) ] ;
% merge all observations together
A = [ a25 ; a50] ;
V = [vel25 ; vel50] ;
% find the minimum domain on which data can be interpolated
% (anything outside of that will return NaN)
Astart = max( [min(a25) min(a50)] ) ;
Astop = min( [max(a25) max(a50)] ) ;
% use the function 'griddata'
[TI,AI] = meshgrid( 25:50 , linspace(Astart,Astop,10) ) ;
VI = griddata(T,A,V,TI,AI) ;
% plot all the intermediate curves
plot(AI,VI)
hold on
% the original curves
plot(a25,vel25,'--k','linewidth',2)
plot(bkp_a50,bkp_vel50,'--k','linewidth',2)
% Highlight the curve at T = 30 ;
c30 = find( TI(1,:) == 30 ) ;
plot(AI(:,c30),VI(:,c30),'--r','linewidth',2)
There were quite a few issues with your code that's why it was not executing properly. I have just made very little changes to your code and made it running,
clc
%13
a50=[1.05
0.931818182
0.932
0.968181818
1.045454545
1.136363636
1.354545455
1.568181818
1.718181818
1.945454545
2.159090909
2.454545455
2.772727273
];
%13
vel50=[0.85
0.705555556
0.605555556
0.533333333
0.472222222
0.45
0.427777778
0.45
0.477777778
0.533333333
0.611111111
0.711111111
0.827777778
];
%13
a25=[0.5
0.613636364
0.686363636
0.795454545
0.918181818
0.963636364
1.090909091
1.236363636
1.304545455
1.431818182
1.545454545
1.659090909
1.818181818
];
%13
vel25=[0.425555556
0.354444444
0.302222222
0.266666667
0.233333333
0.226666667
0.211111111
0.222222222
0.237777778
0.266666667
0.311111111
0.35
0.402222222
];
plot(a25,vel25,'b-');
hold on
plot(a50,vel50,'g-');
minX = min([a25 a50])
maxX = max([a25 a50])
%xx = linspace(minX,maxX);
xx = linspace(0.5,2.7727,100);
vel25_inter = interp1(a25,vel25,xx);
vel50_inter = interp1(a50,vel50,xx);
val = 30; % The interpolated point
interpVel = vel25_inter + ((val-25).*(vel50_inter-vel25_inter))./(50-25);
plot(xx,interpVel,'r-');
There issues were
The interval on which you wanted to interpolate is xx = linspace(minX,maxX); but it gives an error of the type,
Matrix dimensions must agree.
Because you were assigning two values for the starting point and two for the ending point. So I replace it with xx = linspace(0.5,2.7727,100); where the starting point is being the minimum of the two minimum minX and the same for maxX
There are values of a50 (0.931818182) that repeat which was producing the following error
The grid vectors are not strictly monotonic increasing.
I change one of the value and replaced it with 0.932
The output is not that promising but is suppose the one you want?
I have a .csv file with data on each line in the format (x,y,z,t,f), where f is the value of some function at location (x,y,z) at time t. So each new line in the .csv gives a new set of coordinates (x,y,z,t), with accompanying value f. The .csv is not sorted.
I want to use imagesc to create a video of this data in the xy-plane, as time progresses. The way I've done this is by reformatting M into something more easily usable by imagesc. I'm doing three nested loops, roughly like this
M = csvread('file.csv');
uniqueX = unique(M(:,1));
uniqueY = unique(M(:,2));
uniqueT = unique(M(:,4));
M_reformatted = zeros(length(uniqueX), length(uniqueY), length(uniqueT));
for i = 1:length(uniqueX)
for j = 1:length(uniqueY)
for k = 1:length(uniqueT)
M_reformatted(i,j,k) = M( ...
M(:,1)==uniqueX(i) & ...
M(:,2)==uniqueY(j) & ...
M(:,4)==uniqueT(k), ...
5 ...
);
end
end
end
once I have M_reformatted, I can loop through timesteps k and use imagesc on M_reformatted(:,:,k). But doing the above nested loops is very slow. Is it possible to vectorize the above? If so, an outline of the approach would be very helpful.
edit: as noted in answers/comments below, I made a mistake in that there are several possible z-values, which I haven't taken into account. If only a single z-value, the above would be ok.
This vectorized solution allows for negative values of x and y and is many times faster than the non-vectorized solution (close to 20x times for the test case at the bottom).
The idea is to sort the x, y, and t values in lexicographical order using sortrows and then using reshape to build the time slices of M_reformatted.
The code:
idx = find(M(:,3)==0); %// find rows where z==0
M2 = M(idx,:); %// M2 has only the rows where z==0
M2(:,3) = []; %// delete z coordinate in M2
M2(:,[1 2 3]) = M2(:,[3 1 2]); %// change from (x,y,t,f) to (t,x,y,f)
M2 = sortrows(M2); %// sort rows by t, then x, then y
numT = numel(unique(M2(:,1))); %// number of unique t values
numX = numel(unique(M2(:,2))); %// number of unique x values
numY = numel(unique(M2(:,3))); %// number of unique y values
%// fill the time slice matrix with data
M_reformatted = reshape(M2(:,4), numY, numX, numT);
Note: I am assuming y refers to the columns of the image and x refers to the rows. If you want these flipped, use M_reformatted = permute(M_reformatted,[2 1 3]) at the end of the code.
The test case I used for M (to compare the result to other solutions) has a NxNxN space with T times slices:
N = 10;
T = 10;
[x,y,z] = meshgrid(-N:N,-N:N,-N:N);
numPoints = numel(x);
x=x(:); y=y(:); z=z(:);
s = repmat([x,y,z],T,1);
t = repmat(1:T,numPoints,1);
M = [s, t(:), rand(numPoints*T,1)];
M = M( randperm(size(M,1)), : );
I don't think you need to vectorize. I think you change your algorithm.
You only need one loop to step through the lines of the CSV file. For every line, you have (x,y,z,t,f) so just store it in M_reformatted where it belongs. Something like this:
M_reformatted = zeros(max(M(:,1)), max(M(:,2)), max(M(:,4)));
for line = 1:size(M,2)
z = M(line, 3);
if z ~= 0, continue; end;
x = M(line, 1);
y = M(line, 2);
t = M(line, 4);
f = M(line, 5);
M_reformatted(x, y, t) = f;
end
Also note that pre-allocating M_reformatted is a very good idea, but your code may have been getting the size wrong (depending on the data). I think using max like I did will always do the right thing.
I have a set of three vectors (stored into a 3xN matrix) which are 'entangled' (e.g. some value in the second row should be in the third row and vice versa). This 'entanglement' is based on looking at the figure in which alpha2 is plotted. To separate the vector I use a difference based approach where I calculate the difference of one value with respect the three next values (e.g. comparing (1,i) with (:,i+1)). Then I take the minimum and store that. The method works to separate two of the three vectors, but not for the last.
I was wondering if you guys can share your ideas with me how to solve this problem (if possible). I have added my coded below.
Thanks in advance!
Problem in figures:
clear all; close all; clc;
%%
alpha2 = [-23.32 -23.05 -22.24 -20.91 -19.06 -16.70 -13.83 -10.49 -6.70;
-0.46 -0.33 0.19 2.38 5.44 9.36 14.15 19.80 26.32;
-1.58 -1.13 0.06 0.70 1.61 2.78 4.23 5.99 8.09];
%%% Original
figure()
hold on
plot(alpha2(1,:))
plot(alpha2(2,:))
plot(alpha2(3,:))
%%% Store start values
store1(1,1) = alpha2(1,1);
store2(1,1) = alpha2(2,1);
store3(1,1) = alpha2(3,1);
for i=1:size(alpha2,2)-1
for j=1:size(alpha2,1)
Alpha1(j,i) = abs(store1(1,i)-alpha2(j,i+1));
Alpha2(j,i) = abs(store2(1,i)-alpha2(j,i+1));
Alpha3(j,i) = abs(store3(1,i)-alpha2(j,i+1));
[~, I] = min(Alpha1(:,i));
store1(1,i+1) = alpha2(I,i+1);
[~, I] = min(Alpha2(:,i));
store2(1,i+1) = alpha2(I,i+1);
[~, I] = min(Alpha3(:,i));
store3(1,i+1) = alpha2(I,i+1);
end
end
%%% Plot to see if separation worked
figure()
hold on
plot(store1)
plot(store2)
plot(store3)
Solution using extrapolation via polyfit:
The idea is pretty simple: Iterate over all positions i and use polyfit to fit polynomials of degree d to the d+1 values from F(:,i-(d+1)) up to F(:,i). Use those polynomials to extrapolate the function values F(:,i+1). Then compute the permutation of the real values F(:,i+1) that fits those extrapolations best. This should work quite well, if there are only a few functions involved. There is certainly some room for improvement, but for your simple setting it should suffice.
function F = untangle(F, maxExtrapolationDegree)
%// UNTANGLE(F) untangles the functions F(i,:) via extrapolation.
if nargin<2
maxExtrapolationDegree = 4;
end
extrapolate = #(f) polyval(polyfit(1:length(f),f,length(f)-1),length(f)+1);
extrapolateAll = #(F) cellfun(extrapolate, num2cell(F,2));
fitCriterion = #(X,Y) norm(X(:)-Y(:),1);
nFuncs = size(F,1);
nPoints = size(F,2);
swaps = perms(1:nFuncs);
errorOfFit = zeros(1,size(swaps,1));
for i = 1:nPoints-1
nextValues = extrapolateAll(F(:,max(1,i-(maxExtrapolationDegree+1)):i));
for j = 1:size(swaps,1)
errorOfFit(j) = fitCriterion(nextValues, F(swaps(j,:),i+1));
end
[~,j_bestSwap] = min(errorOfFit);
F(:,i+1) = F(swaps(j_bestSwap,:),i+1);
end
Initial solution: (not that pretty - Skip this part)
This is a similar solution that tries to minimize the sum of the derivatives up to some degree of the vector valued function F = #(j) alpha2(:,j). It does so by stepping through the positions i and checks all possible permutations of the coordinates of i to get a minimal seminorm of the function F(1:i).
(I'm actually wondering right now if there is any canonical mathematical way to define the seminorm so we get our expected results... I initially was going for the H^1 and H^2 seminorms, but they didn't quite work...)
function F = untangle(F)
nFuncs = size(F,1);
nPoints = size(F,2);
seminorm = #(x,i) sum(sum(abs(diff(x(:,1:i),1,2)))) + ...
sum(sum(abs(diff(x(:,1:i),2,2)))) + ...
sum(sum(abs(diff(x(:,1:i),3,2)))) + ...
sum(sum(abs(diff(x(:,1:i),4,2))));
doSwap = #(x,swap,i) [x(:,1:i-1), x(swap,i:end)];
swaps = perms(1:nFuncs);
normOfSwap = zeros(1,size(swaps,1));
for i = 2:nPoints
for j = 1:size(swaps,1)
normOfSwap(j) = seminorm(doSwap(F,swaps(j,:),i),i);
end
[~,j_bestSwap] = min(normOfSwap);
F = doSwap(F,swaps(j_bestSwap,:),i);
end
Usage:
The command alpha2 = untangle(alpha2); will untangle your functions:
It should even work for more complicated data, like these shuffled sine-waves:
nPoints = 100;
nFuncs = 5;
t = linspace(0, 2*pi, nPoints);
F = bsxfun(#(a,b) sin(a*b), (1:nFuncs).', t);
for i = 1:nPoints
F(:,i) = F(randperm(nFuncs),i);
end
Remark: I guess if you already know that your functions will be quadratic or some other special form, RANSAC would be a better idea for larger number of functions. This could also be useful if the functions are not given with the same x-value spacing.
I'm looking for a way to speed up some simple two port matrix calculations. See the below code example for what I'm doing currently. In essence, I create a [Nx1] frequency vector first. I then loop through the frequency vector and create the [2x2] matrices H1 and H2 (all functions of f). A bit of simple matrix math including a matrix left division '\' later, and I got my result pb as a [Nx1] vector. The problem is the loop - it takes a long time to calculate and I'm looking for way to improve efficiency of the calculations. I tried assembling the problem using [2x2xN] transfer matrices, but the mtimes operation cannot handle 3-D multiplications.
Can anybody please give me an idea how I can approach such a calculation without the need for looping through f?
Many thanks: svenr
% calculate frequency and wave number vector
f = linspace(20,200,400);
w = 2.*pi.*f;
% calculation for each frequency w
for i=1:length(w)
H1(i,1) = {[1, rho*c*k(i)^2 / (crad*pi); 0,1]};
H2(i,1) = {[1, 1i.*w(i).*mp; 0, 1]};
HZin(i,1) = {H1{i,1}*H2{i,1}};
temp_mat = HZin{i,1}*[1; 0];
Zin(i,1) = temp_mat(1,1)/temp_mat(2,1);
temp_mat= H1{i,1}\[1; 1/Zin(i,1)];
pb(i,1) = temp_mat(1,1); Ub(i,:) = temp_mat(2,1);
end
Assuming that length(w) == length(k) returns true , rho , c, crad, mp are all scalars and in the last line is Ub(i,1) = temp_mat(2,1) instead of Ub(i,:) = temp_mat(2,1);
temp = repmat(eyes(2),[1 1 length(w)]);
temp1(1,2,:) = rho*c*(k.^2)/crad/pi;
temp2(1,2,:) = (1i.*w)*mp;
H1 = permute(num2cell(temp1,[1 2]),[3 2 1]);
H2 = permute(num2cell(temp2,[1 2]),[3 2 1]);
HZin = cellfun(#(a,b)(a*b),H1,H2,'UniformOutput',0);
temp_cell = cellfun(#(a,b)(a*b),H1,repmat({[1;0]},length(w),1),'UniformOutput',0);
Zin_cell = cellfun(#(a)(a(1,1)/a(2,1)),temp_cell,'UniformOutput',0);
Zin = cell2mat(Zin);
temp2_cell = cellfun(#(a)({[1;1/a]}),Zin_cell,'UniformOutput',0);
temp3_cell = cellfun(#(a,b)(pinv(a)*b),H1,temp2_cell);
temp4 = cell2mat(temp3_cell);
p(:,1) = temp4(1:2:end-1);
Ub(:,1) = temp4(2:2:end);