Related
I have an n x m data matrix with n samples and m measurements per sample. I'm dealing with data from mass spectrometry, measuring the concentration of different metabolites. Each column is the concentrations of a single metabolite. The rows are the samples. Some of the samples have a few metabolite measurements that are much higher than the rest of the samples.
I want to find these outlier values, and replace them with NaN. Is there a way to do this automatically, maybe by looking for values higher than X column SDs and making them NaN? I have found relevant questions for R and Python, but not for MATLAB.
Addendum: dfri's solution worked perfectly for me. However, I couldn't use the column SD as a cutoff-measure, because the outliers made the SD so large that the outlier values were still within the threshold (they were 10 000 times larger than the rest). I ended up using 100 x the column median as a threshold for removal.
You can compare elements in your data for some threshold to identify your outliers, and use the resulting indices to replace outlier values by NaN. E.g.
data = randi(4,5); %// values in {1, 2, 3, 4}
threshold = 3; %// decide upon your threshold
data(data > threshold) = NaN
data =
NaN 3 NaN 2 2
3 1 3 2 2
2 2 2 NaN 3
3 1 NaN NaN 3
1 1 1 1 NaN
If you want to replace outliers w.r.t. some threshold column per column, you can make use of e.g. bsxfun (thanks #Dan):
data = randi(4,5) %// values in {1, 2, 3, 4}
threshold = mean(data)+1*std(data) %// per column
data(bsxfun(#(x, y) x > y, data, threshold)) = NaN
%// example:
threshold =
4.7416 3.7416 4.0000 2.8954 1.9477
data =
4 3 2 NaN NaN
4 NaN 3 1 1
1 3 4 1 NaN
4 1 4 1 1
4 1 2 NaN 1
Note that the most important (non-matlab-technical) part in your case, as mentioned by #Dan in his comments above, is to decide upon how you create your threshold values for each of the columns. The simple thresholds in the example above has only been included to show the technical aspects of how to "remove" outliers (set to NaN) given an array of thresholds for the columns.
this is my problem, for example i have an equation x + y =2, so using matlab i want to know how to determine all the possible combination of values of x and y when you add it, and will give sum of 2 (ex: x1 = 0.98, y1 =0.12; x2=0.94 y2=0.16, and etc)
i think i need to use for loop?
for x = 2-y
end
for y =2-x
end
Values of x and y
x y
0 2
0.1 1.9
0.2 1.8
0.3 1.7
0.4 1.6
0.5 1.5
0.6 1.4
0.7 1.3
0.8 1.2
0.9 1.1
1 1
so guys i need your help thanks
To get all possible combinations of x and y between 0 and 2 with a step size of 0.1 you don't even need a for loop. You can create a vector x which contains all possible x values and then calculate the corresponding y's:
x = 0:0.1:2; % Create a vector of values between 0 and 2 in steps 0f 0.1
y = 2 - x;
This will give you two (row) vectors containing all possible combinations which add up to 2.
I have a Nx3-matrix in matlab, where I have a degree value from 0 to 360 in the first column, a radius value from 0 to 0.5 in the second and an integer in the third. Every combination out of (M(n,1),M(n,2)) is unique with M the matrix and n a random number between 1 and N, but it is possible that there is a value in M(:,1) or M(:,2) more than once. M is sorted, first after the first row, then after the second.
My target is now to reshape this matrix into a 360xV-matrix, with V the amount of unique values in M(:,2). If there is a value in M(:,3) at the position M(o,1) and M(p,2) with 1 <= o, p <= N, it should be placed at the position (o,p), if there is no value, then there should a NaN-value placed instead.
Is there a simple way to do this, or do I have to write my own function for that?
Edit:
Desired input:
0 0.25 1
0 0.43 4
1 0.25 2
2 0.03 5
2 0.43 2
Desired output:
NaN 1 4
NaN 2 NaN
5 NaN 2
You can use an approach of finding unique indices for the first and second columns from the input arrays and then using those to set elements in an appropriately (discussed in more detail inside the code as comments) sized output array with the elements from the third column. Here's the implementation -
%// Input array
A = [
0 0.25 1
0 0.43 4
1 0.25 2
2 0.03 5
2 0.43 2 ]
%// Find unique indices for col-1,2
[~,~,idx1] = unique(A(:,1)) %// these would form the row indices of output
[~,~,idx2] = unique(A(:,2)) %// these would form the col indices of output
%// Decide the size of output array based on the "extents" of those indices
nrows = max(idx1)
ncols = max(idx2)
%// Initialize output array with NaNs
out = NaN(nrows,ncols)
%// Set the elements in output indexed by those indices to values from
%// col-3 of input array
out((idx2-1)*nrows + idx1) = A(:,3)
Code run -
out =
NaN 1 4
NaN 2 NaN
5 NaN 2
Is there a simple way to do this, or do I have to write my own function for that?
You'll need to write a method, seeing that what you've described is utterly specific to your problem. There's methods to find unique values, so this will help you when designing your for loop.
Let's say I have a 2 by 9 matrix. I want to replace the 2 by 3 matrices inside this matrix with respect to descending sort of a(2,3), a(2,6), and a(2,9) elements. For example:
a =
0.4 0.4 0.5 0.6 0.2 0.2 0.6 0.2 0.6
0.5 0.8 0.9 0.9 0.6 0.6 0.1 0.2 0.8
[b i] = sort(a(2,3:3:end),2,'descend')
b =
0.9 0.8 0.6
i =
1 3 2
So, I want to have the following matrix:
a =
0.4 0.4 0.5 0.6 0.2 0.6 0.6 0.2 0.6
0.5 0.8 0.9 0.1 0.2 0.8 0.9 0.6 0.6
Try converting to a cell matrix first and then using your i to rearrange the cells
[b i] = sort(a(2,3:3:end),2,'descend')
A = mat2cell(a, 2, 3*ones(1,3));
cell2mat(A(i))
If for whatever reason you don't want to convert the whole of a into a cell matrix, you can do it by extending your indexing vector i to index all the columns. In your case you'd need:
I = [1,2,3,7,8,9,4,5,6]
which you could generate using a loop or else use bsxfun to get
[1 7 4
2 8 5
3 9 6]
and then "flatten" using reshape:
I = reshape(bsxfun(#plus, 3*s-2, (0:2)'), 1, [])
and then finally
a(:,I)
Typically, when a 2d matrix is separated into blocks, best practice ist to use more dimensions:
a=reshape(a,size(a,1),3,[]);
Now you can access each block via a(:,:,1)
To sort use:
[~,idx]=sort(a(2,3,:),'descend')
a=a(:,:,idx)
If you really need a 2d matrix, change back:
a=reshape(a,2,[])
sortrows-based approach:
n = 3; %// number of columns per block
m = size(a,1);
a = reshape(sortrows(reshape(a, m*n, []).', -m*n).', m, []);
This works by reshaping each block into a row, sorting rows according to last column, and reshaping back.
Summary
I have a working that solves my problem as described below, but would like to improve on it by eliminating the loop.
Description
I have a matrix B in Matlab of dimension (2*5)x3which is a concatenation of 2 matrices of 5 rows and where the ikth element reports an index from {1,2,3,4,5}(breaks added for clarity) . The indices can be repeated across rows. For each submatrix of dimension 5x3 of B the listed indices and the order in which they are listed coincide.
B=[0.1 0.2 |1;
0.3 0.4 |2;
0.5 0.6 |2;
0.7 0.8 |3;
0.9 1.1 |1;
---------
1.2 1.3 |1;
1.4 1.5 |2;
1.6 1.7 |2;
1.8 1.9 |3;
2.1 2.2 |1;]
Inside each submatrix of dimension 5x3 of B I would like to sum the rows with same index and get
C = [1 1.3 |1;
0.8 1 |2;
0.7 0.8 |3;
---------
3.3 3.5 |1;
3 3.2 |2;
1.8 1.9 |3;]
without looping.
What have I tried
My incomplete, but working, attempt with one loop:
D = permute(reshape(B,5,2,[]),[1 3 2]);
sum=zeros(3,2,2);
for i=1:2
E=D(:,:,i);
[b,c] = size(E);
rows = ceil(1/(c-1):1/(c-1):b);
cols = repmat(1:c-1,1,b);
sum(:,:,i) = full(sparse(E(rows,end), cols, E(:,1:end-1).'));
end