Eliminate/Remove duplicates from array Matlab - matlab

How can I remove any number that has duplicate from an array.
for example:
b =[ 1 1 2 3 3 5 6]
becomes
b =[ 2 5 6]

Use unique function to extract unique values then compute histogram of data for unique values and preserve those that have counts of 1.
a =[ 1 1 2 3 3 5 6];
u = unique(a)
idx = hist(a, u) ==1;
b = u(idx)
result
2 5 6
for multi column input this can be done:
a = [1 2; 1 2;1 3;2 1; 1 3; 3 5 ; 3 6; 5 9; 6 10] ;
[u ,~, uid] = unique(a,'rows');
idx = hist(uid,1:size(u,1))==1;
b= u(idx,:)

You can first sort your elements and afterwards remove all elements which have the same value as one of its neighbors as follows:
A_sorted = sort(A); % sort elements
A_diff = diff(A_sorted)~=0; % check if element is the different from the next one
A_unique = [A_diff true] & [true A_diff]; % check if element is different from previous and next one
A = A_sorted(A_unique); % obtain the unique elements.
Benchmark
I will benchmark my solution with the other provided solutions, i.e.:
using diff (my solution)
using hist (rahnema1)
using sum (Jean Logeart)
using unique (my alternative solution)
I will use two cases:
small problem (yours): A = [1 1 2 3 3 5 6];
larger problem
rng('default');
A= round(rand(1, 1000) * 300);
Result:
Small Large Comments
----------------|------------|------------%----------------
using `diff` | 6.4080e-06 | 6.2228e-05 % Fastest method for large problems
using `unique` | 6.1228e-05 | 2.1923e-04 % Good performance
using `sum` | 5.4352e-06 | 0.0020 % Only fast for small problems, preserves the original order
using `hist` | 8.4408e-05 | 1.5691e-04 % Good performance
My solution (using diff) is the fastest method for somewhat larger problems. The solution of Jean Logeart using sum is faster for small problems, but the slowest method for larger problems, while mine is almost equally fast for the small problem.
Conclusion: In general, my proposed solution using diff is the fastest method.
timeit(#() usingDiff(A))
timeit(#() usingUnique(A))
timeit(#() usingSum(A))
timeit(#() usingHist(A))
function A = usingDiff (A)
A_sorted = sort(A);
A_unique = [diff(A_sorted)~=0 true] & [true diff(A_sorted)~=0];
A = A_sorted(A_unique);
end
function A = usingUnique (A)
[~, ia1] = unique(A, 'first');
[~, ia2] = unique(A, 'last');
A = A(ia1(ia1 == ia2));
end
function A = usingSum (A)
A = A(sum(A==A') == 1);
end
function A = usingHist (A)
u = unique(A);
A = u(hist(A, u) ==1);
end

Related

Row-wise "ismember" without for-loop

I am trying to perform a row-wise "ismember" in MATLAB in order to find out where, in Set, each element of Input is.
Here is my work so far.
function lia = ismemberrow(Input, Set)
lia = false(size(Input)); % Pre-assign the output lia
for J = 1 : size(Input,1)
% The J-th row of "lia" is the list of locations of the
% elements of Input(J,:) in Set
lia(J,:) = find(ismember(Set, Input(J,:)));
end
end
For example, if the variables Input and Set are defined as follows
Input = [1 4;
4 2;
4 3;
2 4;
1 2;
3 2];
Set = [3 2 4 1];
The the output lia of lia = ismemberrow(Input,Set) will be:
lia = [4 3;
3 2;
3 1;
2 3;
4 2;
1 2];
My function works accurately so far, but this function is called many times in my project so I am thinking of that if I can reduce the for-loop so that it spends less time. May I have some opinions on it?
A single call to ismember (no loop necessary) will give you what you want in the second output argument:
>> [~, lia] = ismember(Input, Set)
lia =
4 3
3 2
3 1
2 3
4 2
1 2
I'd go with ismember as in #gnovice's answer. But here are some alternatives, just for the fun of it.
If the values in Input are guaranteed to be in Set:
[ind, ~] = find(bsxfun(#eq, Set(:), Input(:).'));
result = reshape(ind, size(Input));
If they are not guaranteed to be:
[ind, val] = max(bsxfun(#eq, Set(:), permute(Input, [3 1 2])));
result = permute(ind.*val, [2 3 1]);
If your inputs are positive integers you simply can use indexing
m(Set)=1:numel(Set);
result = m(Input)
If the range of input is large you can use sparse matrix:
s = sparse(Set,1,1:numel(Set));
result = s(Input)
Result:
4 3
3 2
3 1
2 3
4 2
1 2

Matlab index matrix by vector specifying column

Let A be of size [n,m], i.e. it has n rows and m columns. Given I of size [n,1] with max(I)<=m, what is the fastest way to return B of size [n,1], such that B(i)=A(i,I(i))?
Example:
A =
8 1 6
3 5 7
4 9 2
and
I =
1
2
2
I want B to look like
B =
8
5
9
There obviously exist several ways to implement this, but in my case n is in the order of 1e6 and m in the order of 1e2, which is why I'm interested in the fastest implementation. I would like to avoid ind2sub or sub2ind since they both appear to be too slow as well. Any idea is greatly appreciated! Thanks!
You can replicate the behavior of sub2ind yourself. This gives me a speedup in my test:
clear
%% small example
A = rand(4,6)
I = [3 2 2 1]
inds = (I-1)*size(A,1) + (1:length(I));
B = A(inds)
%% timing
n = 1e4;
m = 1e2;
A = rand(n, m);
I = ceil(rand(1,n) * m);
% sub2ind
F = #() A(sub2ind(size(A), 1:size(A,1), I));
timeit(F)
% manual
F = #() A((I-1)*size(A,1) + (1:length(I)));
timeit(F)
You can also use something like this:
A(meshgrid(1:size(A,2),1:size(A,1)) == repmat(I,1,size(A,2)))
which will give you the same result, with no loop and no sub2ind.

Find unique rows of a cell array considering all possible permutations on each row

I have cell array A of dimension m * k.
I want to keep the rows of A unique up to an order of the k cells.
The "tricky" part is "up to an order of the k cells": consider the k cells in the ith row of A, A(i,:); there could be a row j of A, A(j,:), that is equivalent to A(i,:) up to a re-ordering of its k cells, meaning that for example if k=4it could be that:
A{i,1}=A{j,2}
A{i,2}=A{j,3}
A{i,3}=A{j,1}
A{i,4}=A{j,4}
What I am doing at the moment is:
G=[0 -1 1; 0 -1 2; 0 -1 3; 0 -1 4; 0 -1 5; 1 -1 6; 1 0 6; 1 1 6; 2 -1 6; 2 0 6; 2 1 6; 3 -1 6; 3 0 6; 3 1 6];
h=7;
M=reshape(G(nchoosek(1:size(G,1),h),:),[],h,size(G,2));
A=cell(size(M,1),2);
for p=1:size(M,1)
A{p,1}=squeeze(M(p,:,:));
left=~ismember(G, A{p,1}, 'rows');
A{p,2}=G(left,:);
end
%To find equivalent rows up to order I use a double loop (VERY slow).
indices=[];
for j=1:size(A,1)
if ismember(j,indices)==0 %if we have not already identified j as a duplicate
for i=1:size(A,1)
if i~=j
if (isequal(A{j,1},A{i,1}) || isequal(A{j,1},A{i,2}))...
&&...
(isequal(A{j,2},A{i,1}) || isequal(A{j,2},A{i,2}))...
indices=[indices;i];
end
end
end
end
end
A(indices,:)=[];
It works but it is too slow. I am hoping that there is something quicker that I can use.
I'd like to propose another idea, which has some conceptual resemblance to erfan's. My idea uses hash functions, and specifically, the GetMD5 FEX submission.
The main task is how to "reduce" each row in A to a single representative value (such as a character vector) and then find unique entries of this vector.
Judging by the benchmark vs. the other suggestions, my answer doesn't perform as well as one of the alternatives, but I think its raison d'être lies in the fact that it is completely data-type agnostic (within the limitations of the GetMD51), that the algorithm is very straightforward to understand, it's a drop-in replacement as it operates on A, and that the resulting array is exactly equal to the one obtained by the original method. Of course this requires a compiler to get working and has a risk of hash collisions (which might affect the result in VERY VERY rare cases).
Here are the results from a typical run on my computer, followed by the code:
Original method timing: 8.764601s
Dev-iL's method timing: 0.053672s
erfan's method timing: 0.481716s
rahnema1's method timing: 0.009771s
function q39955559
G=[0 -1 1; 0 -1 2; 0 -1 3; 0 -1 4; 0 -1 5; 1 -1 6; 1 0 6; 1 1 6; 2 -1 6; 2 0 6; 2 1 6; 3 -1 6; 3 0 6; 3 1 6];
h=7;
M=reshape(G(nchoosek(1:size(G,1),h),:),[],h,size(G,2));
A=cell(size(M,1),2);
for p=1:size(M,1)
A{p,1}=squeeze(M(p,:,:));
left=~ismember(G, A{p,1}, 'rows');
A{p,2}=G(left,:);
end
%% Benchmark:
tic
A1 = orig_sort(A);
fprintf(1,'Original method timing:\t\t%fs\n',toc);
tic
A2 = hash_sort(A);
fprintf(1,'Dev-iL''s method timing:\t\t%fs\n',toc);
tic
A3 = erfan_sort(A);
fprintf(1,'erfan''s method timing:\t\t%fs\n',toc);
tic
A4 = rahnema1_sort(G,h);
fprintf(1,'rahnema1''s method timing:\t%fs\n',toc);
assert(isequal(A1,A2))
assert(isequal(A1,A3))
assert(isequal(numel(A1),numel(A4))) % This is the best test I could come up with...
function out = hash_sort(A)
% Hash the contents:
A_hashed = cellfun(#GetMD5,A,'UniformOutput',false);
% Sort hashes of each row:
A_hashed_sorted = A_hashed;
for ind1 = 1:size(A_hashed,1)
A_hashed_sorted(ind1,:) = sort(A_hashed(ind1,:));
end
A_hashed_sorted = cellstr(cell2mat(A_hashed_sorted));
% Find unique rows:
[~,ia,~] = unique(A_hashed_sorted,'stable');
% Extract relevant rows of A:
out = A(ia,:);
function A = orig_sort(A)
%To find equivalent rows up to order I use a double loop (VERY slow).
indices=[];
for j=1:size(A,1)
if ismember(j,indices)==0 %if we have not already identified j as a duplicate
for i=1:size(A,1)
if i~=j
if (isequal(A{j,1},A{i,1}) || isequal(A{j,1},A{i,2}))...
&&...
(isequal(A{j,2},A{i,1}) || isequal(A{j,2},A{i,2}))...
indices=[indices;i];
end
end
end
end
end
A(indices,:)=[];
function C = erfan_sort(A)
STR = cellfun(#(x) num2str((x(:)).'), A, 'UniformOutput', false);
[~, ~, id] = unique(STR);
IC = sort(reshape(id, [], size(STR, 2)), 2);
[~, col] = unique(IC, 'rows');
C = A(sort(col), :); % 'sort' makes the outputs exactly the same.
function A1 = rahnema1_sort(G,h)
idx = nchoosek(1:size(G,1),h);
%concatenate complements
M = [G(idx(1:size(idx,1)/2,:),:), G(idx(end:-1:size(idx,1)/2+1,:),:)];
%convert to cell so A1 is unique rows of A
A1 = mat2cell(M,repmat(h,size(idx,1)/2,1),repmat(size(G,2),2,1));
1 - If more complicated data types need to be hashed, one can use the DataHash FEX submission instead, which is somewhat slower.
Stating the problem: The ideal choice in identifying unique rows in an array is to use C = unique(A,'rows'). But there are two major problems here, preventing us from using this function in this case. First is that you want to count in all the possible permutations of each row when comparing to other rows. If A has 5 columns, it means checking 120 different re-arrangements per row! Sounds impossible.
The second issue is related to unique itself; It does not accept cells except cell arrays of character vectors. So you cannot simply pass A to unique and get what you expect.
Why looking for an alternative? As you know, because currently it is very slow:
With nested loop method:
------------------- Create the data (first loop):
Elapsed time is 0.979059 seconds.
------------------- Make it unique (second loop):
Elapsed time is 14.218691 seconds.
My solution:
Generate another cell array containing same cells, but converted to string (STR).
Find the index of all unique elements there (id).
Generate the associated matrix with the unique indices and sort rows (IC).
Find unique rows (rows).
Collect corresponding rows of A (C).
And this is the code:
disp('------------------- Create the data:')
tic
G = [0 -1 1; 0 -1 2; 0 -1 3; 0 -1 4; 0 -1 5; 1 -1 6; 1 0 6; ...
1 1 6; 2 -1 6; 2 0 6; 2 1 6; 3 -1 6; 3 0 6; 3 1 6];
h = 7;
M = reshape(G(nchoosek(1:size(G,1),h),:),[],h,size(G,2));
A = cell(size(M,1),2);
for p = 1:size(M,1)
A{p, 1} = squeeze(M(p,:,:));
left = ~ismember(G, A{p,1}, 'rows');
A{p,2} = G(left,:);
end
STR = cellfun(#(x) num2str((x(:)).'), A, 'UniformOutput', false);
toc
disp('------------------- Make it unique (vectorized):')
tic
[~, ~, id] = unique(STR);
IC = sort(reshape(id, [], size(STR, 2)), 2);
[~, col] = unique(IC, 'rows');
C = A(sort(col), :); % 'sort' makes the outputs exactly the same.
toc
Performance check:
------------------- Create the data:
Elapsed time is 1.664119 seconds.
------------------- Make it unique (vectorized):
Elapsed time is 0.017063 seconds.
Although initialization needs a bit more time and memory, this method is extremely faster in finding unique rows with the consideration of all permutations. Execution time is almost insensitive to the number of columns in A.
It seems that G is a misleading point.
Here is result of nchoosek for a small number
idx=nchoosek(1:4,2)
ans =
1 2
1 3
1 4
2 3
2 4
3 4
first row is complement of the last row
second row is complement of one before the last row
.....
so if we extract rows {1 , 2} from G then its complement will be rows {3, 4} and so on. In the other words if we assume number of rows of G to be 4 then G(idx(1,:),:) is complement of G(idx(end,:),:).
Since rows of G are all unique then all A{m,n}s always have the same size.
A{p,1} and A{p,2} are complements of each other. and size of unique rows of A is size(idx,1)/2
So no need to any loop or further comparison:
h=7;
G = [0 -1 1; 0 -1 2; 0 -1 3; 0 -1 4; 0 -1 5; 1 -1 6; 1 0 6; ...
1 1 6; 2 -1 6; 2 0 6; 2 1 6; 3 -1 6; 3 0 6; 3 1 6];
idx = nchoosek(1:size(G,1),h);
%concatenate complements
M = [G(idx(1:size(idx,1)/2,:).',:), G(idx(end:-1:size(idx,1)/2+1,:).',:)];
%convert to cell so A1 is unique rows of A
A1 = mat2cell(M,repmat(h,size(idx,1)/2,1),repmat(size(G,2),2,1));
Update: Above method works best however if the idea is to get A1 from A other than G I suggest following method based of erfan' s. Instead of converting array to string we can directly work with the array:
STR=reshape([A.'{:}],numel(A{1,1}),numel(A)).';
[~, ~, id] = unique(STR,'rows');
IC = sort(reshape(id, size(A, 2),[]), 1).';
[~, col] = unique(IC, 'rows');
C1 = A(sort(col), :);
Since I use Octave I can not currently run mex file then I cannot test Dev-iL 's method
Result:
erfan method (string): 4.54718 seconds.
rahnema1 method (array): 0.012639 seconds.
Online Demo

Vector of the occurence number

I have a vector a=[1 2 3 1 4 2 5]'
I am trying to create a new vector that would give for each row, the occurence number of the element in a. For instance, with this matrix, the result would be [1 1 1 2 1 2 1]': The fourth element is 2 because this is the first time that 1 is repeated.
The only way I can see to achieve that is by creating a zero vector whose number of rows would be the number of unique elements (here: c = [0 0 0 0 0] because I have 5 elements).
I also create a zero vector d of the same length as a. Then, going through the vector a, adding one to the row of c whose element we read and the corresponding number of c to the current row of d.
Can anyone think about something better?
This is a nice way of doing it
C=sum(triu(bsxfun(#eq,a,a.')))
My first suggestion was this, a not very nice for loop
for i=1:length(a)
F(i)=sum(a(1:i)==a(i));
end
This does what you want, without loops:
m = max(a);
aux = cumsum([ ones(1,m); bsxfun(#eq, a(:), 1:m) ]);
aux = (aux-1).*diff([ ones(1,m); aux ]);
result = sum(aux(2:end,:).');
My first thought:
M = cumsum(bsxfun(#eq,a,1:numel(a)));
v = M(sub2ind(size(M),1:numel(a),a'))
on a completely different level, you can look into tabulate to get info about the frequency of the values. For example:
tabulate([1 2 4 4 3 4])
Value Count Percent
1 1 16.67%
2 1 16.67%
3 1 16.67%
4 3 50.00%
Please note that the solutions proposed by David, chappjc and Luis Mendo are beautiful but cannot be used if the vector is big. In this case a couple of naïve approaches are:
% Big vector
a = randi(1e4, [1e5, 1]);
a1 = a;
a2 = a;
% Super-naive solution
tic
x = sort(a);
x = x([find(diff(x)); end]);
for hh = 1:size(x, 1)
inds = (a == x(hh));
a1(inds) = 1:sum(inds);
end
toc
% Other naive solution
tic
x = sort(a);
y(:, 1) = x([find(diff(x)); end]);
y(:, 2) = histc(x, y(:, 1));
for hh = 1:size(y, 1)
a2(a == y(hh, 1)) = 1:y(hh, 2);
end
toc
% The two solutions are of course equivalent:
all(a1(:) == a2(:))
Actually, now the question is: can we avoid the last loop? Maybe using arrayfun?

Matlab swap

I am trying to create a function that will swap a specific number in a matrix with a specific number in the same matrix. For examlpe, if I start with A = [1 2 3;1 3 2], I want to be able to create B = [2 1 3; 2 3 1], simply by telling matlab to swap the 1's with the 2's. Any advice would be appreciated. Thanks!
If you have the following matrix:
A = [1 2 3; 1 3 2];
and you want all the ones to become twos and the twos to become ones, the following would be the simplest way to do it:
B = A;
B(find(A == 1)) = 2;
B(find(A == 2)) = 1;
EDIT:
As Kenny suggested, this can even be further simplified as:
B = A;
B(A == 1) = 2;
B(A == 2) = 1;
Another way to deal with the original problem is to create a permutation vector indicating to which numbers should the original entries be mapped to. For the example, entries [1 2 3] should be mapped respectively to [2 1 3], so that we can write
A = [1 2 3; 1 3 2];
perm = [2 1 3];
B = perm(A)
(advantage here is that everything is done in one step, and that it also works for operations more complicated than swaps ; drawback is that all elements of A must be positive integers with a known maximum)
Not sure why you would to perform that particular swap (row/column interchanges are more common). Matlab often denotes ':' to represent all of something. Here's how to swap rows and columns:
To swap rows:
A = A([New order of rows,,...], :)
To Swap columns:
A = A(:, [New order of columns,,...])
To change the entire i-th column:
A(:, i) = [New; values; for; i-th; column]
For example, to swap the 2nd and 3rd columns of A = [1 2 3;1 3 2]
A = A(:, [1, 3, 2])
A = [1 2 3; 1 3 2]
alpha = 1;
beta = 2;
indAlpha = (A == alpha);
indBeta = (A == beta);
A(indAlpha) = beta;
A(indBeta ) = alpha
I like this solution, it makes it clearer what is going on. Less magic numbers, could easily be made into a function. Recycles the same matrix if that is important.
I don't have a copy of MatLab installed, but I think you can do some thing like this;
for i=1:length(A)
if (A(i)=1), B(i) = 2, B(i)=A(i)
end
Note, that's only convert 1's to 2's and it looks like you also want to convert 2's to 1's, so you'll need to do a little more work.
There also probably a much more elegant way of doing it given you can do this sort of thing in Matlab
>> A = 1:1:3
A = [1,2,3]
>> B = A * 2
B = [2,4,6]
There might be a swapif primitive you can use, but I haven't used Matlab in a long time, so I'm not sure the best way to do it.
In reference to tarn's more elegant way of swapping values you could use a permutation matrix as follows:
>> a =[1 2 3];
>> T = [1 0 0;
0 0 1;
0 1 0];
>> b = a*T
ans =
1 3 2
but this will swap column 2 and column 3 of the vector (matrix) a; whereas the question asked about swapping the 1's and 2's.
Update
To swap elements of two different values look into the find function
ind = find(a==1);
returns the indices of all the elements with value, 1. Then you can use Mitch's suggestion to change the value of the elements using index arrays. Remeber that find returns the linear index into the matrix; the first element has index 1 and the last element of an nxm matrix has linear index n*m. The linear index is counted down the columns. For example
>> b = [1 3 5;2 4 6];
>> b(3) % same as b(1,2)
ans = 3
>> b(5) % same as b(1,3)
ans = 5
>> b(6) % same as b(2,3)
ans = 6