I am working with a sparse matrix of very large size:
U = sparse(a,b) % a and b are very large
On the hand, there exists the cell Ind which has 'a' rows. In each row, there exists a 'variate' number of elements, e.g. :
Ind{1} = [1 3 5 19 1000 1340]
Ind{2} = [9 100 1500 1600 8000 b]
...
Ind{a} = [3 5 6 90 1000 4300 5712 9480]
as could be seen the maximum index number in Ind{i} can be 'b'. For each of these index vector also exists a content matrix like 'c' :
c = [2 3 1 6 3 5 1 3 4 1 2 ... 5]
Here is the question, for each element in Ind{i}, I want to fill the 'row = i' and the 'col=Ind{i}' with c(Ind{i}), i.e.
for i = 1 : a
U(i,Ind{i}) = c(Ind{i}) ;
end
the problem is 'a' is very large and the loop takes long time to be computed. Any idea to avoid looping?
I'm not sure if there is a way to avoid the loop, but I do get a factor of 2-to-20 speed increase (I ranged a from 3 to 5,000 with b fixed at 10,000) by building three large vectors (two for row and column indices and one for values) and building the sparse matrix after the loop:
strides = cellfun(#numel,Ind);
n = sum(strides);
I(n,1) = 0;
J(n,1) = 0;
S(n,1) = 0;
bot = 1;
for k = 1:a
top = bot + strides(k) - 1 ;
mask = bot:top ;
%
I(mask) = k ;
J(mask) = Ind{k} ;
S(mask) = c(Ind{k}) ;
%
bot = top + 1;
end
U = sparse(I,J,S,a,b);
This is the recommend usage of sparse because assignments to a sparse matrix are more costly than regular arrays.
Related
How to generate a sequence of random pairs without repeating pairs?
The following code already generates the pairs, but does not avoid repetitions:
for k=1:8
Comb=[randi([-15,15]) ; randi([-15,15])];
T{1,k}=Comb;
end
When running I got:
T= [-3;10] [5;2] [1;-5] [10;9] [-4;-9] [-5;-9] [3;1] [-3;10]
The pair [-3,10] is repeated, which cannot happen.
PS : The entries can be positive or negative.
Is there any built in function for this? Any sugestion to solve this?
If you have the Statistics Toolbox, you can use randsample to sample 8 numbers from 1 to 31^2 (where 31 is the population size), without replacement, and then "unpack" each obtained number into the two components of a pair:
s = -15:15; % population
M = 8; % desired number of samples
N = numel(s); % population size
y = randsample(N^2, M); % sample without replacement
result = s([ceil(y/N) mod(y-1, N)+1]); % unpack pair and index into population
Example run:
result =
14 1
-5 7
13 -8
15 4
-6 -7
-6 15
2 3
9 6
You can use ind2sub:
n = 15;
m = 8;
[x y]=ind2sub([n n],randperm(n*n,m));
Two possibilities:
1.
M = nchoosek(1:15, 2);
T = datasample(M, 8, 'replace', false);
2.
T = zeros(8,2);
k = 1;
while (k <= 8)
t = randi(15, [1,2]);
b1 = (T(:,1) == t(1));
b2 = (T(:,2) == t(2));
if ~any(b1 & b2)
T(k,:) = t;
k = k + 1;
end
end
The first method is probably faster but takes up more memory and may not be practicable for very large numbers (ex: if instead of 15, the max was 50000), in which case you have to go with 2.
How can I remove any number that has duplicate from an array.
for example:
b =[ 1 1 2 3 3 5 6]
becomes
b =[ 2 5 6]
Use unique function to extract unique values then compute histogram of data for unique values and preserve those that have counts of 1.
a =[ 1 1 2 3 3 5 6];
u = unique(a)
idx = hist(a, u) ==1;
b = u(idx)
result
2 5 6
for multi column input this can be done:
a = [1 2; 1 2;1 3;2 1; 1 3; 3 5 ; 3 6; 5 9; 6 10] ;
[u ,~, uid] = unique(a,'rows');
idx = hist(uid,1:size(u,1))==1;
b= u(idx,:)
You can first sort your elements and afterwards remove all elements which have the same value as one of its neighbors as follows:
A_sorted = sort(A); % sort elements
A_diff = diff(A_sorted)~=0; % check if element is the different from the next one
A_unique = [A_diff true] & [true A_diff]; % check if element is different from previous and next one
A = A_sorted(A_unique); % obtain the unique elements.
Benchmark
I will benchmark my solution with the other provided solutions, i.e.:
using diff (my solution)
using hist (rahnema1)
using sum (Jean Logeart)
using unique (my alternative solution)
I will use two cases:
small problem (yours): A = [1 1 2 3 3 5 6];
larger problem
rng('default');
A= round(rand(1, 1000) * 300);
Result:
Small Large Comments
----------------|------------|------------%----------------
using `diff` | 6.4080e-06 | 6.2228e-05 % Fastest method for large problems
using `unique` | 6.1228e-05 | 2.1923e-04 % Good performance
using `sum` | 5.4352e-06 | 0.0020 % Only fast for small problems, preserves the original order
using `hist` | 8.4408e-05 | 1.5691e-04 % Good performance
My solution (using diff) is the fastest method for somewhat larger problems. The solution of Jean Logeart using sum is faster for small problems, but the slowest method for larger problems, while mine is almost equally fast for the small problem.
Conclusion: In general, my proposed solution using diff is the fastest method.
timeit(#() usingDiff(A))
timeit(#() usingUnique(A))
timeit(#() usingSum(A))
timeit(#() usingHist(A))
function A = usingDiff (A)
A_sorted = sort(A);
A_unique = [diff(A_sorted)~=0 true] & [true diff(A_sorted)~=0];
A = A_sorted(A_unique);
end
function A = usingUnique (A)
[~, ia1] = unique(A, 'first');
[~, ia2] = unique(A, 'last');
A = A(ia1(ia1 == ia2));
end
function A = usingSum (A)
A = A(sum(A==A') == 1);
end
function A = usingHist (A)
u = unique(A);
A = u(hist(A, u) ==1);
end
I have a matrix D of distances between 3 places and 4 persons
example D(2,3) = 10 means person 3 is far away from place 2 of 10 units.
D=[23 54 67 32
32 5 10 2
3 11 13 5]
another matrix A with the same number of rows (3 places) and where A(i,:) correspond to the persons that picked place i
example for place 1, persons 1 and 3 picked it
no one picked place 2
and persons 2 and 4 picked place 3
A=[1 3 0
0 0 0
2 4 0]
I want to reorder each row of A by the persons who are closest to the place it represents.
In this example, for place 1, person 1 is closer to it than person 3 based on D so nothing to do.
nothing to do for place 2
and there is a change for place 3 since person 4 is closer than 2 to place 3 D(3,2)>D(3,4)
The result should be
A=[1 3
0 0
4 2 ]
each row(place) in A can have 0 or many non zeros elements in it (persons that picked it)
Basically, I want to reorder elements in each row of A based on the rows of D (the closest to the location comes first), something like this but here A and D are not of the same size (number of columns).
[SortedD,Ind] = sort(D,2)
for r = 1:size(A,1)
A(r,:) = A(r,Ind(r,:));
end
There is another Matlab function sortrows(C,colummn_index) that can do the trick. It can sort rows based on the elements in a particular column. So if you transpose your matrix A (C = A') and extend the result by adding to the end the proper column, according to which you want to sort a required row, then you will get what you want.
To be more specific, you can do something like this:
clear all
D=[23 54 67 32;
32 5 10 2;
3 11 13 5];
A=[1 0;
3 0;
4 2 ];
% Sort elements in each row of the matrix A,
% because indices of elements in each row of the matrix D are always
% ascending.
A_sorted = sort(A,2);
% shifting all zeros in each row to the end
for i = 1:length(A_sorted(:,1))
num_zeros = sum(A_sorted(i,:)==0);
if num_zeros < length(A_sorted(i,:))
z = zeros(1,num_zeros);
A_sorted(i,:) = [A_sorted(i,num_zeros+1:length(A_sorted(i,:))) z];
end;
end;
% Prelocate in memory an associated array of the corresponding elements in
% D. The matrix Dr is just a reduced derivation from the matrix D.
Dr = zeros(length(A_sorted(:,1)),length(A_sorted(1,:)));
% Create a matrix Dr of elements in D corresponding to the matrix A_sorted.
for i = 1:length(A_sorted(:,1)) % i = 1:3
for j = 1:length(A_sorted(1,:)) % j = 1:2
if A_sorted(i,j) == 0
Dr(i,j) = 0;
else
Dr(i,j) = D(i,A_sorted(i,j));
end;
end;
end;
% We don't need the matrix A_sorted anymore
clear A_sorted
% In order to use the function SORTROWS, we need to transpose matrices
A = A';
Dr = Dr';
% The actual sorting procedure starts here.
for i = 1:length(A(1,:)) % i = 1:3
C = zeros(length(A(:,1)),2); % buffer matrix
C(:,1) = A(:,i);
C(:,2) = Dr(:,i);
C = sortrows(C,2);
A(:,i) = C(:,1);
% shifting all zeros in each column to the end
num_zeros = sum(A(:,i)==0);
if num_zeros < length(A(:,i))
z = zeros(1,num_zeros);
A(:,i) = [A(num_zeros+1:length(A(:,i)),i) z]';
end;
end;
% Transpose the matrix A back
A = A';
clear C Dr z
I need to replace the zeros (or NaNs) in a matrix with the previous element row-wise, so basically I need this Matrix X
[0,1,2,2,1,0;
5,6,3,0,0,2;
0,0,1,1,0,1]
To become like this:
[0,1,2,2,1,1;
5,6,3,3,3,2;
0,0,1,1,1,1],
please note that if the first row element is zero it will stay like that.
I know that this has been solved for a single row or column vector in a vectorized way and this is one of the nicest way of doing that:
id = find(X);
X(id(2:end)) = diff(X(id));
Y = cumsum(X)
The problem is that the indexing of a matrix in Matlab/Octave is consecutive and increments columnwise so it works for a single row or column but the same exact concept cannot be applied but needs to be modified with multiple rows 'cause each of raw/column starts fresh and must be regarded as independent. I've tried my best and googled the whole google but coukldn’t find a way out. If I apply that same very idea in a loop it gets too slow cause my matrices contain 3000 rows at least. Can anyone help me out of this please?
Special case when zeros are isolated in each row
You can do it using the two-output version of find to locate the zeros and NaN's in all columns except the first, and then using linear indexing to fill those entries with their row-wise preceding values:
[ii jj] = find( (X(:,2:end)==0) | isnan(X(:,2:end)) );
X(ii+jj*size(X,1)) = X(ii+(jj-1)*size(X,1));
General case (consecutive zeros are allowed on each row)
X(isnan(X)) = 0; %// handle NaN's and zeros in a unified way
aux = repmat(2.^(1:size(X,2)), size(X,1), 1) .* ...
[ones(size(X,1),1) logical(X(:,2:end))]; %// positive powers of 2 or 0
col = floor(log2(cumsum(aux,2))); %// col index
ind = bsxfun(#plus, (col-1)*size(X,1), (1:size(X,1)).'); %'// linear index
Y = X(ind);
The trick is to make use of the matrix aux, which contains 0 if the corresponding entry of X is 0 and its column number is greater than 1; or else contains 2 raised to the column number. Thus, applying cumsum row-wise to this matrix, taking log2 and rounding down (matrix col) gives the column index of the rightmost nonzero entry up to the current entry, for each row (so this is a kind of row-wise "cummulative max" function.) It only remains to convert from column number to linear index (with bsxfun; could also be done with sub2ind) and use that to index X.
This is valid for moderate sizes of X only. For large sizes, the powers of 2 used by the code quickly approach realmax and incorrect indices result.
Example:
X =
0 1 2 2 1 0 0
5 6 3 0 0 2 3
1 1 1 1 0 1 1
gives
>> Y
Y =
0 1 2 2 1 1 1
5 6 3 3 3 2 3
1 1 1 1 1 1 1
You can generalize your own solution as follows:
Y = X.'; %'// Make a transposed copy of X
Y(isnan(Y)) = 0;
idx = find([ones(1, size(X, 1)); Y(2:end, :)]);
Y(idx(2:end)) = diff(Y(idx));
Y = reshape(cumsum(Y(:)), [], size(X, 1)).'; %'// Reshape back into a matrix
This works by treating the input data as a long vector, applying the original solution and then reshaping the result back into a matrix. The first column is always treated as non-zero so that the values don't propagate throughout rows. Also note that the original matrix is transposed so that it is converted to a vector in row-major order.
Modified version of Eitan's answer to avoid propagating values across rows:
Y = X'; %'
tf = Y > 0;
tf(1,:) = true;
idx = find(tf);
Y(idx(2:end)) = diff(Y(idx));
Y = reshape(cumsum(Y(:)),fliplr(size(X)))';
x=[0,1,2,2,1,0;
5,6,3,0,1,2;
1,1,1,1,0,1];
%Do it column by column is easier
x=x';
rm=0;
while 1
%fields to replace
l=(x==0);
%do nothing for the first row/column
l(1,:)=0;
rm2=sum(sum(l));
if rm2==rm
%nothing to do
break;
else
rm=rm2;
end
%replace zeros
x(l) = x(find(l)-1);
end
x=x';
I have a function I use for a similar problem for filling NaNs. This can probably be cutdown or sped up further - it's extracted from pre-existing code that has a bunch more functionality (forward/backward filling, maximum distance etc).
X = [
0 1 2 2 1 0
5 6 3 0 0 2
1 1 1 1 0 1
0 0 4 5 3 9
];
X(X == 0) = NaN;
Y = nanfill(X,2);
Y(isnan(Y)) = 0
function y = nanfill(x,dim)
if nargin < 2, dim = 1; end
if dim == 2, y = nanfill(x',1)'; return; end
i = find(~isnan(x(:)));
j = 1:size(x,1):numel(x);
j = j(ones(size(x,1),1),:);
ix = max(rep([1; i],diff([1; i; numel(x) + 1])),j(:));
y = reshape(x(ix),size(x));
function y = rep(x,times)
i = find(times);
if length(i) < length(times), x = x(i); times = times(i); end
i = cumsum([1; times(:)]);
j = zeros(i(end)-1,1);
j(i(1:end-1)) = 1;
y = x(cumsum(j));
y = 0;
for m = 0:variable
for n = 0:m
y = y + f(n,m);
end
end
I vectorized the inner loop this way,
y = 0;
for m = 0:variable
n = 0:m
y = y + f(n,m);
end
This resulted in around 60% speed increase for my code. How do I also vectorize the outer loop?
You are probably looking for the meshgrid function. It is designed to fill in the sort of m by n combinations that it looks like you need. For example:
>> m = 1:4;
>> n = 1:3;
>> [mGridValues, nGridValues] = meshgrid(m,n)
mGridValues =
1 2 3 4
1 2 3 4
1 2 3 4
nGridValues =
1 1 1 1
2 2 2 2
3 3 3 3
This is a little more complicated since your inner loop depends on the value of your outer loop. So you will need to mask out the undesired [n, m] pairs (see below).
Modifying the prototype code that you have provided, you would end up with something like this:
[mValues, nValues] = meshgrid(0:variable, 0:variable); %Start with a full combination of values
mask = mValues >= nValues; %Identify all values where m >= n
mValues = mValues(mask); % And then remove pairs which do not
nValues = nValues(mask); % meet this criteria
y = f(nValues, mValues ); %Perform whatever work you are performing here