i'm running the following code, where M is a ~200,000 by ~200,000 sparse matrix and points is ~200,000 by 2 matrix
inds=sub2ind(size(M),points(:,1),points(:,2));
M(inds)=M(inds)+1;
the problem is that the second line takes very long to run (15-90 seconds).
the operation takes longer depending on how many of the indices in inds are 'new' (i.e. that don't already have a value in the sparse matrix)
is there a more efficient way to do this?
Here's an idea:
M = M + sparse(points(:,1),points(:,2),1,size(M,1),size(M,2),size(points,1));
Just so you know,
S = sparse(i,j,s,m,n,nzmax) uses vectors i, j, and s to generate an
m-by-n sparse matrix such that S(i(k),j(k)) = s(k), with space
allocated for nzmax nonzeros. Vectors i, j, and s are all the same
length. Any elements of s that are zero are ignored, along with the
corresponding values of i and j. Any elements of s that have duplicate
values of i and j are added together.
For the curious:
M = sprand(200000,200000,1e-6);
points = [randperm(200000) ; randperm(200000)]'; %'//Initialization over
Mo = M;
tic;
inds=sub2ind(size(Mo),points(:,1),points(:,2));
Mo(inds) = Mo(inds)+1;
toc
tic;
M = M + sparse(points(:,1),points(:,2),1,size(M,1),size(M,2),size(points,1));
toc
Related
I need to construct a huge sparse matrix in iterations. The code is as follow:
function Huge_Matrix = Create_Huge_Matrix(len, Weight, Index)
k = size(Weight,1);
Huge_Matrix = spalloc(len, len,floor(len*k));
parfor i = 1:len
temp = sparse(1,len);
ind = Index(:,i);
temp(ind) = Weight(:,i);
Huge_Matrix(i,:) = temp;
end
Huge_Matrix = Huge_Matrix + spdiags(-k*ones(len,1),0,len,len);
end
As is shown, len is size of the height * weight of the input image, for 200*200 image, the len is 40000! And I am assigning the Weight into this huge matrix according the position stored in Index. Even though I use parfor to accerlate the loop, the speed is very slow.
I also try to create full matrix at first, it seems that the code can becomes faster, but memory is limited. Is there any other way to speed up the code? Thanks in advance!
As #CrisLuengo says in the comments, there is probably a better way to do what you're trying to do than to create a 40kx40k matrix, but if you have to create a large sparse matrix, it's better to let MATLAB do it for you.
The sparse function has a signature that takes lists of rows, columns and the corresponding values for the nonzero elements of the matrix:
S = sparse(i,j,v) generates a sparse matrix S from the triplets i, j, and v such that S(i(k),j(k)) = v(k). The max(i)-by-max(j) output matrix has space allotted for length(v) nonzero elements. sparse adds together elements in v that have duplicate subscripts in i and j.
If the inputs i, j, and v are vectors or matrices, they must have the same number of elements. Alternatively, the argument v and/or one of the arguments i or j can be scalars.
So, we can simply pass Index as the row indices and Weight as the values, so all we need is an array of column indices the same size as Index:
col_idx = repmat(1:len, k, 1);
Huge_Matrix = sparse(Index, col_idx, Weight, len, len);
(The last two parameters specify the size of the sparse matrix.)
The next step is to create another large sparse matrix and add it to the first. That seems kind of wasteful, so why not just add those entries to the existing arrays before creating the matrix?
Here's the final function:
function Huge_Matrix = Create_Huge_Matrix(len, Weight, Index)
k = size(Weight,1);
% add diagonal indices/weights to arrays
% this avoids creating second huge sparse array
Index(end+1, :) = [1:len];
Weight(end+1, :) = -k*ones(1,len);
% create array of column numbers corresponding to each Index
% make k+1 rows because we've added the diagonal
col_idx = repmat(1:len, k+1, 1);
% let sparse do the work
Huge_Matrix = sparse(Index, col_idx, Weight, len, len);
end
I've written a function that generates a sparse matrix of size nxd
and puts in each column 2 non-zero values.
function [M] = generateSparse(n,d)
M = sparse(d,n);
sz = size(M);
nnzs = 2;
val = ceil(rand(nnzs,n));
inds = zeros(nnzs,d);
for i=1:n
ind = randperm(d,nnzs);
inds(:,i) = ind;
end
points = (1:n);
nnzInds = zeros(nnzs,d);
for i=1:nnzs
nnzInd = sub2ind(sz, inds(i,:), points);
nnzInds(i,:) = nnzInd;
end
M(nnzInds) = val;
end
However, I'd like to be able to give the function another parameter num-nnz which will make it choose randomly num-nnz cells and put there 1.
I can't use sprand as it requires density and I need the number of non-zero entries to be in-dependable from the matrix size. And giving a density is basically dependable of the matrix size.
I am a bit confused on how to pick the indices and fill them... I did with a loop which is extremely costly and would appreciate help.
EDIT:
Everything has to be sparse. A big enough matrix will crash in memory if I don't do it in a sparse way.
You seem close!
You could pick num_nnz random (unique) integers between 1 and the number of elements in the matrix, then assign the value 1 to the indices in those elements.
To pick the random unique integers, use randperm. To get the number of elements in the matrix use numel.
M = sparse(d, n); % create dxn sparse matrix
num_nnz = 10; % number of non-zero elements
idx = randperm(numel(M), num_nnz); % get unique random indices
M(idx) = 1; % Assign 1 to those indices
I am trying to find the entry of matrix Athat has the maximum value.
I have generated matrix A, how can I ask MATLAB to return the four indices in addition to the maximum value of the entry within matrix A
for i = 1:size(CB,2)
for j=1:size(CB,2)
for k=1:size(CB,2)
for l=1:size(CB,2)
A(i,j,k,l)= (abs( conj(transpose([CB(:,i); CB(:,j)]))*MATRIX* [CB(:,k); CB(:,l)])^2);
end
end
end
end
You can use a combination of max and ind2sub:
a = rand(5, 5, 5, 5);
[maxa, maxidx] = max(a(:));
[I, J, K, L] = ind2sub(size(a), maxidx);
Which we can test:
>> a(I, J, K, L) == maxa
ans =
1
The way this works is that we receive a linear index from the second output of the max command. I used the colon operator with max so our input is really one long column vector of a, and the output is the maximum value of the entire matrix, maxa, along with the location of that value in the column vector, maxidx. You can then use ind2sub with size to convert that linear index into subscripts for your matrix.
Use 1-D indexing:
[M,I] = max(A(:));
I is then the index in A where M resides (i.e., M = A(I))
Then you need to use the following to convert from 1D indexing to 4D indexing:
[a,b,c,d] = ind2sub(size(A),I);
Question title explains what I would like. For example, if there are 6 elements in a particular column, how do I randomly pick 1 element from that column. Please keep it simple if possible.
Thanks for the help.
Suppose you have a matrix A of size m-by-n. You wish to pick one element from each of the n columns at random:
>> rows = randsample( m, n ); % sample n times from integers 1:m
Now rows has n values, each represent a random entry at the corresponding column.
To access those values
>> sampledValues = A( sub2ind( size(A), rows, 1:n ) );
For more information see the doc on randsample and sub2ind.
You can use randi if your version of MATLAB is > R2008a
samples = A(sparse(randi(size(A,1),size(A,2),1), 1:size(A,2), true));
or,
[m, n] = size(A);
samples = A(sparse(randi(m,n,1), 1:n, true));
However for older versions you can replace randi with randsample but that requires Statistics Toolbox. Or introduce:
randi = #(imax, m, n) floor(1+rand(m,n)*imax);
Here is an easy way to do it. Note that a version without loops should be more efficient.
Assuming your variable is x loop over its n columns:
selected = zeros(1,n);
for c = 1:n
selected(c) = x(randperm(6,1),n);
end
I have a non-fixed dimensional matrix M, from which I want to access a single element.
The element's indices are contained in a vector J.
So for example:
M = rand(6,4,8,2);
J = [5 2 7 1];
output = M(5,2,7,1)
This time M has 4 dimensions, but this is not known in advance. This is dependent on the setup of the algorithm I'm writing. It could likewise be that
M = rand(6,4);
J = [3 1];
output = M(3,1)
so I can't simply use
output=M(J(1),J(2))
I was thinking of using sub2ind, but this also needs its variables comma separated..
#gnovice
this works, but I intend to use this kind of element extraction from the matrix M quite a lot. So if I have to create a temporary variable cellJ every time I access M, wouldn't this tremendously slow down the computation??
I could also write a separate function
function x= getM(M,J)
x=M(J(1),J(2));
% M doesn't change in this function, so no mem copy needed = passed by reference
end
and adapt this for different configurations of the algorithm. This is of course a speed vs flexibility consideration which I hadn't included in my question..
BUT: this is only available for getting the element, for setting there is no other way than actually using the indices (and preferably the linear index). I still think sub2ind is an option. The final result I had intended was something like:
function idx = getLinearIdx(J, size_M)
idx = ...
end
RESULTS:
function lin_idx = Lidx_ml( J, M )%#eml
%LIDX_ML converts an array of indices J for a multidimensional array M to
%linear indices, directly useable on M
%
% INPUT
% J NxP matrix containing P sets of N indices
% M A example matrix, with same size as on which the indices in J
% will be applicable.
%
% OUTPUT
% lin_idx Px1 array of linear indices
%
% method 1
%lin_idx = zeros(size(J,2),1);
%for ii = 1:size(J,2)
% cellJ = num2cell(J(:,ii));
% lin_idx(ii) = sub2ind(size(M),cellJ{:});
%end
% method 2
sizeM = size(M);
J(2:end,:) = J(2:end,:)-1;
lin_idx = cumprod([1 sizeM(1:end-1)])*J;
end
method 2 is 20 (small number of index sets (=P) to convert) to 80 (large number of index sets (=P)) times faster than method 1. easy choice
For the general case where J can be any length (which I assume always matches the number of dimensions in M), there are a couple options you have:
You can place each entry of J in a cell of a cell array using the num2cell function, then create a comma-separated list from this cell array using the colon operator:
cellJ = num2cell(J);
output = M(cellJ{:});
You can sidestep the sub2ind function and compute the linear index yourself with a little bit of math:
sizeM = size(M);
index = cumprod([1 sizeM(1:end-1)]) * (J(:) - [0; ones(numel(J)-1, 1)]);
output = M(index);
Here is a version of gnovices option 2) which allows to process a whole matrix of subscripts, where each row contains one subscript. E.g for 3 subscripts:
J = [5 2 7 1
1 5 2 7
4 3 9 2];
sizeM = size(M);
idx = cumprod([1 sizeX(1:end-1)])*(J - [zeros(size(J,1),1) ones(size(J,1),size(J,2)-1)]).';