I have to create a very big 3D matrix (such as: 500000x60x60). Is there any way to do this in matlab?
When I try
omega = zeros(500000,60,60,'single');
I get an out-of-memory error.
The sparse function is no option since it is only meant for 2D matrices. So is there any alternative to that for higher dimensional matrices?
Matlab only has support for sparse matrices (2D). For 3D tensors/arrays, you'll have to use a workaround. I can think of two:
linear indexing
cell arrays
Linear indexing
You can create a sparse vector like so:
A = spalloc(500000*60*60, 1, 100);
where the last entry (100) refers to the amount of non-zeros eventually to be assigned to A. If you know this amount beforehand it makes memory usage for A more efficient. If you don't know it beforehand just use some number close to it, it'll still work, but A can consume more memory in the end than it strictly needs to.
Then you can refer to elements as if it is a 3D array like so:
A(sub2ind(size(A), i,j,k))
where i, j and k are the indices to the 1st, 2nd and 3rd dimension, respectively.
Cell arrays
Create each 2D page in the 3D tensor/array as a cell array:
a = cellfun(#(x) spalloc(500000, 60, 100), cell(60,1), 'UniformOutput', false);
The same story goes for this last entry into spalloc. Then concatenate in 3D like so:
A = cat(3, a{:});
then you can refer to individual elements like so:
A{i,j,k}
where i, j and k are the indices to the 1st, 2nd and 3rd dimension, respectively.
Since your matrix is sparse, try to use ndsparse (N-dimensional sparse arrays FEX)
Related
I want to sum elements of array which every element is a matrix.
I wrote below but not working:
AA={[1 2;3 4],[5 6;7 8]}
i=1:2;
sum(AA{i})
If you are wanting to perform operations across a set of 2-D matrices, all of which are the same size (and not too huge), it's easiest to store them as a 3-D matrix instead. See here for discussion/examples.
If you already have your matrices in a cell array, as in your example, you can concatenate them into a 3-D matrix using cat and sum across the third dimension without a for loop using sum:
mat = sum(cat(3, AA{:}), 3);
I attempted to use the solution from this post: Multiply a 3D matrix with a 2D matrix, to vectorize the multiplication of a one dimensional with a three dimensional, but to no avail. Any help would be greatly appreciated!
for s = 1: s_length
for a = 1: a_length
for g = g_length
two_dim(s,a) = two_dim(s,a) + one_dim(g) * three_dim(s,a,g);
end
end
end
I think this does what you want.
two_dim = reshape(three_dim, [], size(three_dim,3))*one_dim(:);
two_dim = reshape(two_dim, size(three_dim,1), size(three_dim,2));
This works as follows:
First line reshapes your 3D array into a matrix, collapsing the first two dimensions into one. That way the operation you want is standard multiplication of the resulting matrix times a column vector.
Second line reshapes the result into matrix shape.
mtimes does not work with inputs with dimension larger than 2, so you have to find a way to do the multiplication by yourself. The solution of Luis Mendo is a nice solution; here is another one using bsxfun:
two_dim = two_dim + squeeze(sum(bsxfun(#times, three_dim, reshape(one_dim,[1 1 g_length])),3));
Here is how it works:
reshape makes the vector one_dim looking like a 3D array. This must be done because the multiplication between the vector and the 3D array is performed along the 3rd dimension, so Matlab need a hint on sizes.
bsxfun perfoms the element-wise multiplcation, so the result must be sumed up along the 3rd dimension (and squeezed to be compliant with a 2D matrix format)
I need a 3D matrix in matlab, I have another 2D matrix (7570x3) too,
The 3D matrix should have zero number except of all dimensions in 2D matrix that should have 1 value. How can I do that.
i.e. 2D matrix (1,:) = 28,64,27 then 3d(27,64,27) should be 1
how can i do that?
Assuming a is your 2-d matrix, and b is the 3-d one, use sub2ind as follows:
b=false(max(a)); % preallocate memory for a logical zeros matrix b
b(sub2ind(size(b),a(:,1),a(:,2),a(:,3))) = 1;
Check to see what max(a) gives you to see if you can host a 3-d matrix of size(max(a) to begin with. Since you are interested in a logcial matrix (ones and zeros), the size of that matrix in memory is the same as the # of elements, n*m*l, so a 1000x1000x1000 will take 1 GB.
Note, it very well may be that b is very sparse, if that is the case you can refer to this thread to see how to deal with it. Know that at the moment, and to the best of my knowledge, matlab doesn't support 3d sparse matrices. So you may want to check this option from the FEX. When I think of it, you already have a sparse look-up table of the 3D matrix! it is just your 2D matrix you started with...
Thanks a lot #natan
for non-integer matrix also can use:
b=false(floor(max(a))); % preallocate memory for a logical zeros matrix b
b(sub2ind(size(b),floor(a(:,1)),floor(a(:,2)),floor(a(:,3)))) = 1;
or use round function:
b=false(round(max(a))); % preallocate memory for a logical zeros matrix b
b(sub2ind(size(b),round(a(:,1)),round(a(:,2)),round(a(:,3)))) = 1;
preface: As the matlab guiderules state, Usually, when one wants to efficiently populate a sparse matrix in matlab, he should create a vector of indexes into the matrix and a vector of values he wants to assign, and then concentrate all the assignments into one atomic operation, so as to allow matlab to "prepare" the matrix in advance and optimize the assignment speed. A simple example:
A=sparse([]);
inds=some_index_generating_method();
vals=some_value_generating_method();
A(inds)=vals;
My question: what can I do in the case where inds contain overlapping indexes, i.e inds=[4 17 8 17 9] where 17 repeats twice.
In this case, what I would want to happen is that the matrix would be assigned the addition of all the values that are mapped to the same index, i.e for the previous example
A(17)=vals(2)+vals(4) %as inds(2)==inds(4)
Is there any straightforward and, most importantly, fast way to achieve this? I have no way of generating the indexes and values in a "smarter" way.
This might help:
S = sparse(i,j,s,m,n,nzmax) uses vectors i, j, and s to generate an m-by-n sparse matrix such that S(i(k),j(k)) = s(k), with space allocated for nzmax nonzeros. Vectors i, j, and s are all the same length. Any elements of s that are zero are ignored, along with the corresponding values of i and j. Any elements of s that have duplicate values of i and j are added together.
See more at MATLAB documentation for sparse function
A question about matlab and randomisation of a 3d matrix respecting the rows and columns.
I have a n x n x s matrix M and I want to mess it up a bit, but with some control.
I can achieve my wish with a for loop
for j=1:size(M,3)
r=randperm(size(M,1));
random_M(:,:,j)=M(r,r,j);
end
Is there a way to perform this without having to loop over j? I need many randomisation iterations and could afford the benefits of indexing.
Cheers!
edit: Some more thoughts following Alexandrew's comments
I have created a function that randomises a squeezed version of M:
function randomMat=randomiseMat(Mat)
[rows,cols]=size(Mat);
r=randperm(rows);
randomMat=Mat(r,r);
then, using arrayfun I seem to get what I want:
randomM=arrayfun(#(x) randomiseMat(M(:,:,x)),1:size(M,3),'UniformOutput', false)
however, randomM is now a cell array of size (1,size(M,3)) with each cell containing randomised array.
Is there a way to make it in a 3d matrix just like the input M?
You can calculate all the values for r in one go, and then use arrayfun:
[nRows,nCols,nPages] = size(M);
[~,r]=sort(rand(nRows,nPages));
%# you should test on a realistic example whether a for-loop
%# isn't faster here
outCell = arrayfun(#(x) M(r(:,x),r(:,x),x), 1:nPages,'UniformOutput',false);
randomM = cat(3,outCell{:});