I've got a really big matrix which I should "upscale" (i.e.: create another matrix where the elements of the first are grouped 40-by-40). For every 40-by-40 group I should evaluate a series of parameters (i.e.: frequencies, average and standard deviation).
I'm quite sure I can make such thing with a loop, but I was wondering if there was a more elegant vectorized method...
You might find blockproc useful. This command allows you to apply a function (e.g. #mean, #std etc.) to each distinct block in a 2D matrix.
Related
I have a 2D 2401*266 matrix K which corresponds to x values (t: stored in a 1*266 array) and y values(z: stored in a 1*2401 array).
I want to extrapolate the matrix K to predict some future values (corresponding to t(1,267:279). So far I have extended t so that it is now a 1*279 matrix using a for loop:
for tq = 267:279
t(1,tq) = t(1,tq-1)+0.0333333333;
end
However I am stumped on how to extrapolate K without fitting a polynomial to each individual row?
I feel like there must be a more efficient way than this??
There are countless of extrapolation methods in the literature, "fitting a polynomial to each row" would be just one of them, not necessarily invalid, not sure why you mention that you do no wan't to do it. For 2D data perhaps fitting a surface would lead to better results though.
However, if you want an easy, simple way (that might or might not work with your problem), you can always use the function interp2, for interpolation. If you chose spline or makima as interpolation functions, it will also extrapolate for any query point outside the domain of K.
I have two matrices X and Y, both of order mxn. I want to create a new matrix O of order mxm such that each i,j th entry in this new matrix is computed by applying a function to ith and jth row of X and Y respectively. In my case m = 10000 and n = 500. I tried using a loop but it takes forever. Is there an efficient way to do it?
I am targeting two functions dot product -- dot(row_i, row_j) and exp(-1*norm(row_i-row_j)). But I was wondering if there is a general way so that I can plugin any function.
Solution #1
For the first case, it looks like you can simply use matrix multiplication after transposing Y -
X*Y'
If you are dealing with complex numbers -
conj(X*ctranspose(Y))
Solution #2
For the second case, you need to do a little more work. You need to use bsxfun with permute to re-arrange dimensions and employ the raw form of norm calculations and finally squeeze to get a 2D array output -
squeeze(exp(-1*sqrt(sum(bsxfun(#minus,X,permute(Y,[3 2 1])).^2,2)))
If you would like to avoid squeeze, you can use two permute's -
exp(-1*sqrt(sum(bsxfun(#minus,permute(X,[1 3 2]),permute(Y,[3 1 2])).^2,3)))
I would also advise you to look into this problem - Efficiently compute pairwise squared Euclidean distance in Matlab.
In conclusion, there isn't a common most efficient way that could be employed for every function to ith and jth row of X. If you are still hell bent on that, you can use anonymous function handles with bsxfun, but I am afraid it won't be the most efficient technique.
For the second part, you could also use pdist2:
result = exp(-pdist2(X,Y));
I have a huge sparse matrix (1,000 x 1,000,000) that I cannot load on matlab (not enough RAM).
I want to visualize this matrix to have an idea of its sparsity and of the differences of the values.
Because of the memory constraints, I want to proceed as follows:
1- Divide the matrix into 4 matrices
2- Load each matrix on matlab and visualize it so that the colors give an idea of the values (and of the zeros particularly)
3- "Stick" the 4 images I will get in order to have a global idea for the original matrix
(i) Is it possible to load "part of a matrix" in matlab?
(ii) For the visualization tool, I read about spy (and daspect). However, this function only enables to visualize the non-zero values indifferently of their scales. Is there a way to add a color code?
(iii) How can I "stick" plots in order to make one?
If your matrix is sparse, then it seems that the currently method of storing it (as a full matrix in a text file) is very inefficient, and certainly makes loading it into MATLAB very hard. However, I suspect that as long as it is sparse enough, it can still be leaded into MATLAB as a sparse matrix.
The traditional way of doing this would be to load it all in at once, then convert to sparse representation. In your case, however, it would make sense to read in the text file, one line at a time, and convert to a MATLAB sparse matrix on-the-fly.
You can find out if this is possible by estimating the sparsity of your matrix, and using this to see if the whole thing could be loaded into MATLAB's memory as a sparse matrix.
Try something like: (untested code!)
% initialise sparse matrix
sparse_matrix = sparse(num_rows, num_cols);
row_num = 1;
fid = fopen(filename);
% read each line of text file in turn
while ~feof(fid)
this_line = fscanf(fid, '%f');
% add row to sparse matrix (note transpose, which I think is required)
sparse_matrix(row_num, :) = this_line';
row_num = row_num + 1;
end
fclose(fid)
% visualise using spy
spy(sparse_matrix)
Visualisation
With regards to visualisation: visualising a sparse matrix like this via a tool like imagesc is possible, but I believe it may internally create the full matrix – maybe someone can confirm if this is true or not. If it does, then it's going to cause you memory problems.
All spy is really doing is plotting in 2D the locations of the non-zero elements. You can fairly easily write your own spy function, which can have different coloured or sized points depending on the values at each location. See this answer for some examples.
Saving sparse matrices
As I say above, the method your matrix is saved as is pretty inefficient – for a matrix with 10% sparsity, around 95% of your text file will be a zero or a space. I don't know where this data has come from, but if you have any control over its creation (e.g. it comes from another program you have written) it would make much more sense to save only the non-zero elements in the format row_idx, col_idx, value.
You can then use spconvert to import the sparse matrix directly.
One of the simplest methods (if you can actually store the full sparse matrix in RAM) is to use gnuplot to visualize the sparisty pattern.
I was able to spy matrices of size 10-20GB using gnuplot without problems. But make sure you use png or jpeg formats to output the image. Note that you don't need the value of the non-zero entry only the integers (row, col). And plot them "plot "row_col.dat" using 1:2 with points".
This chooses your row as x axis and cols as your y axis and start plotting the non-zero entries. It is very easy to do this. This is the most scalable solution I know. Gnuplot works at decent speed even for very large datasets (>10GB of [row, cols]), but Matlab just hangs (with due respect)
I use imagesc() to visualise arrays. It scales the values in array to values between 0 and 1, then plots the array like a greyscale bitmap image (of course you can change the colormap to make it easier to see detail).
I need to convolve a matrix with many other matrices with few calls to convn.
for example: I have size(MyMat)=[fm, fm ,1, bSize] and size(masks)=[s, s, maskNum]
I want res(:,:,k,:) to be the product of convolving masks(:,:,k) with MyMat
res(:,:,k,:)=convn(MyMat,masks(:,:,k));
since the convolution takes up over 80% of the running time for my script and is called hundreds of thousands of times, I don't want to use a loop.
I'm looking for the fastest way to do this. basically, you could say I have bSize matrices, and I want to apply convolution masks masks to all of them with as few calls as possible to convolution.
The matrices are all small,non-sparse, fft-based convolution will probably slow it down (as a commentor here verified :) )
(The reason I have a 1 in the size of MyMat is because I actually have more elements in that dimension, but I compute the convolution for each element in that dimension in a loop)
The main goal is simply to eliminate the need for the following loop, or make it parallel with very little overhead, if possible:
for i=1:length
res(:,:,:,i)=convn(MyArray,convMask(:,:,i));
end
parallelizing for the GPU would be great if there's a way to do this with less overhead than the usual parfor
Thank you!
I assume that you are preallocating the array res correctly? Without a simple demo of what your doing and an idea of the size of fm, s, etc., one can only make guesses to help you. If the sizes of your matrices are large enough you might look into FFT-based convolution methods (there are some for convn on the Matlab File Exchange). If the data is sparse (> 50% zeros), you could try converting this to matrix multiplication and use sparse data types. You could also try gpuArray/convn if you have a decent one.
I have a dataset Sig of size 65536 x 192 in Matlab. If I want to take the one-dimensional fft along the second dimension, I could either do a for loop:
%pre-allocate ect..
for i=1:65536
F(i,:) = fft(Sig(i,:));
end
or I could specify the dimension and do it without the for loop:
F = fft(Sig,[],2);
which is about 20 times faster for my dataset.
I have looked for something similar for the discrete wavelet transform (dwt), but been unable to find it. So I was wondering if anyone knows a way to do dwt across a specified dimension in Matlab? Or do I have to use for loops?
In your loop FFT example, it seems you operate on lines. Matlab use a Column-major order. It may explain the difference of performance. Is the performance the same if you operate on columns ?
If this is the right explanation, you could use dwt in a loop.
A solution if you really need performance is to do your own MEX calling a C discrete wavelet transform library the way you want.
I presume you're using the function from the Wavelet Toolbox: http://www.mathworks.co.uk/help/toolbox/wavelet/ref/dwt.html
The documentation doesn't seem to describe acting on an array, so it's probably not supported. If it does allow you to input an array, then it will operate on the first non-singleton dimension or it will ignore the shape and treat it as a vector.