Related
Say I have a long list A of values (say of length 1000) for which I want to compute the std in pairs of 100, i.e. I want to compute std(A(1:100)), std(A(2:101)), std(A(3:102)), ..., std(A(901:1000)).
In Excel/VBA one can easily accomplish this by writing e.g. =STDEV(A1:A100) in one cell and then filling down in one go. Now my question is, how could one accomplish this efficiently in Matlab without having to use any expensive for-loops.
edit: Is it also possible to do this for a list of time series, e.g. when A has dimensions 1000 x 4 (i.e. 4 time series of length 1000)? The output matrix should then have dimensions 901 x 4.
Note: For the fastest solution see Luis Mendo's answer
So firstly using a for loop for this (especially if those are your actual dimensions) really isn't going to be expensive. Unless you're using a very old version of Matlab, the JIT compiler (together with pre-allocation of course) makes for loops inexpensive.
Secondly - have you tried for loops yet? Because you should really try out the naive implementation first before you start optimizing prematurely.
Thirdly - arrayfun can make this a one liner but it is basically just a for loop with extra overhead and very likely to be slower than a for loop if speed really is your concern.
Finally some code:
n = 1000;
A = rand(n,1);
l = 100;
for loop (hardly bulky, likely to be efficient):
S = zeros(n-l+1,1); %//Pre-allocation of memory like this is essential for efficiency!
for t = 1:(n-l+1)
S(t) = std(A(t:(t+l-1)));
end
A vectorized (memory in-efficient!) solution:
[X,Y] = meshgrid(1:l)
S = std(A(X+Y-1))
A probably better vectorized solution (and a one-liner) but still memory in-efficient:
S = std(A(bsxfun(#plus, 0:l-1, (1:l)')))
Note that with all these methods you can replace std with any function so long as it is applies itself to the columns of the matrix (which is the standard in Matlab)
Going 2D:
To go 2D we need to go 3D
n = 1000;
k = 4;
A = rand(n,k);
l = 100;
ind = bsxfun(#plus, permute(o:n:(k-1)*n, [3,1,2]), bsxfun(#plus, 0:l-1, (1:l)')); %'
S = squeeze(std(A(ind)));
M = squeeze(mean(A(ind)));
%// etc...
OR
[X,Y,Z] = meshgrid(1:l, 1:l, o:n:(k-1)*n);
ind = X+Y+Z-1;
S = squeeze(std(A(ind)))
M = squeeze(mean(A(ind)))
%// etc...
OR
ind = bsxfun(#plus, 0:l-1, (1:l)'); %'
for t = 1:k
S = std(A(ind));
M = mean(A(ind));
%// etc...
end
OR (taken from Luis Mendo's answer - note in his answer he shows a faster alternative to this simple loop)
S = zeros(n-l+1,k);
M = zeros(n-l+1,k);
for t = 1:(n-l+1)
S(t,:) = std(A(k:(k+l-1),:));
M(t,:) = mean(A(k:(k+l-1),:));
%// etc...
end
What you're doing is basically a filter operation.
If you have access to the image processing toolbox,
stdfilt(A,ones(101,1)) %# assumes that data series are in columns
will do the trick (no matter the dimensionality of A). Note that if you also have access to the parallel computing toolbox, you can let filter operations like these run on a GPU, although your problem might be too small to generate noticeable speedups.
To minimize number of operations, you can exploit the fact that the standard deviation can be computed as a difference involving second and first moments,
and moments over a rolling window are obtained efficiently with a cumulative sum (using cumsum):
A = randn(1000,4); %// random data
N = 100; %// window size
c = size(A,2);
A1 = [zeros(1,c); cumsum(A)];
A2 = [zeros(1,c); cumsum(A.^2)];
S = sqrt( (A2(1+N:end,:)-A2(1:end-N,:) ...
- (A1(1+N:end,:)-A1(1:end-N,:)).^2/N) / (N-1) ); %// result
Benchmarking
Here's a comparison against a loop based solution, using timeit. The loop approach is as in Dan's solution but adapted to the 2D case, exploting the fact that std works along each column in a vectorized manner.
%// File loop_approach.m
function S = loop_approach(A,N);
[n, p] = size(A);
S = zeros(n-N+1,p);
for k = 1:(n-N+1)
S(k,:) = std(A(k:(k+N-1),:));
end
%// File bsxfun_approach.m
function S = bsxfun_approach(A,N);
[n, p] = size(A);
ind = bsxfun(#plus, permute(0:n:(p-1)*n, [3,1,2]), bsxfun(#plus, 0:n-N, (1:N).')); %'
S = squeeze(std(A(ind)));
%// File cumsum_approach.m
function S = cumsum_approach(A,N);
c = size(A,2);
A1 = [zeros(1,c); cumsum(A)];
A2 = [zeros(1,c); cumsum(A.^2)];
S = sqrt( (A2(1+N:end,:)-A2(1:end-N,:) ...
- (A1(1+N:end,:)-A1(1:end-N,:)).^2/N) / (N-1) );
%// Benchmarking code
clear all
A = randn(1000,4); %// Or A = randn(1000,1);
N = 100;
t_loop = timeit(#() loop_approach(A,N));
t_bsxfun = timeit(#() bsxfun_approach(A,N));
t_cumsum = timeit(#() cumsum_approach(A,N));
disp(' ')
disp(['loop approach: ' num2str(t_loop)])
disp(['bsxfun approach: ' num2str(t_bsxfun)])
disp(['cumsum approach: ' num2str(t_cumsum)])
disp(' ')
disp(['bsxfun/loop gain factor: ' num2str(t_loop/t_bsxfun)])
disp(['cumsum/loop gain factor: ' num2str(t_loop/t_cumsum)])
Results
I'm using Matlab R2014b, Windows 7 64 bits, dual core processor, 4 GB RAM:
4-column case:
loop approach: 0.092035
bsxfun approach: 0.023535
cumsum approach: 0.0002338
bsxfun/loop gain factor: 3.9106
cumsum/loop gain factor: 393.6526
Single-column case:
loop approach: 0.085618
bsxfun approach: 0.0040495
cumsum approach: 8.3642e-05
bsxfun/loop gain factor: 21.1431
cumsum/loop gain factor: 1023.6236
So the cumsum-based approach seems to be the fastest: about 400 times faster than the loop in the 4-column case, and 1000 times faster in the single-column case.
Several functions can do the job efficiently in Matlab.
On one side, you can use functions such as colfilt or nlfilter, which performs computations on sliding blocks. colfilt is way more efficient than nlfilter, but can be used only if the order of the elements inside a block does not matter. Here is how to use it on your data:
S = colfilt(A, [100,1], 'sliding', #std);
or
S = nlfilter(A, [100,1], #std);
On your example, you can clearly see the difference of performance. But there is a trick : both functions pad the input array so that the output vector has the same size as the input array. To get only the relevant part of the output vector, you need to skip the first floor((100-1)/2) = 49 first elements, and take 1000-100+1 values.
S(50:end-50)
But there is also another solution, close to colfilt, more efficient. colfilt calls col2im to reshape the input vector into a matrix on which it applies the given function on each distinct column. This transforms your input vector of size [1000,1] into a matrix of size [100,901]. But colfilt pads the input array with 0 or 1, and you don't need it. So you can run colfilt without the padding step, then apply std on each column and this is easy because std applied on a matrix returns a row vector of the stds of the columns. Finally, transpose it to get a column vector if you want. In brief and in one line:
S = std(im2col(X,[100 1],'sliding')).';
Remark: if you want to apply a more complex function, see the code of colfilt, line 144 and 147 (for v2013b).
If your concern is speed of the for loop, you can greatly reduce the number of loop iteration by folding your vector into an array (using reshape) with the columns having the number of element you want to apply your function on.
This will let Matlab and the JIT perform the optimization (and in most case they do that way better than us) by calculating your function on each column of your array.
You then reshape an offseted version of your array and do the same. You will still need a loop but the number of iteration will only be l (so 100 in your example case), instead of n-l+1=901 in a classic for loop (one window at a time).
When you're done, you reshape the array of result in a vector, then you still need to calculate manually the last window, but overall it is still much faster.
Taking the same input notation than Dan:
n = 1000;
A = rand(n,1);
l = 100;
It will take this shape:
width = (n/l)-1 ; %// width of each line in the temporary result array
tmp = zeros( l , width ) ; %// preallocation never hurts
for k = 1:l
tmp(k,:) = std( reshape( A(k:end-l+k-1) , l , [] ) ) ; %// calculate your stat on the array (reshaped vector)
end
S2 = [tmp(:) ; std( A(end-l+1:end) ) ] ; %// "unfold" your results then add the last window calculation
If I tic ... toc the complete loop version and the folded one, I obtain this averaged results:
Elapsed time is 0.057190 seconds. %// windows by window FOR loop
Elapsed time is 0.016345 seconds. %// "Folded" FOR loop
I know tic/toc is not the way to go for perfect timing but I don't have the timeit function on my matlab version. Besides, the difference is significant enough to show that there is an improvement (albeit not precisely quantifiable by this method). I removed the first run of course and I checked that the results are consistent with different matrix sizes.
Now regarding your "one liner" request, I suggest your wrap this code into a function like so:
function out = foldfunction( func , vec , nPts )
n = length( vec ) ;
width = (n/nPts)-1 ;
tmp = zeros( nPts , width ) ;
for k = 1:nPts
tmp(k,:) = func( reshape( vec(k:end-nPts+k-1) , nPts , [] ) ) ;
end
out = [tmp(:) ; func( vec(end-nPts+1:end) ) ] ;
Which in your main code allows you to call it in one line:
S = foldfunction( #std , A , l ) ;
The other great benefit of this format, is that you can use the very same sub function for other statistical function. For example, if you want the "mean" of your windows, you call the same just changing the func argument:
S = foldfunction( #mean , A , l ) ;
Only restriction, as it is it only works for vector as input, but with a bit of rework it could be made to take arrays as input too.
In MATLAB we can perform cross-validated LASSO with
[w, FitInfo] = lasso(X, y, 'CV', 3);
and obtain the best weights with
w(:,FitInfo.IndexMinMSE)
How can we find the coefficient of determination?
If I understand well the MATLAB documentation and the coefficient of determination definition, I think the following code should do the job:
optimal_weights = w(:,FitInfo.IndexMinMSE);
SStot = var(y)*length(y);
predicted_values = X*optimal_weights;
SSres = sum( (y(:)-predicted_values(:)).^2 );
R2 = 1 - SSres/SStot;
Note SStot could be computed with sum(), but I think using var() is a bit faster.
The above answer, by Yellows is correct, but you have to make sure to include the intercept term ==>
Predicted_values = X*2(:,fitInfo.IndexMinMSE) + fitInfo.Intercept(fitInfo.IndexMinMSE)
I am attempting to fit some UV/Vis absorbance spectra to published reference standards. In general, the absorbance one obtains from the spectrometer is equal to a linear combination of the concentration of each absorber multiplied by the cross-section of absorption for each molecule at each wavelength (and multiplied by the pathlength of the spectrometer).
That said, not all spectrometers are precise in the x axis (wavelength), so some adjustment may be necessary to fit one's experimental data to the reference standards.
In this script, I am adjusting the index of my wavelength and spectral intensity to see if integer steps of my spectra result in a better fit to the reference standards (each step is 0.08 nm). Of course, I need to save the output of the fit parameters; however, since each fit has a different set of dimensions, I'm having difficulty just throwing them into a structure(k) (commented out in the following code snippet).
If anyone has a tip or hint, I'd be very appreciative. The relevant portion of my sample code follows:
for i = -15:15
lengthy = length(wavelengthy)
if i >= 0
xvalue = (1:lengthy - abs(i))
yvalue = (1+abs(i):lengthy)
else
xvalue = (1+abs(i):lengthy)
yvalue = (1: lengthy - abs(i))
end
Phi = #(k,wavelengthy) ( O3standard(yvalue) .* k(1) + Cl2standard(yvalue) .* k(2) + ClOstandard(yvalue) .* k(3) + OClOstandard(yvalue) .* k(4));
[khat, resnorm, residual, exitflag, output, lambda, jacobian] = lsqcurvefit(Phi,k0,wavelengthy(xvalue),workingspectra(yvalue), lowerbound, upperbound, options);
%parameters.khat(k,:) = khat;
%parameters.jacobian(k,:) = jacobian;
%parameters.exitflag(k,:) = exitflag;
%parameters.output(k,:) = output;
%parameters.residuals(k,:) = residual
%concentrations(:,k) = khat./pathlength
k=k+1
end
I want to find how similar a picture is to some samples that I have (black and white).
I use the sum of absolute difference code, but because I'm new to MATLAB I didn't find out how to use it. How does this algorithm work? Does it give a measure of how similar the pics are?
I= imread('img1.jpg');
image2= imread('img2.jpg');
% J = uint8(filter2(fspecial('gaussian'), I));
K = imabsdiff(I,image2);
figure, imshow(K,[])
Well I think you pretty much answered your question yourself. It is the sum of the absolute difference. So let say you have img1 and img2 which are the same size and type.
To find the difference, do subtraction
img1-img2
To find the absolute difference, use the absolute value function abs
abs(img1-img2)
To find the sum, use the sum function. Note that you will need to do this for each "dimension" your image has. If you are not sure, type size(img1) and see if there are 2 or 3 numbers that show up, this corresponds to how many sum(...) you need to use.
For a color image (3 dimensions):
sum(sum(sum(abs(img1-img2))))
^^ Is the sum of the absolute differences. Whichever has the smallest sum can be considered the closest image.
If you have different sized images, you need to use the normxcorr2 function. This function will return a matrix of the same size with how well the template (small) image fits into the big image at each different point. Find the maximum value of that matrix and that is how well that image fits.
For instance:
correlation = normxcorr2(smallImg, bigImg);
compareMe = max(correlation(:))
It is best practice to use MATLAB's build-in function imabsdiff. In contrast to the other suggested answers, it takes care of the range boundaries if your image is formatted as uint8. Consider:
img1 = uint8(10);
img2 = uint8(20);
sum(abs(img1(:)-img2(:)))
gives you 0, whereas
imabsdiff(img1(:),img2(:))
correctly gives 10.
You should use the command im2col in MATLAB you will be able to do so in Vectorized manner.
Just arrange each neighborhood in columns (For each frame).
Put them in 3D Matrix and apply you operation on the 3rd dimension.
Code Snippet
I used Wikipedia's definition of "Sum of Absolute Differences".
The demo script:
```
% Sum of Absolute Differences Demo
numRows = 10;
numCols = 10;
refBlockRadius = 1;
refBlockLength = (2 * refBlockRadius) + 1;
mImgSrc = randi([0, 255], [numRows, numCols]);
mRefBlock = randi([0, 255], [refBlockLength, refBlockLength]);
mSumAbsDiff = SumAbsoluteDifferences(mImgSrc, mRefBlock);
```
The Function SumAbsoluteDifferences:
```
function [ mSumAbsDiff ] = SumAbsoluteDifferences( mInputImage, mRefBlock )
%UNTITLED2 Summary of this function goes here
% Detailed explanation goes here
numRows = size(mInputImage, 1);
numCols = size(mInputImage, 2);
blockLength = size(mRefBlock, 1);
blockRadius = (blockLength - 1) / 2;
mInputImagePadded = padarray(mInputImage, [blockRadius, blockRadius], 'replicate', 'both');
mBlockCol = im2col(mInputImagePadded, [blockLength, blockLength], 'sliding');
mSumAbsDiff = sum(abs(bsxfun(#minus, mBlockCol, mRefBlock(:))));
mSumAbsDiff = col2im(mSumAbsDiff, [blockLength, blockLength], [(numRows + blockLength - 1), (numCols + blockLength - 1)]);
end
```
Enjoy...
Is there an elegant way in Matlab to get the output of normxcorr2 cropped to the size of the image or cropped only to the part of the matrix that does not use zero padded edges in computation?
To understand what I mean, consider the conv2 command. There is an optional parameter called shape that can be set to same or valid.
C = conv2(A,B,'same');
C = conv2(A,B,'valid');
For example:
size( conv2( rand(50,50) , rand(6,6), 'valid') )
ans =
45 45
size( conv2( rand(50,50) , rand(6,6), 'same') )
ans =
50 50
size( conv2( rand(50,50) , rand(6,6)) )
ans =
55 55
Currently I wrote my own function, that does something like this:
function I = normxcorr2e(template,im,shape)
switch shape
case 'same'
I = normxcorr2(template,im);
r = size(I,1)-size(im,1);
c = size(I,2)-size(im,2);
m1=floor(r/2);
n1=floor(c/2);
m2=ceil(r/2);
n2=ceil(c/2);
I(1:m2,:) = [];
I(end-m1+1:end,:) = [];
I(:,1:n2) = [];
I(:,end-n1+1:end) = [];
case 'full'
%Do nothing
case 'valid'
%TODO - write this case...
otherwise
throw(Mexception('normxcorr2e:BadInput','shape %s is not recognized',shape));
end
end
Do you have a better idea? The main criterion for a successful answer will be the elegance of the proposed solution.
Edit(1) First of all, thanks for all of your answers. All of them are good and upvoted by me. I still haven't decided which is the best. By the way, I am thinking recently about the case where the template is large compared to the image. In that case, it makes sense to speedup the computation by cropping the image argument before running normxcorr2.
Here's a variant that has a few extra features relative to the other answers:
It allows you to omit the shape argument (default is 'full').
It only calls normxcorr2 when shape is a valid string.
It performs the indexing in one line using logical indexing. The sizes of the lead padding and desired center region is used to create index vectors of true and false values. The trailing padding doesn't need to be specified since a logical index that is shorter than the dimension it indexes will simply be padded with false values.
And here's the code:
function I = normxcorr2e(template, im, shape)
if (nargin == 2) || strcmp(shape,'full')
I = normxcorr2(template, im);
return
end
switch shape
case 'same'
pad = floor(size(template)./2);
center = size(im);
case 'valid'
pad = size(template) - 1;
center = size(im) - pad;
otherwise
throw(Mexception('normxcorr2e:BadInput',...
'SHAPE must be ''full'', ''same'', or ''valid''.'));
end
I = normxcorr2(template, im);
I = I([false(1,pad(1)) true(1,center(1))], ...
[false(1,pad(2)) true(1,center(2))]);
end
This would be much more concise. I hope it's what you're looking for:
function I = normxcorr2e(template,im,shape)
args={'full','same','valid'};
cropSize=(find(strcmp(shape,args))-1)*size(template);
crop=#(x,r) x(1+floor(r(1)/2):end-ceil(r(1)/2),1+floor(r(2)/2):end-ceil(r(2)/2))
I=crop(normxcorr2(template,im),cropSize);
There is not much elegance here - you run the correlation, then you remove what you can't use. But it works.
function I = normxcorr2e(template,im,shape)
%# perform cross correlation with automated zero-padding
I = normxcorr2(template,im);
switch shape
case 'same'
%# if we were guaranteed to have odd-sized templates only
%# we would only need padLow
templateSize = size(template);
padLow = floor(templateSize/2);
padHigh = templateSize - padLow - 1;
I = I( (1+padLow(1)):(end-padHigh(1)), (1+padLow(2)):(end-padHigh(2)) );
case 'full'
%Do nothing
case 'valid'
%# with even size, we need to remove the larger of the two pad sizes
%# i.e. padLow, on all sides
templateSize = size(template);
padLow = templateSize/2;
I = I( (2*padLow(1)):(end-2*padLow(1)+1), (2*padLow(2)):(end-2*padLow(2)+1) );
otherwise
throw(Mexception('normxcorr2e:BadInput','shape %s is not recognized',shape));
end