I know it is not possible to create a true binary variable in Matlab. Even 'true(8,1)' has a size of 8 bytes, instead of 8 bits. Very memory inefficient.
In Jon Bentley's Programming Pearls he poses this problem:
input: a file containing at most n positive integers, each less then n, where n = 1E7. No integer exists twice.
output: a sorted list in increasing order of input integers
constraints: At most roughly 1MB of storage available in main memory. Ample disk storage.
In the answer he claims the program should be able to solve it lightning fast using a binary map, or bit-vector. I tried to implement this bit-vector in matlab:
classdef binar
properties (Access=protected)
byteslist uint8
end
methods
function obj = binar(N)
N=ceil(N/8);
obj.byteslist=zeros([N,1],'uint8');
end
function obj = setbit(obj,N,bit) %N from 0 to N-1
v=rem(N,8)+1; %byte position [1 8]
B = (N-v+1)/8+1; %byte nr [1 ..]
obj.byteslist(B)=bitset(obj.byteslist(B),v,bit);
end
function bit = getbit(obj,N)
v=rem(N,8)+1;
B = (N-v)/8+1;
bit = bitget(obj.byteslist(B),v);
end
end
end
To my surprise setting bits is very slow. I timed this code:
N=1E7;
B = binar(N);
itr=1E5;
tic
for ct = 1:itr
pos = uint32(randi([0,N-1],1));
B = setbit(B,pos,1);
end
toc
From the size perspective this works fine. 'S=whos('B');S.bytes/2^20' returns 1.19MB. However it does not seem to be very fast. Running bitset 1E5 times takes 0.367s, where I would think this is literally the simplest operation you can ask of a computer.
Is there a way to make this faster in Matlab?
update: I tried with bitand, and a predeclared vector
obj.byte=uint8([1 2 4 8 16 32 64 128]); %declared on creation of obj
obj.byteslist(B)=bitand(obj.byteslist(B),obj.byte(v));
Then i tried:
obj.byteslist(B)=obj.byteslist(B)+obj.byte(v);
There is hardly a speed difference with bitset.
Say I have a long list A of values (say of length 1000) for which I want to compute the std in pairs of 100, i.e. I want to compute std(A(1:100)), std(A(2:101)), std(A(3:102)), ..., std(A(901:1000)).
In Excel/VBA one can easily accomplish this by writing e.g. =STDEV(A1:A100) in one cell and then filling down in one go. Now my question is, how could one accomplish this efficiently in Matlab without having to use any expensive for-loops.
edit: Is it also possible to do this for a list of time series, e.g. when A has dimensions 1000 x 4 (i.e. 4 time series of length 1000)? The output matrix should then have dimensions 901 x 4.
Note: For the fastest solution see Luis Mendo's answer
So firstly using a for loop for this (especially if those are your actual dimensions) really isn't going to be expensive. Unless you're using a very old version of Matlab, the JIT compiler (together with pre-allocation of course) makes for loops inexpensive.
Secondly - have you tried for loops yet? Because you should really try out the naive implementation first before you start optimizing prematurely.
Thirdly - arrayfun can make this a one liner but it is basically just a for loop with extra overhead and very likely to be slower than a for loop if speed really is your concern.
Finally some code:
n = 1000;
A = rand(n,1);
l = 100;
for loop (hardly bulky, likely to be efficient):
S = zeros(n-l+1,1); %//Pre-allocation of memory like this is essential for efficiency!
for t = 1:(n-l+1)
S(t) = std(A(t:(t+l-1)));
end
A vectorized (memory in-efficient!) solution:
[X,Y] = meshgrid(1:l)
S = std(A(X+Y-1))
A probably better vectorized solution (and a one-liner) but still memory in-efficient:
S = std(A(bsxfun(#plus, 0:l-1, (1:l)')))
Note that with all these methods you can replace std with any function so long as it is applies itself to the columns of the matrix (which is the standard in Matlab)
Going 2D:
To go 2D we need to go 3D
n = 1000;
k = 4;
A = rand(n,k);
l = 100;
ind = bsxfun(#plus, permute(o:n:(k-1)*n, [3,1,2]), bsxfun(#plus, 0:l-1, (1:l)')); %'
S = squeeze(std(A(ind)));
M = squeeze(mean(A(ind)));
%// etc...
OR
[X,Y,Z] = meshgrid(1:l, 1:l, o:n:(k-1)*n);
ind = X+Y+Z-1;
S = squeeze(std(A(ind)))
M = squeeze(mean(A(ind)))
%// etc...
OR
ind = bsxfun(#plus, 0:l-1, (1:l)'); %'
for t = 1:k
S = std(A(ind));
M = mean(A(ind));
%// etc...
end
OR (taken from Luis Mendo's answer - note in his answer he shows a faster alternative to this simple loop)
S = zeros(n-l+1,k);
M = zeros(n-l+1,k);
for t = 1:(n-l+1)
S(t,:) = std(A(k:(k+l-1),:));
M(t,:) = mean(A(k:(k+l-1),:));
%// etc...
end
What you're doing is basically a filter operation.
If you have access to the image processing toolbox,
stdfilt(A,ones(101,1)) %# assumes that data series are in columns
will do the trick (no matter the dimensionality of A). Note that if you also have access to the parallel computing toolbox, you can let filter operations like these run on a GPU, although your problem might be too small to generate noticeable speedups.
To minimize number of operations, you can exploit the fact that the standard deviation can be computed as a difference involving second and first moments,
and moments over a rolling window are obtained efficiently with a cumulative sum (using cumsum):
A = randn(1000,4); %// random data
N = 100; %// window size
c = size(A,2);
A1 = [zeros(1,c); cumsum(A)];
A2 = [zeros(1,c); cumsum(A.^2)];
S = sqrt( (A2(1+N:end,:)-A2(1:end-N,:) ...
- (A1(1+N:end,:)-A1(1:end-N,:)).^2/N) / (N-1) ); %// result
Benchmarking
Here's a comparison against a loop based solution, using timeit. The loop approach is as in Dan's solution but adapted to the 2D case, exploting the fact that std works along each column in a vectorized manner.
%// File loop_approach.m
function S = loop_approach(A,N);
[n, p] = size(A);
S = zeros(n-N+1,p);
for k = 1:(n-N+1)
S(k,:) = std(A(k:(k+N-1),:));
end
%// File bsxfun_approach.m
function S = bsxfun_approach(A,N);
[n, p] = size(A);
ind = bsxfun(#plus, permute(0:n:(p-1)*n, [3,1,2]), bsxfun(#plus, 0:n-N, (1:N).')); %'
S = squeeze(std(A(ind)));
%// File cumsum_approach.m
function S = cumsum_approach(A,N);
c = size(A,2);
A1 = [zeros(1,c); cumsum(A)];
A2 = [zeros(1,c); cumsum(A.^2)];
S = sqrt( (A2(1+N:end,:)-A2(1:end-N,:) ...
- (A1(1+N:end,:)-A1(1:end-N,:)).^2/N) / (N-1) );
%// Benchmarking code
clear all
A = randn(1000,4); %// Or A = randn(1000,1);
N = 100;
t_loop = timeit(#() loop_approach(A,N));
t_bsxfun = timeit(#() bsxfun_approach(A,N));
t_cumsum = timeit(#() cumsum_approach(A,N));
disp(' ')
disp(['loop approach: ' num2str(t_loop)])
disp(['bsxfun approach: ' num2str(t_bsxfun)])
disp(['cumsum approach: ' num2str(t_cumsum)])
disp(' ')
disp(['bsxfun/loop gain factor: ' num2str(t_loop/t_bsxfun)])
disp(['cumsum/loop gain factor: ' num2str(t_loop/t_cumsum)])
Results
I'm using Matlab R2014b, Windows 7 64 bits, dual core processor, 4 GB RAM:
4-column case:
loop approach: 0.092035
bsxfun approach: 0.023535
cumsum approach: 0.0002338
bsxfun/loop gain factor: 3.9106
cumsum/loop gain factor: 393.6526
Single-column case:
loop approach: 0.085618
bsxfun approach: 0.0040495
cumsum approach: 8.3642e-05
bsxfun/loop gain factor: 21.1431
cumsum/loop gain factor: 1023.6236
So the cumsum-based approach seems to be the fastest: about 400 times faster than the loop in the 4-column case, and 1000 times faster in the single-column case.
Several functions can do the job efficiently in Matlab.
On one side, you can use functions such as colfilt or nlfilter, which performs computations on sliding blocks. colfilt is way more efficient than nlfilter, but can be used only if the order of the elements inside a block does not matter. Here is how to use it on your data:
S = colfilt(A, [100,1], 'sliding', #std);
or
S = nlfilter(A, [100,1], #std);
On your example, you can clearly see the difference of performance. But there is a trick : both functions pad the input array so that the output vector has the same size as the input array. To get only the relevant part of the output vector, you need to skip the first floor((100-1)/2) = 49 first elements, and take 1000-100+1 values.
S(50:end-50)
But there is also another solution, close to colfilt, more efficient. colfilt calls col2im to reshape the input vector into a matrix on which it applies the given function on each distinct column. This transforms your input vector of size [1000,1] into a matrix of size [100,901]. But colfilt pads the input array with 0 or 1, and you don't need it. So you can run colfilt without the padding step, then apply std on each column and this is easy because std applied on a matrix returns a row vector of the stds of the columns. Finally, transpose it to get a column vector if you want. In brief and in one line:
S = std(im2col(X,[100 1],'sliding')).';
Remark: if you want to apply a more complex function, see the code of colfilt, line 144 and 147 (for v2013b).
If your concern is speed of the for loop, you can greatly reduce the number of loop iteration by folding your vector into an array (using reshape) with the columns having the number of element you want to apply your function on.
This will let Matlab and the JIT perform the optimization (and in most case they do that way better than us) by calculating your function on each column of your array.
You then reshape an offseted version of your array and do the same. You will still need a loop but the number of iteration will only be l (so 100 in your example case), instead of n-l+1=901 in a classic for loop (one window at a time).
When you're done, you reshape the array of result in a vector, then you still need to calculate manually the last window, but overall it is still much faster.
Taking the same input notation than Dan:
n = 1000;
A = rand(n,1);
l = 100;
It will take this shape:
width = (n/l)-1 ; %// width of each line in the temporary result array
tmp = zeros( l , width ) ; %// preallocation never hurts
for k = 1:l
tmp(k,:) = std( reshape( A(k:end-l+k-1) , l , [] ) ) ; %// calculate your stat on the array (reshaped vector)
end
S2 = [tmp(:) ; std( A(end-l+1:end) ) ] ; %// "unfold" your results then add the last window calculation
If I tic ... toc the complete loop version and the folded one, I obtain this averaged results:
Elapsed time is 0.057190 seconds. %// windows by window FOR loop
Elapsed time is 0.016345 seconds. %// "Folded" FOR loop
I know tic/toc is not the way to go for perfect timing but I don't have the timeit function on my matlab version. Besides, the difference is significant enough to show that there is an improvement (albeit not precisely quantifiable by this method). I removed the first run of course and I checked that the results are consistent with different matrix sizes.
Now regarding your "one liner" request, I suggest your wrap this code into a function like so:
function out = foldfunction( func , vec , nPts )
n = length( vec ) ;
width = (n/nPts)-1 ;
tmp = zeros( nPts , width ) ;
for k = 1:nPts
tmp(k,:) = func( reshape( vec(k:end-nPts+k-1) , nPts , [] ) ) ;
end
out = [tmp(:) ; func( vec(end-nPts+1:end) ) ] ;
Which in your main code allows you to call it in one line:
S = foldfunction( #std , A , l ) ;
The other great benefit of this format, is that you can use the very same sub function for other statistical function. For example, if you want the "mean" of your windows, you call the same just changing the func argument:
S = foldfunction( #mean , A , l ) ;
Only restriction, as it is it only works for vector as input, but with a bit of rework it could be made to take arrays as input too.
I am having issues with a code of mine dealing with decay. The error "Subscript indices must either be real positive integers or logicals" continues to occur no matter how many times I attempt to fix the line of code: M=M(t)+h.*F
Here is the complete code so that it may be easier to solve the issue:
M=10000;
M=#(t) M*exp(-4.5*t);
F=-4.5*M(t);
h=.1;
t(1)=0;
tmax=20;
n=(tmax-t(1))/h;
i=1;
while h<=.5
while i<=n
t=t+h;
M=M(t)+h.*F;
data_out=[t,M];
dlmwrite('single_decay_euler_h.txt',data_out,'delimiter','\t','-append');
i=i+1;
end
h=h+.1;
end
Thanks for any help.
In the start, you're setting M = 5000;. In the following line, you're creating an anonymous function also called M:
M=#(t) M*exp(-4.5*t);
Now, your initial M = 5000 variable has been overwritten, and is substituted by the function:
M(t) = 5000 * exp(-4.5*t); %// Note that the first M is used to get 5000
Thereafter you do F = -4.5*M(t). I don't know what the value t is here, but you're giving F the value -4.5 * 5000 * exp(-4.5*t), for some value of t. You are not creating a function F.
In the first iteration of the loop, M=M(t)+h.*F; is interpreted as:
M = 5000 * exp(-4.5*0) + 0.1*F %// Where F has some value determined by previous
%// the function above and the previous value of t
%// -4.5*0 is because t = 0
M is now no longer a function, but a single scalar value. The next iteration t = 0.1. When you do: M=M(t)+h.*F; now, it interprets both the first and second M as a variable, not a function. t is therefore used as an index, instead of being an input parameter to the function M (since you have overwritten it).
When you are writing M(t), you are trying to access the 0.1'th element of the 1x1 matrix (scalar) M, which obviously isn't possible.
Additional notes:
The outer while loop has no purpose as it stands now, since i isn't reset after the inner loop. When you're finished with the first iteration of the outer loop, i is already >n, so it will never enter the inner loop again.
You shouldn't mix variable and function names (as you do with M. Use different names, always. Unless you have a very good reason not to.
data_out=[t,M]; is a growing vector inside a loop. This is considered very bad practice, ans is very slow. It's better to pre-allocate memory for the vector, for instance using data_out = zeros(k,1), and insert new values using indexes, data_out(ii) = M.
It's recommended not to use i and j as variable names in MATLAB as these also represent the imaginary unit sqrt(-1). This might cause some strange bugs if you're not paying attention to it.
You can almost certainly do what you're trying to do without loops. However, the function you have written is not functioning, and it's not explained all too well what you're trying to do, so it's hard to give advice as to how you can get what you want (but I'll give it a try). I'm skipping the dlmwrite-part, because I don't really understand what you want to output.
M = 5000;
t0 = 0;
tmax = 20;
h = 0.1; %// I prefer leading zeros in decimal numbers
t = t0: h: tmax;
data_out = M .* exp(-4.5 * t);
The problem is caused by M(t) in your code, because t is not an integer or logical (t=1,1.1,1.2,...)
You need to change your code to pass an integer as a subscript. Either multiply t by 10, or don't use the matrix M if you don't need it.
I have the following bytes stored in a vector:
data = [189 33 136 147]
These 4 bytes represent a single float in Big-endian order. How can I get this number in MATLAB?
I will need to concatenate and convert. I tried:
x = typecast(str2num(sprintf('%d%d%d%d',data(1),data(2),data(3),data(4))), 'single')
To no avail (I got x = []).
great example here:
>> dataL = typecast(uint8([189, 33, 136, 147]), 'uint32')
dataL =
2475172285
>> dataF = double(dataL)
dataF =
2.4752e+09
big to little, try swapbytes
>> dataLbig = swapbytes(dataL)
dataLbig =
3173091475
>> dataFbig = double(dataLbig)
dataFbig =
3.1731e+09
Is this what you were expecting?
I'll leave this here in case it's useful for anyone. As #MarkMikofski showed, using typecast and swapbytes is the standard method for this. However, if you your data is already floating-point, these functions can be inefficient in some cases. I use the following utility function in my video-encoding/decoding tools:
function x = Bit32toDouble(y)
n = numel(y);
if n >= 65536
x = double(swapbytes(typecast(uint8(y(:)),'uint32'))).';
elseif n > 4
x = sum(bsxfun(#times,[16777216;65536;256;1],reshape(y(:),4,n/4)));
else
x = sum([16777216;65536;256;1].*y(:));
end
There are separate cases depending on the number of bytes passed in. Only when a large amount of data is processed at once is the typecast/swapbytes most efficient. If the function is called repeatedly with smaller inputs, as is common in my application, the other cases are much faster because they do every thing in Matlab's native floating-point.
Hey guys I am trying to find the Power Spectral Density of a .wav signal i recorded which is essentially a sine immersed in noise. The function that i have written is supposed to take records all of 1024 points in length and use it to find the Gxx of the signal by finding Gxx per record and then adding them and dividing them by the number of records better explained in the algorithm below:
a. Step through the wav file and extract the first record length (e.g. 1 to 1024 points). (Note that the record length is your new “N”, hence the frequency spacing changes in accordance with this, NOT the total length of the wav file).
b. Perform the normal PSD function on this record.
c. Store this vector.
d. Extract the next 1024 points in the wav file (e.g. 1025:2048) and perform PSD on this record.
e. Add this to the previously stored record and continue through steps c to e until you reach the end of your wav file or the total number of records you want. (Remember that total records*record length must be less than the total length of the wavfile!)
f. Divide the PSD by the number of averages (or number of records).
This is your averaged PSD
The function I created is as follows:
%Function to plot PSD
function[f1, GxxAv] = HW3_A_Fn_811003472_RCT(x,fs,NumRec)
Gxx = 0;
GxxAv = 0;
N = 1024;
df = fs/N;
f1 = 0:df:fs/2;
dt = 1/fs;
T = N*dt;
q = 0;
e = 1;
for i = 1:NumRec;
for r = (1+q):(N*e);
L = x(1+q:N*e);
M = length(L);
Xm = fft(L).*dt;
aXm = abs(Xm);
Gxx(1)=(1/T).*(aXm(1).*aXm(1));
for k = 2:(M/2);
Gxx(k) = (2/T) *(aXm(k).*(aXm(k)));
%Gxx = Gxx + Gxx1(k);
end
Gxx((M/2)+1)= (1/T)*(aXm((M/2)+1)).*(aXm((M/2)+1));
q = q+1024;
e = e+1;
%Gxx = Gxx + Gxx1((M/2)+1);
end
GxxAv = GxxAv + Gxx;
%Gxx = Gxx + Gxx1;
end
GxxAv = GxxAv/NumRec;
And the code I used to call this function is as follows:
[x,fs] = wavread('F:\Final\sem1Y3\Acoustics\sinenoise5s.wav');
[f1,GxxAv] = HW3_A_Fn_811003472_RCT(x,fs,100); %where 100 is the number of records to generated
plot(f1,GxxAv)
xlabel ('Frequency / Hz', 'fontsize', 18)
ylabel ('Amplitude Squared per Frequency / WU^2/Hz', 'fontsize', 18)
title ('Plot of the single sided PSD, using Averaging', 'fontsize', 18)
grid on
When Trying to plot this graph the following error was observed:
??? Index exceeds matrix dimensions.
Error in ==> HW3_A_Fn_811003472_RCT at 19
L = x(1+q:N*e);
Error in ==> HW3_A_3_811003472_RCT at 3
[f1,GxxAv] = HW3_A_Fn_811003472_RCT(x,fs,100); %where 100 is the number of records to generated
I am not sure how to fix it and i have tried many different methods but still i get this error. I am not too familiar with Matlab but all I really want to do for line 19 is to go like:
x(1:1024), x(1025:2048), x(2049:3072), x(3072:4096)...etc to 100 records
Any ideas??? Thanks
This is obviously homework, so I am not gonna do your work for you. But there quite some things wrong with your code. Start by fixing all of those first:
Use more appropriate function names, homework123 is not a good name to describe what the function does.
Use more appropriate variable names. More standard in this context would be nfft instead of N and n_average instead of NumRec. I don't care about the exact thing you use, but it should describe exactly what the variable does.
Your error message clearly hints that you are trying to index x in some illegal way. Start with making a loop that just prints the right indices (1..1024, 1025..2048, ...) and make sure it follows your instruction E. Only when this works as expected add the rest of the code.
you use a triple-nested for-loop. You only need a single for-loop or while-loop to solve this problem.