I am working with binary information, read from file. This is a sequence of numbers, in 40 bit format, where first 8bit should be neglected in my case, and another 32 bits are "shuffled" 32bit single precision IEEE 754 format. This "shuffle" is pretty straightforward: the correct IEEE 754 binary32 I get, when I take following order of bits: 24-32, 17-24, 9-16
All this is simulated with code below.
Question: How can I improve the code below to make it faster, get rid of "for" loop and use efficient MATLAB matrix operations?
a = (1:5*8*1000000)'; % represent indices of bits binary information, read from file
tic
a_reshaped = reshape(a, 8, [])';
toc %Elapsed time is 0.176375 seconds.
n_elem = size(a_reshaped,1)/5;
result = zeros(n_elem,8*4);
for i = 1:n_elem
result(i,:) = [a_reshaped(5*i,:) a_reshaped(5*i-1,:) a_reshaped(5*i-2,:) a_reshaped(5*i-3,:)];
end
toc %Elapsed time is 4.243868 seconds.
Try this:
ind = size(a_reshaped,1):-1:1;
ind(end:-5:1) = []; %remove the indices for rows you don't need
a_reduced = a_reshaped(ind,:);
result = flipud(reshape(a_reduced',8*4,[])');
the "for" loop can be replaced within following one line code, which is much faster:
result(1:n_elem,:) = [a_reshaped(5*(1:n_elem),:) a_reshaped(5*(1:n_elem)-1,:) a_reshaped(5*(1:n_elem)-2,:) a_reshaped(5*(1:n_elem)-3,:)]; %Elapsed time is 0.766840 seconds.
Related
I have made this scripts that calculates the frequency of a given dataset, but matlab is not precise enough, is it possible to make matlab read in more accurat numbers and not cut off the numbers? I want it to use 8 digits (0.12345678) instead of 4 (0.1234) that is does now
fid = fopen('forceCoeffs.dat','rt');
A = textscan(fid, '%f%f%f%f%f%f', 'HeaderLines',9,'Collect', 9);
A = A{1};
fclose(fid);
t = A(:,1);
Fs = 1/(A(1,1));
x = A(:,2)
x = detrend(x,0);
xdft = fft(x);
freq = 0:Fs/length(x):Fs/2;
xdft = xdft(1:length(x)/2+1);
plot(freq,abs(xdft));
[~,I] = max(abs(xdft));
fprintf('Maximum occurs at %d Hz.\n',freq(I));
File: https://drive.google.com/file/d/0B9CEsYCSSZUSb1JmcHRkbFdWYUU/view?usp=sharing
Thank you for including the forceCoeffs.dat file as it allowed me to run your code. Here is an explanation of what you are seeing.
First I want to point out that MATLAB is not rounding anything. You can check the data type of A to ensure you have enough precision.
>> class(A)
ans =
double
And since you are reading in the file using %f for each column, MATLAB will use all the bits provided by the double type. Ok, now take a look at the contents of your file. The first column has only 2 decimals of precision at most.
0.05 -7.013874e-09 1.410717e+02 -6.688450e-02 -3.344226e-02 -3.344224e-02
...
349.95 -1.189524e-03 1.381022e+00 -2.523909e-01 -1.273850e-01 -1.250059e-01
350 -1.423947e-03 1.380908e+00 -2.471767e-01 -1.250123e-01 -1.221644e-01
Since no more is needed MATLAB only prints four decimal places when you look at the variable in the variable explorer. Try looking at one of the other columns to see what I am talking about. I commented out the A = A{1} part of your code and looked at the second column. When clicking on the number you see the full precision.
You can use a long type to display 16 digits
To get more than 4 digits precision, you can use
format long
However, to get exactly 8 digits, you need to round it. If your number is a then let use:
format long
round(1e8*a)*1e-8
Suppose I have a long data vector y, plus some indices into it. I want to extract a short snippet or window around every index.
For example, suppose I want to construct a matrix containing 64 samples before and 64 samples after every value that is below three. This is trivial to do in a for-loop:
WIN_SIZE = 64;
% Sample data with padding
data = [nan(WIN_SIZE,1); randn(1e6,1); nan(WIN_SIZE,1)];
% Sample events, could be anything
index = find(data < 3);
snippets = nan(length(index), 2*WIN_SIZE + 1);
for ii=1:length(index)
snippets(ii,:) = data((index(ii)-WIN_SIZE):(index(ii)+WIN_SIZE));
end
However,this is not blazingly fast. Is there any way to vectorize (or otherwise speed up) this operation?
(In case this is unclear, the index could be anything and may not necessarily be a property of the data; I just wanted something simple to illustrate the idea.)
Use bsxfun -
snippets = data(bsxfun(#plus,index(:),[-WIN_SIZE:WIN_SIZE]))
I have a square matrix A (nxn). I would like to create a series of k powers of this matrix into an nxnxk multidimensional matrix (Not element-wise but actual powers of the matrix), i.e.getting [A^0 A^1 A^2..A^k]. It's sort of a varied vandermonde for matrix case.
I am able to do it with loops but it is annoying and slow. I tried using bsxfun but no luck since I am probably missing something here.
Here is a simple loop that I did:
for j=1:1:100
final(:,:,j)=A^(j-1);
end
You are trying to perform cummulative version of mpower with a vector of k values.
Sadly, bsxfun hasn't evolved yet to handle such a case. So, the best I could suggest at this point would be having a running storage that accumulates the matrix-product at each iteration to be used at the next one.
Your original loop code looked something like this -
final = zeros([size(A),100]);
for j=1:1:100
final(:,:,j)=A^(j-1);
end
So, with the suggestion, the modified loopy code would be -
final = zeros([size(A),100]);
matprod = A^0;
final(:,:,1) = matprod;
for j=2:1:100
matprod = A*matprod;
final(:,:,j)= matprod;
end
Benchmarking -
%// Input
A = randi(9,200,200);
disp('---------- Original loop code -----------------')
tic
final = zeros([size(A),100]);
for j=1:1:100
final(:,:,j)=A^(j-1);
end
toc
disp('---------- Modified loop code -----------------')
tic
final2 = zeros([size(A),100]);
matprod = A^0;
final2(:,:,1) = matprod;
for j=2:1:100
matprod = A*matprod;
final2(:,:,j)= matprod;
end
toc
Runtimes -
---------- Original loop code -----------------
Elapsed time is 1.255266 seconds.
---------- Modified loop code -----------------
Elapsed time is 0.205227 seconds.
yesterday I implemented my first bootstrap in MATLab. (and yes, I know, for loops are evil.):
%data is an mxn matrix where the data should be sampled per column but there
can be a NaNs Elements
%from the array (a column of data) n values are sampled nReps times
function result = bootstrap_std(data, n, nReps,quantil)
result = zeros(1,size(data,2));
for i=1:size(data,2)
bootstrap_data = zeros(n,nReps);
values = find(~isnan(data(:,i)));
if isempty(values)
bootstrap_data(:,:) = NaN;
else
for k=1:nReps
bootstrap_data(:,k) = datasample(data(values,i),n);
end
end
stat = zeros(1,nReps);
for k=1:nReps
stat(k) = nanstd(bootstrap_data(:,k));
end
sort(stat);
result(i) = quantile(stat,quantil);
end
end
As one can see, this version works columnwise. The algorithm does what it should but is really slow when the data size increaes. My question is now: Is it possible to implement this logic without using for loops? My problem is here that I could not find a version of datasample which does the sampling columnwise. Or is there a better function to use?
I am happy for any hint or idea how I can speed up this implementation.
Thanks and best regards!
stephan
The bottlenecks in your implementation are
The function spends a lot of time inside nanstd which is unnecessary since you exclude NaN values from your sample anyway.
There are a lot of functions that operate column-wise, but you spend time looping over the columns and calling them many times.
You make many calls to datasample which is a relatively slow function. It's much faster to create a random vector of indices using randi and use that instead.
Here's how I would write the function (actually I probably wouldn't put in this many comments, and I wouldn't use so many temp variables, but I'm doing it now so you can see what all the steps of the computation are).
function result = bootstrap_std_new(data, n, nRep, quantil)
result = zeros(1, size(data,2));
for i = 1:size(data,2)
isbad = isnan(data(:,i)); %// Vector of NaN values
if all(isbad)
result(i) = NaN;
else
data0 = data(~isbad, i); %// Temp copy of this column for indexing
index = randi(size(data0,1), n, nRep); %// Create the indexing vector
bootstrapdata = data0(index); %// Sample the data
stdevs = std(bootstrapdata); %// Stdev of sampled data
result(i) = quantile(stdevs, quantil); %// Find the correct quantile
end
end
end
Here are some timings
>> data = randn(100,10);
>> data(randi(1000, 50, 1)) = NaN;
>> tic, bootstrap_std(data, 50, 1000, 0.5); toc
Elapsed time is 1.359529 seconds.
>> tic, bootstrap_std_new(data, 50, 1000, 0.5); toc
Elapsed time is 0.038558 seconds.
So this gives you about a 35x speedup.
Your main issue seems to be that you may have varying numbers/positions of NaN in each column, so can't work on the full matrix unless you're okay with also sampling NaNs. However, some of the inner loops could be simplified.
for k=1:nReps
bootstrap_data(:,k) = datasample(data(values,i),n);
end
Since you're sampling with replacement, you should be able to just do:
bootstrap_data = datasample(data(values,i), n*nReps);
bootstrap_data = reshape(bootstrap_data, [n nReps]);
Also nanstd can work on a full matrix so no need to loop:
stat = nanstd(bootstrap_data); % or nanstd(x,0,2) to change dimension
It would also be worth just looking over your code with profile to see where the bottlenecks are.
I am writing some matlab code and have written an algorithm that works but I don't think its particularly efficient. Since I am trying to improve my programming skills I would like to know if there is a more efficient way of doing this.
I have a (reasonably large ~ E07) matrix of values which are unordered, but fall within the range [-100, 100]. I want to create a second matrix based on the first, by using the following rules:
If the value of the point is > 70, then the value of the point should be set to 70.
If the value of the point is < -70, then the value of the point should be set to -70.
All other values should be rounded to the nearest multiple of 5.
Here is what I am currently doing:
data = 100*(-1+2*rand(1,10000000)); % create random dataset for stackoverflow
new_data = zeros(1,length(data));
for i = 1:length(data)
if (data(i) > 70)
new_data(i) = 70;
elseif (data(i) < -70)
new_data(i) = -70;
else
new_data(i) = round(data(i)/5.0)*5.0;
end
end
Is there a more efficient method? I think there should be a way to do this using logical indexes but those are a new discovery for me...
You do not need a loop at all:
data = 100*(-1+2*rand(1,10000000)); % create random dataset for stackoverflow
new_data = zeros(1,length(data)); % note that this memory allocation is not necessary at this point
new_data = round(data/5.0)*5.0;
new_data(data>70) = 70;
new_data(data<-70) = -70;
Even easier is to use max and min. Do it in one simple line.
new_data = round(5*max(-70,min(70,data)))/5;
The two answers by H.Muster and woodchips are of course the way to do it, but there still are small improvements to be found. If you are after performance you might want to exploit specifics of your problem. For example, your output data is integers -100 <= x <= 100. This obviously qualifies for 8-bit signed integer data type. This code (note explicit cast to int8 from arbitrary double precision data)
% your double precision input data
data = 100*(-1+2*rand(1,10000000));
% cast to int8 - matlab does usual round here
data = int8(data);
new_data = 5*(max(-70,min(70,data))/5);
is the fastest for two reasons:
1 data element takes 1 byte, not 8. Memory bandwidth is a limiting factor here, so you get a lot of improvement
round is no longer necessary
Here are some timings from the codes of H.Muster, woodchips, and my small modification:
H.Muster Elapsed time is 0.235885 seconds.
woodchips Elapsed time is 0.167659 seconds.
my code Elapsed time is 0.023061 seconds.
The difference is quite striking. Although MATLAB uses doubles everywhere, you should try to use integer data types when possible..
Edit This works because of how matlab implements integer arithmetic. Differently than in C, a cast of double to int implies a round operation:
a = 0.1;
int8(a)
ans =
0
a = 0.9;
int8(a)
ans =
1