I am reading a 40 MB file with three different ways. But the first one is way faster than the 2 others. Do you guys have an idea why ? I would rather implement condition in loops or whiles to separate data than load everything with the first quick method and separate them then - memory saving -
LL=10000000;
fseek(fid,startbytes, 'bof');
%% Read all at once %%%%%%%%%%%%%%%%%%%%%%
tic
AA(:,1)=int32(fread(fid,LL,'int32'));
toc
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
fseek(fid,startbytes,'bof');
%% Read all using WHILE loop %%%%%%%%%%%%%
tic
i=0;
AA2=int32(zeros(LL,1));
while i<LL
i=i+1;
AA2(i,1)=fread(fid,1,'int32');
end
toc
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
fseek(fid,startbytes,'bof');
%% Read all using FOR loop %%%%%%%%%%%%%%%
tic
AA3=int32(zeros(LL,1));
for i=1:LL
AA3(i,1)=fread(fid,1,'int32');
end
toc
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Elapsed time is 0.312916 seconds.
Elapsed time is 138.811520 seconds.
Elapsed time is 116.799286 seconds.
Here are my two cents on this:
Is the JIT accelerator enabled?
Since MATLAB is an interpreted language, for loops can be quite slow. while loops may be even slower, because the termination condition is re-evaluted in each iteration (unlike for loops that iterate a predetermined number of times). Nevertheless, this is not so true with JIT acceleration, which can significantly boost their performance.
I'm not near MATLAB at the moment, so I cannot reproduce this scenario myself, but you can check yourself whether you have JIT acceleration turned on by typing the following in the command window:
feature accel
If the result is 0 it means that it's disabled, and this is probably the reason for the huge reduction in performance.
Too many system calls?
I'm not familiar with the internals of fread, but I can only assume that one fread call to read the entire file invokes less system calls than multiple fread calls. System calls are usually expensive, so this can, to some extent, account for the slowdown.
Related
Help for 'PAUSE' says
PAUSE(n) pauses for n seconds before continuing, where n can also be a
fraction. The resolution of the clock is platform specific. Fractional
pauses of 0.01 seconds should be supported on most platforms.
But in my case pause(0.01) doesn't do anything at all (pause, pause(n) with whole number n works)
Is there any way to make a millisecond level delay (50 ms, 100 ~ 500 ms) delay in matlab?
Matlab version is
MATLAB Version 7.9.0.529 (R2009b)
64 bit on a Windows 10 64 bit Home edition
I see two options. Let's call them the looping option and the native option. The first is just using a while loop to check if your desired time to wait is already reached. You can do this with MATLAB's stopwatch timer tic and toc. This is the normal time (not the CPU-time). As you are writing a loop, which runs at maximum speed, you might encounter a high CPU-usage but this should be OK if it is only for a couple of milliseconds.
%% looping
% desired time to wait
dt_des = 0.001; % 1 ms
% initialize clock
t_st = tic;
% looping
while toc(t_st) < dt_des
end
% read clock
toc(t_st)
The native option is using pause (make sure that you have enabled it once with pause('on')). I assumed from your question that this does not always work -- however, it does on my system (R2019b, windows 10).
%% use pause()
tic
pause(0.001)
toc
The results of both are
Elapsed time is 0.001055 seconds.
Elapsed time is 0.001197 seconds.
It's not too accurate but you might get better results if you tune the numbers on your PC.
you can also use
java.lang.Thread.sleep(10);
if you are using an old matlab version, see discussion here.
Say I have many (around 1000) large matrices (about 1000 by 1000) and I want to add them together element-wise. The very naive way is using a temp variable and accumulates in a loop. For example,
summ=0;
for ii=1:20
for jj=1:20
summ=summ+ rand(400);
end
end
After searching on the Internet for some while, someone said it's better to do with the help of sum(). For example,
sump=zeros(400,400,400);
count=0;
for ii=1:20
for j=1:20
count=count+1;
sump(:,:,count)=rand(400);
end
end
sum(sump,3);
However, after I tested two ways, the result is
Elapsed time is 0.780819 seconds.
Elapsed time is 1.085279 seconds.
which means the second method is even worse.
So I am just wondering if there any effective way to do addition? Assume that I am working on a computer with very large memory and a GTX 1080 (CUDA might be helpful but I don't know whether it's worthy to do so since communication also takes time.)
Thanks for your time! Any reply will be highly appreciated!.
The fastes way is to not use any loops in matlab at all.
In many cases, the internal functions of matlab all well optimized to use SIMD or other acceleration techniques.
An example for using the build in functionalities to create matrices of the desired size is X = rand(sz1,...,szN).
In your explicit case sum(rand(400,400,400),3) should give you then the fastest result.
I ran this code in a computer with 44 workers. However, each iteration in parallel is slower than in serial mode, though the total execution time for the loop as a whole goes down.
template=cell(31,1);
for i=1:31
template{i}=rand(i+24);
end
parfor i=1:5
img=rand(800,1280+i); % It's an other function that gives me the values of img ,here it's just an example
tic
cellfun (#(t) normxcorr2 ( t ,img),template,'UniformOutput',0);
toc
end
As a result, elapsed time in each loop is approximately 18s. When I change the parfor to for, the time elapsed is approximately 6.7s in each loop.
Can you explain me why in this case the parfor loop is slower than the for loop?
I checked the MATLAB documentation and also similar questions, but it didn't help me.
Note : the total time of execution of the script is faster for the parfor version, I just want to understand why the function cellfun is 3 times slower in parallel version.
Check CPU usage.
I believe main reason here is that stuff like fft (which is a part of xcorr most likely) will already use more than a single core. I can't test parfor right now, but ordinary for loop already has about 70% CPU utilization on my 4C/4T CPU with your code. So, parfor can at best fill the remaining 30% (on my computer), but will obviously then run each instance slower.
I am trying to use matlab for data acquisition with a licor820 instrument. The instrument outputs data at 2 hertz.
I have tried many different methods using infinite loops with asynchronous sampling (readasync) and timed readings but I am unable to get 2 hertz data. I am getting reads in the .51 s range. here are three examples of my methods. Any advice on what I may be doing wrong or how to properly sample at the highest frequency would be greatly appreciated!
example1: using readasync
tinit=tic; %initialization timer
s=serial('COM4') %,'InputBufferSize',40);
fopen(s)
while toc(tinit)<2 %allow time to initialize
end
while 1<2 %infinite loop for continuous sampling
readasync(s)
data=fscanf(s)
toc %allows me to see time between data acquisitions
tic
end
example 2: using bytes available.
My thinking here is to acquire data when I have the minimum amount of bytes necessary. Although I am unsure exactly how to determine how many bytes are necessary for my instrument, besides through visually looking at the data and narrowing it down to around 40 bytes:
while 1<2 %infinite loop for continuous sampling
if s.BytesAvailable >35
scandata=fscanf(s);
toc
tic
end
end
example 3: time forcing.
Since I need 2 hertz data my thinking here was to just force read the buffer every .49 seconds. The weird thing I see here is that it initially provides samples every .49 seconds, but while I monitor the bytes available at the port I see it steady dropping from 512 until it gets to 0 and then I stop getting .49 second samples. I guess I don't really understand how to use serial efficiently.
while 1<2 %infinite loop
if toc(t2)>=.49 %only sample after .49 seconds have passed
t2=tic; %reinitiate the tic for this forced time loop
bytes=s.BytesAvailable %to monitor how many bytes there are at the port
scandata=fscanf(s);
if ~isempty(scandata) && length(scandata)== 3 %checks for successful read
toc
tic
end
end
end
I feel there must be some way to sample completely in sync with the an instrument but I can't figure it out. Any help, suggestions, or ideas would be greatly appreciated! Thanks!
Dont rely on tic and toc. These functions use the time supplied by the OS calls. Mathworks claims to use high resolution timers, but do not rely on this! If you do not use a realtime OS these measurements are subject to unknown variation.
Sampling should be performed by realtime capable hardware. In your case I suspect that your sampling rate is actually controlled by your instrument. The output of the instrument is buffered by your serial interface. Therefore it seems to me that Matlab does not influence the sampling rate at all. (As long as the buffer does not overflow)
Try to acquire about 2000 samples or more and see how long it takes. Then divide the total time by the number of samples (-1) and compare this to the expected 0.5 s. If there is a difference, try adjusting the configuration of your instrument.
Is there any way to show the execution progress (even a rough estimation) of a Matlab instruction?
For example let's say I am computing distances using pdist:
D = pdist(my_matrix,'cosine');
and this computation takes hours, does Matlab provide any way to show periodically the execution progress?
Not intrinsically. You can of course do after-the-fact checking with the profiler or tic/toc.
If this is something you will be doing a lot for a single function, you could consider modifying the function and saving it in your path with a new name (I have a directory called "Modified Builtin" for just this purpose). In the case of pdist.m, you might save pdist_updates.m. Looking a the function, the actual distances are calculated starting around line 250 with a series of nested loops. Add in a line like:
disp(sprintf('Processing pair %d of %d',i,n-1));
at line 265. If you really wanted to get fancy, you could use tic and toc to time each loop and provide an estimate of how long the entire calculation would take so you would know how long you had to run to the coffee machine.
Of course this will cause problems if you end up cancelling your Statistics Toolbox license or if Mathworks upgrades the toolbox and changes the functionality, so use this approach sparingly.