I have the following code which works on matlab R2015 until the concatenation of I2D but stops at imadjust because I'm out of memory. I'm treating 2249 images of 1984*1984 pixels on a 16 Go RAM and 64 bit.
When I run the same code a more powerful computer (24 Go RAM) but with the R2009 version of Matlab, the code stops at the cat step saying the arguments dimensions are not consistent.
I don't get why it is working on R2015 and not on R2009. Notice that I will have to run the code on R2009 (but that's another story). Therefore, the problem of out of memory will maybe not be important but I don't know yat because it is not working on the R2009 version (24 Go RAM).
%reading data and inverting the GV
for k=1:n
I = imread(fullfile(rep,list1(k).name),ext(3:end));
for i=1:Row
for j=1:Col
I(i,j)=255-I(i,j);
end
end
nomFichier=sprintf('Xa1_3D_%04d.bmp',k);
imwrite(I,fullfile(rep2,nomFichier),ext(3:end));
end
%Now, we want to read all the images at once to have all the available
%informations for the histogram adjustement
chemin = fullfile(rep2,ext);
list2 = dir(chemin);
o=numel(list2);
I2{o}=ones;
for k=1:o
I2{k} = imread(fullfile(rep2,list2(k).name),ext(3:end));
end
%Combination of images in 2D matrix and hist adjustement
I2D=cat(2,I2{:});
I2D=imadjust(I2D);
Related
I am working on an algorithm (in MATLAB) that needs to find Hu-Moments of overlapping blocks in an image. I am converting image in column matrix (im2col(...,'sliding')) and then calculating Hu-Moments for each column individually. For calculating the Hu Moments for the blocks of an Image of 512X512 my system is taking 14-15 minutes. Code is as given below:
d=im2col(A,[m n],'sliding');
[mm nn]=size(d);
for j=1:nn
d_temp=d(:,j);
d_pass_temp=col2im(d_temp,[n n], [n n], 'distinct');
[mn_t vr_t]=new_hu_moment(d_pass_temp);
[mn]=[mn mn_t];
[vr]=[vr vr_t];
end
'new_hu_moment' is my own made function returning mean and variance of hu moments for the respective block.
My sys configuration is I3 processor with 6GB RAM.
please suggest for performance up-gradation of this code.
is there any function in matlab that can calculate 7 Hu moments for overlapping blocks.
1.preallocate mn and vr before the loop, something like
% I suppose that new_hu_moment returns row of known length nvals
mn=zeros(nn,nvals);
vr=zeros(nn,nvals);
for k=1:nn,
...
[mn(k,:) vr(k,:)]=new_hu_moment(d_pass_temp);
...
end
I strongly recomend not using i and j as variable names as they are synonyms for imaginary unit. In older MATLAB it can cause errors
2.If you have enough memory and for-loop processes independent data chunks, use parfor or cellfun/arrayfun
3.If code is not as fast you want it to be, use profiler to find the performance bottleneck
I wrote a MATLAB code for finding seismic signal (ex. P wave) from SAC(seismic) file (which is read via another code). This algorithm is called STA/LTA trigger algorithm (actually not that important for my question)
Important thing is that actually this code works well, but since my seismic file is too big (1GB, which is for two months), it takes almost 40 minutes for executing to see the result. Thus, I feel the need to optimize the code.
I heard that replacing loops with advanced functions would help, but since I am a novice in MATLAB, I cannot get an idea about how to do it, since the purpose of code is scan through the every time series.
Also, I heard that preallocation might help, but I have mere idea about how to actually do this.
Since this code is about seismology, it might be hard to understand, but my notes at the top might help. I hope I can get useful advice here.
Following is my code.
function[pstime]=classic_LR(tseries,ltw,stw,thresh,dt)
% This is the code for "Classic LR" algorithm
% 'ns' is the number of measurement in STW-used to calculate STA
% 'nl' is the number of measurement in LTW-used to calculate LTA
% 'dt' is the time gap between measurements i.e. 0.008s for HHZ and 0.02s for BHZ
% 'ltw' and 'stw' are long and short time windows respectively
% 'lta' and 'sta' are long and short time windows average respectively
% 'sra' is the ratio between 'sta' and 'lta' which will be each component
% for a vector containing the ratio for each measurement point 'i'
% Index 'i' denotes each measurement point and this will be converted to actual time
nl=fix(ltw/dt);
ns=fix(stw/dt);
nt=length(tseries);
aseries=abs(detrend(tseries));
sra=zeros(1,nt);
for i=1:nt-ns
if i>nl
lta=mean(aseries(i-nl:i));
sta=mean(aseries(i:i+ns));
sra(i)=sta/lta;
else
sra(i)=0;
end
end
[k]=find(sra>thresh);
if ~isempty(k)
pstime=k*dt;
else
pstime=0;
end
return;
If you have MATLAB 2016a or later, you can use movmean instead of your loop (this means you also don't need to preallocate anything):
lta = movmean(aseries(1:nt-ns),nl+1,'Endpoints','discard');
sta = movmean(aseries(nl+1:end),ns+1,'Endpoints','discard');
sra = sta./lta;
The only difference here is that you will get sra with no leading and trailing zeros. This is most likely to be the fastest way. If for instance, aseries is 'only' 8 MB than this method takes less than 0.02 second while the original method takes almost 6 seconds!
However, even if you don't have Matlab 2016a, considering your loop, you can still do the following:
Remove the else statement - sta(i) is already zero from the preallocating.
Start the loop from nl+1, instead of checking when i is greater than nl.
So your new loop will be:
for i=nl+1:nt-ns
lta = mean(aseries(i-nl:i));
sta = mean(aseries(i:i+ns));
sra(i)=sta/lta;
end
But it won't be so faster.
I have some problems with matlab 2015a Win10 X64 16GB Ram.
There is a bunch of images (1280x960x8bit) and I want to load them into a 3d matrix. In theory that matrix should store ~1.2GB for 1001 images.
What I have so far is:
values(:,:,:)= zeros(960, 1280, 1001, 'uint8');
for i = Start:Steps:End
file = strcat(folderStr, filenameStr, num2str(i), '.png');
img = imread(file);
values(:,:,i-Start+1) = img;
end
This code is working for small amount of images but using it for all 1001 images I get "Out of memory" error.
Another problem is the speed.
Reading 50 images and save them takes me ~2s, reading 100 images takes ~48s.
What I thought this method does is allocating the memory and change the "z-elements" of the matrix picture by picture. But obviously it holds more memory than needed to perform that single task.
Is there any method I can store the grey values of a 2d picture sequence to a 3d matrix in matlab without wasting that much time and ressources?
Thank you
The only possibility I can see is that your idexes are bad. But I can only guess because the values of start step and End are not given. If End is 100000000, Start is 1 and step is 100000000, you are only reading 2 images, but you are accessing values(:,:,100000000) thus making the variable incredibly huge. That is, most likely, your problem.
To solve this, create a new variable:
imagenames=Start:Step:End; %note that using End as a variable sucks, better ending
for ii=1:numel(imagenames);
file = strcat(folderStr, filenameStr, num2str(imagenames(ii)), '.png');
img = imread(file);
values(:,:,ii) = img;
end
As Shai suggests, have a look to fullfile for better filename accessing
Why does the code below cause Matlab to keep increasing the amount of memory used? Nothing is being stored from one iteration to the next? Yet Matlab keeps using up my system memory until the whole machine grinds to a halt. I am running on windows 7 professional with about 16GB of physical ram. y is just a 100*1 vector and x a 100*7 matrix of data. The system start to struggle as Matlab gets over 15GB which in understandable what is not so understandable is why Matlab needs so much memory for the program below.
clc;
iter=100000000;
b_OIRE=[1,0,1,1,1,1,1];
nsims=2;
for t=1:nsims
y=y_store(:,t);
[b_GIREII]=GIREII(y,x,b_OIRE,iter);
end
function [b_GIREII MSE]=GIREII(y,x,b_OIRE,iter) % [## "iter" denotes the iteration number]
[n, p]=size(x);
dim=1;
b=x\y;
b_GIREII=b;
sigma_sq=((y-x*b)'*(y-x*b))/(n-p);
econFlag=0;
[U,sigma,V] = svd(x,econFlag);
U1=U(:,1:p);
d=zeros(p,1);
k=zeros(p,1);
alpha=V'*b_GIREII;
Delta=sigma.^1; % [Error! not sigma.^2 but sigma.^1]
Delta=diag(Delta);
f=Delta.*alpha;
F=diag(f);
Theta=sum(f);
c=p^2*sigma_sq+p*Theta^2;
g=Theta*sum(alpha);
one=ones(p,1);
b=F*alpha;
I=eye(p);
A=sigma_sq*I+F.^2;
G=sigma_sq*I+Theta*F;
H=sigma_sq*I+f*f';
q=(p-1)/p;
k1=0;
k2=0;
d1=0;
d2=0;
for ii=1:p
k1=k1+alpha(ii)/(q*sigma_sq+Delta(ii)^2*alpha(ii)^2);
k2=k2+1/(q*sigma_sq+Delta(ii)^2*alpha(ii)^2);
d1=d1+alpha(ii);
d2=d2+Delta(ii)^2*alpha(ii)^2;
end
for ii=1:p
k(ii)=(Delta(ii)*alpha(ii)^2)/((q*sigma_sq+Delta(ii)^2*alpha(ii)^2))-(k1/k2)*(Delta(ii)*alpha(ii)/(q*sigma_sq+Delta(ii)^2*alpha(ii)^2));
d(ii)=(1/p)*(d1/(sigma_sq+d2)*Delta(ii)*alpha(ii)-k(ii));
end
K=diag(k);
D=one*d';
b_GIREII= V*(K+D)*U1'*y;
MSE=(k'*A*k)+(2*k'*H*d)-(2*b'*k)+(p*d'*H*d)-((2*g/Theta)*f'*d)+alpha'*alpha;
best_GIREII_MSE=MSE;
best_b_GIREII=b_GIREII;
best_index=1;
for jj=1:iter % [## "iter" denotes the iteration number]
alpha=V'*b_GIREII;
Alpha_store(:,jj)=alpha;
f=Delta.*alpha;
f_store(:,jj)=f;
F=diag(f);
Theta=sum(f);
c=p^2*sigma_sq+p*Theta^2;
g=Theta*sum(alpha);
b=F*alpha;
A=sigma_sq*I+F.^2;
A_store(:,:,jj)=A;
G=sigma_sq*I+Theta*F;
H=sigma_sq*I+f*f';
k1=0;
k2=0;
d1=0;
d2=0;
for ii=1:p
k1=k1+alpha(ii)/(q*sigma_sq+Delta(ii)^2*alpha(ii)^2);
k2=k2+1/(q*sigma_sq+Delta(ii)^2*alpha(ii)^2);
d1=d1+alpha(ii);
d2=d2+Delta(ii)^2*alpha(ii)^2;
end
for ii=1:p
k(ii)=(Delta(ii)*alpha(ii)^2)/((q*sigma_sq+Delta(ii)^2*alpha(ii)^2))-(k1/k2)*(Delta(ii)*alpha(ii)/(q*sigma_sq+Delta(ii)^2*alpha(ii)^2));
d(ii)=(1/p)*(d1/(sigma_sq+d2)*Delta(ii)*alpha(ii)-k(ii));
end
D=one*d';
K=diag(k);
b_GIREII= V*(K+D)*U1'*y;
MSE=(k'*A*k)+(2*k'*H*d)-(2*b'*k)+(p*d'*H*d)-((2*g/Theta)*f'*d)+alpha'*alpha;
if(MSE<best_GIREII_MSE)
best_b_GIREII=b_GIREII;
best_GIREII_MSE=MSE;
best_index=jj+1;
end
if rem(jj,10000000)==0
disp('10000000 interations complete')
end
end
end
In the function GIREII.m, you are putting following three variables in a loop which runs for 100000000 times.
1. Alpha_store(:,jj)=alpha; %%jj goes from 1:10^8
2. f_store(:,jj)=f;
3. A_store(:,:,jj)=A;
Interestingly, you are not using either of them. I guess you can just remove them. Please pay attention to the warnings in the MATLAB editor, which are underlined in orange color.
It seems like #Rody already found the problem with your code, but in fact it is not hard to track this kind of problem down in general.
Here are the three steps to get there:
Turn on dbstop if error
Wait till the code breaks down, or hit Ctrl+C once you see the memory usage growing beyond what you expected, to investigate in the even more general case you could also set a breakpoint.
Use whos and see whether you either have a huge number of variables, or some huge variables in your workspace.
I have written the following code in MATLAB to process large images of the order of 3000x2500 pixels. Currently the operation takes more than half hour to complete. Is there any scope to improve the code to consume less time? I heard parallel processing can make things faster, but I have no idea on how to implement it. How do I do it, given the following code?
function dirvar(subfn)
[fn,pn] = uigetfile({'*.TIF; *.tiff; *.tif; *.TIFF; *.jpg; *.bmp; *.JPG; *.png'}, ...
'Select an image', '~/');
I = double(imread(fullfile(pn,fn)));
ld = input('Enter the lag distance = '); % prompt for lag distance
fh = eval(['#' subfn]); % Function handles
I2 = uint8(nlfilter(I, [7 7], fh));
imshow(I2); % Texture Layer Image
imwrite(I2,'result_mat.tif');
% Zero Degree Variogram
function [gamma] = ewvar(I)
c = (size(I)+1)/2; % Finds the central pixel of moving window
EW = I(c(1),c(2):end); % Determines the values from central pixel to margin of window
h = length(EW) - ld; % Number of lags
gamma = 1/(2 * h) * sum((EW(1:ld:end-1) - EW(2:ld:end)).^2);
end
The input lag distance is usually 1.
You really need to use the profiler to get some improvements out of it. My first guess (as I haven't run the profiler, which you should as suggested already), would be to use as little length operations as possible. Since you are processing every image with a [7 7] window, you can precalculate some parts,
such that you won't repeat these actions
function dirvar(subfn)
[fn,pn] = uigetfile({'*.TIF; *.tiff; *.tif; *.TIFF; *.jpg; *.bmp; *.JPG; *.png'}, ...
'Select an image', '~/');
I = double(imread(fullfile(pn,fn)));
ld = input('Enter the lag distance = '); % prompt for lag distance
fh = eval(['#' subfn]); % Function handles
%% precalculations
wind = [7 7];
center = (wind+1)/2; % Finds the central pixel of moving window
EWlength = (wind(2)+1)/2;
h = EWlength - ld; % Number of lags
%% calculations
I2 = nlfilter(I, wind, fh);
imshow(I2); % Texture Layer Image
imwrite(I2,'result_mat.tif');
% Zero Degree Variogram
function [gamma] = ewvar(I)
EW = I(center(1),center(2):end); % Determines the values from central pixel to margin of window
gamma = 1/(2 * h) * sum((EW(1:ld:end-1) - EW(2:ld:end)).^2);
end
end
Note that by doing so, you trade performance for clearness of your code and coupling (between the function dirvar and the nested function ewvar). However, since I haven't profiled your code (you should do that yourself using your own inputs), you can find what line of your code consumes the most time.
For batch processing, I would also recommend to leave out any input, imshow, imwrite and uigetfile. Those are commands that you typically call from a more high-level function/script and that will force you to enter these inputs even when you want them to stay the same. So instead of that code, make each of the variables they produce (/process) a parameter (/return value) for your function. That way, you could leave MATLAB running during the weekend to process everything (without having manually enter to all those values), even if you are unable to speed up the code.
A few general purpose tricks:
1 - use the MATLAB profiler to determine all the computational bottlenecks
2 - parallel processing can make things faster and there are a lot of tools that you can use, but it depends on how your entire code is set up and whether the code is optimized for it. By far the easiest trick to learn is parfor, where you can replace the top level for loop by parfor. This does mean you must open the MATLAB pool with matlabpool open.
3 - If you have a rather recent Nvidia GPU as well as MATLAB 2011, you can also write some CUDA code.
All in all 30 mins to me is peanuts, so don't fret it too much.
First of all, I strongly suggest you follow the advice by #Egon: Write a separate function that collects a list of files (the excellent UIPICKFILES from the FEX is your friend here), and then runs your filtering code in a loop for each image. Note that you should definitely keep the call to imwrite in your filtering code: In case the analysis crashes at image 48 (e.g. due to power failure), you don't want to lose all the previous work.
Running thusly in batch mode has two big advantages: (1) you can start running your code and go home for the week-end, and (2) you can easily parallelize this outside loop using PARFOR. However, with only a dual-core machine, it is unlikely that you get any significant improvements from parallelization - your OS also wants to run stuff at times, and the overhead of parallelization might be more than the gain from running two workers. Also, 2.5GB of RAM is seriously limiting.
As to your specific code: in my experience using IM2COL is often faster than NLFILTER. im2col creates a nElementsInMask-by-nMasks array out of your image, so that you can apply the filtering in one single operation. With a 7x7 window, the output of im2col will be 3000*2500*49 bytes, which is close to 400MB. Thus, it should just work. All that you need to do is rewrite ewvar so that it works on a 49x1 array of pixels that make up the pixels your mask, which will require some index juggling, if I understand your code correctly.