I have some problems with matlab 2015a Win10 X64 16GB Ram.
There is a bunch of images (1280x960x8bit) and I want to load them into a 3d matrix. In theory that matrix should store ~1.2GB for 1001 images.
What I have so far is:
values(:,:,:)= zeros(960, 1280, 1001, 'uint8');
for i = Start:Steps:End
file = strcat(folderStr, filenameStr, num2str(i), '.png');
img = imread(file);
values(:,:,i-Start+1) = img;
end
This code is working for small amount of images but using it for all 1001 images I get "Out of memory" error.
Another problem is the speed.
Reading 50 images and save them takes me ~2s, reading 100 images takes ~48s.
What I thought this method does is allocating the memory and change the "z-elements" of the matrix picture by picture. But obviously it holds more memory than needed to perform that single task.
Is there any method I can store the grey values of a 2d picture sequence to a 3d matrix in matlab without wasting that much time and ressources?
Thank you
The only possibility I can see is that your idexes are bad. But I can only guess because the values of start step and End are not given. If End is 100000000, Start is 1 and step is 100000000, you are only reading 2 images, but you are accessing values(:,:,100000000) thus making the variable incredibly huge. That is, most likely, your problem.
To solve this, create a new variable:
imagenames=Start:Step:End; %note that using End as a variable sucks, better ending
for ii=1:numel(imagenames);
file = strcat(folderStr, filenameStr, num2str(imagenames(ii)), '.png');
img = imread(file);
values(:,:,ii) = img;
end
As Shai suggests, have a look to fullfile for better filename accessing
Related
If i create a cell with 1000 matrices ( size of each matrix 800*1280), clearing each matrix after using it will speed up calculations ?
Example
A=cell(1000,1);
for i=1:1000
A{i}=rand(800,1280);
end
image=A{1};
image2=A{2}; % I will use image and image2 with other functions
A{1}=[];
A{2}=[];
EDIT
The real use of the cell will be like :
A=cell(1000,1);
parfor i=1:1000
A{i}=function_that_creates_image(800,1280); % image with size 800*1280 px
end
for i=1:number_of_images % number_of_images=1000 in this case
image1=A{1};
image2=A{2};
A{1}=[];
A{2}=[];
% image1 and image 2 will be used then in the next lines
%next lines of code
end
I noticed that calculating components of A in a parfor loop is faster than calculating each component for each loop inside the for
If you want to use less memory and speed up calculations, it's wiser to avoid using cells. Luckily, it's very easy in your case, since all your matrices are the same size, so you could use an ND-array.
A = zeros(800,1280,1000);
for k = 1:size(A,3)
A(:,:,k) = function_that_creates_image(800,1280);
end
image = A(:,:,1);
image2 = A(:,:,2); % I will use image and image2 with other functions
EDIT:
If you want to further process each image, I would save them to a file within the parfor, so you will have 1000 .mat files at the end of the first loop:
parfor k = 1:number_of_images
A = function_that_creates_image(800,1280);
save(['images_dir\image' num2str(k) '.mat'],'A');
end
then you can load them as needed for processing using load:
for k = 1:number_of_images-1
image1 = load(['images_dir\image' num2str(k) '.mat']);
image2 = load(['images_dir\image' num2str(k+1) '.mat'];
% do what you want with those images...
end
This way you only keep 2 images in memory each time, and on the next iteration they a replaced by the next images.
If you fit everything in memory (you need at least 16GB to hold data and do some work on parts of it, to work on the full beast at the same time you should be having 32GB), clearing these won't change anything at all. If you don't, I would assume/hope Matlab and Windows are smart enough to optimize which chunk is held in memory and which is put on the disk, so again deleting won't help. But you might not want to rely on that.
What you can do is to have A{i} = 'path-to-file';, then load it in memory just for the time when needed. Why do you even need to first load all the images and then do work on them one by one? It would be much better for memory to simply have image1 = rand(...);, image2 = rand(...); in the loop itself, and reuse these image1 and image2. No need to even have this A.
In general, tall arrays are your memory friendly solution you should be using if you want to have tons of data at the same time. https://www.mathworks.com/help/matlab/tall-arrays.html
I'm using MATLAB_R2011a_student. I have some image stacks saved as 32 bit tiffs, some over 1000 frames. I would like to be able to pull out a specific frame from the stack and save it as a 32 bit tiff or some readable format where there would be no data loss from the original. Currently my code looks like this:
clear, clc;
k=163;
image=('/Users/me/Filename.tiff');
A = uint8(imread(image, k));
B=A(:,:,1);
J=imadjust(B,stretchlim(B),[]);
imwrite(J,'/Users/me/163.tif','tif');
(I'm assuming reading it as 8 bit, and the way I'm saving are not the best way to do this)
Either way this code works for a seemingly random number of frames (for example in one file.tiff the above code works for frames 1-165 but none of the frames after 165, for a different file.tiff the code works for frames 1-8 but none of the frames after 8) I'm also getting a strange horizontal line in the output image when this does work:
??? Error using ==> rtifc
Invalid TIFF image index specified.
Error in ==> readtif at 52
[X, map, details] = rtifc(args);
Error in ==> imread at 443
[X, map] = feval(fmt_s.read, filename, extraArgs{:});
Thanks!
The best way (in my opinion) to handle tiff stacks is to use the Tiff library available since a few years. I must admit that I don't know much about OOP but I managed to understand enough to load a tiff stack and manipulate it.That's the kind of simple demo I wish I had seen a year ago haha.
I the following example I load a single stack and store it all into a 3D array. I use imfinfo to fetch infos about the images, notably the number of images/stack and the actual image dimensions. If you want you can choose to load only one image using appropriate indices. Please try the code below and play around with it; you'll understand what I mean.
clear
clc
%// Get tiff files you wish to open
hFiles = dir('*.tif');
%// Here I only have 1 multi-tiff file containing 30 images. Hence hInfo is a 30x1 structure.
hInfo = imfinfo(hFiles(1).name);
%// Set parameters.
ImageHeight = hInfo(1).Height;
ImageWidth = hInfo(1).Width;
SliceNumber = numel(hInfo);
%// Open Tiff object
Stack_TiffObject = Tiff(hFiles.name,'r');
%// Initialize array containing your images.
ImageMatrix = zeros(ImageHeight,ImageWidth,SliceNumber,'uint32');
for k = 1:SliceNumber
%// Loop through each image
Stack_TiffObject.setDirectory(k)
%// Put it in the array
ImageMatrix(:,:,k) = Stack_TiffObject.read();
end
%// Close the Tiff object
Stack_TiffObject.close
Hope that helps.
I have a large stack of 800 16bit gray scale images with 2048x2048px. They are read from a single BigTIFF file and the whole stack barely fits into my RAM (8GB).
Now I need do a median projection. That means I want to compute the median of each pixel across all 800 frames. The Matlab median function fails because there is not enough memory left make a copy of the whole array for the function call. What would be an efficient way to compute the median?
I have tried using a for loop to compute the median one pixel at a time, but this is still terribly slow.
Iterating over blocks, as #Shai suggests, may be the most straightforward solution. If you do have this problem frequently, you may want to consider converting the image to a mat-file, so that you can access the pixels as n-d array directly from disk.
%# convert to mat file
matObj = matfile('dest.mat','w');
matObj.data(2048,2048,numSlices) = 0;
for t = 1:numSlices
matObj.data(:,:,t) = imread(tiffFile,'index',t);
end
%# load a block of the matfile to take median (run as part of a loop)
medianOfBlock = median(matObj.data(1:128,1:128,:),3);
I bet that the distributions of the individual pixel values over the stack (i.e. the histograms of the pixel jets) are sparse.
If that's the case, the amount of memory needed to keep all the pixel histograms is much less than 2K x 2K x 64k: you can use a compact hash map to represent each histogram, and update them loading the images one at a time. When all updates are done, you go through your histograms and compute the median of each.
If you have access to the Image Processing Toolbox, Matlab has a set of tool to handle large images called Blockproc
From the docs :
To avoid these problems, you can process large images incrementally: reading, processing, and finally writing the results back to disk, one region at a time. The blockproc function helps you with this process.
I will try my best to provide help (if any), because I don't have an 800-stack TIFF image, nor an 8GB computer, but I want to see if my thinkings can form a solution.
First, 800*2048*2048*8bit = 3.2GB, not including the headers. With your 8GB RAM it should not be too difficult to store it at once; there might be too many programs running and chopping up the contiguous memories. Anyway, let's treat the problem as Matlab can't load it as a whole into the memory.
As Jonas suggests, imread supports loading a TIFF image by index. It also supports a PixelRegion parameter, so you can also consider accessing parts of the image by this parameter if you want to utilize Shai's idea.
I came up with a median algo that doesn't use all the data at the same time; it barely scans through a sequence of un-ordered data, one at each time; but it does keep a memory of 256 counters.
_
data = randi([0,255], 1, 800);
bins = num2cell(zeros(256,1,'uint16'));
for ii = 1:800
bins{data(ii)+1} = bins{data(ii)+1} + 1;
end
% clearvars data
s = cumsum(cell2mat(bins));
if find(s==400)
med = ( find(s==400, 1, 'first') + ...
find(s>400, 1, 'first') ) /2 - 1;
else
med = find(s>400, 1, 'first') - 1;
end
_
It's not very efficient, at least because it uses a for loop. But the benefit is instead of keeping 800 raw data in memory, only 256 counters are kept; but the counters need uint16, so actually they are roughly equivalent to 512 raw data. But if you are confident that for any pixel the same grayscale level won't count for more than 255 times among the 800 samples, you can choose uint8, and hence reduce the memory by half.
The above code is for one pixel. I'm still thinking how to expand it to a 2048x2048 version, such as
for ii = 1:800
img_data = randi([0,255], 2048, 2048);
(do stats stuff)
end
By doing so, for each iteration, you only need these kept in memory:
One frame of image;
A set of counters;
A few supplemental variables, with size comparable to one frame of image.
I use a cell array to store the counters. According to this post, a cell array can be pre-allocated while its elements can still be stored in memory non-contigously. That means the 256 counters (512*2048*2048 bytes) can be stored separately, which is quite reasonable for your 8GB RAM. But obviously my sample code does not make use of it since bins = num2cell(zeros(....
I have a MATLAB variable that is a 3x6 cell array. One of the columns of the cell array holds at most 150-200 small RGB images, like 16x20 pixel size (again, at most). The rest of the columns are:
an equal number of labels that are strings of a 3 or 4 characters,
an image mask, which is about 350x200
3 integers
For some reason saving this object is taking a very long time, or at least for the size of the object. It has already been 10 minutes(which isn't too bad, but I plan on expanding the size of the object to hold several thousand of those small images) and MATLAB doesn't seem to be making any progress. In fact, when I open the containing directory of the variable, its size is cycling between 0 bytes to about 120kB. (i.e. it will increase to 120 in steps of 30 or 40 kB, then restart).
Is this normal behavior? Do MATLAB variables always take so long to save? What's going on here?
Mistake: I'm saving AllData, not my SVM variable. AllData has the same data as the SVM keeper, less the actual SVM itself and one integer.
What particular points of the code would be helpful to show for solving this? The code itself is a few hundred lines and broken up in several functions. What would be important to consider to troubleshoot this? When the variable is created? when it's saved? The way I create the smaller images?
Hate to be the noob who takes a picture of their desktop. but the machine I'm working has problems taking screenshots. Anyway, here it is
Alldata/curdata are just subsets of the 3x7 array... actually it's a 3x8, but the last is just an int.
Interesting side point: I interrupted the saving process and the variable seemed to save just fine. I trained a new svm on the saved data and it works just fine. I'd like to not do that in the future though.
Using whos:
Name Size Bytes Class Attributes
AllData 3x6 473300 cell
Image 240x352x3 253440 uint8
RESTOREDEFAULTPATH_EXECUTED 1x1 1 logical
SVMKeeper 3x8 2355638 cell
ans 3x6 892410 cell
curData 3x6 473300 cell
dirpath 1x67 134 char
im 240x352x3 1013760 single
s 1x1 892586 struct
Updates:
1.Does this always happen, or did you only do it once?
-It always happens
2.Does it take the same time when you save it to a different (local) drive?
-I will investigate this more when I get back to that computer
3.How long does it take to save a 500kb matrix to that folder?
-Almost instantaneous
4.And as asked above, what is the function call that you use?
-Code added below
(Image is a rgb image)
MaskedImage(:,:,1)=Image(:,:,1).*Mask;
MaskedImage(:,:,2)=Image(:,:,2).*Mask;
MaskedImage(:,:,3)=Image(:,:,3).*Mask;
MaskedImage=im2single(MaskedImage);
....
(I use some method to create a bounding box around my 16x20 image)
(this is in a loop of that occurs about 100-200 times)
Misfire=input('is this a misfire?','s');
if (strcmpi(Misfire,'yes'))
curImageReal=MaskedImage(j:j+Ybound,i:i+Xbound,:);
Training{curTrainingIndex}=curImageReal; %Training is a cell array of images
Labels{curTrainingIndex}='ncr';
curTrainingIndex=curTrainingIndex+1;
end
(the loop ends)...
SaveAndUpdate=input('Would you like to save this data?(say yes,definitely)','s');
undecided=1;
while(undecided)
if(strcmpi(SaveAndUpdate,'yes,definitely'))
AllData{curSVM,4}=Training;
AllData{curSVM,5}=Labels;
save(strcat(dirpath,'/',TrainingName),'AllData'); <---STUCK HERE
undecided=0;
else
DontSave=input('Im not going to save. Say YESNOSAVE to NOT SAVE','s')
if(strcmpi(DontSave,'yesnosave'))
undecided=0;
else
SaveAndUpdate=input('So... save? (say yes,definitely)','s');
end
end
end
It is a bit unclear if you are doing some custom file saving or not. If it is the first, I'm guessing that you have a really slow save loop going on, maybe some hardware issues. Try to save the data using MATLAB's save function:
tic
save('test.mat', 'AllData')
toc
if that works fine try to work your way from there e.g. saving one element at a time etc.
You can profile your code by using the profiler, open it with the command profile viewer and then type in the code, script or function that you want to profile in the input text field.
This isn't a great answer, but it seems that the problem was that I was saving the version of my image after I had converted it to a single. I don't know why this would cause such a dramatic slowdown (after removing this line of code it worked instantly) so if someone could edit my answer to shed more light on the situation, that would be appreciated.
Here is the desired workflow:
I want to load 100 images into MATLAB workspace
Run a bunch of my code on the images
Save my output (the output returned by my code is an integer array) in a new array
By the end I should have a data structure storing the output of the code for images 1-100.
How would I go about doing that?
If you know the name of the directory they are in, or if you cd to that directory, then use dir to get the list of image names.
Now it is simply a for loop to load in the images. Store the images in a cell array. For example...
D = dir('*.jpg');
imcell = cell(1,numel(D));
for i = 1:numel(D)
imcell{i} = imread(D(i).name);
end
BEWARE that these 100 images will take up too much memory. For example, a single 1Kx1K image will require 3 megabytes to store, if it is uint8 RGB values. This may not seem like a huge amount.
But then 100 of these images will require 300 MB of RAM. The real issue comes about if your operations on these images turn them into doubles, then they will now take up 2.4 GIGAbytes of memory. This will quickly eat up the amount of RAM you have, especially if you are not using a 64 bit version of MATLAB.
Assuming that your images are named in a sequential way, you could do this:
N = 100
IMAGES = cell(1,N);
FNAMEFMT = 'image_%d.png';
% Load images
for i=1:N
IMAGES{i} = imread(sprintf(FNAMEFMT, i));
end
% Run code
RESULT = cell(1,N);
for i=1:N
RESULT{i} = someImageProcessingFunction(IMAGES{i});
end
The cell array RESULT then contains the output for each image.
Be aware that depending on the size of your images, prefetching the images might make you run out of memory.
As many have said, this can get pretty big. Is there a reason you need ALL of these in memory when you are done? Could you write the individual results out as files when you are done with them such that you never have more than the input and output images in memory at a given time?
IMWRITE would be good to get them out of memory when you are done.