I am reading some data from a csv or text file ( which consists of thousands of rows with each row consists of fixed number of columns - e.g.: 20).
I am keeping the above details in matlab with a structure as follows.
initial_var(firs).second_var(sec).third_var(thir).time(end+1, :) = [];
initial_var(firs).second_var(sec).third_var(thir).scan(end+1, :) = [];
initial_var(firs).second_var(sec).third_var(thir).time(end+1, :) = val1;
initial_var(firs).second_var(sec).third_var(thir).scan(end+1, :) = val2;
where first, sec, thir, val1, val2 are filled from the csv/text file.
There are multiple fields available other than time and scan but I have not included them here.
While running the program, I am getting the warning
The variable initial_var appears to change size on every loop iteration. Consider preallocating for speed.
I know this can be solved by preallocating and initializing.
But my question here is, what is the better way of keeping the above data rather than the above mentioned structure type?
These lines won't do anything:
initial_var(firs).second_var(sec).third_var(thir).time(end+1, :) = [];
initial_var(firs).second_var(sec).third_var(thir).scan(end+1, :) = [];
It means "delete the row after the end of this array".
You might like to look at a multi-dimensional structure:
vars(firs,sec,thr).time(end+1, :) = val1
vars(firs,sec,thr).scan(end+1, :) = val2
Should be easier to initialise too.
Also, when loading the data, you might like to look at textscan.
Typically, the fastest and most flexible way to read data is with fscanf. (See also csvread for a convenience wrapper for csv files.) For example:
data = randn(1e4, 20);
save data.txt data -ASCII
tic
h = fopen('data.txt')
data_read = fscanf(h, '%f');
data_read = reshape(data_read, 1e4, []);
toc
Elapsed time is 0.089097 seconds.
If the data are all numeric, then it is fastest to store and operate on simple matrices.
Also, if you post some specific data and reproducible code, we might be able to give more specific answers...
Related
I have a piece of MATLAB code that works fine, but I wanted to know is there any faster way of performing the same task, where each .csv file is a 768*768 dimension matrix
Current code:
for k = 1:143
matFileName = sprintf('ang_thresholded%d.csv', k);
matData = load(matFileName);
imshow(matData)
end
Any help in this regard will be very helpful. Thank You!
In general, its better to separate the loading, computational and graphical stuff.
If you have enough memory, you should try to change your code to:
n_files=143;
% If you know the size of your images a priori:
matData=zeros( 768, 768,n_files); % prealocate for speed.
for k = 1:n_files
matFileName = sprintf('ang_thresholded%d.csv', k);
matData(:,:,k) = load(matFileName);
end
seconds=0.01;
for k=1:n_Files
%clf; %Not needed in your case, but needed if you want to plot more than one thing (hold on)
imshow(matData(:,:,k));
pause(seconds); % control "framerate"
end
Note the use of pause().
Here is another option using Matlab's data stores which are designed to work with large datasets or lots of smaller sets. The TabularTextDatastore is specifically for this kind of text based data.
Something like the following. However, note that since I don't have any test files it is sort of notional example ...
ttds = tabularTextDatastore('.\yourDirPath\*.csv'); %Create the data store
while ttds.hasdata %This turns false after reading the last file.
temp = read(ttds); %Returns a Matlab table class
imshow(temp.Variables)
end
Since it looks like your filenames' numbering is not zero padded (e.g. 1 instead of 001) then the file order might get messed up so that may need addressed as well. Anyway I thought this might be a good alternative approach worth considering depending on what else you want to do with the data and how much of it there might be.
I have a loop as below
for chnum=1:300
PI=....
area=....
save ('Result.mat' ,'chnum' ,'PI' ,'area',' -append') %-append
%% I like to have sth like below
% 1, 1.2,3.7
% 2, 1,8, 7.8
% .....
end
but it doesn't save. Do you have any idea why?
Best
Analysis of the Problem
The matlab help page for save states that the -append option will append new variables to the saved file. It will not append new rows to the already saved matrices.
Solution
To achieve what you intended you have to store your data in matrices and save the whole matrice with a single call to save().
PI = zeros(300,1);
area = zeros(300,1);
for chnum=1:300
PI(chnum)=.... ;
area(chnum)=.... ;
end
save ('Result.mat' ,'chnum' ,'PI' ,'area');
For nicer memory management I have added a pre-allocation of the arrays.
Well, even if it's not part of the question, I don't think that you are using a good approach to save your calculations. Reading/writing operations performed on the disk (saving data on a file is falls in this case) are very expensive in terms of time. This is why I suggest you to proceed as follows:
res = NaN(300,2)
for chnum = 1:300
PI = ...
area = ...
res(chnum,:) = [PI area]; % saving chnum looks a bit like an overkill since you can retrieve it just using size(res,1) when you need it...
end
save('Result.mat','res');
Basically, instead of processing a row and saving it into the file, then processing another row and saving it into the file, etc... you just save your whole data into a matrix and you just save your final result to file.
I am trying to loop through multiple structures at once, extract variables of interest, and combine them into a single cell array. The problem: all of the variables have the same names. I have a working pseudocode--here it is:
Let's say I load i structures in my workspace. Now I want to loop through each structure, and extract time and position data from each structure.
First, I load my structures. Something like...
data_1
data_2
data_3
Then, I create appropriately sized cell arrays.
time{i,:} = zeros(size(structures));
position{i,:} = zeros(size(structures));
Finally, I loop through my structures to extract cell arrays and create a single array.
for i = 1:size(structures)
time_i= data_i.numbers.time;
position_i= data_i.numbers.position;
time {i,:} = time_i;
position{i,:} = position_i;
end
I want to end with a cell array containing a concatenation of all the variables in a single cell structure.
Could you please help convert my pseudo code/ideas into a script, or point me to resources that might help?
Thanks!
You're likely going to be better off loading your data internal to the loop and storing it into a cell or structure rather than trying to deal with iteratively named variables in your workspace. eval is, in nearly all cases, a significant code smell, not least of which because MATLAB's JIT compiler ignores eval statements so you get none of the engine's optimizations. eval statements are also difficult to parse, debug, and maintain.
An example of a stronger approach:
for ii = 1:nfiles
tmp = load(thefilenames{ii}); % Or use output of dir
trialstr = sprintf('trial_%u', ii); % Generate trial string
data.(trialstr).time = tmp.numbers.time;
data.(trialstr).position = tmp.numbers.position;
end
Which leaves you with a final data structure of:
data
trial_n
time
position
Which is far easier to iterate through later.
My final script for anyone interested:
for i = 1:4 %for 4 structures that I am looping through
eval(['time_',num2str(i),'= data_',num2str(i),'.numbers.time;']);
eval(['position_',num2str(i),'= data_',num2str(i),'.numbers.position;']);
%concatenate data into a single cell array here
time{i} = {eval(['time_',num2str(i)])};
position{i} = {eval(['position_',num2str(i)])};
end
...
eval(['time_',num2str(i),'= data_',num2str(i),'.numbers.time;'])
eval(['position_',num2str(i),'= data_',num2str(i),'.numbers.position;'])
...
I have a problem with increasing the size of the structure iteratively. I could not find its solution.
What I need is to have 200 vectors (named EVE1-EVE200), as fields of a structure.
I wrote the following code in matlab:
for i=1:200;
events=struct(['EVE' num2str(i)],[]);
end
But it just forms the last structure.
Would anyone please help me?
Is this way of initializing a data base is the best way anyways?
For what you ask for, it is best to use the following approach:
events = struct;
for i=1:200
events.( ['EVE' num2str(i)] ) = [];
end
But why not use a cell array instead of a struct?
events = {};
for i=1:200
events{i} = [];
end
The latter is much clearer and serves the same purpose.
I want to find the average of all the matrix:
Data=(Data{1}+......+Data{n})/n)
where Data{n} is a matrix of m*n..
Thank you sooo much
First, you convert your cell array into a 3D array, then you can take the average, like this:
tmp = cat(3,Data{:}); %# catenates the data, so that it becomes a m*n*z (or m*1*n)
averageData = mean(tmp,3); %# takes average along 3rd dimension
Note: if you get memory problems this way, and if you don't need to keep the variable Data around, you can replace tmp with Data and all will work fine.
Alternatively, if Data is simply a m*n numeric array
averageData = mean(Data,2);
If your cell array is really big, you might want to keep away from the above solution because of its memory usage. I'd then suggest using the utility mtimesx which is available from Matlab Central, here.
N = length(Data);
b = cell(N,1);
b(:) = {1};
averageData = mtimesx(Data,b)/N;
In the above example, I assumed that Data is a line-shaped cell array. I have never used personally mtimesx, this solution comes from there, where timing issues are also discussed.
Hope this helps.
A.