I have several data named ET1_A_C1_l1, ET1_A_C2_l1, ET1_A_C3_l1, ..., ET1_A_C63_l1 in Workspace. Besides that,I also have another sets of data named ET1_H_C1_l1, ET1_H_C2_l1, ..., ET1_A_C63_l1
Now I need to combine 2 set of data into one named Total_data.mat; For example,
Total_data=[ET1_A_C1_l1 ET1_A_C2_l1 ET1_A_C3_l1 ..... ET1_A_63_l1;ET1_H_C1_l1 ET1_H_C2_l1 ....ET1_H_C63_l1]
and need to take a huge of time to type the code one by one. Is there any idea using the loop to do this??
Thanks.
Rather than jumping on my wagon straight away, I'll start with the solution (which has been set up with an example):
%# State the size of each matrix
T = 6; N = 2;
%# State the number of matrices in category A and H (63 in your case - but 2 in my example)
K = 2;
%# Set up some example matrices
ET1_A_C1_l1 = rand(T, N); ET1_A_C2_l1 = 1 + rand(T, N);
ET1_H_C1_l1 = 2 + rand(T, N); ET1_H_C2_l1 = 3 + rand(T, N);
%# Preallocate a matrix to hold the output
M = NaN(2 * T, K * N);
%# Loop over the variables and add them to the matrix using the evil eval
for k = 1:K
M(1:T, (k*N)-1:k*N) = eval(['ET1_A_C', num2str(k), '_l1']);
M(T+1:2*T, (k*N)-1:k*N) = eval(['ET1_H_C', num2str(k), '_l1']);
end
%# Save to a mat file
save('Total_Data.mat', 'M');
Now, wagon time: If you've been given the data in the form that you have it now, and there was nothing you could do about it, and you realize what a terrible situation it is to be in, then you can stop reading now.
But, if you were responsible for creating all those E_blah variables in the first place, then you need to take a look at the answer of #jerad and start thinking about different ways of storing data. A cell array or a structure is one way to go about it. Or start with one big matrix in the first place. But remember the following two general rules:
1) If you have more than 20 variables in your workspace, then you're probably doing it wrong.
2) If you find yourself frequently using the evil function eval then you're almost definitely doing it wrong.
Having this kind of problem suggests to me that you're not yet comfortable with the other data structures available in matlab... like cell arrays and structures. You could easily solve this problem by storing your data in a less arrays and then indexing them properly when needed.
Read about structures (this tutorial is excellent) in the matlab documentation and then try to use one to store all of your data. I think that will solve this problem and many others you didn't know you had.
You should be using something like the following.
ET = struct;
ET.A.C(1) = ET1_A_C1;
ET.A.C(2) = ET1_A_C2;
...
ET.A.C(N) = ET1_A_CN;
ET.H.C(1) = ET1_H_C1;
ET.H.C(2) = ET1_H_C2;
...
ET.H.C(N) = ET1_H_CN;
Now every thing is one variable which you can save without typing anything extra.
filename=Total_data.mat;
for i=1:63
J(i,1)=ET1_A_C{i};
J(i,2)=ET1_H_C{i};
end
save(filename,'J(1:63,1)','J(1:63,2)');
Related
In Matlab (R2021b) I am using some given function, which reads time-dependent values of several variables and returns them in a combined matrix together with a time vector. In the data matrix each column represents one vector of time-dependent values for one variable.
[data,time] = function_reading_data_of_several_values('filename');
For readability of the following code where the variables are further processed, I would like to store these columns in separate vector variables. I am doing it like that:
MomentX = data(1,:);
MomentY = data(2,:);
MomentZ = data(3,:);
ForceX = data(4,:);
ForceY = data(5,:);
ForceZ = data(6,:);
That is working. But is there some simpler (or shorter) way of assigning the column of the matrix to individual vectors? I am asking because in real program I have more than the 6 columns as in the example. Code is getting quite long. I was thinking of something similar to the line below, but that does not work:
[MomentX,MomentY,MomentZ,ForceX,ForceY,ForceZ] = data; %does not work
Do you have any idea? Thanks for help!
Update:
Thanks to the hint here in the group to use tables, a solution could be like this:
...
[data,time] = function_reading_data_of_several_values('filename');
% data in matrix. Each column representing a stime dependent variable
varNames = {'MomentX', 'MomentX',...}; % Names of columns
T=array2table(data','VariableNames',varNames); % Transform to Table
Stress = T.MomentX/W + T.ForceY/A %accesing table columns
...
This seems to work fine and readable to me.
Solution 1: In industrial solutions like dSpace, it is very common to do it in struct arrays:
mydata.X(1).time = [0.01 0.02 0.03 0.04];
mydata.Y(1).name = 'MomentX';
mydata.Y(1).data = [1 2 3 4];
mydata.Y(2).name = 'MomentY';
mydata.Y(2).data = [2 3 4 5];
Solution 2: It is also very common to create tables
See: https://de.mathworks.com/help/matlab/ref/table.html
As already commented, it is probably better to use a table instead of separate variables may not be a good idea. But if you want to, it can be done this way:
A = magic(6): % example 6-column matrix
A_cell = num2cell(A, 1); % separate columns in cells
[MomentX, MomentY, MomentZ, ForceX, ForceY, ForceZ] = A_cell{:};
This is almost the same as your
[MomentX,MomentY,MomentZ,ForceX,ForceY,ForceZ] = data; %does not work
except that the right-hand side needs to be a comma-separated list, which in this case is obtained from a cell array.
Suppose I have n .mat files and each are named as follows: a1, a2, ..., an
And within each of these mat files there is a variable called: var (nxn matrix)
I would like to create a matrix: A = [a1.var a2.var, ..., an.var] without writing it all out because there are many .mat files
A for-loop comes to mind, something like this:
A = []
for i = 1:n
[B] = ['a',num2str(i),'.mat',var];
A = [A B]
end
but this doesn't seem to work or even for the most simple case where I have variables that aren't stored as a(i) but rather 'a1', 'a2' etc.
Thank you very much!
load and concatenate 'var' from each of 'a(#).mat':
n = 10;
for i = n:-1:1 % 1
file_i = sprintf('a%d.mat', i); % 2
t = load(file_i, 'var');
varsCell{i} = t.var; % 3
end
A = [varsCell{:}]; % concatenate each 'var' in one step.
Here are some comment on the above code. All the memory-related stuff isn't very important here, but it's good to keep in mind during larger projects.
1)
In MATLAB, it is rarely a good idea or necessary to grow variables during a for loop. Each time an element is added, MATLAB must find and allocate a new block of RAM. This can really slow things down, especially for long loops or large variables. When possible, pre-allocate your variables (A = zeros(n,n*n)). Alternatively, it sometimes works to count backwards in the loop. MATLAB pre-allocates the whole array, since you're effectively telling it the final size.
2)
Equivalent to file_i = ['a',num2str(i),'.mat'] in this case, sprintf can be clearer and more powerful.
3)
Store each 'var' in a cell array. This is a balance between allocating all the needed memory and the complication of indexing into the correct places of a preallocated array. Internally, the cell array is a list of pointers to the location of each loaded 'var' matrix.
to create a test set...
generate 'n' matrices of n*n random doubles
save each as 'a(#).mat' in current directory
for i = 1:n
var = rand(n);
save(sprintf('a%d.mat',i), 'var');
end
Code
%%// The final result, A would have size nX(nXn)
A = zeros(n,n*n); %%// Pre-allocation for better performance
for k =1:n
load(strcat('a',num2str(k),'.mat'))
A(1:n,(k-1)*n+1:(k-1)*n+n) = var;
end
I have this file which is a series of x, y, z coordinates of over 34 million particles and I am reading them in as follows:
parfor i = 1:Ntot
x0(i,1)=fread(fid, 1, 'real*8')';
y0(i,1)=fread(fid, 1, 'real*8')';
z0(i,1)=fread(fid, 1, 'real*8')';
end
Is there a way to read this in without doing a loop? It would greatly speed up the read in. I just want three vectors with x,y,z. I just want to speed up the read in process. Thanks. Other suggestions welcomed.
I do not have a machine with Matlab and I don't have your file to test either but I think coordinates = fread (fid, [3, Ntot], 'real*8') should work fine.
Maybe fread is the function you are looking for.
You're right. Reading data in larger batches is usually a key part of speeding up file reads. Another part is pre-allocating the destination variable zeros, for example, a zeros call.
I would do something like this:
%Pre-allocate
x0 = zeros(Ntot,1);
y0 = zeros(Ntot,1);
z0 = zeros(Ntot,1);
%Define a desired batch size. make this as large as you can, given available memory.
batchSize = 10000;
%Use while to step through file
indexCurrent = 1; %indexCurrent is the next element which will be read
while indexCurrent <= Ntot
%At the end of the file, we may need to read less than batchSize
currentBatch = min(batchSize, Ntot-indexCurrent+1);
%Load a batch of data
tmpLoaded = fread(fid, currentBatch*3, 'read*8')';
%Deal the fread data into the desired three variables
x0(indexCurrent + (0:(currentBatch-1))) = tmpLoaded(1:3:end);
y0(indexCurrent + (0:(currentBatch-1))) = tmpLoaded(2:3:end);
z0(indexCurrent + (0:(currentBatch-1))) = tmpLoaded(3:3:end);
%Update index variable
indexCurrent = indexCurrent + batchSize;
end
Of course, make sure you test, as I have not. I'm always suspicious of off-by-one errors in this sort of work.
I need to calculate the mean, standard deviation, and other values for a number of variables and I was wondering how to use a loop to my advantage. I have 5 electrodes of data. So to calculate the mean of each I do this:
mean_ch1 = mean(ch1);
mean_ch2 = mean(ch2);
mean_ch3 = mean(ch3);
mean_ch4 = mean(ch4);
mean_ch5 = mean(ch5);
What I want is to be able to condense that code into a line or so. The code I tried does not work:
for i = 1:5
mean_ch(i) = mean(ch(i));
end
I know this code is wrong but it conveys the idea of what I'm trying to accomplish. I want to end up with 5 separate variables that are named by the loop or a cell array with all 5 variables within it allowing for easy recall. I know there must be a way to write this code I'm just not sure how to accomplish it.
You have a few options for how you can do this:
You can put all your channel data into one large matrix first, then compute the mean of the rows or columns using the function MEAN. For example, if each chX variable is an N-by-1 array, you can do the following:
chArray = [ch1 ch2 ch3 ch4 ch5]; %# Make an N-by-5 matrix
meanArray = mean(chArray); %# Take the mean of each column
You can put all your channel data into a cell array first, then compute the mean of each cell using the function CELLFUN:
meanArray = cellfun(#mean,{ch1,ch2,ch3,ch4,ch5});
This would work even if each chX array is a different length from one another.
You can use EVAL to generate the separate variables for each channel mean:
for iChannel = 1:5
varName = ['ch' int2str(iChannel)]; %# Create the name string
eval(['mean_' varName ' = mean(' varName ');']);
end
If it's always exactly 5 channels, you can do
ch = {ch1, ch2, ch3, ch4, ch5}
for j = 1:5
mean_ch(j) = mean(ch{j});
end
A more complicated way would be
for j = 1:nchannels
mean_ch(j) = eval(['mean(ch' num2str(j) ')']);
end
Apart from gnovice's answer. You could use structures and dynamic field names to accomplish your task. First I assume that your channel data variables are all in the format ch* and are the only variables in your MATLAB workspace. The you could do something like the following
%# Move the channel data into a structure with fields ch1, ch2, ....
%# This could be done by saving and reloading the workspace
save('channelData.mat','ch*');
chanData = load('channelData.mat');
%# Next you can then loop through the structure calculating the mean for each channel
flds = fieldnames(chanData); %# get the fieldnames stored in the structure
for i=1:length(flds)
mean_ch(i) = mean(chanData.(flds{i});
end
I am trying to deal with a very large dataset. I have k = ~4200 matrices (varying sizes) which must be compared combinatorially, skipping non-unique and self comparisons. Each of k(k-1)/2 comparisons produces a matrix, which must be indexed against its parents (i.e. can find out where it came from). The convenient way to do this is to (triangularly) fill a k-by-k cell array with the result of each comparison. These are ~100 X ~100 matrices, on average. Using single precision floats, it works out to 400 GB overall.
I need to 1) generate the cell array or pieces of it without trying to place the whole thing in memory and 2) access its elements (and their elements) in like fashion. My attempts have been inefficient due to reliance on MATLAB's eval() as well as save and clear occurring in loops.
for i=1:k
[~,m] = size(data{i});
cur_var = ['H' int2str(i)];
%# if i == 1; save('FileName'); end; %# If using a single MAT file and need to create it.
eval([cur_var ' = cell(1,k-i);']);
for j=i+1:k
[~,n] = size(data{j});
eval([cur_var '{i,j} = zeros(m,n,''single'');']);
eval([cur_var '{i,j} = compare(data{i},data{j});']);
end
save(cur_var,cur_var); %# Add '-append' when using a single MAT file.
clear(cur_var);
end
The other thing I have done is to perform the split when mod((i+j-1)/2,max(factor(k(k-1)/2))) == 0. This divides the result into the largest number of same-size pieces, which seems logical. The indexing is a little more complicated, but not too bad because a linear index could be used.
Does anyone know/see a better way?
Here's a version that combines going fast with using minimal memory.
I use fwrite/fread so that you still can use parfor (and this time, I made sure it works :) )
%# assume data is loaded an k is known
%# find the index pairs for comparisons. This could be done more elegantly, I guess.
%# I'm constructing a lower triangular array, i.e. an array that has ones wherever
%# we want to compare i (row) and j (col). Then I use find to get i and j
[iIdx,jIdx] = find(tril(ones(k,k),-1));
%# create a directory to store the comparisons
mkdir('H_matrix_elements')
savePath = fullfile(pwd,'H_matrix_elements');
%# loop through all comparisons in parallel. This way there may be a bit more overhead from
%# the individual function calls. However, parfor is most efficient if there are
%# a lot of relatively similarly fast iterations.
parfor ct = 1:length(iIdx)
%# make the comparison - do double b/c there shouldn't be a memory issue
currentComparison = compare(data{iIdx(ct)},data{jIdx{ct});
%# create save-name as H_i_j, e.g. H_104_23
saveName = fullfile(savePath,sprintf('H_%i_%i',iIdx(ct),jIdx(ct)));
%# save. Since 'save' is not allowed, use fwrite to write the data to disk
fid = fopen(saveName,'w');
%# for simplicity: save data as vector, add two elements to the beginning
%# to store the size of the array
fwrite(fid,[size(currentComparison)';currentComparison(:)]); % ' #SO formatting
%# close file
fclose(fid)
end
%# to read e.g. comparison H_104_23
fid = fopen(fullfile(savePath,'H_104_23'),'r');
tmp = fread(fid);
fclose(fid);
%# reshape into 2D array.
data = reshape(tmp(3:end),tmp(1),tmp(2));
You can get rid of the eval and clear calls by assigning the filename separately.
for i=1:k
[~,m] = size(data{i});
file_name = ['H' int2str(i)];
cur_var = cell(1, k-i);
for j=i+1:k
[~,n] = size(data{j});
cur_var{i,j} = zeros(m, n, 'single');
cur_var{i,j} = compare(data{i}, data{j});
end
save(file_name, cur_var);
end
If you need the saved variables to take different names, use the -struct option to save.
str.(file_name);
save(file_name, '-struct', str);