I have a C program which repeatedly executes an algorithm and outputs its intermediate result after each iteration. The data will be processed using Matlab R2019a.
I'm using the C Matrix API to create .MAT files, and I can write a matrix to the .MAT file:
MATFile *m = matOpen("matlab.MAT", "w");
mxArray *a = mxCreateDoubleMatrix(1, 1, mxREAL);
*mxGetPr(a) = nxm;
matPutVariable(m, "nxm", a);
mxDestroyArray(a);
matClose(m);
However, the documentation for [maxPutVariable](https://www.mathworks.com/help/matlab/apiref/matputvariable.html) states that if I use the same variable name twice, the second will overwrite the first.
I do not want to store in memory all of my intermediate values. Perhaps I could read in the matrix, extend it to include a new value, and write it out again.
Is there a decent way to do this using the C API, or should I just write Matlab code to parse a different output format?
You can save multiple named variables in a single MAT-file. And when you add a new variable to a MAT-file, it's appended to the end of the file, so that's not bad on I/O even if your existing file is relatively large. So what I'd do is have each loop iteration of your C program store its output into a new named nxm<NNN> variable in the MAT-file, and then have your Matlab program read the file and concatenate them all together.
In your C code:
MATFILE *m = matOpen("matlab.mat", "w");
[...]
char var_name[1024];
for (i = 0; i < n_iterations; i++) {
[... do work to produce nxm ...]
mxArray *a = mxCreateDoubleMatrix(1, 1, mxREAL);
*mxGetPr(a) = nxm;
sprintf(var_name, "nxm%d", i);
matPutVariable(m, var_name, a);
mxDestroyArray(a);
}
matClose(m);
Then on the Matlab side after that's all done:
s = load('matlab.mat');
c = struct2cell(s);
all_data = cat(1, c{:});
You could also write your code so that each pass reads in the previous pass's output, extends the array, and writes it back to the same named variable. But that would be a lot more coding in C, and (I think) would perform less well if your intermediate results were large, instead of these single numbers.
For that matter, since it looks like you're working with a small data set - nxm is a scalar double - you could just fprintf("%d\n", nxm) your nxm values to a text file with a single number on each line, and then read it in to Matlab using fscanf or str2double.
Related
I am currently working with a script that saves matrices as .mat files from other .mat files. I need to save 96 separate files so I have a loop that goes through the matrix names. I need to have the matrices saved with specific titles, that I have saved the titles in cell arrays {}. However, when I use the save(filename,variable) function, I get an error saying:
Error using save
Must be a text scalar.
Error in File_Creator (line 35)
save(name,fname);
My matrices need to be named 'PHI_Af','PHI_Am' (so on until) 'SLR_EF' (so every cr value needs to have a matrix with every par value. Here is what I am currently attempting:
cr = {'Af','Am','As','Aw','BS','BW','Cs','Cw','Cf','Ds','Dw','Df','ET','EF'};
par = {'PHI','BLD','KS','LAMBDA','PSIS','SLR'};
underscore = {'_'};
%% i and j are parameters in a loop where i = 1:length(par) and j = 1:length(cr)
%% f is the variable currently storing the matrix
s.(horzcat(par{i},underscore{1},cr{j})) = f;
name = string(strcat(par{i},'_',cr{j},'.mat'));
fname = string(s.(horzcat(par{i},underscore{1},cr{j})));
save(name,fname);
When I replace 'fname' with a generic string e.g. 'f', then the command runs but all the matrices save as the same thing ('f'), which makes it extremely difficult to run them all in the same script later.
I hope somebody can tell me what I'm doing wrong or provide me with a better solution. Please let me know if I can provide any more information.
Thank you
Assuming that the matrix, f, changes in each iteration of the loop (due to some other code you didn't post), it seems like this is all the code you need:
cr = {'Af','Am','As','Aw','BS','BW','Cs','Cw','Cf','Ds','Dw','Df','ET','EF'};
par = {'PHI','BLD','KS','LAMBDA','PSIS','SLR'};
for i = 1:length(par)
for j = 1:length(cr)
% add code here that loads the matrix f
name = [par{i}, '_', cr{j}, '.mat'];
save(name, 'f');
end
end
My code is posted below. It does exactly what I need it to do.
It reads in a file and plots the data that I need. If I want to read in another file and have it go through the same code, without having to write the whole thing a second time with different variables, is that possible? I would like to store the matrices from each loop.
As you can see the file I get is called: Oxygen_1keV_300K.xlsx
I have another file called: Oxygen_1keV_600K.xlsx
and so on.
How can I loop through these files without having to re-code the whole thing? I then want to plot them all on the same graph. It would be nice to store the final matrix Y and Ymean for each file so they are not overwritten.
clear
clc
files = ['Oxygen_1keV_300K','Oxygen_1keV_300K','Oxygen_1keV_600K','Oxygen_1keV_900K'];
celldata = cellstr(file)
k = cell(1,24);
for k=1:24
data{k} = xlsread('C:\Users\Ben\Desktop\Oxygen_1keV_300K.xlsx',['PKA', num2str(k)]);
end
for i=1:24
xfinal{i}=data{1,i}(end,1);
xi{i}=0:0.001:xfinal{i};
xi{i}=transpose(xi{i});
x{i}=data{1,i}(:,1);
y{i}=data{1,i}(:,4);
yi{i} = interp1(x{i},y{i},xi{i});
end
Y = zeros(10001, numel(data));
for ii = 1 : numel(data)
Y(:, ii) = yi{ii}(1 : 10001);
end
Ymean = mean(Y, 2);
figure (1)
x=0:0.001:10;
semilogy(x,Ymean)
Cell arrays make it very easy to store a list of strings that you can access as part of a for loop. In this case, I would suggest putting your file paths in a cell array as a substitute for the string used in your xlsread call
For example,
%The first file is the same as in your example.
%I just made up file names for the next two.
%Use the full file path if the file is not in your current directory
filepath_list = {'C:\Users\Ben\Desktop\Oxygen_1keV_300K.xlsx', 'file2.xlsx', 'file3.xlsx'};
%To store separate results for each file, make Ymean a cell array or matrix too
YMean = zeros(length(filepath_list), 1);
%Now use a for loop to loop over the files
for ii=1:length(filepath_list)
%Here's where your existing code would go
%I only include the sections which change due to the loop
for k=1:24
%The change is that on this line you use the cell array variable to load the next file path
data{k} = xlsread(filepath_list{ii},['PKA', num2str(k)]);
end
% ... do the rest of your processing
%You'll need to index into Ymean to store your result in the corresponding location
YMean(ii) = mean(Y, 2);
end
Cell arrays are a basic matlab variable type. For an introduction, I recommend the documentation for creating and accessing data in cell arrays.
If all your files are in the same directory, you can also use functions like dir or ls to populate the cell array programatically.
I have a 2d matrix (A=80,42), I am trying to split it into (80,1) 42 times and save it with a different name. i.e.
M_n1, M_n2, M_n3, … etc (representing the number of column)
I tried
for i= 1:42
M_n(i)=A(:,i)
end
it didn't work
How can I do that without overwrite the result and save each iteration in a file (.txt) ?
You can use eval
for ii = 1:size(A,2)
eval( sprintf( 'M_n%d = A(:,%d);', ii, ii ) );
% now you have M_n? var for you to process
end
However, the use of eval is not recommanded, you might be better off using cell array
M_n = mat2cell( A, [size(A,1)], ones( 1, size(A,2) ) );
Now you have M_n a cell array with 42 cells one for each column of A.
You can access the ii-th column by M_n{ii}
Generally, doing if you consider doing this kind of things: don't.
It does not scale up well, and having them in one array is usually far more convenient.
As long as the results have the same shape, you can use a standard array, if not you can put each result in a cell array eg. :
results = cell(nTests,1)
result{1} = runTest(inputs{1})
or even
results = cellfun(#runTest,inputs,'UniformOutput',false); % where inputs is a cell array
And so on.
If you do want to write the numbers to a file at each iteration, you could do it without the names with csvwrite or the like (since you're only talking about 80 numbers a time).
Another option is using matfile, which lets you write directly to a variable in a .mat file. Consult help matfile for the specifics.
Apparently Simulink supports only few datatypes.
So, how to keep something other? I want to produces images from a directory, how to keep directory list and current position?
The following code causes error
function DoPostPropSetup(block)
block.NumDworks = 1;
block.Dwork(1).Name = 'Filelist';
block.Dwork(1).Dimensions = 1;
% block.Dwork(1).DatatypeID = -10; % MATLAB Array % does not work
block.Dwork(1).DatatypeID = 0; % double
block.Dwork(1).Complexity = 'Real'; % real
block.Dwork(1).UsedAsDiscState = true;
block.Dwork(1).Data = dir(block.DialogPrm(1).Data);
You cannot store non-numeric types in level-2 s-function Dwork. For your use, you may need to convert your list of files to a character string with a path separator and use char type to store it. You need a different Dwork to store the position.
Another approach is to store the list as char matrix with smaller file names padded with spaces or any character which is not part of valid file name. Storing as matrix will make it easier to index into the list.
I am trying to deal with a very large dataset. I have k = ~4200 matrices (varying sizes) which must be compared combinatorially, skipping non-unique and self comparisons. Each of k(k-1)/2 comparisons produces a matrix, which must be indexed against its parents (i.e. can find out where it came from). The convenient way to do this is to (triangularly) fill a k-by-k cell array with the result of each comparison. These are ~100 X ~100 matrices, on average. Using single precision floats, it works out to 400 GB overall.
I need to 1) generate the cell array or pieces of it without trying to place the whole thing in memory and 2) access its elements (and their elements) in like fashion. My attempts have been inefficient due to reliance on MATLAB's eval() as well as save and clear occurring in loops.
for i=1:k
[~,m] = size(data{i});
cur_var = ['H' int2str(i)];
%# if i == 1; save('FileName'); end; %# If using a single MAT file and need to create it.
eval([cur_var ' = cell(1,k-i);']);
for j=i+1:k
[~,n] = size(data{j});
eval([cur_var '{i,j} = zeros(m,n,''single'');']);
eval([cur_var '{i,j} = compare(data{i},data{j});']);
end
save(cur_var,cur_var); %# Add '-append' when using a single MAT file.
clear(cur_var);
end
The other thing I have done is to perform the split when mod((i+j-1)/2,max(factor(k(k-1)/2))) == 0. This divides the result into the largest number of same-size pieces, which seems logical. The indexing is a little more complicated, but not too bad because a linear index could be used.
Does anyone know/see a better way?
Here's a version that combines going fast with using minimal memory.
I use fwrite/fread so that you still can use parfor (and this time, I made sure it works :) )
%# assume data is loaded an k is known
%# find the index pairs for comparisons. This could be done more elegantly, I guess.
%# I'm constructing a lower triangular array, i.e. an array that has ones wherever
%# we want to compare i (row) and j (col). Then I use find to get i and j
[iIdx,jIdx] = find(tril(ones(k,k),-1));
%# create a directory to store the comparisons
mkdir('H_matrix_elements')
savePath = fullfile(pwd,'H_matrix_elements');
%# loop through all comparisons in parallel. This way there may be a bit more overhead from
%# the individual function calls. However, parfor is most efficient if there are
%# a lot of relatively similarly fast iterations.
parfor ct = 1:length(iIdx)
%# make the comparison - do double b/c there shouldn't be a memory issue
currentComparison = compare(data{iIdx(ct)},data{jIdx{ct});
%# create save-name as H_i_j, e.g. H_104_23
saveName = fullfile(savePath,sprintf('H_%i_%i',iIdx(ct),jIdx(ct)));
%# save. Since 'save' is not allowed, use fwrite to write the data to disk
fid = fopen(saveName,'w');
%# for simplicity: save data as vector, add two elements to the beginning
%# to store the size of the array
fwrite(fid,[size(currentComparison)';currentComparison(:)]); % ' #SO formatting
%# close file
fclose(fid)
end
%# to read e.g. comparison H_104_23
fid = fopen(fullfile(savePath,'H_104_23'),'r');
tmp = fread(fid);
fclose(fid);
%# reshape into 2D array.
data = reshape(tmp(3:end),tmp(1),tmp(2));
You can get rid of the eval and clear calls by assigning the filename separately.
for i=1:k
[~,m] = size(data{i});
file_name = ['H' int2str(i)];
cur_var = cell(1, k-i);
for j=i+1:k
[~,n] = size(data{j});
cur_var{i,j} = zeros(m, n, 'single');
cur_var{i,j} = compare(data{i}, data{j});
end
save(file_name, cur_var);
end
If you need the saved variables to take different names, use the -struct option to save.
str.(file_name);
save(file_name, '-struct', str);