How to solve the .mat file sorting? - matlab

I have three .mat files, A.mat, B.mat and C.mat. And the content of the .mat files look like this:
![enter image description here][1]
It is in r2 = [rs2,cs2,sortedValues2] format. How can I sort rs2(e.g, 3468, 3909...) of the three
.mat files together in increasing order, and count the number of appearance of each element of rs2?
Anyone can give me any suggestion?

Original
First, you're going to want load each files r2, then pull out it's rs(1, :) value into a column vector, then sort that column vector. Once you have the vector, use logical indexing to determine how many times each element repeats. Finally, attach those tow columns together to attach each element to it's corresponding count. The output vector will have duplicate rows, but I assume that won't be a problem.
allData = [load('A.mat',r2); load('B.mat',r2)l; load('C.mat',r2)];
colVector = allData(:, 1);
sortedVec = sort(colVector);
countVec = zeros(size(sortedVec));
for ii = 1:length(sortedVec)
countVec(ii) = sum(sortedVec==sortedVec(ii));
end
outputVec = [sortedVec, countVec]
Edit
Since your edited question is easy, and almost the same as your original, I'll answer it here. Most of the code is the same, you just need to get the data out of the cell array instead of the files. Like so:
colVector = [yourCellArray{:}];
sortedVec = sort(colVector);
countVec = zeros(size(sortedVec));
for ii = 1:length(sortedVec)
countVec(ii) = sum(sortedVec==sortedVec(ii));
end
outputVec = [sortedVec, countVec]

load A
rs2_1 = r2(:,1);
clearvars r2
load B
rs2_2 = r2(:,1);
clearvars r2
load C
rs2_3 = r2(:,1);
clearvars r2
% to combine
rs2_combined = [rs2_1;rs2_2;rs2_3];
% to sort
rs2_sorted = sort(rs2_combined);
% to count for appearance
rs2_count = hist(rs2_combined, min(rs2_combined):1:max(rs2_combined));
EDIT: using cell arrays
% recreate your situation
R = cell(11,10);
R = cellfun(#(x) [randi(3000,50,1),ones(50,1),ones(50,1)*-.008], R,'UniformOutput', false);
% extract rs2
r = cell2mat( reshape(R, [1,1,11,10]) );
rs2 = reshape( r(:,1,:,:), [50*11*10,1] );
% do what you want
rs2_sorted = sort(rs2);
rs2_count = hist(rs2, min(rs2):1:max(rs2));
note - I assumed you have 50x3 arrays. If merely 50x1, then reshape(R, [1,11,10]) and reshape( r, [50*11*10,1] ); also works.
hist put all numbers into different bins of different values. It's equivalent to do
rs2_scale = min(rs2_combined):1:max(rs2_combined);
rs2_count = zeros(1, length(rs2_scale));
for ii = 1:length(rs2_scale)
rs2_count(ii) = sum( rs2_combined == rs2_scale(ii) );
end
To remove zero-count numbers -
rs2_count(rs2_count==0) = [];
Then you can calculate the probability -
rs2_prob = rs2_count / sum(rs2_count);
Validate this answer by
>> sum(rs2_prob)
ans =
1.0000

Related

Function call with variable number of input arguments when number of input arguments is not explicitly known

I have a variable pth which is a cell array of dimension 1xn where n is a user input. Each of the elements in pth is itself a cell array and length(pth{k}) for k=1:n is variable (result of another function). Each element pth{k}{kk} where k=1:n and kk=1:length(pth{k}) is a 1D vector of integers/node numbers of again variable length. So to summarise, I have a variable number of variable-length vectors organised in a avriable number of cell arrays.
I would like to try and find all possible intersections when you take a vector at random from pth{1}, pth{2}, {pth{3}, etc... There are various functions on the File Exchange that seem to do that, for example this one or this one. The problem I have is you need to call the function this way:
mintersect(v1,v2,v3,...)
and I can't write all the inputs in the general case because I don't know explicitly how many there are (this would be n above). Ideally, I would like to do some thing like this;
mintersect(pth{1}{1},pth{2}{1},pth{3}{1},...,pth{n}{1})
mintersect(pth{1}{1},pth{2}{2},pth{3}{1},...,pth{n}{1})
mintersect(pth{1}{1},pth{2}{3},pth{3}{1},...,pth{n}{1})
etc...
mintersect(pth{1}{1},pth{2}{length(pth{2})},pth{3}{1},...,pth{n}{1})
mintersect(pth{1}{1},pth{2}{1},pth{3}{2},...,pth{n}{1})
etc...
keep going through all the possible combinations, but I can't write this in code. This function from the File Exchange looks like a good way to find all possible combinations but again I have the same problem with the function call with the variable number of inputs:
allcomb(1:length(pth{1}),1:length(pth{2}),...,1:length(pth{n}))
Does anybody know how to work around this issue of function calls with variable number of input arguments when you can't physically specify all the input arguments because their number is variable? This applies equally to MATLAB and Octave, hence the two tags. Any other suggestion on how to find all possible combinations/intersections when taking a vector at random from each pth{k} welcome!
EDIT 27/05/20
Thanks to Mad Physicist's answer, I have ended up using the following which works:
disp('Computing intersections for all possible paths...')
grids = cellfun(#(x) 1:numel(x), pth, 'UniformOutput', false);
idx = cell(1, numel(pth));
[idx{:}] = ndgrid(grids{:});
idx = cellfun(#(x) x(:), idx, 'UniformOutput', false);
idx = cat(2, idx{:});
valid_comb = [];
k = 1;
for ii = idx'
indices = reshape(num2cell(ii), size(pth));
selection = cellfun(#(p,k) p{k}, pth, indices, 'UniformOutput', false);
if my_intersect(selection{:})
valid_comb = [valid_comb k];
endif
k = k+1;
end
My own version is similar but uses a for loop instead of the comma-separated list:
disp('Computing intersections for all possible paths...')
grids = cellfun(#(x) 1:numel(x), pth, 'UniformOutput', false);
idx = cell(1, numel(pth));
[idx{:}] = ndgrid(grids{:});
idx = cellfun(#(x) x(:), idx, 'UniformOutput', false);
idx = cat(2, idx{:});
[n_comb,~] = size(idx);
temp = cell(n_pipes,1);
valid_comb = [];
k = 1;
for k = 1:n_comb
for kk = 1:n_pipes
temp{kk} = pth{kk}{idx(k,kk)};
end
if my_intersect(temp{:})
valid_comb = [valid_comb k];
end
end
In both cases, valid_comb has the indices of the valid combinations, which I can then retrieve using something like:
valid_idx = idx(valid_comb(1),:);
for k = 1:n_pipes
pth{k}{valid_idx(k)} % do something with this
end
When I benchmarked the two approaches with some sample data (pth being 4x1 and the 4 elements of pth being 2x1, 9x1, 8x1 and 69x1), I got the following results:
>> benchmark
Elapsed time is 51.9075 seconds.
valid_comb = 7112
Elapsed time is 66.6693 seconds.
valid_comb = 7112
So Mad Physicist's approach was about 15s faster.
I also misunderstood what mintersect did, which isn't what I wanted. I wanted to find a combination where no element present in two or more vectors, so I ended writing my version of mintersect:
function valid_comb = my_intersect(varargin)
% Returns true if a valid combination i.e. no combination of any 2 vectors
% have any elements in common
comb_idx = combnk(1:nargin,2);
[nr,nc] = size(comb_idx);
valid_comb = true;
k = 1;
% Use a while loop so that as soon as an intersection is found, the execution stops
while valid_comb && (k<=nr)
temp = intersect(varargin{comb_idx(k,1)},varargin{comb_idx(k,2)});
valid_comb = isempty(temp) && valid_comb;
k = k+1;
end
end
Couple of helpful points to construct a solution:
This post shows you how to construct a Cartesian product between arbitrary arrays using ndgrid.
cellfun accepts multiple cell arrays simultaneously, which you can use to index specific elements.
You can capture a variable number of arguments from a function using cell arrays, as shown here.
So let's get the inputs to ndgrid from your outermost array:
grids = cellfun(#(x) 1:numel(x), pth, 'UniformOutput', false);
Now you can create an index that contains the product of the grids:
index = cell(1, numel(pth));
[index{:}] = ndgrid(grids{:});
You want to make all the grids into column vectors and concatenate them sideways. The rows of that matrix will represent the Cartesian indices to select the elements of pth at each iteration:
index = cellfun(#(x) x(:), index, 'UniformOutput', false);
index = cat(2, index{:});
If you turn a row of index into a cell array, you can run it in lockstep over pth to select the correct elements and call mintersect on the result.
for i = index'
indices = num2cell(i');
selection = cellfun(#(p, i) p{i}, pth, indices, 'UniformOutput', false);
mintersect(selection{:});
end
This is written under the assumption that pth is a row array. If that is not the case, you can change the first line of the loop to indices = reshape(num2cell(i), size(pth)); for the general case, and simply indices = num2cell(i); for the column case. The key is that the cell from of indices must be the same shape as pth to iterate over it in lockstep. It is already generated to have the same number of elements.
I believe this does the trick. Calls mintersect on all possible combinations of vectors in pth{k}{kk} for k=1:n and kk=1:length(pth{k}).
Using eval and messing around with sprintf/compose a bit. Note that typically the use of eval is very much discouraged. Can add more comments if this is what you need.
% generate some data
n = 5;
pth = cell(1,n);
for k = 1:n
pth{k} = cell(1,randi([1 10]));
for kk = 1:numel(pth{k})
pth{k}{kk} = randi([1 100], randi([1 10]), 1);
end
end
% get all combs
str_to_eval = compose('1:length(pth{%i})', 1:numel(pth));
str_to_eval = strjoin(str_to_eval,',');
str_to_eval = sprintf('allcomb(%s)',str_to_eval);
% use eval to get all combinations for a given pth
all_combs = eval(str_to_eval);
% and make strings to eval in intersect
comp = num2cell(1:numel(pth));
comp = [comp ;repmat({'%i'}, 1, numel(pth))];
str_pattern = sprintf('pth{%i}{%s},', comp{:});
str_pattern = str_pattern(1:end-1); % get rid of last ,
strings_to_eval = cell(length(all_combs),1);
for k = 1:size(all_combs,1)
strings_to_eval{k} = sprintf(str_pattern, all_combs(k,:));
end
% and run eval on all those strings
result = cell(length(all_combs),1);
for k = 1:size(all_combs,1)
result{k} = eval(['mintersect(' strings_to_eval{k} ')']);
%fprintf(['mintersect(' strings_to_eval{k} ')\n']); % for debugging
end
For a randomly generated pth, the code produces the following strings to evaluate (where some pth{k} have only one cell for illustration):
mintersect(pth{1}{1},pth{2}{1},pth{3}{1},pth{4}{1},pth{5}{1})
mintersect(pth{1}{1},pth{2}{1},pth{3}{1},pth{4}{2},pth{5}{1})
mintersect(pth{1}{1},pth{2}{1},pth{3}{1},pth{4}{3},pth{5}{1})
mintersect(pth{1}{1},pth{2}{1},pth{3}{2},pth{4}{1},pth{5}{1})
mintersect(pth{1}{1},pth{2}{1},pth{3}{2},pth{4}{2},pth{5}{1})
mintersect(pth{1}{1},pth{2}{1},pth{3}{2},pth{4}{3},pth{5}{1})
mintersect(pth{1}{2},pth{2}{1},pth{3}{1},pth{4}{1},pth{5}{1})
mintersect(pth{1}{2},pth{2}{1},pth{3}{1},pth{4}{2},pth{5}{1})
mintersect(pth{1}{2},pth{2}{1},pth{3}{1},pth{4}{3},pth{5}{1})
mintersect(pth{1}{2},pth{2}{1},pth{3}{2},pth{4}{1},pth{5}{1})
mintersect(pth{1}{2},pth{2}{1},pth{3}{2},pth{4}{2},pth{5}{1})
mintersect(pth{1}{2},pth{2}{1},pth{3}{2},pth{4}{3},pth{5}{1})
mintersect(pth{1}{3},pth{2}{1},pth{3}{1},pth{4}{1},pth{5}{1})
mintersect(pth{1}{3},pth{2}{1},pth{3}{1},pth{4}{2},pth{5}{1})
mintersect(pth{1}{3},pth{2}{1},pth{3}{1},pth{4}{3},pth{5}{1})
mintersect(pth{1}{3},pth{2}{1},pth{3}{2},pth{4}{1},pth{5}{1})
mintersect(pth{1}{3},pth{2}{1},pth{3}{2},pth{4}{2},pth{5}{1})
mintersect(pth{1}{3},pth{2}{1},pth{3}{2},pth{4}{3},pth{5}{1})
mintersect(pth{1}{4},pth{2}{1},pth{3}{1},pth{4}{1},pth{5}{1})
mintersect(pth{1}{4},pth{2}{1},pth{3}{1},pth{4}{2},pth{5}{1})
mintersect(pth{1}{4},pth{2}{1},pth{3}{1},pth{4}{3},pth{5}{1})
mintersect(pth{1}{4},pth{2}{1},pth{3}{2},pth{4}{1},pth{5}{1})
mintersect(pth{1}{4},pth{2}{1},pth{3}{2},pth{4}{2},pth{5}{1})
mintersect(pth{1}{4},pth{2}{1},pth{3}{2},pth{4}{3},pth{5}{1})
As Madphysicist pointed out, I misunderstood the initial structure of your initial cell array, however the point stands. The way to pass an unknown number of arguments to a function is via comma-separated-list generation, and your function needs to support it by being declared with varargin. Updated example below.
Create a helper function to collect a random subcell from each main cell:
% in getRandomVectors.m
function Out = getRandomVectors(C) % C: a double-jagged array, as described
N = length(C);
Out = cell(1, N);
for i = 1 : length(C)
Out{i} = C{i}{randi( length(C{i}) )};
end
end
Then assuming you already have an mintersect function defined something like this:
% in mintersect.m
function Intersections = mintersect( varargin )
Vectors = varargin;
N = length( Vectors );
for i = 1 : N; for j = 1 : N
Intersections{i,j} = intersect( Vectors{i}, Vectors{j} );
end; end
end
Then call this like so:
C = { { 1:5, 2:4, 3:7 }, {1:8}, {2:4, 3:9, 2:8} }; % example double-jagged array
In = getRandomVectors(C); % In is a cell array of randomly selected vectors
Out = mintersect( In{:} ); % Note the csl-generator syntax
PS. I note that your definition of mintersect differs from those linked. It may just be you didn't describe what you want too well, in which case my mintersect function is not what you want. What mine does is produce all possible intersections for the vectors provided. The one you linked to produces a single intersection which is common to all vectors provided. Use whichever suits you best. The underlying rationale for using it is the same though.
PS. It is also not entirely clear from your description whether what you're after is a random vector k for each n, or the entire space of possible vectors over all n and k. The above solution does the former. If you want the latter, see MadPhysicist's solution on how to create a cartesian product of all possible indices instead.

how to write vector with different length in a file?

I have 7 vectors (L_mean_argil, L_Min_argil, L_Max_argil, R_mean_argil, R_Min_argil, R_Max_argil, Zero_argil) with different lengths and I want to create a .sae file with all the vectors. Here my code:
fileID = fopen('BN_for_prediciting_zeros_argilliti_v1.sae','w');
fprintf(fileID,'L_mean_argil, L_Min_argil, L_Max_argil, R_mean_argil, R_Min_argil, R_Max_argil, Zero_argil\n');
fprintf(fileID,'%6.4f, %6.4f, %6.4f, %6.4f, %6.4f, %6.4f, %6.4f \n', L_mean_argil, L_Min_argil, L_Max_argil, R_mean_argil, R_Min_argil, R_Max_argil, Zero_argil');
fclose(fileID);
It doesn't write vectors properly. Someone can help me?
I expect this:
L_mean_argil(1) L_Min_argil(1) ... Zero_argil(1)
L_mean_argil(2) L_Min_argil(2) ... Zero_argil(2)
...
L_mean_argil(end) L_Min_argil(end) ... Zero_argil(end)
But in the final part of some vectors, there will be empty cells associated to others with values
Here is an answer assuming you want to leave blanks in rows with missing data.
%Sample Data all of different lengths
a = 1:4;
b = (1:6)+1;
c = (1:5)+2;
%Get max length
maxSize = max([numel(a) numel(b) numel(c)]);
%Convert to Cell array if it isn't already
a = num2cell(a,1);
b = num2cell(b,1);
c = num2cell(c,1);
%Initialize a cell to hold everything. Init to the max size
output = cell(3,maxSize);
output(1,1:numel(a)) = a;
output(2,1:numel(b)) = b;
output(3,1:numel(c)) = c;
%Print...
fprintf('%0.1f,%0.1f,%0.1f\n',output{:})
Results in the this:
1.0,2.0,3.0
2.0,3.0,4.0
3.0,4.0,5.0
4.0,5.0,6.0
,6.0,7.0
,7.0,

How do I load multiple files in MatLab?

I have the following code to load a single matrix file in matlab
filename='1';
filetype='.txt';
filepath='D:\20170913\';
fidi = fopen(strcat(filepath,filename,filetype));
Datac = textscan(fidi, repmat('%f', 1, 640), 'HeaderLines',1,
'CollectOutput',1);
f1 = Datac{1};
sum(sum(f1))
how can I load many files, say 1-100.
Thanks in advance.
Just include everything in a for loop that loops from 1 up to N_files, which is the number of files that you have. I used the function num2str() to convert the index i to a string. I also included the matrix sum in an array file_sums and a loaded_matrices cell array that stores all the loaded matrices. If all the loaded matrices have a known and same dimensions, you could use a 2D array (ex. loaded_matrices = zeros(N_rows, N_columns, N_files); and then load the data as loaded_matrices(:,:,i) = Datac{1};).
% N_files - the number of files that you have
N_files = 100;
file_sums = zeros(1,N_files);
loaded_matrices = cell(1,N_files);
for i=1:1:N_files
filename=num2str(i);
filetype='.txt';
filepath='';
fidi = fopen(strcat(filepath,filename,filetype));
Datac = textscan(fidi, repmat('%f', 1, 640), 'HeaderLines',1,...
'CollectOutput',1);
loaded_matrices{i} = Datac{1};
file_sums(i) = sum(sum(loaded_matrices{i}));
end

How to store .csv data and calculate average value in MATLAB

Can someone help me to understand how I can save in matlab a group of .csv files, select only the columns in which I am interested and get as output a final file in which I have the average value of the y columns and standard deviation of y axes? I am not so good in matlab and so I kindly ask if someone to help me to solve this question.
Here what I tried to do till now:
clear all;
clc;
which_column = 5;
dirstats = dir('*.csv');
col3Complete=0;
col4Complete=0;
for K = 1:length(dirstats)
[num,txt,raw] = xlsread(dirstats(K).name);
col3=num(:,3);
col4=num(:,4);
col3Complete=[col3Complete;col3];
col4Complete=[col4Complete;col4];
avgVal(K)=mean(col4(:));
end
col3Complete(1)=[];
col4Complete(1)=[];
%columnavg = mean(col4Complete);
%columnstd = std(col4Complete);
% xvals = 1 : size(columnavg,1);
% plot(xvals, columnavg, 'b-', xvals, columnavg-columnstd, 'r--', xvals, columnavg+columstd, 'r--');
B = reshape(col4Complete,[5000,K]);
m=mean(B,2);
C = reshape (col4Complete,[5000,K]);
S=std(C,0,2);
Now I know that I should compute mean and stdeviation inside for loop, using mean()function, but I am not sure how I can use it.
which_column = 5;
dirstats = dir('*.csv');
col3Complete=[]; % Initialise as empty matrix
col4Complete=[];
avgVal = zeros(length(dirstats),2); % initialise as columnvector
for K = 1:length(dirstats)
[num,txt,raw] = xlsread(dirstats(K).name);
col3=num(:,3);
col4=num(:,4);
col3Complete=[col3Complete;col3];
col4Complete=[col4Complete;col4];
avgVal(K,1)=mean(col4(:)); % 1st column contains mean
avgVal(K,2)=std(col4(:)); % 2nd column contains standard deviation
end
%columnavg = mean(col4Complete);
%columnstd = std(col4Complete);
% xvals = 1 : size(columnavg,1);
% plot(xvals, columnavg, 'b-', xvals, columnavg-columnstd, 'r--', xvals, columnavg+columstd, 'r--');
B = reshape(col4Complete,[5000,K]);
meanVals=mean(B,2);
I didn't change much, just initialised your arrays as empty arrays so you do not have to delete the first entry later on and made avgVal a column vector with the mean in column 1 and the standard deviation in column 1. You can of course add two columns if you want to collect those statistics for your 3rd column in the csv as well.
As a side note: xlsread is rather heavy for reading files, since Excel is horribly inefficient. If you want to read a structured file such as a csv, it's faster to use importdata.
Create some random matrix to store in a file with header:
A = rand(1e3,5);
out = fopen('output.csv','w');
fprintf(out,['ColumnA', '\t', 'ColumnB', '\t', 'ColumnC', '\t', 'ColumnD', '\t', 'ColumnE','\n']);
fclose(out);
dlmwrite('output.csv', A, 'delimiter','\t','-append');
Load it using csvread:
data = csvread('output.csv',1);
data now contains your five columns, without any headers.

MATLAB memory allocation: loop over mat-files

I have n mat-files containing parts of a single huge matrix. I want to load the mat-files and concatenate the row,column,value-vectors to build matrix W. Right now, I'm doing the following, but it's really slow and I know I shouldn't dynamically increase the size of rows,cols,vals:
rows=[];cols=[];vals=[];
for ii=1:n
var = load( sprintf('w%d.mat', ii) );
[r,c,v] = find( var.w );
rows = [rows; r];
cols = [cols; c];
vals = [vals; v];
end
W = sparse( rows, cols, vals );
Do you know a better way? Thanks in advance!
Solution
Based on Daniel R's suggestion, I solved it like this:
rows=zeros(maxSize,1);
cols=zeros(maxSize,1);
vals=zeros(maxSize,1);
idx = 1;
for ii=1:n
var = load( sprintf('w%d.mat', ii) );
[r,c,v] = find( var.w );
len = size(r,1);
rows(idx:(idx-1+len))=r;
cols(idx:(idx-1+len))=c;
vals(idx:(idx-1+len))=v;
idx = idx+len;
end
W = sparse( rows, cols, vals );
Extremely fast, thanks a lot!
You need to preallocate an array. Assuming r,c and v have the size a*b each.
In total, you need a*n rows and b columns, thus preallocate using rows=nan(a*n,b)
To write the Data into the array, you have to set the correct indices: rows((ii-1)*a+1:ii*a,1:end)=r
A few ideas.
It looks to me the matrices have the same sizes.
Another bottleneck might be the find function.
could it happen that two matrices have different values at the same index? That might lead to problems.
You could do:
var = load( 'w1.mat' );
W = var.w;
for ii = 2:n
var = load( sprintf('w%d.mat', ii) );
ig = var.w~=0;
W(ig) = var.w(ig);
end
in case var.w is not sparse add a W = sparse(W)
Here is a trick that sometimes helps to speed things up as if your variables were initialized. However, sometimes it just does not help so the only way to find out is to try:
Replace lines like:
rows = [rows; r];
by lines like:
rows(end+(1:numel(r))) = r;