Load Multiple .mat Files to Matlab workspace - matlab

I'm trying to load several .mat files to the workspace. However, they seem to overwrite each other. Instead, I want them to append. I am aware that I can do something like:
S=load(file1)
R=load(file2)
etc.
and then append the variables manually.
But there's a ton of variables, and making an append statement for each one is extremely undesirable (though possible as a last resort). Is there some way for me to load .mat files to the workspace (by using the load() command without assignment) and have them append?

Its not entirely clear what you mean by "append" but here's a way to get the data loaded into a format that should be easy to deal with:
file_list = {'file1';'file2';...};
for file = file_list'
loaded.(char(file)) = load(file);
end
This makes use of dynamic field references to load the contents of each file in the list into its own field of the loaded structure. You can iterate over the fields and manipulate the data however you'd like from here.

It sounds like you have a situation in which each file contains a matrix variable A and you want to load into memory the concatenation of all these matrices along some dimension. I had a similar need, and wrote the following function to handle it.
function var = loadCat( dim, files, varname )
%LOADCAT Concatenate variables of same name appearing in multiple MAT files
%
% where dim is dimension to concatenate along,
% files is a cell array of file names, and
% varname is a string containing the name of the desired variable
if( isempty( files ) )
var = [];
return;
end
var = load( files{1}, varname );
var = var.(varname);
for f = 2:numel(files),
newvar = load( files{f}, varname );
if( isfield( newvar, varname ) )
var = cat( dim, var, newvar.(varname) );
else
warning( 'loadCat:missingvar', [ 'File ' files{f} ' does not contain variable ' varname ] );
end
end
end

Clark's answer and function actually solved my situation perfectly... I just added the following bit of code to make it a little less tedious. Just add this to the beginning and get rid of the "files" argument:
[files,pathname] = uigetfile('*.mat', 'Select MAT files (use CTRL/COMM or SHIFT)', ...
'MultiSelect', 'on');
Alternatively, it could be even more efficient to just start with this bit:
[pathname] = uigetdir('C:\');
files = dir( fullfile(pathname,'*.mat') ); %# list all *.mat files
files = {files.name}'; %# file names
data = cell(numel(files),1); %# store file contents
for i=1:numel(files)
fname = fullfile(pathname,files{i}); %# full path to file
data{i} = load(fname); %# load file
end
(modified from process a list of files with a specific extension name in matlab).
Thanks,
Jason

Related

Finding pair of files in a folder

I have a folder with tens of thousand of files. Every file in the folder should have a pair matching except the initial few letter, for example:
X_Date_Time_Place.dat
Y_Date_Time_Place.dat
Each X_* and Y_* combine to make one pair of files.
However, there always be some thousand of files extra which need to be eliminated from the folder. The extra files are also of the same type but without pair. For example, there may be more 'X_Date_Time_Place.dat' then 'Y_Date_Time_Place.dat'. Only variables in the file names are 'Date', 'Time' and 'Place'.
I have written a simple script (using for loop) that takes the name of one file and check all the other files in a loop until it finds its match. However, it is taking enormous amount of time to find a pair.
Is there any faster and more efficient way to do it?
You can split to two lists:
xlist = dir( fullfile( path_to_folder, 'X_*.dat') );
ylist = dir( fullfile( path_to_folder, 'Y_*.dat') );
%// remove prefixes
xlist = cellfun(#(x) x(3:end), {xlist.name}, 'uni', false);
ylist = cellfun(#(y) y(3:end), {ylist.name}, 'uni', false);
common = intersect(xlist, ylist);
Using intersect to find the common suffixes leaves you with common holding all Date_Time_Place.dat for which you have BOTH X_Date_Time_Place.dat and Y_Date_Time_Place.dat.
To get all pairs:
allParis = cellfun(#(c) {fullfile(path_to_folder,['X_',c]),
fullfile(path_to_folder,['Y_',c])}, common, 'uni', false);
You can use the function dir and specify a string and/or an extension that you want your filename to contain :
In your example :
I=dir('* _Date_Time_Place *.dat')
Will return a struct I whose fields will be all the filenames containing the string *_Date_Time_Place* and having the extension .dat .
You can then access to the elements in the struct with calls to I(1), I(2).
Minor note :
For this to work, your current folder must be the one where your files are.
Well, I don't have 10,000 files formatted like this but here is what I would do.
Xfiles = dir('X*.dat');
filenames = {Xfiles.name};
% Here I would determine how many pairs I am looking for (the unique X's)
% I am assuming that your X files are unique.
% remove the "X" from the file name
filenames2 = cellfun(#(x) repexprep(x, 'X',''));
keys = filenames2;
values = 1:length(filenames2);
fileMap = containers.Map(keys, values);
% for each Y look for the filename
Yfiles = dir('Y*.dat');
Yfiles2 = cellfun(#(x) repexprep(x, 'Y',''));
pairs = cell(length(Yfiles2),2);
% this assumes that for every Y there must be an X
% if this is not true then handle the empty idx case.
for x = 1:length(Yfiles2)
idx = fileMap(Yfiles2{x});
pairs(x,:) = {Xfiles(idx), Yfiles(idx)};
end

for loop+structure allocation in matlab

This is a problem I am working on in Matlab.
I am looping through a series of *.ALL files and stripping the field name by a delimiter '.'. The first field is the network, second station name, third location code and fourth component. I pre-allocate my structure based on the number of files (3) I run through which for this example is a 3x3x3 structure that I would like to define as struct(station,loc code,component). You can see these definitions in my code example below.
I would like to loop through the station, loc code, and component and fill their values in the structure. The only problem is for some reason the way I've defined the loop it's actually looping through the files more than once. I only want to loop through each file once and extract the station, comp, and loc code from it and put it inside the structure. Because it's looping through the files more than once it's taken like 10 minutes to fill the structure. This is not very efficient at all. Can someone point out the culprit line for me or tell me what I'm doing incorrectly?
Here's my code below:
close all;
clear;
[files]=dir('*.ALL');
for i = 1:length(files)
fields=textscan(files(i).name, '%s', 'Delimiter', '.');
net{i,1}=fields{1}{:};
sta{i,1}=fields{1}{2};
loc{i,1}=fields{1}{3};
comp{i,1}=fields{1}{4};
data = [];
halfhour(1:2) = struct('data',data);
hour(1:24) = struct('halfhour',halfhour);
day(1:366) = struct('hour',hour);
PSD_YE_DBT(1:length(files),1:length(files),1:length(files)) =
struct('sta','','loc','','comp','','allData',[],'day',day);
end
for s=1:1:length(sta)
for l=1:1:length(loc)
for c=1:1:length(comp)
tempFileName = strcat(net{s},'.',sta{s},'.',loc{l},'.',comp{c},'.','ALL');
fid = fopen(tempFileName);
PSD_YE_DBT(s,l,c).sta = sta{s};
PSD_YE_DBT(s,l,c).loc = loc{l};
PSD_YE_DBT(s,l,c).comp = comp{c};
end
end
end
Example file names for the three files I'm working with are:
XO.GRUT.--.HHE.ALL
XO.GRUT.--.HHN.ALL
XO.GRUT.--.HHZ.ALL
Thanks in advance!

dir(myFiles{m}) does't work MATLAB

myFiles = 1x7 cell
when I try
for m =1:numel(myFiles )
fil{m} = dir(myFiles {m});
fil{m}.bytes ;
end
This is not working
I got the error :
function is not defined for 'cell' inputs.
First of all you should mention the error message you get.
Now, besides that are some obvious problems:
myFiles {ii}
This is not valid syntax to index into a cell array. Perhaps removing the space helps.
Furthermore you loop over m and then use ii as an index.
Lastly you assign to fil everytime. In practice this means only the last result is stored. Perhaps assigning to fil(m) would suit your needs better.
The command dir will show you the content of a folder. As your variable is named "myFiles" I assume it contains filenames and not foldernames. So I think you're rather looking for a loop like this:
for ii = 1:numel(myFiles)
fil{ii} = which( myFiles{ii} )
end
which gives you an array with the full paths to your files. Or are you looking for the folders containing the files in "myFiles"? Then you can use:
for ii = 1:numel(myFiles)
fil{ii} = fileparts( which( myFiles{ii} ) )
end
returning you the corresponding folders.
regarding your comments:
the existence of the files/folders in "myFiles" is the only purpose?
Then you could do that:
for ii = 1:numel(myFiles)
fil(ii) = exist( which(myFiles{ii}), 'file' );
end
existMyFiles = logical(fil);
returning a logical array specifying the existence of your files.

Import datasheet iteration matlab - name to exist outside of function

So have some simple collumated data in text files named like:
Hm_slit_01.txt...Hm_slit_21.txt; Hs_slit_01.txt...Hs_slit_23.txt; Sm_slit_01.txt...Sm_slit_27.txt; Ss_slit_01.txt...Ss_slit_32.txt etc...
and I need to import it as datasheets into matlab, idealy with the same name but no .txt.
So I have created a function that takes a prefix Hm, Hs ... and a final number to iterate the file naming up to. BUT it doesn't save the datasheet externally in the workplace.
Here is the code.
function [datasheet] = imp_vrad(prefix,numslits)
%[data] = imp_vrad(prefix,numslits)
% imports data with filename <prefix>_slit_<##>.txt
% to be a matlab datasheet data = <prefix>_slit_<##>
% here ## starts from '01' and goes to 'numslits'
% FILES MUST BE IN WORKING DIRECTORY!
for currslit=1:numslits
if currslit < 10
filename = [prefix '_slit_0' num2str(currslit) '.txt']; %names file
data = [prefix '_slit_0' num2str(currslit)]; %names datasheet
else
filename = [prefix '_slit_' num2str(currslit) '.txt']; %names file
data = [prefix '_slit_' num2str(currslit)]; %names datasheet
end
disp(['importing ' filename ' as ' data])
importdata(filename); %imports the data
end
end
The line in question is the last line. I have also tried data=importdata(filename); to no avail.
Thanks in advance!
I'm not sure if there's a way to change the scope of a variable in matlab. You can define them dynamically using eval, but that still won't get us past the scope problem.
You have two options: if the names mater to you, you could put the imported tables in a struct, akin to what import does; or, if you only care about the index, you could import them into an array (perhaps multidimensional).
For the former you could do something like
function [datasheet] = imp_vrad(prefix,numslits)
%[data] = imp_vrad(prefix,numslits)
% imports data with filenames <prefix>_slit_<##>.txt
% into a struct with labels <prefix>_slit_<##>
% here ## starts from '01' and goes to 'numslits'
% FILES MUST BE IN WORKING DIRECTORY!
%
datasheet = struct();
for currslit=1:numslits
dataname = sprintf('%s_slit_%02d',prefix,currslit);
filename = [dataname,'.txt'];
disp(['importing ' filename ' as ' data])
data = importdata(filename); %imports the data
datasheet.(dataname) = data;
end
end

MATLAB Reading several images from folder

I have a folder called BasePics within a folder called Images. Inside BasePics there are 30 JPEG images. I'm wondering if the following is possible: Can a script be written that reads all of these images using the imread() command. The names of the images are somewhat sequential: C1A_Base.jpg, C1B_Base.jpg, C1C_Base.jpg, C2A_Base.jpg, C2B_Base.jpg, C2C_Base.jpg, etc.... all the way up to C10C_Base.jpg
Can a loop be used somehow:
file = dir('Images\BasePics');
NF = length(file);
for k = 1:NF
images(k) = imread(fullfile('ImagesBasePics',file(k))
imagesc(images(k))
end
This is a rough idea of what I want to do, but I'm wondering if it can be done with the current naming format I have in the Images folder. I would also like to have each image being read be its own variable with the same or similar name as it is named in the folder Images\BasePics currently, rather than have an concatenated array of 30 images all under the one variable images. I would like to have 30 separate variables, with names such as A1, A2,A3,B1,B2,B3 etc...
Also when I just ask for:
dir images\BasePics
Matlab outputs 33 files, instead of 30. There are two extra files at the beginning of the folder: '.' and '..' and one at the end: 'Thumbs.db' These do not exist when I look at the folder separately, is there a way to programically have Matlab skip over these?
Thanks!!
Since you know the names of the files in advance, you can skip the dir and go ahead and read the files:
for l = 'ABC'
for n=1:10
nm = sprintf('C%d%c_Base.jpg', n, l );
fnm = sprintf('%c%d', l, n );
imgs.(fnm) = imread( fullfile('images','BasePics', nm ) );
end
end
Now you have a struct imgs with fields A1...C10 for each image.
You are very close. I would just use dir('Images\BasePics\*.jpg') to get rid of the extraneous files.
The naming system you want will not lend itself to additional batch processing (do you really want to type all of A1, A2, etc?). I would either keep it sequential, and store a list of the filenames to match, or use a struct array, like images.C1A, etc.
dirlist = dir('Images\BasePics\*.jpg');
for k = 1:length(dirlist);
fname = dirlist(k).name;
[path,name,ext] = fileparts(fname); % separate out base name of file
images.(name) = imread(fullfile('Images\BasePics', fname));
end