Related
So I have an iterative loop that extracts data from .csv files in MATLAB's active folder and plots it. I would like to take it one step further and run the script on two folders, each with their own .csv files.
One folder is called stress and the other strain. As the name implies, they contain .csv files for stress and strain data for several samples, each of which is called E3-01, E3-02, E3-03, etc. In other words, both folders have the same number of files and the same names.
The way I see it, the process would have the following steps:
Look in the stress folder, look inside file E3-01, extract the data in the column labelled Stress
Look in the strain folder, look inside file E3-01, extract the data in the column labelled Strain
Combine the data together for sample E3-01 and plot it
Repeat steps 1-3 for all files in the folders
Like I said, I already have a script that can find the right column and extract the data. What I'm not sure about is how to tell MATLAB to alternate the folder that the script is being run on.
Instead of a script, would a function be better? Something that accepts 4 inputs: the names of the two folders and the columns to extract?
EDIT: Apologies, here's the code I have so far:
clearvars;
files = dir('*.csv');
prompt = {'Plot name:','x label:','y label:','x values:','y values:','Points to eliminate:'};
dlg_title = 'Input';
num_lines = 1;
defaultans = {'Title','x label','y label','Surface component 1.avg(epsY) [True strain]','Stress','0'};
answer = inputdlg(prompt,dlg_title,num_lines,defaultans);
name_plot = answer{1};
x_label = answer{2};
y_label = answer{3};
x_col = answer{4};
y_col = answer{5};
des_cols = {y_col,x_col};
smallest_n = 100000;
points_elim = answer{6};
avg_x_values = [];
avg_y_values = [];
for file = files'
M=xlsread(file.name);
[row,col]=size(M);
if smallest_n > row
smallest_n = row;
end
end
smallest_n=smallest_n-points_elim;
avg_x_values = zeros(smallest_n,size(files,1));
avg_y_values = zeros(smallest_n,size(files,1));
hold on;
set(groot, 'DefaultLegendInterpreter', 'none');
set(gca,'FontSize',20);
ii = 0;
for file = files'
ii = ii + 1;
[n,s,r] = xlsread(file.name);
colhdrs = s(1,:);
[row, col] = find(strcmpi(s,x_col));
x_values = n(1:end-points_elim,col);
[row, col] = find(strcmpi(s,y_col));
y_values = n(1:end-points_elim,col);
plot(x_values,y_values,'DisplayName',s{1,1});
avg_x_values(:,ii)=x_values(1:smallest_n);
avg_y_values(:,ii)=y_values(1:smallest_n);
end
ylabel({y_label});
xlabel({x_label});
title({name_plot});
colormap(gray);
hold off;
avg_x_values = mean(avg_x_values,2);
avg_y_values = mean(avg_y_values,2);
plot(avg_x_values,avg_y_values);
set(gca,'FontSize',20);
ylabel({y_label});
xlabel({x_label});
title({name_plot});
EDIT 2: #Adriaan I tried to write the following function to get a column from a file:
function [out_col] = getcolumn(col,file)
file = dir(file);
[n,s,r] = xlsread(file.name);
colhdrs = s(1,:);
[row, col] = find(strcmpi(s,col));
out_col = n(1:end,col);
end
but I get the error
Function 'subsindex' is not defined for values of class 'struct'.
Error in getcolumn (line 21)
y = x(:,n);
not sure why.
You can do both, of course, and it depends on preference mainly, provided you're the sole user of the script. If others are going to use it as well, use functions instead, as they can contain a proper help file and calling help functionname will then give you that help.
For instance:
folders1 = dir(../strain/*)
folders2 = dir(../stress/*)
for ii 1 = 1:numel(folders)
operand1 = folders1{ii};
operand2 = folders2{ii};
%... rest of script
%
% Or function:
data = YourFunction(folders1{ii},folders2{ii})
end
So all in all you can use both, although from experience I find functions easier to use in the end, as you just pass parameters and don't need to trawl through the complete code to change the parameters each run.
Additionally you can partition off small parts of your program which do a fix task. If you nest your functions, and finally call just a single function in your scripts, you don't have to look at hundreds of lines of code each time you run the script, but rather can just run a single function (which can also be inside a script or function, ad infinitum).
Finally, a function has its own scope; meaning that any variables that are in that function stay within that function unless you explicitly set them as output (apart from global variables, but those are problematic anyway). This can be a good thing, or a bad thing, depending on the rest of your code. If you function would output ~20 variables for further processing, the function probably should include more steps. It'd be a good thing if you create lots of intermediate variables (I always do), because when the function's finished running, the scope of that function will be removed from memory, saving you clear tmpVar1 tmpVar2 tmpVar3 etc every few lines in your script.
For the script the argument in favour would be that it is easier to debug; you don't need dbstop on error and can step a bit easier through the script, keeping check of all your variables. But, after the debugging has been completed, this argument becomes moot, and thus in general I'd start with writing a script, and once it performs as desired, I rework it to a function at minimal extra effort.
I have a cell array containing file names. I want to check for the existence of all of these files in the subject folder, and if any one does not exist I wish to send a continue to the top-most for-loop (see mock code). Is there a way to do this in a one or two liner, instead of 1) using a for-loop and a double if-statement, or 2) building a function that for-loops over exist().
subjects = {'/data/subject01','/data/subject02','/data/subject03'};
files = {'a.txt','b.txt','c.txt'};
for ii = 1:numel(subjects)
for jj = 1:numel(files)
fileExists = exist([subject{ii} '/' file{jj}],'file')
if ~fileExists
continue
end
end
if ~fileExists
continue
end
% Some code to execute if all files exist.
end
The *fun functions are just loops internally and are generally slower than the explicit loop. They also very often unnecessarily obfuscate the intent and behavior of the code.
You can use ismember with all and dir to make the approach clearer and remove the unnecessary loop:
subjects = {'./data/subject01','./data/subject02'};
files = {'a.txt','b.txt','c.txt'};
for ii = 1:numel(subjects)
filelist = dir(fullfile(subjects{ii}, '*.txt'));
foundfilenames = {filelist(:).name};
if all(ismember(files, foundfilenames))
fprintf('All %u files are here: %s\n', numel(files), subjects{ii})
else
fprintf('All %u files are not here: %s\n', numel(files), subjects{ii})
end
end
With my folder structure:
/data
/subject01
a.txt
b.txt
/subject02
a.txt
b.txt
c.txt
I see the following, as expected:
All 3 files are not here: ./data/subject01
All 3 files are here: ./data/subject02
You could remove the loop by iterating over all combinations of the two arrays:
subjects = {'/data/subject01','/data/subject02','/data/subject03'};
files = {'a.txt','b.txt','c.txt'};
a=numel(subjects);
b=numel(files);
k=a*b;
paths = arrayfun(#(ii)[subjects{mod(ii-1,a)+1} '/' files{ceil(ii/b)}],1:k,'uniformoutput',0);
checkExist = cellfun(#exist, paths, repmat({'file'},1,k));
if all(checkExist)
% Some code to execute if all files exist
end
Resolved it with a cellfun and by string arrays. Technically there is still a for-loop but it resolves the double if-statement. I will leave this question open for better solutions.
subjects = {'/data/subject01','/data/subject02','/data/subject03'};
files = string({'a.txt','b.txt','c.txt'});
for ii = 1:numel(subjects)
paths = subject{ii} + files;
checkExist = cellfun(#exist, cellstr(paths), repmat({'file'},size(paths))
if ~all(checkExist(:))
continue
end
% Some code to execute if all files exist.
end
the function dir returns an array like
.
..
Folder1
Folder2
and every time I have to get rid of the first 2 items, with methods like :
for i=1:numel(folders)
foldername = folders(i).name;
if foldername(1) == '.' % do nothing
continue;
end
do_something(foldername)
end
and with nested loops it can result in a lot of repeated code.
So can I avoid these "folders" by an easier way?
Thanks for any help!
Edit:
Lately I have been dealing with this issue more simply, like this :
for i=3:numel(folders)
do_something(folders(i).name)
end
simply disregarding the first two items.
BUT, pay attention to #Jubobs' answer. Be careful for folder names that start with a nasty character that have a smaller ASCII value than .. Then the second method will fail. Also, if it starts with a ., then the first method will fail :)
So either make sure you have nice folder names and use one of my simple solutions, or use #Jubobs' solution to make sure.
A loop-less solution:
d=dir;
d=d(~ismember({d.name},{'.','..'}));
TL; DR
Scroll to the bottom of my answer for a function that lists directory contents except . and ...
Detailed answer
The . and .. entries correspond to the current folder and the parent folder, respectively. In *nix shells, you can use commands like ls -lA to list everything but . and ... Sadly, MATLAB's dir doesn't offer this functionality.
However, all is not lost. The elements of the output struct array returned by the dir function are actually ordered in lexicographical order based on the name field. This means that, if your current MATLAB folder contains files/folders that start by any character of ASCII code point smaller than that of the full stop (46, in decimal), then . and .. willl not correspond to the first two elements of that struct array.
Here is an illustrative example: if your current MATLAB folder has the following structure (!hello and 'world being either files or folders),
.
├── !hello
└── 'world
then you get this
>> f = dir;
>> for k = 1 : length(f), disp(f(k).name), end
!hello
'world
.
..
Why are . and .. not the first two entries, here? Because both the exclamation point and the single quote have smaller code points (33 and 39, in decimal, resp.) than that of the full stop (46, in decimal).
I refer you to this ASCII table for an exhaustive list of the visible characters that have an ASCII code point smaller than that of the full stop; note that not all of them are necessarily legal filename characters, though.
A custom dir function that does not list . and ..
Right after invoking dir, you can always get rid of the two offending entries from the struct array before manipulating it. Moreover, for convenience, if you want to save yourself some mental overhead, you can always write a custom dir function that does what you want:
function listing = dir2(varargin)
if nargin == 0
name = '.';
elseif nargin == 1
name = varargin{1};
else
error('Too many input arguments.')
end
listing = dir(name);
inds = [];
n = 0;
k = 1;
while n < 2 && k <= length(listing)
if any(strcmp(listing(k).name, {'.', '..'}))
inds(end + 1) = k;
n = n + 1;
end
k = k + 1;
end
listing(inds) = [];
Test
Assuming the same directory structure as before, you get the following:
>> f = dir2;
>> for k = 1 : length(f), disp(f(k).name), end
!hello
'world
a similar solution from the one suggested by Tal is:
listing = dir(directoryname);
listing(1:2)=[]; % here you erase these . and .. items from listing
It has the advantage to use a very common trick in Matlab, but assumes that you know that the first two items of listing are . and .. (which you do in this case). Whereas the solution provided by Tal (which I did not try though) seems to find the . and .. items even if they are not placed at the first two positions within listing.
Hope that helps ;)
If you're just using dir to get a list of files and and directories, you can use Matlab's ls function instead. On UNIX systems, this just returns the output of the shell's ls command, which may be faster than calling dir. The . and .. directories won't be displayed (unless your shell is set up to do so). Also, note that the behavior of this function is different between UNIX and Windows systems.
If you still want to use dir, and you test each file name explicitly, as in your example, it's a good idea to use strcmp (or one of its relations) instead of == to compare strings. The following would skip all hidden files and folder on UNIX systems:
listing = dir;
for i = 1:length(listing)
if ~strcmp(listing(i).name(1),'.')
% Do something
...
end
end
You may also wanna exclude any other files besides removing dots
d = dir('/path/to/parent/folder')
d(1:2)=[]; % removing dots
d = d([d.isdir]) % [d.isdir] returns a logical array of 1s representing folders and 0s for other entries
I used: a = dir(folderPath);
Then used two short code that return struct:
my_isdir = a([a.isdir]) Get a struct which only has folder info
my_notdir = a(~[a.isdir]) Get a struct which only has non-folder info
Combining #jubobs and #Tal solutions:
function d = dir2(folderPath)
% DIR2 lists the files in folderPath ignoring the '.' and '..' paths.
if nargin<1; folderPath = '.'; elseif nargin == 1
d = dir(folderPath);
d = d(~ismember({d.name},{'.','..'}));
end
None of the above puts together the elements as I see the question having being asked - obtain a list only of directories, while excluding the parents.
Just combining the elements, I would go with:
function d = dirsonly(folderPath)
% dirsonly lists the unhidden directories in folderPath ignoring '.' and '..'
% creating a simple cell array without the rest of the dir struct information
if nargin<1; folderPath = '.'; elseif nargin == 1
d = dir(folderPath);
d = {d([d.isdir] & [~ismember({d.name},{'.','..'})]).name}.';
end
if hidden folders in general aren't wanted the ismember line could be replaced with:
d = {d([d.isdir] & [~strncmp({d.name},'.',1)]).name}.';
if there were very very large numbers of interfering non-directory files it might be more efficient to separate the steps:
d = d([d.isdir]);
d = {d([~strncmp({d.name},'.',1)]).name}.';
We can use the function startsWith
folders = dir("folderPath");
folders = string({folders.name});
folders = folders(~startsWith(folders,"."))
Potential Solution - just remove the fields
Files = dir;
FilesNew = Files(3:end);
You can just remove them as they are the first two "files" in the structure
Or if you are actually looking for specific file types:
Files = dir('*.mat');
I have a folder called BasePics within a folder called Images. Inside BasePics there are 30 JPEG images. I'm wondering if the following is possible: Can a script be written that reads all of these images using the imread() command. The names of the images are somewhat sequential: C1A_Base.jpg, C1B_Base.jpg, C1C_Base.jpg, C2A_Base.jpg, C2B_Base.jpg, C2C_Base.jpg, etc.... all the way up to C10C_Base.jpg
Can a loop be used somehow:
file = dir('Images\BasePics');
NF = length(file);
for k = 1:NF
images(k) = imread(fullfile('ImagesBasePics',file(k))
imagesc(images(k))
end
This is a rough idea of what I want to do, but I'm wondering if it can be done with the current naming format I have in the Images folder. I would also like to have each image being read be its own variable with the same or similar name as it is named in the folder Images\BasePics currently, rather than have an concatenated array of 30 images all under the one variable images. I would like to have 30 separate variables, with names such as A1, A2,A3,B1,B2,B3 etc...
Also when I just ask for:
dir images\BasePics
Matlab outputs 33 files, instead of 30. There are two extra files at the beginning of the folder: '.' and '..' and one at the end: 'Thumbs.db' These do not exist when I look at the folder separately, is there a way to programically have Matlab skip over these?
Thanks!!
Since you know the names of the files in advance, you can skip the dir and go ahead and read the files:
for l = 'ABC'
for n=1:10
nm = sprintf('C%d%c_Base.jpg', n, l );
fnm = sprintf('%c%d', l, n );
imgs.(fnm) = imread( fullfile('images','BasePics', nm ) );
end
end
Now you have a struct imgs with fields A1...C10 for each image.
You are very close. I would just use dir('Images\BasePics\*.jpg') to get rid of the extraneous files.
The naming system you want will not lend itself to additional batch processing (do you really want to type all of A1, A2, etc?). I would either keep it sequential, and store a list of the filenames to match, or use a struct array, like images.C1A, etc.
dirlist = dir('Images\BasePics\*.jpg');
for k = 1:length(dirlist);
fname = dirlist(k).name;
[path,name,ext] = fileparts(fname); % separate out base name of file
images.(name) = imread(fullfile('Images\BasePics', fname));
end
how to write a program in matlab that reads a certain number of images let's say 20 for example which are saved in a given directory (C:) such that later i can use them. suppose that the images are saved by numbers. later, i am gonna use them.
I'd have the code look something like this. Assuming cell array im holds your images.
Write out:
IMG_DIR = 'C:\';
filename_root = 'image';
IMG_EXT = '.jpg';
NUM_IMAGES = 20;
for i = 1:NUM_IMAGES
imwrite(im{i}, [IMG_DIR filename_root num2str(i) IMG_EXT]);
end
Read in:
for i = 1:NUM_IMAGES
im{i} = imread([IMG_DIR filename_root num2str(i) IMG_EXT]);
end
If you don't know how many there are, you can also use ls command (works differently in Windows vs. Linux).
If you don't know, in advance, which files will be in there, but you know that they have the string in them, 'rawImage' (like 'rawImage001.jpg' etc.) you can do something like
a = dir('c:\temp');
requiredBaseFileName = 'rawImage'; % you want them to contain the substring 'rawImage'
for i = 1:length(a),
fileName = a(i).name;
if(isempty(strfind(fileName,'.jpg')) & isempty(strfind(fileName,'.png')))
continue;
end
if(isempty(strfind(fileName,requiredBaseFileName)))
continue;
end
% do your processing here
end