Loading .csv files based on file name in Matlab - matlab

I have a large number of csv files which are named sequentially based on date in the same folder.
example name (d_log_yyyymmddhhmmss). All files have the same number of columns and headers and ultimately I am concatanating.
I am loading them using the following which seems to be fast and efficient:. However, this works for limited files and as I have large amounts I want to split them in cells based on the date they include in their name. For example within a month it would create 30 cell arrays with individual files and 30 concatanated files. Any help appreciated.
d = uigetdir();
filePattern = fullfile (d, '*.csv');
file = dir (filePattern);
Data =cell (1, numel(file));
for k=1 : numel(file)
filename = file(k).name;
fullfilename = fullfile (d, filename);
Tables {k} = readtable (fullfilename);
fprintf ('read file %s\n', fullfilename);
end
bigdata = vertcat (Data{:});
EDIT
d = uigetdir();
filePattern = fullfile (d, '*.csv');
files = dir (filePattern);
unique_months_list = {};
for i = 1:numel(files)
file_month = files(i).name(07:12); % yyyymm part
basefilename = files(i).name;
fullfilename = fullfile (d, basefilename);
if ismember(file_month, unique_months_list);
TablesSep {i} = readtable (fullfilename);
fprintf ('read file %s\n', fullfilename);
else unique_months_list {end+1} =file_month;
fprintf ('read file %s\n', fullfilename);
TablesOct {i} = readtable (fullfilename);
% elseif
% unique_months_list {end+2} =file_month;
% disp('new'); % do your thing if another month
%TablesNov {i} = readtable (fullfilename);
end
end

Edited according to your comment:
files = dir('d_log_*');
unique_months_list = {};
for i = 1:numel(files)
file_month = files(i).name(7:12); % yyyymm part
if ismember(file_month, unique_months_list)
disp('same'); % do your thing if same month
else
unique_months_list {end+1} = file_month;
disp('new'); % do your thing if another month
end
end

Related

How to delete a string in multiple files by using the function eraseBetween (Matlab R2019a)

I am trying without success to delete a specific string (everything between xmln= and the next space) in multiple text files (xml files) by using the Matlab function eraseBetween. The files would then be moved to a new folder. The code so far is below. Any help welcome.
modified_xml = fullfile(pwd, 'modified_xml')
if exist([pwd '\modified_xml'])~=7
mkdir(modified_xml);
end
InputOldFiles = dir(fullfile('*.xml'));
OldFilesNames = {InputOldFiles.name};
for k = 1 : length(InputOldFiles)
OldFiles = OldFilesNames{k};
FID = fopen(OldFiles, 'r');
if FID == -1, error('Cannot open file'), end
% Delete specific string (everything between xmlns= and the next empty space)
text = textscan(FID, '%s', 'delimiter', '\n', 'whitespace', '');
new_text = eraseBetween(text,"xmlns="," ");
NewFiles = fopen(OldFiles, 'w');
if NewFiles == -1, error('Cannot open file'), end
% Save the file
fprintf(NewFiles, '%s\n', new_text{:});
fclose(NewFiles);
% Copy the files
copyfile(OldFiles, NewFiles);
end
Here's how I did it:
% Little function to edit multiple XML files
% New files are copied into the Modified_XML_Files folder
% and original files are not overwritten
% Creation of a new folder
Modified_XML_Files = fullfile(pwd, 'Modified_XML_Files')
if exist([pwd '\Modified_XML_Files'])~=7
mkdir(Modified_XML_Files);
end
% Copy XML files and move to new folder
copyfile('*.xml', 'Modified_XML_Files')
cd Modified_XML_Files
% List of XML files to modify
XML_Files = dir(fullfile('*.xml'));
XML_Files_Names = {XML_Files.name};
% Loop and delete text between xmlns= and the next space inclusively
for k = 1 : length(XML_Files)
Old_File = XML_Files_Names{k};
file_text = textread(Old_File, '%s', 'delimiter', '\n', 'whitespace', '');
file_text = eraseBetween(file_text, "xmlns=", " ", 'Boundaries', 'inclusive');
New_File = fopen(Old_File, 'w');
for i=1:length(file_text)
fprintf(New_File, '%s\n', file_text{i});
end
New_File = fclose(New_File);
end

Read images from sub folders and save into another folder

I have a code here.
start_path = fullfile(matlabroot, '\toolbox\images\imdemos');
% Ask user to confirm or change.
topLevelFolder = uigetdir(start_path);
if topLevelFolder == 0
return;
end
% Get list of all subfolders.
allSubFolders = genpath(topLevelFolder);
% Parse into a cell array.
remain = allSubFolders;
listOfFolderNames = {};
while true
[singleSubFolder, remain] = strtok(remain, ';');
if isempty(singleSubFolder)
break;
end
listOfFolderNames = [listOfFolderNames singleSubFolder];
end
numberOfFolders = length(listOfFolderNames)
% Process all image files in those folders.
for k = 1 : numberOfFolders
% Get this folder and print it out.
thisFolder = listOfFolderNames{k};
fprintf('Processing folder %s\n', thisFolder);
% Get PNG files.
filePattern = sprintf('%s/*.png', thisFolder);
baseFileNames = dir(filePattern);
% Add on TIF files.
filePattern = sprintf('%s/*.tif', thisFolder);
baseFileNames = [baseFileNames; dir(filePattern)];
% Add on JPG files.
filePattern = sprintf('%s/*.jpg', thisFolder);
baseFileNames = [baseFileNames; dir(filePattern)];
numberOfImageFiles = length(baseFileNames);
% Now we have a list of all files in this folder.
if numberOfImageFiles >= 1
% Go through all those image files.
for f = 1 : numberOfImageFiles
fullFileName = fullfile(thisFolder, baseFileNames(f).name);
pathname = strcat('C:\\xampp\\htdocs\\PACS_Client\\cbir_matlab\\ano\\');
outputBaseFileName = sprintf('%3.3d.jpg',f);
outputFullFileName = fullfile(pathname, outputBaseFileName);
fprintf('Processing image file %s\n', fullFileName);
im=imread(fullFileName);
imshow(im);
data = im;
imwrite(data,[pathname,outputBaseFileName]);
end
else
fprintf(' Folder %s has no image files in it.\n', thisFolder);
end
end
i tried to save all the images from sub folders into another folder.But could not get all the images.Only few numbers of images are saved.I want to save all the images.can anyone help me with this code?
i updated your code a bit please check if this works for you. One issue being your base file name always is '%3.3d.jpg' so every picture will be a '.jpg' even if its not. Also you are loading and showing images, but you only need to copy them, so you can go for copyfile. 3rd you always setting every image 001.jpg which will overwrite the the last 001.jpg from the previous folder. you have to add the number so the next folder starts with higher numbers.
start_path = fullfile(matlabroot, '\toolbox\images\imdemos');
% Ask user to confirm or change.
topLevelFolder = uigetdir(start_path);
if topLevelFolder == 0
return;
end
%dir where everything should go. if the destination is not the
%topLevelFolder
%destinationpath = strcat('D:\\pics\\');
destinationpath = topLevelFolder;
% Get list of all subfolders.
allSubFolders = genpath(topLevelFolder);
% Parse into a cell array.
remain = allSubFolders;
listOfFolderNames = {};
%while true
% [singleSubFolder, remain] = strtok(remain, ';');
% if isempty(singleSubFolder)
% break;
% end
% listOfFolderNames = [listOfFolderNames singleSubFolder];
%end
%your while worked fine, but try to avoid 'while true' with break
for i=1:sum(strfind(allSubFolders,';'))
[singleSubFolder, remain] = strtok(remain, ';');
listOfFolderNames = [listOfFolderNames singleSubFolder];
end
numberOfFolders = length(listOfFolderNames)
%set inital count
picturecount=0;
% Process all image files in those folders.
for k = 1 : numberOfFolders
% Get this folder and print it out.
thisFolder = listOfFolderNames{k};
fprintf('Processing folder %s\n', thisFolder);
% Get PNG files.
filePattern = sprintf('%s/*.png', thisFolder);
baseFileNames = dir(filePattern);
% Add on TIF files.
filePattern = sprintf('%s/*.tif', thisFolder);
baseFileNames = [baseFileNames; dir(filePattern)];
% Add on JPG files.
filePattern = sprintf('%s/*.jpg', thisFolder);
baseFileNames = [baseFileNames; dir(filePattern)];
numberOfImageFiles = length(baseFileNames);
% Now we have a list of all files in this folder.
if numberOfImageFiles >= 1
% Go through all those image files.
for f = 1 : numberOfImageFiles
fullFileName = fullfile(thisFolder, baseFileNames(f).name);
[~,~,ext] = fileparts(baseFileNames(f).name); %get extension
outputBaseFileName = sprintf(['%3.3d' ext],f+picturecount);%create name based on picturecount
outputFullFileName = fullfile(destinationpath, outputBaseFileName);
%fprintf('Processing image file %s\n', fullFileName);
%im=imread(fullFileName);
%imshow(im);
%data = im;
%imwrite(data,[pathname,outputBaseFileName]);
%you dont need it in matlab just copy the file
copyfile(fullFileName,outputFullFileName);
end
picturecount=picturecount+numberOfImageFiles;%set picturecount for next k
else
fprintf(' Folder %s has no image files in it.\n', thisFolder);
end
end
The problem is in baseFileNames = dir(filePattern), where you reset the list each time the loop is on a new folder. This is why at the end you will only have the images of the last folder. Simply add baseFileNames = [] just before the for loop and then replace baseFileNames = dir(filePattern) with baseFileNames = [baseFileNames; dir(filePattern)].

Accessing multiple folder matlab and store in same mat

I am newbie to Matlab. I have code like below. How to read multiple folder. I know by using loop but somehow I google it I cannot find it. Also,what i have wrote below is for myA folder. How to put featureVector in same mat AllTrain for myA, myB, myC? The folder contain of image.
clear all;
clc;
trainlabel = [];
featureVector = [];
AllTrain = [];
% Specify the folder
myA = 'C:\Users\NotComplex\a';
myB = 'C:\Users\NotComplex\b';
myC = 'C:\Users\NotComplex\c';
for k = 1 : length(theFiles)
baseFileName = theFiles(k).name;
fullFileName = fullfile(myA, baseFileName);
fprintf(1, 'Now reading %s\n', fullFileName);
% Now imread the file
imageArray = imread(fullFileName);
imageEdge = edge(imageArray, 'canny', 0.4);
a_inv_mom = Hu_Moments(imageEdge);
format short
a_inv_mom_normal = -sign(a_inv_mom).*(log10(abs(a_inv_mom)));
featureVector = cat(1,a_inv_mom_normal);
AllTrain(k,:) = [featureVector k];
% imshow(imageEdge); % Display image.
drawnow % Force display to update immediately.
end
To be compact:
folderCell = {'C:\Users\NotComplex\a', 'C:\Users\NotComplex\b', 'C:\Users\NotComplex\c'};
theFiles = cellfun(#(x) fullfile(x, '*.pgm'), folderCell, 'UniformOutput', false);
Then you can continue with your loop, theFiles will contain all of the .pgm files from all of the three directories

saving outputs from a loop in Matlab in different folders

I am using the codes below which save the output images in the same directory as is the m-file. I am trying to automate to save the output images in different folders with the folder name same as the 'file' name.
clear all;
files = dir('*.dat');
for k = 1:length(files);
filename = files(k).name;
data1 = fopen(filename,'r');
data2 = textscan(data1, '%f%f','headerlines',1,'CollectOutput',1);
data = data2{:,1};
x = data(:,1);
y = data(:,2);
plot(x, y);
[pathstr, name, ext] = fileparts(filename);
temp = ['fig', num2str(k), '.eps'];
print(gcf, '-depsc', temp);
fclose(data1);
end
Any help will be very much appreciated.
You have to create a subfolder (named after your filename), then print in this folder.
folderName = filename;
mkdir(folderName);
print( gcf , [folderName '/' filename] ); %or use `filesep` function to replace /

Creat a loop to generate a a mean value of data which I got form TXT files?

I trying to do script to evalute my lab data into matlab, I have lots of txt files for many samples, each sample has 30 txt files. I did a function to get the data from these files and store them into a structure that contains data and labels.
I would like to know if it is possible to load all the 30 files using loop instead of line by line .
function s = try1(fn)
% adapt for other file format according to your needs
fid = fopen(fn);
s.data = [];
% skip first lines
for k=1:6
l = fgetl(fid);
% read header
a = regexp(l,',','split');
s.labels = a;
l = fgetl(fid);
k=0;
end
while( (l~=-1)&(k<130) )
l = l(1:end-1);
a = regexp(l,', ','split');
a = regexpre
p(a, ',','.');
s.data = [s.data; str2double(a(2:4))];
l = fgetl(fid);
k = k+1;
end
fclose(fid);
end
To do what you want with a data directory and processing each file in a loop you can use a combination of fullfile(...) and a loop. This will likely work:
dataDirectory = 'c:/myDirectory';
allFilesDir = dir(fullfile(dataDirectory , '*.txt'));
allFN = {allFilesDir.name}
% Or do this:
% allFN = {'file1.txt', 'file2.txt'};
result = [];
for ind = 1:length(allFN)
myFN = fullfile(dataDirectory, allFN{ind});
result(ind) = try1(myFN);
end
% Now get the mean:
myMean = mean([result.data])