I have more than 10,000 csv file in one folder and file names are 0,1,2,3... like that. I would like to read them and write into one file for further processing.I tried this
files= dir('C:\result\*.csv');
outs = cell(numel(files),1)
for i = 1:numel(files)
out{i} = csvread('%i',2,0)
end
but it didn't work.
Rather than reading them in as csv files, I would just read in the raw files and write them out again. This will likely be much faster.
files = dir('C:\result\*.csv');
filenames = fullfile('C:\result', {files.name});
% Sort the files based on their number
[~, ind] = sort(str2double(regexp(filenames, '[0-9]+(?=\.csv$)', 'match', 'once')));
filenames = filenames(ind);
% Open the file that you want to combine them into
outfile = 'output.csv';
outfid = fopen(outfile, 'wb');
for k = 1:numel(filenames)
% Open each file
fid = fopen(filenames{k}, 'rb');
% Read in contents and remove any trailing newlines
contents = strtrim(fread(fid, '*char'));
% Write out the content and add a newline
fprintf(outfid, '%s\n', contents);
% Close the input file
fclose(fid);
end
fclose(outfid);
Related
I have been working on making a database which contains images and their preset values and other important parameters. But unfortunately, I'm not being able to save the initial data of say 10 images in one .csv file. I have made the code that runs fine with creating .csv file but saving the last value and overwriting all the previous values. I gave also once modified that is comment down in the code using sprintf but it make .csv file for every iteration separately. But i want to make one .csv file containing 7 column with all the respective values.
My code is below and output of my code is attached Output.
Please someone guide me how to make single .csv file with 10 values for instance (could be increased to hundreds in final database) to save in 1 .csv file.
clc
clear all
myFolder = 'C:\Users\USER\Desktop\PixROIDirectory\PixelLabelData_1';
filePattern = fullfile(myFolder, '*.png'); % Change to whatever pattern you need
theFiles = dir(filePattern);
load('gTruthPIXDATA.mat','gTruth')
gTruth.LabelDefinitions;
for i=1:10
%gTruth.LabelData{i,1};
baseFileName = theFiles(i).name;
fullFileName = fullfile(myFolder, baseFileName);
fprintf(1, 'Now reading %s\n', fullFileName);
imageArray = imread(fullFileName);
oUt = regionprops(imageArray,'BoundingBox');
Y = floor(oUt.BoundingBox);
X_axis = Y(1);
Y_axis = Y(2);
Width = Y(3);
Height = Y(4);
CLASS = gTruth.LabelDefinitions{1,1};
JPG = gTruth.DataSource.Source{i,1};
PNG = gTruth.LabelData{i,1};
OUTPUT = [JPG X_axis Y_axis Width Height CLASS PNG]
% myFile = sprintf('value%d.csv',i);
% csvwrite(myFile,OUTPUT);
end
Try fprintf (https://www.mathworks.com/help/matlab/ref/fprintf.html).
You will need to open your output file to be written, then you can append lines to it through each iteration
Simple example:
A = [1:10]; % made up a matrix of numbers
fid = fopen('test.csv','w'); % open a blank csv and set as writable
for i = 1:length(A) % loop through the matrix
fprintf(fid,'%i\n',A(i)); % print each integer, then a line break \n
end
fclose(fid); % close the file for writing
I have two text files: file1.txt & file2.txt
Contents of file1.txt:
Required contents of file;
Required contents of file;
Required contents of file;
Required contents of file;
Required contents of file;
My old contents(1);
My old contents(2);
My old contents(3);
My old contents(4);
Required contents of file;
Required contents of file;
Required contents of file;
Required contents of file;
Contents of file2.txt:
My new var(1);
My new var(2);
My new var(3);
My new var(4);
I have an updateFile.m function: which is trying to replace the old contents from file1.txt with new var respectively
function updateFile(file)
% Read the new contents
fid = fopen('file2.txt', 'r');
c1 = onCleanup(#()fclose(fid));
newVars = textscan(fid,'%s','Delimiter','\n');
newVars = newVars{1};
% Save the testfile in to a cellaray variable
fid = fopen(file, 'r');
c2 = onCleanup(#()fclose(fid));
oldContent = textscan(fid,'%s','Delimiter','\n');
% Search for specific strings
oldContentFound = strfind(oldContent{1},'My old contents(1);');
oldContentRowNo = find(~cellfun('isempty',oldContentFound));
% Move the file position marker to the correct line
fid = fopen(file, 'r+');
c3 = onCleanup(#()fclose(fid));
for k=1:(oldContentRowNo-1)
fgetl(fid);
end
% Call fseek between read and write operations
fseek(fid, 0, 'cof');
for idx = 1:length(newVars)
fprintf(fid, [newVars{idx} '\n']);
end
end
The problem I am facing is that, file1.txt still contains some old contents which are not required. Any help would be appreciated.
Thanks
You can add some characters into the file but you can't remove some. You have to erase your file with a new one that has the correct content.
Here is a rewriting of your function that does the job:
function updateFile(file)
% Read the new contents
fid = fopen('file2.txt', 'r');
newVars = textscan(fid,'%s','Delimiter','\n');
fclose(fid);
newVars = newVars{1};
% Read the old content
fid = fopen(file, 'r');
f1 = textscan(fid, '%s', 'Delimiter', '\n');
fclose(fid);
f1 = f1{1};
% Find pattern start line
[~,k] = ismember('My old contents(1);', f1);
% Replace pattern
for i = 1:numel(newVars)
f1{k+i-1} = newVars{i};
end
% Replace initial file
fid = fopen(file, 'w');
cellfun(#(x) fprintf(fid, '%s\n', x), f1);
fclose(fid);
Best,
File access is inherently byte-based. There's no such thing as inserting or deleting content at the middle of a file. In most cases, the easiest approach is to edit the content in memory, then rewrite the entire file.
In your case, after reading the input file, find the start and end rows in the cell array, and simply replace them with the new data. Then open your file again for normal writing, and output the entire cell array.
I have a MATLAB script that reads a line from a text file. Each line of the text file contains the filename of a CSV. I need to keep track of what line MATLAB is working on so that I can save the data for that line in a cell array. How can I do that?
To illustrate, the first few lines of my .dat file looks like this:
2006-01-003-0010.mat
2006-01-027-0001.mat
2006-01-033-1002.mat
2006-01-051-0001.mat
2006-01-055-0011.mat
2006-01-069-0004.mat
2006-01-073-0023.mat
2006-01-073-1003.mat
2006-01-073-1005.mat
2006-01-073-1009.mat
2006-01-073-1010.mat
2006-01-073-2006.mat
2006-01-073-5002.mat
2006-01-073-5003.mat
I need to save the variable site_data from each of these .mat files into a different cell of O3_data. Therefore, I need to have a counter so that O3_data{1} is the data from the first line of the text file, O3_data{2} is the data from the second line, etc.
This code works, but it's done without using the counter so I only get the data for one of the files I'm reading in:
year = 2006:2014;
for y = 1:9
flist = fopen(['MDA8_' num2str(year(y)) '_mat.dat']); % Open the list of file names - CSV files of states with data under consideration
nt = 0; % Counter will go up one for each file loaded
while ~feof(flist) % While end of file has not been reached
fname = fgetl(flist);
disp(fname); % Stores name as string in fname
fid = fopen(fname);
while ~feof(fid)
currentLine = fgetl(fid);
load (fname, 'site_data'); % Load current file. It is all the data for one site for one year
O3_data = site_data;
% Do other stuff
end
fclose(fid);
end
fclose(flist);
end
If I add the time index part, MATLAB is telling me that Subscript indices must either be real positive integers or logicals. nt is an integer so I don't know what I'm doing wrong. I need the time index so that I can have O3_data{i} in which each i is one of the files I'm reading in.
year = 2006:2014;
for y = 1:9
flist = fopen(['MDA8_O3_' num2str(year(y)) '_mat.dat']); % Open the list of file names - CSV files of states with data under consideration
nt = 0;
while ~feof(flist) % While end of file has not been reached
fname = fgetl(flist);
fid = fopen(fname);
while ~feof(fid)
currentLine = fgetl(fid);
nt = nt+1; % Time index
load (fname, 'site_data'); % Load current file. It is all the data for one site for one year
O3_data{nt} = site_data;
% Do other stuff
end
fclose(fid);
end
fclose(flist);
end
Try the following:
years = 2006:2014;
for y=1:numel(years)
% read list of filenames for this year (as a cell array of strings)
fid = fopen(sprintf('MDA8_O3_%d_mat.dat',years(y)), 'rt');
fnames = textscan(fid, '%s');
fnames = fnames{1};
fclose(fid);
% load data from each MAT-file
O3_data = cell(numel(fnames),1);
for i=1:numel(fnames)
S = load(fnames{i}, 'site_data');
O3_data{i} = S.site_data;
end
% do something with O3_data cell array ...
end
Try the following - note that since there is an outer for loop, the nt variable will have to be initialized outside of that loop so that we don't overwrite data from previous years (or previous j's). We can avoid the inner while loop since the just read file is a *.mat file and we are using the load command to load its single variable into the workspace.
year = 2006:2014;
nt = 0;
data_03 = {}; % EDIT added this line to initialize to empty cell array
% note also the renaming from 03_data to data_03
for y = 1:9
% Open the list of file names - CSV files of states with data under
% consideration
flist = fopen(['MDA8_O3_' num2str(year(y)) '_mat.dat']);
% make sure that the file identifier is valid
if flist>0
% While end of file has not been reached
while ~feof(flist)
% get the name of the *.mat file
fname = fgetl(flist);
% load the data into a temp structure
data = load(fname,'site_data');
% save the data to the cell array
nt = nt + 1;
data_03{nt} = data.site_data;
end
fclose(flist); % EDIT moved this in to the if statement
end
end
Note that the above assumes that each *.dat file contains a list of *.mat files as illustrated in your above example.
Note the EDITs in the above code from the previous posting.
I am trying to have MATLAB read in a list of files. However, I have several lists of these files and they are all different lengths. How can I set a for loop so that the i goes from 1 to the end of the file? (The line I'm talking about is the 'for i = 1:END OF FILE' part.
To illustrate, the first few lines of my .dat file looks like this:
2006-01-003-0010.mat
2006-01-027-0001.mat
2006-01-033-1002.mat
2006-01-051-0001.mat
2006-01-055-0011.mat
My code looks like this:
for y = 1:9
flist = fopen([num2str(year(y))'_MDA8_mat.dat']); % Open the list of file names - CSV files of states with data under consideration
nt = 0; % Counter will go up one for each file loaded
while ~feof(flist) % While end of file has not been reached
for i = 1:END OF FILE % Number of files CHECK EACH TIME FILE IS MODIFIED
fname = fgetl(flist); % Reads next line of list, which is the name of the next data file
disp(fname); % Stores name as string in fname
nt = nt+1; % Time index
load (fname, 'site_data'); % Load current file. It is all the data for one site for one year
O3_data{i} = site_data;
% Stuff
end
end
Try this code :
for y = 1:9
flist = fopen([num2str(year(y))'_MDA8_mat.dat']); % Open the list of file names - CSV files of states with data under consideration
while ~feof(flist) % While end of file has not been reached
newFileName = fgetl(flist);
fid = fopen(newFileName);
while ~feof(fid)
currentLine = fgetl(fid);
% Do stuff on each sub-file
end
fclose(fid);
end
flcose(flist);
end
There another while check you can do like this :
flist = fopen([num2str(year(y))'_MDA8_mat.dat']);
line = fgetl(flist);
while(ischar(line))
%Do process (like open next file with the line variable)
%read the next line
line = fgetl(flist);
end
I am currently having 20 text files naming start from file1 to file20. I am reading them into matlab using
filePattern = fullfile(myFolder, '*.txt');
dataFiles = dir(filePattern);
for k = 1:length(dataFiles)
baseFileName = dataFiles(k).name;
fullFileName = fullfile(myFolder, baseFileName);
fid = fopen(fullFileName, 'r');
line = fgetl( fid );
while ischar( line )
tks = regexp( line, '\[([^,]+),([^\]]+)\]([^\[]+)\[([^\]]+)\]([^\[]+)', 'tokens' );
for ii = 1:numel(tks)
j=j+1;
mat( j ,: ) = str2double( tks{ii} );
end
line = fgetl( fid );
end
fclose( fid );
end
It is working perfectly, but I need to retain the same order the text files appear in the folder. The data from file1 next file2 next file3 till file20 into Matlab.
But it is rearranging into file1 file10 file11 file12 ... file2 file20 and reading. dataFiles is a structure and in that the files are loaded alphabetically. How to prevent that?
I'd recommend using sort_nat (available on Matlab Central) for this task.
Run this in an empty folder:
% create sample files
for i = 1:20
filename = sprintf('file%d.txt',i);
fclose(fopen(filename, 'w'));
end
% obtain folder contents
files = dir('*.txt');
%{files.name} % -> list of files might be in alphabetical order (depends on OS)
% sort_nat sorts strings containing digits in a way such that the numerical value
% of the digits is taken into account
[~,order] = sort_nat({files.name});
files = files(order);
% check output is in numerical order
{files.name}