I have a set of days of log files that I need to parse and look at in matlab.
The log files look like this:
LOG_20120509_120002_002.csv
(year)(month)(day)_(hour)(minute)(second)_(log part number)
The logs increment hourly, but sometimes the seconds are one or two seconds off (per hour) which means i need to ignore what they say to do loadcsv.
I also have another file:
LOG_DATA_20120509_120002.csv
which contains data for the whole hour (different data).
The overall objective is to:
loop through each day
loop through each hour
read in LOG_DATA for whole hour
loop through each segment
read in LOG for each segment
compile a table of all the data
I guess the question is then, how do i ignore the minutes of the day if they are different? I suspect it will be by looping through all the files in the folder, in which case how do i do that?
Looping through all the files in the folder is relatively easy:
files = dir('*.csv');
for file = files'
csv = load(file.name);
% Do some stuff
end
At first, you must specify your path, the path that your *.csv files are in there
path = 'f:\project\dataset'
You can change it based on your system.
then,
use dir function :
files = dir (strcat(path,'\*.csv'))
L = length (files);
for i=1:L
image{i}=csvread(strcat(path,'\',file(i).name));
% process the image in here
end
pwd also can be used.
Related
In the below code I am trying to sort files based on a string within the name. I've been piecing this together with google searches and community help (I'm very new at matlab). Right now I'm getting two odd errors. First, when I try and make a folder, it creates some file (highlighted filein picture that I can't open and the wav files that should have been moved to the folder disappear.
I'm also having an issue where the code renames the first two data files moved to "01" and "01 (1)" and I have no idea why.
DirIn = 'C:\Folder\Experiment' %set incoming directory
eval(['filelist=dir(''' DirIn '/*.wav'')']) %get file list
for i = 1:length(filelist);
Filename = filelist(i).name
name = strsplit(Filename, '_');
newStr = extractBetween(name,7,8);
if strcmp(newStr,'01')
DirOut = fullfile(DirIn, '01');
mkdir DirIn DirOut
movefile(fullfile(filelist(i).folder, filelist(i).name), DirOut);
end
end
This should work:
DirIn = 'C:\Folder\Experiment'; %set incoming directory
filelist=dir(fullfile(DirIn, '*.wav')); %get file list
DirOut = fullfile(DirIn, '01');
for i = 1:length(filelist);
Filename = filelist(i).name
newStr = Filename(7:8);
if strcmp(newStr,'01')
if ~exist(DirOut)
mkdir(DirOut)
end
movefile(fullfile(filelist(i).folder, filelist(i).name), DirOut);
end
end
Firstly, you don't need eval to get the file list. eval impact performance significantly. The below is what you should have done:
filelist=dir(fullfile(DirIn, '*.wav'));
You don't need strsplit or extractBetween since you only intend to extract a part of the string by indexing, i.e. the 7th and 8th characters, you may do this:
newStr = Filename(7:8);
To use variable as an input, you need to use mkdir as a function rather than console command:
mkdir(DirOut)
Lastly, a bit of optimisation. Since DirOut is constant, you can take it outside the loop. You may also want to check if DirOut has already been created to avoid the warning message and overhead in mkdir.
There is no issue with movefile.
Couple things go wrong, first, it is not recommended to use eval. In this case you can just create a character array to pass to dir as follows:
filelist = dir([DirIn '/*.wav'])
Then, you have a strplit that appears to do nothing, since it looks like your files don't have '_' in them, so name will just return Filename. But that is not the issue, since you are using extractBetween on the Filename.
The following does not what you think it does,
mkdir DirIn DirOut
will create two directories named DirIn and DirOut in the current working directory of Matlab. To create the directories you want, use:
mkdir(DirOut)
Since the output directory did not exist before, I suspect Matlab moved the file to the input directory, and renamed it to 01, if you manually add the extension .wav it should be one of the original files.
I have a directory with two folders: one folder contains several subfolders with multiple .txt files (folder 1). The second folder contains several .hjson files (folder 2).
I would like load each .txt and .hjson files make a several calculation (e.g. velocity, acceleration, curvature) and save in the same .txt file adding news columns with headers (e.g. velocity, acceleration, curvature). So far, I have one code to load .txt files.
My goal is to make a code that reads, loads, computes and automatically save it. Please let me know if you have any suggestion.
%% Read and load
dir_to_search = 'C:\Programs\pedro\Test\';
txtpattern = fullfile(dir_to_search, '*.txt');
dinfo = dir(txtpattern);
for K = 1 : length(dinfo)
thisfilename = fullfile(dir_to_search, dinfo(K).name); %just the name
thisdata = load(thisfilename); %load just this file
End
You can use fprintf to write to a file.
For example:
formatSpec="%9.5f %8.5f\n";
for i=1:n
fprintf(fid,formatSpec, var1(i),var2(i));
end
If I have a group of .wav files and Im trying to pick only month wise or do daily/only night psd(power spectral density) averages etc or choose files belonging to a month how to go about? The following are first 10 .wav files in a .txt file that are read into matlab code-
AMAR168.1.20150823T200235Z.wav
AMAR168.1.20150823T201040Z.wav
AMAR168.1.20150823T201845Z.wav
AMAR168.1.20150823T202650Z.wav
AMAR168.1.20150823T203455Z.wav
AMAR168.1.20150823T204300Z.wav
AMAR168.1.20150823T205105Z.wav
AMAR168.1.20150823T205910Z.wav
AMAR168.1.20150823T210715Z.wav
yyyymmddTHHMMSSZ.wav is part of the format to get sense of some parameters.
Many thanks
You need to be more specific.
Do all files always start with "AMAR168.1." for instance?
Anyway, here's a general approach to get you started:
AllFilenames = fileread ('filenames.dat');
FileNames = strsplit (AllFilenames, '\n');
for i = FileNames
if ~isempty (strfind (i{:}, '20150823')); disp(i{:}); end
end
Your filename examples aren't very useful because they all have the same date, but, anyway, you get the point.
Alternatively, if the filenames always have the same format and size, you could do, e.g.:
AllFilenames = fileread ('filenames.dat');
AllFilenames = strvcat (strsplit (AllFilenames, '\n'));
LogicalIndices = categorical (cellstr (AllFilenames(:,15:16))) == '08';
to obtain all rows where the month is '08' for instance. This assumes that the month is always at position 15 to 16 in the string
I apologize in advance that this question is not specific. But my goal is to take a bunch of image files, which are currently named as: 0.tif, 1.tif, 2.tif, etc... and rename them just as numbers to 000.tif, 001.tif, 002.tif, ... , 010.tif, etc...
The reason I want to do this is because I am trying to load the images into matlab and for batch processing but matlab does not order them correctly. I use the dir command as dir(*.tif) to get all the images and load them into an array of files that I can iterate over and process, but in this array element 1 is 0.tif, element 2 is 1.tif, element 3 is 10.tif, element 4 is 100.tif, and so on.
I want to keep the ordering of the elements as I process them. However, I do not care if I have to change the order of the elements BEFORE processing them (i.e. I can make it work to rename, for example, 2.tif to 10.tif if I had to) but I am looking for a way to convert the file names the way I initially described.
If there is a better way to get matlab to properly order the files when it loads them into the array using dir please let me know because that would be much easier.
Thanks!!
You can do this without having to rename the files, if you want. When you grab the files using dir, you'll have a list of files like so:
files =
'0.tif'
'1.tif'
'10.tif'
...
You can grab just the numeric part using regexp:
nums = regexp(files,'\d+','match');
nums = str2double([nums{:}]);
nums =
0 1 10 11 12 ...
regexp returns its matches as a cell-array, the second line converts it back to actual numbers.
We can now get an actual numeric order by sorting the resulting array:
[~,order] = sort(nums);
and then put the files in the correct order:
files = files(order);
This should (I haven't tested it, I don't have a folder full of numerically labelled files handy) produce a list of files like so:
files=
'0.tif'
'1.tif'
'2.tif'
'3.tif'
...
this is partially dependent on the version of matlab you have. If you have a version with findstr this should work well
num_files_to_rename = numel(name_array);
for ii=1:num_files_to_rename
%in my test i used cells to store my strings you may need to
%change the bracket type for your application
curr_file = name_array{ii};
%locates the period in the file name (assume there is only one)
period_idx = findstr(curr_file ,'.');
%takes everything to the left of the period (excluding the period)
file_name = str2num(curr_file(1:period_idx-1));
%zeropads the file name to 3 spaces using a 0
new_file_name = sprintf('%03d.tiff',file_name)
%you can uncomment this after you are sure it works as you planned
%movefile(curr_file, new_file_name);
end
the actual rename operation movefile is commented out for now. make sure the output names are as you expect before uncommenting it and renaming all the files.
EDIT there is no real error checking in this code, it just assumes every file name has one and only one period, and an actual number as the name
The Batch file below do the rename of the files you want:
#echo off
setlocal EnableDelayedExpansion
for /F "delims=" %%f in ('dir /B *.tif') do (
set "name=00%%~Nf"
ren "%%f" "!name:~-3!.tif"
)
Note that this solution preserve the same order of your original files, even if there are missing numbers in the sequence..
I have several data log files (here: 34) for those I have to calculate some certain values. I wrote a seperate function to publish the results of the calculation in a pdf file. But I only can publish one file after another, so it takes a while to publish all 34 files.
Now I want to automize that with a loop - importing the data, calculate the values and publish the results for every log file in a new pdf file. I want 34 pdf files for every log file at the end.
My problem is, that I couldn't find a way to rename the pdf files during publishing. The pdf file is always named after the script which is calculating the values. Obviously the pdf is overwritten within a loop. So at the end everything is calculated, but I only have the pdf from the last calculated log file.
There was this hacky solution to change the Matlab publish script, but since I don't have admin rights I can't use that:
"This is really hacky, but I would modify publish to accept a new option prefix. Replace line 93
[scriptDir,prefix] = fileparts(fullPathToScript);
with
if ~isfield(options, 'prefix')
[scriptDir,prefix] = fileparts(fullPathToScript);
else
[scriptDir,~] = fileparts(fullPathToScript);
prefix = options.prefix; end
Now you can set options.prefix to whatever filename you want. If you want to be really hardcore, make the appropriate modifications to supplyDefaultOptions and checkOptionFields as well."
Any suggestions?
Thanks in advance,
Martin
Here's one idea using movefile to rename the resultant published PDF on each iteration:
for i = 1:34
file = publish(files(i)); % Replace with your own command(s)
[pathStr,fileName,ext] = fileparts(file);
newFile = [pathStr filesep() fileName '_' int2str(i) ext]; % Example: append _# to each
[success,msg,msgid] = movefile(file,newFile);
if ~success
error(msgid,msg);
end
end
Also used are fileparts and filesep. See this question for other ways to rename and move files.