The Problem
I'm using the (simplified) code below to get information about a repo I have on box.com.
The script should:
access my remote repo via FTP
get a list of all the directories in the repo
iterate through each directory in the list, where in each iteration it:
3a) enters the directory and gets some information about the files there
3b) stores the information in an array
3c) goes back to the root with "../"
I found the following works for about half the directories in my repo (70 of 134):
ftpobj = ftp("ftp.box.com","myUname","myPassword","TLSMode","strict");
% Get dir list
dirList = dir(ftpobj);
numDirs = size(dirList,1);
% Setup out Array
clearvars outArray
outArray = ["directory" "numFiles"];
for i = 1:numDirs
% Select and Move to subfolder
folder = dirList(i,1).name;
cd(ftpobj, folder);
% Get a filelist for dir
files = dir(ftpobj);
numFiles = size(files,1);
% Determine Output and add to OutArray
outLine = [folder numFiles];
outArray = [outArray; outLine];
disp(i + " of " + numDirs + " done");
cd(ftpobj, "../"); % Move back to Root
end
But it drops out around halfway through on the cd(ftpobj,"../"); line, saying:
"ftp://ftp.box.com//2021-11-04/../" is nonexistent or not a directory.
I've Tried:
If I run the cd(ftpobj,"../"); command again in the terminal, it works fine. I can manually increase i and go by step by step again no problem - just not in the loop.
I've tried adding in a try catch over the whole loop, but it still stops working when it gets past 2021-11-04 (dir 71)!
I tried adding in an if statement to the code which skips out the problem directory (i==71), and it instead just tripped up on the next one (dir 72). I then tried changing the for statement to for i = 72:numDirs, without clearing the workspace, and it tripped out again.
A possible solution?
I cleared the workspace completely and ran the whole code again but with for i = 72:numDirs and it carried on perfectly to the end. So I guess I can run this loop in two halves but that seems hacky?
Could it be a bandwidth limit/ structure limit or something? I haven't been able to find anything about that though?
I managed to find a reasonable work-around. All I've done is reset the ftpobj every 30 queries by putting the following in my loop. It's gone through 4 repos and analysed over 700 directories so far without issue so while a bit weird (and possibly slower?) I think it's a viable solution!
if mod(i,30) == 0 % reset the ftp every 30
disp("resetting FTP object at "+ i);
ftpobj = ftp("ftp.box.com","myUname","myPassword","TLSMode","strict");
end
Related
I have written a script which does the following: first it writes a text file to a specified directory, this text file is then called for a Post Processor via a PowerShell command which I also call in MATLAB in order to generate a result *.csv file. Lastly, the script formats the data in this generated *.csv file and plots the results.
The problem is that the script can only proceed once the given files have been placed in the directory - especially in the case of the *.csv file this may take 10-15 seconds. Currently I have simply added a sort of break point which prompts the user to wait, as follows:
fprintf("Check if *.csv is present in directory - then click 'Continue' \n")
keyboard;
I'm guessing there's a much better way to do this - ideally to run the script, automatically wait at a given point until a specified file is present in the directory, and then continue running the code.
How can I automate the check for existence of the *.csv file?
You can check if the file filename exist using isfile (post R2017b) like this:
if isfile(filename)
% File exists.
else
% File does not exist.
end
Pre R2017b you may use if exist(filename, 'file') == 2.
You may combine this with a loop the following way:
t_Start = tic; % Start stopwatch timer (to set a maximum time)
dt = 2; % Check every dt seconds (example dt = 2)
while isfile(filename) == false && toc(t_Start) < 60 % maximum 60 seconds
pause(dt);
time_counter = time_counter + dt;
end
% Continue with your code here.
I'm using a while lopp to check if the file exist, and wait 2 seconds between each check. There is no reason to check every millisecond, unless you're in a hurry. You can of course change this.
t_Start = tic; and toc(t_start) < 60 is a stopwatch that stops the loop after 60 seconds. Just using tic and toc < 60 works too, but that would reset a tic/toc call outside of the loop, if there are any.
Note that isfile(filename) == false is the same as ~isfile(filename), but can be easier to read for beginners.
You can use exist(). This function checks for the existence of a variable, function, file, folder etc. Putting this in an if statement should do the trick:
if exist('your_file.csv')
% Run your code
end
In the below code I am trying to sort files based on a string within the name. I've been piecing this together with google searches and community help (I'm very new at matlab). Right now I'm getting two odd errors. First, when I try and make a folder, it creates some file (highlighted filein picture that I can't open and the wav files that should have been moved to the folder disappear.
I'm also having an issue where the code renames the first two data files moved to "01" and "01 (1)" and I have no idea why.
DirIn = 'C:\Folder\Experiment' %set incoming directory
eval(['filelist=dir(''' DirIn '/*.wav'')']) %get file list
for i = 1:length(filelist);
Filename = filelist(i).name
name = strsplit(Filename, '_');
newStr = extractBetween(name,7,8);
if strcmp(newStr,'01')
DirOut = fullfile(DirIn, '01');
mkdir DirIn DirOut
movefile(fullfile(filelist(i).folder, filelist(i).name), DirOut);
end
end
This should work:
DirIn = 'C:\Folder\Experiment'; %set incoming directory
filelist=dir(fullfile(DirIn, '*.wav')); %get file list
DirOut = fullfile(DirIn, '01');
for i = 1:length(filelist);
Filename = filelist(i).name
newStr = Filename(7:8);
if strcmp(newStr,'01')
if ~exist(DirOut)
mkdir(DirOut)
end
movefile(fullfile(filelist(i).folder, filelist(i).name), DirOut);
end
end
Firstly, you don't need eval to get the file list. eval impact performance significantly. The below is what you should have done:
filelist=dir(fullfile(DirIn, '*.wav'));
You don't need strsplit or extractBetween since you only intend to extract a part of the string by indexing, i.e. the 7th and 8th characters, you may do this:
newStr = Filename(7:8);
To use variable as an input, you need to use mkdir as a function rather than console command:
mkdir(DirOut)
Lastly, a bit of optimisation. Since DirOut is constant, you can take it outside the loop. You may also want to check if DirOut has already been created to avoid the warning message and overhead in mkdir.
There is no issue with movefile.
Couple things go wrong, first, it is not recommended to use eval. In this case you can just create a character array to pass to dir as follows:
filelist = dir([DirIn '/*.wav'])
Then, you have a strplit that appears to do nothing, since it looks like your files don't have '_' in them, so name will just return Filename. But that is not the issue, since you are using extractBetween on the Filename.
The following does not what you think it does,
mkdir DirIn DirOut
will create two directories named DirIn and DirOut in the current working directory of Matlab. To create the directories you want, use:
mkdir(DirOut)
Since the output directory did not exist before, I suspect Matlab moved the file to the input directory, and renamed it to 01, if you manually add the extension .wav it should be one of the original files.
I'm trying to access images in a matlab interface
my code is as follows:
global im2 im
axes(handles.axes4);
[path1, user_cance]= imgetfile();
if user_cance
msgbox(sprintf('Error'), 'Error', 'Error');
return
end
srcFiles = dir('C:\Users\User\Desktop\images test\yale faces\yalefaces\..');
% yale faces is the database folder
for i = 1 : length(srcFiles)
file_name=dir(strcat('C:\Users\User\Desktop\images test\yale faces\yalefaces'));
im2=imread(strcat('C:\Users\User\Desktop\images test\yale faces\yalefaces',file_name(i).name));
%processing of read image
end
the issue is that when I run the code, it gives the following error:
Can't open file "C:\Users\User\Desktop\images test\yale faces\yalefaces" for
reading;
you may not have read permission.
Does anyone know how to solve this issue?
When you do a directory listing (without any wildcards) you are going to get the current directory '.' and parent directory as well '..'. You can't read these like files because they are directories. You will need to filter out the directories prior to trying to read them with imread.
files = dir('C:\Users\User\Desktop\images test\yale faces\yalefaces');
% Remove directories
files = files(~[files.isdir]);
As a side note, it is very hard to tell what your code is doing, but I'm pretty sure it doesn't do what you hope.
It seems like you want to get all images within the database. If that's so, you'll want to do something like.
folder = 'C:\Users\User\Desktop\images test\yale faces\yalefaces';
% Get a list of all files in this folder
files = dir(folder);
files = files(~[files.isdir]);
for k = 1:numel(files)
% Append the folder with the filename to get the path and load
im2 = imread(fullfile(folder, files(k).name));
end
I highly discourage using strcat to construct file paths particularly because it removes trailing/leading whitespace from each input which can corrupt a filename. fullfile was designed for exactly this so please use that.
I have several data log files (here: 34) for those I have to calculate some certain values. I wrote a seperate function to publish the results of the calculation in a pdf file. But I only can publish one file after another, so it takes a while to publish all 34 files.
Now I want to automize that with a loop - importing the data, calculate the values and publish the results for every log file in a new pdf file. I want 34 pdf files for every log file at the end.
My problem is, that I couldn't find a way to rename the pdf files during publishing. The pdf file is always named after the script which is calculating the values. Obviously the pdf is overwritten within a loop. So at the end everything is calculated, but I only have the pdf from the last calculated log file.
There was this hacky solution to change the Matlab publish script, but since I don't have admin rights I can't use that:
"This is really hacky, but I would modify publish to accept a new option prefix. Replace line 93
[scriptDir,prefix] = fileparts(fullPathToScript);
with
if ~isfield(options, 'prefix')
[scriptDir,prefix] = fileparts(fullPathToScript);
else
[scriptDir,~] = fileparts(fullPathToScript);
prefix = options.prefix; end
Now you can set options.prefix to whatever filename you want. If you want to be really hardcore, make the appropriate modifications to supplyDefaultOptions and checkOptionFields as well."
Any suggestions?
Thanks in advance,
Martin
Here's one idea using movefile to rename the resultant published PDF on each iteration:
for i = 1:34
file = publish(files(i)); % Replace with your own command(s)
[pathStr,fileName,ext] = fileparts(file);
newFile = [pathStr filesep() fileName '_' int2str(i) ext]; % Example: append _# to each
[success,msg,msgid] = movefile(file,newFile);
if ~success
error(msgid,msg);
end
end
Also used are fileparts and filesep. See this question for other ways to rename and move files.
I have a set of days of log files that I need to parse and look at in matlab.
The log files look like this:
LOG_20120509_120002_002.csv
(year)(month)(day)_(hour)(minute)(second)_(log part number)
The logs increment hourly, but sometimes the seconds are one or two seconds off (per hour) which means i need to ignore what they say to do loadcsv.
I also have another file:
LOG_DATA_20120509_120002.csv
which contains data for the whole hour (different data).
The overall objective is to:
loop through each day
loop through each hour
read in LOG_DATA for whole hour
loop through each segment
read in LOG for each segment
compile a table of all the data
I guess the question is then, how do i ignore the minutes of the day if they are different? I suspect it will be by looping through all the files in the folder, in which case how do i do that?
Looping through all the files in the folder is relatively easy:
files = dir('*.csv');
for file = files'
csv = load(file.name);
% Do some stuff
end
At first, you must specify your path, the path that your *.csv files are in there
path = 'f:\project\dataset'
You can change it based on your system.
then,
use dir function :
files = dir (strcat(path,'\*.csv'))
L = length (files);
for i=1:L
image{i}=csvread(strcat(path,'\',file(i).name));
% process the image in here
end
pwd also can be used.