I have written a script which does the following: first it writes a text file to a specified directory, this text file is then called for a Post Processor via a PowerShell command which I also call in MATLAB in order to generate a result *.csv file. Lastly, the script formats the data in this generated *.csv file and plots the results.
The problem is that the script can only proceed once the given files have been placed in the directory - especially in the case of the *.csv file this may take 10-15 seconds. Currently I have simply added a sort of break point which prompts the user to wait, as follows:
fprintf("Check if *.csv is present in directory - then click 'Continue' \n")
keyboard;
I'm guessing there's a much better way to do this - ideally to run the script, automatically wait at a given point until a specified file is present in the directory, and then continue running the code.
How can I automate the check for existence of the *.csv file?
You can check if the file filename exist using isfile (post R2017b) like this:
if isfile(filename)
% File exists.
else
% File does not exist.
end
Pre R2017b you may use if exist(filename, 'file') == 2.
You may combine this with a loop the following way:
t_Start = tic; % Start stopwatch timer (to set a maximum time)
dt = 2; % Check every dt seconds (example dt = 2)
while isfile(filename) == false && toc(t_Start) < 60 % maximum 60 seconds
pause(dt);
time_counter = time_counter + dt;
end
% Continue with your code here.
I'm using a while lopp to check if the file exist, and wait 2 seconds between each check. There is no reason to check every millisecond, unless you're in a hurry. You can of course change this.
t_Start = tic; and toc(t_start) < 60 is a stopwatch that stops the loop after 60 seconds. Just using tic and toc < 60 works too, but that would reset a tic/toc call outside of the loop, if there are any.
Note that isfile(filename) == false is the same as ~isfile(filename), but can be easier to read for beginners.
You can use exist(). This function checks for the existence of a variable, function, file, folder etc. Putting this in an if statement should do the trick:
if exist('your_file.csv')
% Run your code
end
Related
The Problem
I'm using the (simplified) code below to get information about a repo I have on box.com.
The script should:
access my remote repo via FTP
get a list of all the directories in the repo
iterate through each directory in the list, where in each iteration it:
3a) enters the directory and gets some information about the files there
3b) stores the information in an array
3c) goes back to the root with "../"
I found the following works for about half the directories in my repo (70 of 134):
ftpobj = ftp("ftp.box.com","myUname","myPassword","TLSMode","strict");
% Get dir list
dirList = dir(ftpobj);
numDirs = size(dirList,1);
% Setup out Array
clearvars outArray
outArray = ["directory" "numFiles"];
for i = 1:numDirs
% Select and Move to subfolder
folder = dirList(i,1).name;
cd(ftpobj, folder);
% Get a filelist for dir
files = dir(ftpobj);
numFiles = size(files,1);
% Determine Output and add to OutArray
outLine = [folder numFiles];
outArray = [outArray; outLine];
disp(i + " of " + numDirs + " done");
cd(ftpobj, "../"); % Move back to Root
end
But it drops out around halfway through on the cd(ftpobj,"../"); line, saying:
"ftp://ftp.box.com//2021-11-04/../" is nonexistent or not a directory.
I've Tried:
If I run the cd(ftpobj,"../"); command again in the terminal, it works fine. I can manually increase i and go by step by step again no problem - just not in the loop.
I've tried adding in a try catch over the whole loop, but it still stops working when it gets past 2021-11-04 (dir 71)!
I tried adding in an if statement to the code which skips out the problem directory (i==71), and it instead just tripped up on the next one (dir 72). I then tried changing the for statement to for i = 72:numDirs, without clearing the workspace, and it tripped out again.
A possible solution?
I cleared the workspace completely and ran the whole code again but with for i = 72:numDirs and it carried on perfectly to the end. So I guess I can run this loop in two halves but that seems hacky?
Could it be a bandwidth limit/ structure limit or something? I haven't been able to find anything about that though?
I managed to find a reasonable work-around. All I've done is reset the ftpobj every 30 queries by putting the following in my loop. It's gone through 4 repos and analysed over 700 directories so far without issue so while a bit weird (and possibly slower?) I think it's a viable solution!
if mod(i,30) == 0 % reset the ftp every 30
disp("resetting FTP object at "+ i);
ftpobj = ftp("ftp.box.com","myUname","myPassword","TLSMode","strict");
end
Given that I write to a workbook, I currently kill all Excel processes so that my code works when I call it in a loop.
xlswrite(path,values);
system('taskkill /F /IM EXCEL.EXE');
This makes me unable to run the code while I am working in another Excel file. How do I make it so that Matlab terminates only the Excel processes that itself created?
This was a "feature" introduced somewhere around R2015b to speed up multiple writes to Excel... not very user/memory friendly!
The xlswrite documentation links to this MathWorks Support Answer to manually write to Excel using actxserver, you can then manually delete the COM object referencing Excel.
You can actually edit xlswrite and see that it uses matlab.io.internal.getExcelInstance, which does the same thing and creates a COM interface with actxserver.
A cheeky option would be to copy xlswrite, add the Excel variable it creates as an output, and Quit and delete it afterwards, as shown below. I don't advocate breaking any of The MathWorks' copyright ownership of that function.
A less cheeky option would be to create a comparable function based on the answer I linked above, for writing data only it would look something like this:
function xlswriteClean( File, Data, Range )
% XLSWRITECELAN writes data like XLSWRITE, but deletes the Excel instance!
% XLSWRITECELAN (FILE,DATA,RANGE) writes the variable in
% DATA to FILE, in the range specified by RANGE.
% RANGE is optional, defaults to "A1"
% Obtain the full path name of the file
% Could handle this more elegantly, i.e.
% this always assumes the current directory, but user might give a full path
file = fullfile(pwd, File);
% Open an ActiveX connection to Excel
h = actxserver('excel.application');
%Create a new work book (excel file)
wb = h.WorkBooks.Add();
% Select the appropriate range
if nargin < 3
Range = 'A1';
end
rng = h.Activesheet.get('Range', Range);
% Write the data to the range
rng.value = Data;
% Save the file with the given file name, close Excel
wb.SaveAs( File );
% Clean up - the point of this function
wb.Close;
h.Quit;
h.delete;
end
You can customise basically everything within the new Excel workbook using the COM object h, so you could add any functionality which you use in xlswrite like sheet naming etc.
You can start the excel process by powershell and get its process id then use the process id to kill the process:
[~, pid] = system('powershell (Start-Process excel.exe -passthru).Id');
% Do your work
% ...
system(['powershell Stop-Process -Id ' pid])
I have compiled my simple MatLab function so I can run it from a Windows console as a command line app. After I enter the command and its single argument, the prompt returns immediately even though the command is still running, as if running in the background.
Can someone tell me why this happens and how do I stop it? I'd like it to behave like a normal application and not return to the prompt until complete.
EDIT: Sorry I didn't post the code previously, I figured I was making a common mistake and would get a quick answer. Anyway, here it is.
It translates a .mat file to a .csv file and then runs another console command via "system" to perform a quick edit on the resulting .csv file. Several minutes of IO but not much else.
I've never compiled MatLab before so probably something simple, I hope.
function WriteMatToCSV( inFile, overWrite )
if ~exist(inFile, 'file')
fprintf('Error File not found: %s\n',inFile);
return;
end
overWriteExisting = 1;
if exist('overWrite','var')
overWriteExisting = overWrite;
end
% Change file extension
outFile = strrep( inFile, '.mat','.csv');
fprintf('In : %s\n',inFile);
fprintf('Out: %s\n',outFile);
% Overwrite if exists?
if exist(outFile, 'file')
if ~overWriteExisting
fprintf('Error File exists: %s\n',outFile);
return;
end
end
% Get variable name in the file
matObj = matfile(inFile);
info = whos(matObj);
name = info.name;
fprintf('Found variable: %s\n',name);
% Load the variable from the file
fprintf('Loading mat...\n');
load(inFile);
% Write the variable to .csv file
fprintf('Writing csv...\n');
export(eval(name),'File',outFile','Delimiter',',');
fprintf('Rewriting to remove MatLab bits...\n');
command = 'RewriteMatLabCSV';
command = [command ' ' outFile];
[status,cmdout] = system(command,'-echo');
fprintf(cmdout);
fprintf('Done\n');
end
I have a set of days of log files that I need to parse and look at in matlab.
The log files look like this:
LOG_20120509_120002_002.csv
(year)(month)(day)_(hour)(minute)(second)_(log part number)
The logs increment hourly, but sometimes the seconds are one or two seconds off (per hour) which means i need to ignore what they say to do loadcsv.
I also have another file:
LOG_DATA_20120509_120002.csv
which contains data for the whole hour (different data).
The overall objective is to:
loop through each day
loop through each hour
read in LOG_DATA for whole hour
loop through each segment
read in LOG for each segment
compile a table of all the data
I guess the question is then, how do i ignore the minutes of the day if they are different? I suspect it will be by looping through all the files in the folder, in which case how do i do that?
Looping through all the files in the folder is relatively easy:
files = dir('*.csv');
for file = files'
csv = load(file.name);
% Do some stuff
end
At first, you must specify your path, the path that your *.csv files are in there
path = 'f:\project\dataset'
You can change it based on your system.
then,
use dir function :
files = dir (strcat(path,'\*.csv'))
L = length (files);
for i=1:L
image{i}=csvread(strcat(path,'\',file(i).name));
% process the image in here
end
pwd also can be used.
When I open MATLAB, it defaults to open to /home/myUser. Whenever I open a file interactively (say I run Simulink and click "Open...") the dialog will start in /home/myUser. Then I might go into /home/myUser/myDir1/myDir2/ before clicking on myModel.mdl.
If I open a different Simulink file with the "Open..." dialog again, it'll kick me back to /home/myUser. Note that this is regardless of file, I'm just using Simulink as an example. I would like to keep it in /home/myUser/myDir1/myDir2, meaning I should be in the same directory as the last file I just opened (or saved).
Programatically, I would set up a callback to cd into whatever directory the file I chose was in after selecting the file using uigetfile. Is this possible to do just with MATLAB's own "Open" or "Save As" commands?
There is a wrapper for uigetfile that remembers the last director on the file exchange. You can get wrappers for the other file dialogs as well from the same author.
EDIT
How to overload the built-in uigetdir etc.
(1) Rename uigetdir2 to uigetdir, and make sure it's in a path that has precedence over the path for the built-in function (this should be the case by default).
(2) Use BUILTIN to ensure that the new function doesn't call itself.
(2) Since uigetdir is implemented as a .m-file, as opposed to being compiled (like sum), the builtin command doesn't work for it. Thus, open uigetdir, find the private function uigetdir_helper (which is private, so we can't call it), and finally uncover the java method (that turns out to have changed from R2011a to R2011b. Yay.). This allows us to finally overload uigetdir at the cost of having to parse the input ourselves.
Here's lines 37 to 67 from the modified uigetdir
%% Check passed arguments and load saved directory
% if start path is specified and not empty call uigetdir with arguments
% from user, if no arguments are passed or start path is empty load saved
% directory from mat file
% parse inputs
parserObj = inputParser;
parserObj.FunctionName = 'uigetdir';
parserObj.addOptional('startPath',[],#(x)exist(x,'dir'));
parserObj.addOptional('dialogTitle','Please choose directory',#ischar);
parserObj.parse(varargin{:});
inputList = parserObj.Results;
% create directory dialog instance - this has changed from 2011a to 2011b
if verLessThan('matlab','7.13')
dirdlg = UiDialog.UiDirDialog();
else
dirdlg = matlab.ui.internal.dialog.FolderChooser();
end
dirdlg.InitialPathName = inputList.startPath;
dirdlg.Title = inputList.dialogTitle;
if nargin > 0 && ~isempty(inputList.startPath)
% call dirdlg instead of uigetdir to avoid infinite recursion
dirdlg.show();
% if dirname empty, return 0 for uigetdir.
directory_name = dirdlg.SelectedFolder;
else
% set default dialog open directory to the present working directory
lastDir = pwd;
% load last data directory
if exist(lastDirMat, 'file') ~= 0
% lastDirMat mat file exists, load it
load('-mat', lastDirMat)
% check if lastDataDir variable exists and contains a valid path
if (exist('lastUsedDir', 'var') == 1) && ...
(exist(lastUsedDir, 'dir') == 7)
% set default dialog open directory
lastDir = lastUsedDir;
end
end
dirdlg.InitialPathName = lastDir;
% call dirdlg instead of uigetdir to avoid infinite recursion
dirdlg.show();
% if dirname empty, return 0 for uigetdir.
directory_name = dirdlg.SelectedFolder;
end