I often call computationally intensive command-line programs from within MATLAB using the system command:
[status, result] = system(cmd_line_for_my_low_level_exe, '-echo');
where the -echo option (supposedly) echoes console output (stdout) generated by low_level_exe in the MATLAB command window.
On Linux machines this works great, with MATLAB echoing the console output in (seemingly) real-time. Users get a nice continuous update on low_level_exe's progress.
On Windows machines this is not the case. It can often be many minutes in-between echoes, and users sometimes get impatient and assume the code has crashed...
Is there a way to increase/control the frequency of MATLAB's -echo, or possibly another, better option entirely? (I'd prefer to stay away from mex files to maintain compatibility with Octave).
Is this actually a MATLAB issue, or just a Linux/Windows incompatibility?
Related
I want to have two MATLAB windows open on the same computer. The desired scenario is as follows: MATLAB window 1 is continuously running a script that has nothing to do with MATLAB window 2. At the same time, MATLAB window 2 is running a script that continuously checks for a certain condition, and if it is met, then it will terminate the script running on MATLAB window 1, and then terminate its own script as well. I want to have two MATLAB windows instead of one since I believe it will be more time efficient for what I am trying to do. I found an interesting "KeyInject" program at http://au.mathworks.com/matlabcentral/fileexchange/40001-keyinject , but I was wondering if there is a simpler way already built into MATLAB.
Do you want simple, or a flexible, infinitely expandable version 1.0? Simple would be to trigger System A via a file created by System B.
Simple would have System B create a file, then System A would check for the file with the command
if exist ( fileName, 'file' )
then do your shutdown commands. On startup, System A would delete the file with
delete ( fileName );
The second option is to use the udp command. UDP allows any data to be sent between processes, whether on the same computer or over a network. (See https://www.mathworks.com/help/instrument/udp.html for more info).
I see several ways:
Restructure to avoid this XY problem
Use (mat) files (as Hoki suggested), possibly using the parallel computing toolbox to keep everything in one MATLAB session.
Write some MEX functions that communicate with each other via a global pipe.
Write an Auto(Hot)key script.
Option 2 is probably easiest. Take a look at events and listeners if you write in OOP, otherwise, you'd have to poll inside a loop
Option 3 is harder and way more time consuming to implement, but allows for much faster detection of the condition, and much faster data transfer between the sessions. Use only if speed is essential...but I guess that doesn't apply :)
Option 4: the AutoHotkey solution is probably the most Horrible Thing® you could do on an already Horrible Construction®, but oh what fun!! In both MATLAB sessions, you create a (hidden) figure with the name Window1 or Window2, respectively. These window names are something that AutoHotkey can easily track. If the conditions are met, you update the corresponding window name, triggering the remainder of the AutoHotkey script: press a button in the other window! If you need to transfer data between the windows: you can create basic edit boxes in both GUIs, and copy-paste the data between them. If you're on Linux: you can use Autokey for the same purpose, but by then you're basically writing Python code doing the heavy lifting, so just use Python.
Or, you know, use KeyInject. Less fun.
I am running a parallelized code courtesy the MATLAB Parallel Computing Toolbox using the spmd command. Specifically, the code is like this:
spmd
out = function(data,labindex);
end
Now the function involves a library (libsvm) which gives me a trained classifier for each iteration. During the training process, there are several debug messages being printed out to the standard output by the library and somehow these are not appearing on my standard terminal - I think this is because the workers are actually on a cluster and hence the debug messages are not visible to me.
Is there anyway to reroute the debug messages ? (possibly other than writing to a file on a shared disk)
One option may be to try the Parallel Command Window. This opens a new special Command Window with one pane per lab. You'll need to run commands from the "P>>" pmode prompt in this window. More here.
Is there a way to call Matlab functions from outside, in particular by the Windows cmd (but also the Linux terminal, LUA-scripts, etc...), WITHOUT opening a new instance of Matlab each time?
for example in cmd:
matlab -sd myCurrentDirectory -r "function(parameters)" -nodesktop -nosplash -nojvm
opens a new instance of Matlab relatively fast and executes my function. Opening and closing of this reduced matlab prompt takes about 2 seconds (without computations) - hence for 4000 executions more than 2 hours. I'd like to avoid this, as the called function is always located in the same workspace. Can it be done in the same instance always?
I already did some research and found the possibility of the MATLAB COM Automation Server, but it seems quite complicated to me and I don't see the essential steps to make it work for my case. Any advices for that?
I'm not familiar with c/c++/c# but I'm thinking about the use of python (but just in the worst case).
Based on the not-working, but well thought, idea of #Ilya Kobelevskiy here the final workaround:
function pipeConnection(numIterations,inputFile)
for i=1:numIterations
while(exist('inputfile','file'))
load inputfile;
% read inputfile -> inputdata
output = myFunction(inputdata);
delete('inputfile');
end
% Write output to file
% Call external application to process output data
% generate new inputfile
end;
Another convenient solution would be to compile an executable of the Matlab function:
mcc -m myfunction
run this .exe-file using cmd:
cd myCurrentDirectory && myfunction.exe parameter1 parameter2
Be aware that the parameters are now passed as strings and the original .m-file needs to be adjusted considering that.
further remarks:
I guess Matlab still needs to be installed on the system, though
it is not necessary to run it.
I don't know how far this method is limited respectively the complexity of the
underlying function.
The speed-up compared to the initial apporach given in the question is
relatively small
Amongst the several methods exposed here, there is one workaround that should reduce the execution time of your multiple matlab calls. The idea is to run a custom function multiple times within on matlab session.
For example, myRand.m function is defined as
function r = myRand(a,b)
r = a + (b-a).*rand;
Within the matlab command window, we generate the single line command like this
S = [1:5; 1:5; 101:105];
cmd_str = sprintf('B(%d) = myRand(%d,%d);', S)
It generates the following command string B(1) = myRand(1,101);B(2) = myRand(2,102);B(3) = myRand(3,103);B(4) = myRand(4,104);B(5) = myRand(5,105); that is executed within a single matlab session with
matlab -nojvm -nodesktop -nosplash -r "copy_the_command_string_here";
One of the limitation is that you need to run your 4000 function calls in a row.
I like approach proposed by Magla, but given the constrains stated in your comment to it, it can be improved to still run single function in one matlab session.
Idea is to pipe your inputs and outputs. For inputs, you can check if certain input file exists, if it does, read input for your function from it, do work, write output to another file to signal script/function processing results that it matlab function is done and is waiting for the next input.
It is very straightforwad to implement using disk files, with some effort it is probably possible to do through memory disk (i.e., open input/output fiels in RAM).
function pipeConnection(numIterations,inputFile,outputFile)
for i=1:numIterations
while(!isfile(inputFile))
sleep(50);
end;
% Read inputs
output = YourFunction(x,y,z);
% Write output to file, go to next iteration
end;
return;
If number of iterations is unknown when you start, you can also encode exit conditions in input file rather than specifying number of iterations right away.
If you're starting up MATLAB from the command line with the -r option in the way you describe, then it will always start a new instance as you describe. I don't believe there's a way around this.
If you are calling MATLAB from a C/C++ application, MATLAB provides the MATLAB engine interface, which would connect to any running instance of MATLAB.
Otherwise the MATLAB Automation Server interface that you mention is the right way to go. If you're finding it complicated, I would suggest posting a separate question detailing what you've tried and what difficulties you're having.
For completeness, I'll mention that MATLAB also has an undocumented interface that can be called directly from Java - however, as it's undocumented it's very difficult to get right, and is subject to change across versions so you shouldn't rely on it.
Edit: As of R2014b, MATLAB makes available the MATLAB Engine for Python, via which you can automate MATLAB from a Python script. And as of R2016b, there is also the MATLAB Engine for Java. If anyone was previously considering the undocumented Java techniques mentioned above, this would now be the way to go.
I have compiled a Matlab routine using the MCR and deployed it to other computers without having them installed matlab. So far, so good. But of course, the routine is not completely error-free, particularly the GUI part. The problem is that when the MCR tries to write the error message to the terminal, it seems to corrupt the terminal so that everything is no longer legible - not even the prompt. Sometimes I also get an extra window, vaguely resembling the matlab editor window, full of illegible ascii characters. Does anyone know what is causing this, or how to avoid it?
My first attempt was a big try-catch block around everything, but whatever it is still seems to get through. The catch block just tries to divert the error to an errordlg rather than the command prompt:
catch e
errordlg({e.message;['in: ',e.stack.name]})
end
MATLAB Compiler does not support command window functions.
Peter Webb tells on Loren's blog:
Certain MATLAB functions cannot be deployed because they act on
objects that are not present in a deployed application. For example,
since deployed applications have no command window, functions that
modify the command window can't be deployed.
So, you probably need to get rid of any function that prints to the command window.
Also, you can check out the mccExcludedFiles.log file.
DOS is always given as an example of single tasking operating system. However when a command is issued through command-line, control switches from the shell to the command and then switches back to shell when the command completes.Thus there are two processes executing simultaneously. Is there something wrong in my understanding ?
No, they weren't executing simultaneously.
COMMAND.COM had a resident portion that was in memory all the time and a transient portion that could be tossed out at will.
When you ran a program, it typically got loaded in place of the transient portion and then run. When the program exited, it did so by calling code in the resident portion which would then reload the transient portion if necessary and continue.
The fact that some of the code remained resident in no way means that it was "running". In a similar way, vast tracts of MS-DOS (the kernel) stayed continuously in memory yet they weren't "running", unless called explicitly by a non-kernel program.
Now there were things can could be said to run concurrently, DOS had plenty of TSR (terminate and stay resident) programs that would run, hook into an interrupt or DOS in some way, then exit but leaving some memory allocated (where its code was).
Then, in response to certain events, that code would be run. Perhaps one of the famous ones was Borland Sidekick which was a personal information manager that would pop up instantly with a keypress.
While the other process is running, the command line processor is not running: it is suspended. The only "multitasking" facility that was available in DOS was "Terminate and Stay Resident".
It doesnt matter whether you are running DOS or Windows or Linux or BSD or whatever on that processor it is all the same. At that period of time you for purposes of this discussion had a single execution unit, a single core executing the instructions, mostly in order. Makes no difference if those instructions wear the name DOS or Linux or windows. Just instructions.
Just like now as then when a windows program decides to terminate it tries to do it nicely with some flavor of exit call. When a linux program terminates it tries to do so nicely with some flavor of exit call to the system. And when a dos program terminates it tries to do so nicely with some flavor of exit call to the system. In a shell, command prompt, etc linux, windows, dos, that shell, which is a program itself, loads and branches to the program you have loaded and your program runs for a while and as mentioned tries to return to the prior program nicely with some flavor of exit. Just like when the shell you were running wants to return when it is done running it tries to do so nicely.
As with linux or windows, easier to see back then, you dont run anything "at the same time" or "in parallel" one instruction stream at a time. (today we have multiple execution units and/or cores that are designed to each be doing something in parallel with something managing them, so today you can actually say "in parallel") You want to switch "tasks" or "threads" or "processes" you needed an interrupt, that switched to you different code, an interrupt handler, and that handler could return to the same program that was interrupted or switch to another. You can put whatever name on it you want it is how you make things look like they are running at the same time. dos, linux, windows, etc, this is typically how you switch from one "program" or bit of code to another. linux and windows have their kernels and operating system behind them that was called during the interrupts, and dos had that as well (dos HAS that, dos is still alive you touch a dos machine every few days most likely (gas pump, atm machine, etc), dos is also still used in the development and testing of x86 motherboards/computers, nothing can compete with it as an embedded x86 platform, nothing has the freedom that dos has to do what you want, this is why bios upgrades are still distributed as a dos program). The interrupt handlers would give time slices to the various bios handlers and dos handlers. task/process/thread switching was not as designed or planned as an operating system like linux or windows, but it was there, for each version of dos there were rules you followed and you could switch tasks (tsrs are a popular term). Just talking to a floppy, hard disk, etc there was code involved in the whole process, it wasnt buried in the hardware, lots of things happened in parallel. no different than a hard disk controller driver in something more complicated like linux or windows. At least one, maybe some, non-microsoft dos clones could multitask.
The short answer, When you have a function bob() that calls a function ted().
int bob ( int something )
{
...some code
...more code
ted();
...some code
...more code
}
is bob() still running? Are they running in parallel? No, the bob() code is still there, somewhere, waiting for the ted() code to finish what it was doing and return. So long as ted() doesnt crash it will return and bob() can continue to execute. bob is suspended while ted executes. Not much different with a shell or command line in an more complicated operating system. There is some function somewhere that has loaded your program into memory and called it, it might be a fork or clone of a command line that you were running so that that command line can continue "in parallel" or the clone can continue in parallel. but the concept is the same.
The difference from a trivial C program like the one above is that the code above can be thought of being resolved at compile time where loading and running a program is definitely runtime, basically self modifying code, the program modifies memory then jumps to it. When it returns that code, cleans up, unwinds, and exits itself or waits for another command depending on the design. DOS was just very very simple, a bunch of system calls, combined with a bunch of BIOS calls, and a very simple command line that could load programs and do a small number of other commands. It didnt have any rules you couldnt get around (windows is a dos program), if the program you launched didnt want to return (you could at least at the time launch linux from dos through an intermediate dos program) well it kind of messes up your question of what happens when the program completes, well linux didnt return, it took over the system.