I have a function in matlab that contains a few sub functions. Currently abort processing is done by looking for a flag before every sub function, then assert. PS: This is not in a GUI.
But some sub functions take longer processing time, maybe due to a loop, or a time consuming function in the sub function. So it takes longer to abort processing sometimes.
Is there any way I can set a thread to be constantly (or at a fixed interval) looking for the flag to abort? Or do I need to go in the sub functions, look for the flag in each loop and inside the time consuming function?
Thank you!
Related
When writing iterative programs, a common situation is that you need to define a condition at which the program will stop execution and return the result. This stop condition can be for example rdd.isEmpty. The problem is that this "condition test" is an action which triggers a job to be executed and therefore schedueling,serialisation and others costs for each iteration
def iterate(layer:RDD[Long])={
layer.cache()
if(layer.isEmpty) return null;
val nextlayer=process(layer)//contains hashjoins, joins, filters, cache
iterate(nextlayer)
}
The timeline will look like:
[isempty][------spacing----][isempty][------spacing----][isempty]
what is the best way for iterative programming in such situtation? we should not be forced to launch a job in each iteration.
is there a method to check for empty rdd without executing an action?
possible solution:
as you can see in the image below, the is empty is now executed every 5 iterations.each iteration is represented by a periodic triplets of blue rectangles. i did this by modifying the stop condition to the following:
if(layer.index%5==0 && layer.isEmpty) return null;
But as you can see in the figure below am still getting actions that get executed as "run at ThreadPoolExecutor.java". A research shows that those actions are happenning because i am doing "broadcast hash joins" of small DFs with larger ones
threadpoolexecutor reason
timeline
You can try using
layer.cache()
layer.isEmpty
This means that the check for empty is going to trigger an action, but the rdd will be cached, so when you pass it to the process method, the things that were done in isEmpty will be "skipped".
For the use case of being able to abort parallel simulations with a MATLAB GUI, I would like to stop all scheduled simulations after the user pressed the Stop button.
All simulations are submitted at once using the parsim command, hence something like a callback to my GUI variables (App Designer) would be the most preferable solution.
Approaches I have considered but were not exactly providing a desirable solution:
The Simulation Manager provides the functionality to close simulations using its own interface. If I only had the code its Stop button executes...
parsim uses the Simulink.SimulationInput class as input to run simulations, allowing to modify the preSimFcn at the beginning of each simulation. I have not found a way to "skip" the simulation at its initialization phase apart from intentionally throwing an error so far.
Thank you for your help!
Update 1: Using the preSimFcn to set the the termination time equal to the start time drastically reduces simulation time. But since the first step still is computed there has to be a better solution.
simin = simin.setModelParameter('StopTime',get_param(mdl,'StartTime'))
Update 2: Intentionally throwing an error executing the preSimFcn, for example by setting it to
simin = simin.setModelParameter('SimulationCommand','stop')
provides the shortest termination times for me so far. Though, it requires catching and identifying the error in the ErrorMessageof the Simulink.SimulationOutput object. As this is exactly the "ugly" implementation I wanted to avoid, the issue is still active.
If you are using 17b or later, parsim provides an option to 'RunInBackground'. It returns an array of Future objects.
F = parsim(in, 'RunInBackground', 'on')
Please note that is only available for parallel simulations. The Simulink.Simulation.Future object F provides a cancel method which will terminate the simulation. You can use the fetchOutputs methods to fetch the output from the simulation.
F.cancel();
I came accross this sentence in MATLAB doc:
The body of a parfor-loop cannot make reference to a nested function. However, it can call a nested function by means of a function handle.
Can someone please explain what this means?
A parfor loop is different from a normal loop, in that the body of the loop has its independent workspace for every iteration. In fact, when you are running the parfor loop on a parallel pool, the variables that need to be transmitted to the loop body are saved and reloaded (that's, by the way, the reason for the "variable x cannot be sliced which may lead to communication overhead" warning: Having to save and reload huge variables may add quite a bit to your processing time).
Consequently, calls to nested functions won't work - the nested function in the parent function no longer shares its workspace with the loop body. Furthermore, nested function calls may alter workspace variables across iterations of a loop, which won't mesh with parallel execution.
In contrast, passing a function handle, or calling a separate function, works fine. The function defined in the function handle, as well as the separate function, have their own workspaces, nothing gets shared across iterations of the parfor body, and thus the iterations can run completely independently.
/aside: Creating a function handle to a nested function may still be able to cause you problems: a live function (as opposed to a function handle stored as string which you "activate" with str2func) handle can carry quite a bit of the existing workspace, including handle objects. Both the size of the workspace and the not-being-passed-by-reference (because of save&reload) may lead to unhappiness.
I have the following question: How do I tell Matlab that it should not wait for the results of a function? Is there a way other than threads?
My problem: I have a function A that is called by a Timer every few seconds. If a specific event is met, another function B is called inside function A. Function B opens a Batch File.
I want function A to go on without waiting for function B to end. Is there a way to easily do it?
I'm sorry if this question was already asked, but I couldn't find a satisfying answer. Please also excuse my bad english.
I would like to thank everyone who answers for their help.
In your function B, just call the batch file with a & at the end of the line.
For example:
!mybatch.bat &
This will run the file mybatch.bat in background mode and will return execution to Matlab immediately after the call.
or if you prefer the complete form:
[status, result] = system('mybatch.bat &')
But in this case it is a bit useless, since the system call mybatch in the background, the result variable is always empty and status is always 0 (whether a file mybatch.bat was found and executed or not)
edit: That is the quick trick in case it is only the batch file execution which is slowing down your program.
If you have more matlab instructions in function B and you really need function A to go on without waiting, you will have to set up a listener object with function B as a callback. Then in your function A, trigger the event (which will activate the listener and call function B).
I have a function that's taking a long time to run. When I profile it, I find that over half the time (26 out of 50 seconds) is not accounted for in the line by line timing breakdown, and I can show that the time is spent after the function finishes running but before it returns control by the following method:
ts1 = tic;
disp ('calling function');
functionCall(args);
disp (['control returned to caller - ', num2str(toc(ts1))]);
The first line of the function I call is ts2 = tic, and the last line is
disp (['last line of function- ', num2str(toc(ts2))]);
The result is
calling function
last line of function - 24.0043
control returned to caller - 49.857
Poking around on the interwebs, I think this is a symptom of the way MATLAB manages memory. It deallocates on function returns, and sometimes this takes a long time. The function does allocate some large (~1 million element) arrays. It also works with handles, but does not create any new handle objects or store handles explicitly. My questions are:
Is this definitely a memory management problem?
Is there any systematic way to diagnose what causes a problem in this function, as opposed to others which return quickly?
Are there general tips for reducing the amount of time MATLAB spends cleaning up on a function exit?
You are right, it seems to be the time spent on garbage collection. I am afraid it is a fundamental MATLAB flaw, it is known since years but MathWorks has not solved it even in the newest MATLAB version 2010b.
You could try setting variables manually to [] before leaving function - i.e. doing garbage collection manually. This technique also helps against memory leaks in previous MATLAB versions. Now MATLAB will spent time not on end but on myVar=[];
You could alleviate problem working without any kind of references - anonymous functions, nested functions, handle classes, not using cellfun and arrayfun.
If you have arrived to the "performance barrier" of MATLAB then maybe you should simply change the environment. I do not see any sense anyway starting today a new project in MATLAB except if you are using SIMULINK. Python rocks for technical computing and with C# you can also do many things MATLAB does using free libraries. And both are real programming languages and are free, unlike MATLAB.
I discovered a fix to my specific problem that may be applicable in general.
The function that was taking a long time to exit was called on a basic object that contained a vector of handle objects. When I changed the definition of the basic object to extend handle, I eliminated the lag on the close of the function.
What I believe was happening is this: When I passed the basic object to my function, it created a copy of that object (MATLAB is pass by value by default). This doesn't take a lot of time, but when the function exited, it destroyed the object copy, which caused it to look through the vector of handle objects to make sure there weren't any orphans that needed to be cleaned up. I believe it is this operation that was taking MATLAB a long time.
When I changed the object I was passing to a handle, no copy was made in the function workspace, so no cleanup of the object was required at the end.
This suggests a general rule to me:
If a function is taking a long time to clean up its workspace on exiting and you are passing a lot of data or complex structures by value, try encapsulating the arguments to that function in a handle object
This will avoid duplication and hence time consuming cleanup on exit. The downside is that your function can now unexpectedly change your inputs, because MATLAB doesn't have the ability to declare an argument const, as in c++.
A simple fix could be this: pre-allocate the large arrays and pass them as args to your functionCall(). This moves the deallocation issue back to the caller of functionCall(), but it could be that you are calling functionCall more often than its parent, in which case this will speed up your code.
workArr = zeros(1,1e6); % allocate once
...
functionCall(args,workArr); % call with extra argument
...
functionCall(args,wokrArr); % call again, no realloc of workArr needed
...
Inside functionCall you can take care of initializing and/or re-setting workArr, for instance
[workArr(:)] = 0; % reset work array