This is a general algorithm question, but my primary environment is Matlab.
I have a function
out=f(arg1,arg2,,.....)
which takes a long time to execute and is expensive to compute (i.e. cluster time). A given argument argn can be a string, integer, vector, and even a function handle
For this reason, I want to avoid calling f(args) for the same argument values. Inside my program, this can occur in ways that are not necessarily controllable by the programmer.
So, I want to call f() once for each possible value of args, and save the results to disk. Then, whenever it is called the next time, check if there is currently a result for those argument values. If so, I would load it from disk.
My current idea is to create a cell variable, with one row for each function call. In the first column is out. In column 2:N are the values of argn, and check the equivalence of each individually.
Since the variable types of the arguments vary, how would I go about doing this?
Is there a better algorithm?
More generally, how do people deal with saving simulation results to disk and storing metadata? (other than cramming everything into a filename!)
You can implement a function that looks something like this:
function result = myfun(input)
persistent cache
if isempty(cache)
cachedInputs = [];
cachedOutputs = [];
cache = {cachedInputs, cachedOutputs};
end
[isCached, idx] = ismember(input, cache{1});
if isCached
result = cache{2}(idx);
else
result = doHardThingOnCluster(input);
cache{1}(end+1) = input;
cache{2}(end+1) = result;
end
This simple example assumes that your inputs and outputs are both scalar numbers that can be stored in an array. If you have to deal with strings, or anything more complicated, you could use a cell array for caching rather than an array. Or in fact, maybe a containers.Map might be even better. Alternatively, if you have to cache really massive results, you might be better off saving it to a file and caching the file name, then loading the file in if you find it's been cached.
Hope that helps!
Related
I have a large Map in matlab and want to traverse all the keys of it.
But I can only store a cell for the keys first and traverse the cell, which calls for memory to store all the keys.
While in python, the generator/iterator can sequentially give the indexes or entries of list back in loops, which does not cost much memory.
In matlab, I tried for loop:
% Let MAP is a containers.Map with 10000000 keys.
keys = MAP.keys;
for keyIdx = 1:size(keys,2)
MAP(keys{1,keyIdx});
end
While in python, the for loop with iterator:
% Let MAP is a dictionary with 10000000 keys.
for key in MAP.iterkeys():
MAP[key]
So, is there generator/iterator like that in python in matlab to save memory when I process the maps?
To answer your main question: No. There is no generator in MATLAB.
Also, the Python code you wrote does not use a Python generator.
The memory footprint of both snippets are similar.
But I think I did not get the details about your question....
Last:
you can use numel(keys) instead of size(keys, 2) since keys is of one dimension
you can use MAP(keys{keyIdx}) instead of MAP(keys{1, keyIdx})
MATLAB uses the lazy copy or copy-on-write machinism. This means that when you do
B = A;
both A and B point to the same memory until you try to modify one of the arrays, at which point MATLAB will copy the data so that modifying one doesn’t affect the other copy.
This means also that, given a Containers.Map object map,
keys = map.keys
will not copy the keys, but instead keys will share data with the keys as they are stored inside the map.
Note that
keys = map.keys;
for keyIdx = 1:numel(keys)
value = map(keys{keyIdx})
end
is the same as
for key = map.keys
value = map(key{1})
end
and the same as
for value = map.values
value = value{1}
end
In none of these cases is data copied!
(I am guessing at the syntax, as I’m typing on my phone right now, I think the {1} indexing is required here, but I might be wrong about it.)
Prologue:
I am in the process of designing/prototyping a piece of code in Matlab.
As at the moment it is not clear to me which matrices should be returned by my functions, I chose, as a general approach, to bind my returned values in containers.Map (as I would do e.g. in python).
Hence, the general setting is
function output = myfoo(args)
output = containers.Map;
...some stuff
output('outname1') = ...
output('outname2') = ...
end
this approach should have the advantage of allowing me to add more returned data without messing up the other code too much or break backwards compatibility.
Issue:
How to deal in a elegant way with matrix slicing?
Say that I need to do something like
output('outname1')(2:end) = ...
(which gives an error as two indexes are not allowed and a boring workaround like
temp = output('outname1')
temp(2:end) = ...
output('outname1') = temp
is required).
Question:
Is there a proficient way to deal with this, avoiding all this referencing/copying job?
No, there is no way to do it without a temporary variable. The only case in which a double index is valid in Matlab is for a cell array. In that case, you can use
output{...}(...)
However, in any other case, a double index results in an error.
I have to make a call to read in a matrix with a name that is created from a strcat, in Matlab. For example
person = 2;
Index_XNY = strcat('Index_X',num2str(person),'Y');
Big_Index = find(Index_XNY(1,:)==1);
This works if I replace Index_XNY with Index_X2Y for this example.
I have tried a number of different things to obtain Big_Index, but so far I have not been successful. Can this be done, as there is a large collection of data elements that need to be called.
You can use the eval function to evaluate a Matlab expression in a string:
expr = strcat('find(Index_X',num2str(person),'Y(1,:)==1)');
Big_Index = eval(expr);
Of course, a lot of alternative ways to do this with eval function exist, depending on how much you would like to put into the string and how much would be left out.
I have a function that returns two values, like so:
[a b] = myfunc(x)
Is there a way to get the second return value without using a temporary variable, and without altering the function?
What I'm looking for is something like this:
abs(secondreturnvalue(myfunc(x)))
not that i know of. subsref doesn't seem to work in this case, possibly because the second variable isn't even returned from the function.
since matlab 2009b it is possible to use the notation
[~, b] = function(x)
if you don't need the first argument, but this still uses a temporary variable for b.
Unless there is some pressing need to do this, I would probably advise against it. The clarity of your code will suffer. Storing the outputs in temporary variables and then passing these variables to another function will make your code cleaner, and the different ways you could do this are outlined here: How to elegantly ignore some return values of a MATLAB function?.
However, if you really want or need to do this, the only feasible way I can think of would be to create your own function secondreturnvalue. Here's a more general example called nth_output:
function value = nth_output(N,fcn,varargin)
[value{1:N}] = fcn(varargin{:});
value = value{N};
end
And you would call it by passing as inputs 1) the output argument number you want, 2) a function handle to myfunc, and 3) whatever input arguments you need to pass to myfunc:
abs(nth_output(2,#myfunc,x))
function y = myfunc(param)
C = param.C;
L = param.L;
Kp = param.Kp;
Ki = param.Ki;
...
Is there a way to generalize the above code? I know how to generalize the structure access using fieldnames() and getfield(), but not how to set variables without calling eval() (which is evil).
for n = fieldnames(param)'
name = n{1};
value = param.(name);
do_something_with(name,value); % ????
never mind, I figured it out; this helper function works:
function vars_pull(s)
for n = fieldnames(s)'
name = n{1};
value = s.(name);
assignin('caller',name,value);
end
The only way to create a variable whose name is determined at run-time is to use a function like eval, evalin, feval, or assignin. (assignin is the least evil choice BTW, at least you don't need to convert your value to a string and back.)
However, I question why you want to do that, why not just access the values through the input structure as you need them. If you want to save typing (speaking from experience, as I am extremely lazy), I usually name my input parameter structure something short, like p. The throughout my code I just access the fields directly, (e.g. p.Kp, and after a while I don't even see the p. anymore.) This also makes it easy to pass the structure into subfunctions as needed.
You can use the excellent submission at FileExchange:
V2STRUCT - Pack & Unpack variables to & from structures with enhanced functionality
Here's a workaround: save the structure to a .mat file using the '-struct' option, and then immediately reload it. Here's an example for struct variable X:
save('deleteme.mat','-struct','X');
load('deleteme.mat');
delete('deleteme.mat');
It's kludgey, but actually pretty fast, at least with an SSD.