I'm trying to speed up the simulation of some panel data in Matlab. I have to simulate first over individuals (loop index ii from 1 to N) and then for each individual over age (loop index jj from 1 to JJ). The code is slow because inside the two loops there is a bilinear interpolation to do.
Since the iterations in the outer loop are independent, I tried to use parfor in the outer loop (the loop indexed by ii), but I get the error message "the parfor cannot run due to the way the variable hsim is used". Could someone explain why and how to solve the problem if possible? Any help is greatly appreciated!
a_sim = zeros(Nsim,JJ);
h_sim = zeros(Nsim,JJ);
% Find point on a_grid corresponding to zero assets
aa0 = find_loc(a_grid,0.0);
% Zero housing
hh0 = 1;
a_sim(:,1) = a_grid(aa0);
h_sim(:,1) = h_grid(hh0);
parfor ii=1:Nsim !illegal
for jj=1:JJ-1
z_c = z_sim_ind(ii,jj);
apol_interp = griddedInterpolant({a_grid,h_grid},apol(:,:,z_c,jj));
hpol_interp = griddedInterpolant({a_grid,h_grid},hpol(:,:,z_c,jj));
a_sim(ii,jj+1) = apol_interp(a_sim(ii,jj),h_sim(ii,jj));
h_sim(ii,jj+1) = hpol_interp(a_sim(ii,jj),h_sim(ii,jj));
end
end
I think #Ben Voigt's suggestion was correct. To spell it out, do something like this:
parfor ii=1:Nsim
a_sim_row = a_sim(ii,:);
h_sim_row = h_sim(ii,:);
for jj=1:JJ-1
z_c = z_sim_ind(ii,jj);
apol_interp = griddedInterpolant({a_grid,h_grid},apol(:,:,z_c,jj));
hpol_interp = griddedInterpolant({a_grid,h_grid},hpol(:,:,z_c,jj));
a_sim_row(jj+1) = apol_interp(a_sim_row(jj),h_sim_row(jj));
h_sim_row(jj+1) = hpol_interp(a_sim_row(jj),h_sim_row(jj));
end
a_sim(ii,:) = a_sim_row;
h_sim(ii,:) = h_sim_row;
end
This is a fairly standard parfor pattern to work around the limitation (in this case, parfor cannot spot that what you're doing is not order-independent as far as the outer loop is concerned) - extract a whole slice, do whatever is needed, then put the whole slice back.
Related
I want to assign one value to a list of objects in Matlab without using a for-loop (In order to increase efficiency)
Basically this works:
for i=1:Nr_of_Objects
Objectlist(i,1).weight=0.2
end
But I would like something like this:
Objectlist(:,1).weight=0.2
Which is not working. I get this error:
Expected one output from a curly brace or dot indexing expression, but there were 5 results.
Writing an array to the right hand side is also not working.
I`m not very familiar with object oriented programming in Matlab, so I would be happy if someone could help me.
Your looking for the deal function:
S(1,1).a = 1
S(2,1).a = 2
S(1,2).a = 3
[S(:,1).a] = deal(4)
Now S(1,1).a and S(2,1).a equal to 4.
In matlab you can concatenate several output in one array using []. And deal(X) copies the single input to all the requested outputs.
So in your case:
[Objectlist(:,1).weight] = deal(0.2)
Should work.
Noticed that I'm not sure that it will be faster than the for loop since I don't know how the deal function is implemented.
EDIT: Benchmark
n = 1000000;
[S(1:n,1).a] = deal(1);
tic
for ii=1:n
S(ii,1).a = 2;
end
toc
% Elapsed time is 3.481088 seconds
tic
[S(1:n,1).a] = deal(2);
toc
% Elapsed time is 0.472028 seconds
Or with timeit
n = 1000000;
[S(1:n,1).a] = deal(1);
g = #() func1(S,n);
h = #() func2(S,n);
timeit(g)
% ans = 3.67
timeit(h)
% ans = 0.41
function func1(S,n)
for ii=1:n
S(ii,1).a = 2;
end
end
function func2(S,n)
[S(1:n,1).a] = deal(2);
end
So it seems that using the deal function reduce the computational time.
Using Matlab R2019a, is there any way to avoid the for-loop in the following code in spite of the dimensions containing different element so that each element has to be checked? M is a vector with indices, and Inpts.payout is a 5D array with numerical data.
for m = 1:length(M)-1
for power = 1:noScenarios
for production = 1:noScenarios
for inflation = 1:noScenarios
for interest = 1:noScenarios
if Inpts.payout(M(m),power,production,inflation,interest)<0
Inpts.payout(M(m+1),power,production,inflation,interest)=...
Inpts.payout(M(m+1),power,production,inflation,interest)...
+Inpts.payout(M(m),power,production,inflation,interest);
Inpts.payout(M(m),power,production,inflation,interest)=0;
end
end
end
end
end
end
It is quite simple to remove the inner 4 loops. This will be more efficient unless you have a huge matrix Inpts.payout, as a new indexing matrix must be generated.
The following code extracts the two relevant 'planes' from the input data, does the logic on them, then writes them back:
for m = 1:length(M)-1
payout_m = Inpts.payout(M(m),:,:,:,:);
payout_m1 = Inpts.payout(M(m+1),:,:,:,:);
indx = payout_m < 0;
payout_m1(indx) = payout_m1(indx) + payout_m(indx);
payout_m(indx) = 0;
Inpts.payout(M(m),:,:,:,:) = payout_m;
Inpts.payout(M(m+1),:,:,:,:) = payout_m1;
end
It is possible to avoid extracting the 'planes' and writing them back by working directly with the input data matrix. However, this yields more complex code.
However, we can easily avoid some indexing operations this way:
payout_m = Inpts.payout(M(1),:,:,:,:);
for m = 1:length(M)-1
payout_m1 = Inpts.payout(M(m+1),:,:,:,:);
indx = payout_m < 0;
payout_m1(indx) = payout_m1(indx) + payout_m(indx);
payout_m(indx) = 0;
Inpts.payout(M(m),:,:,:,:) = payout_m;
payout_m = payout_m1;
end
Inpts.payout(M(m+1),:,:,:,:) = payout_m1;
It seems like there is not a way to avoid this. I am assuming that each for lop independently changes a variable parameter used in the main calculation. Thus, it is required to have this many for loops. My only suggestion is to turn your nested loops into a function if you're concerned about appearance. Not sure if this will help run-time.
Suppose we are running an infinite for loop in MATLAB, and we want to store the iterative values in a vector. How can we declare the vector without knowing the size of it?
z=??
for i=1:inf
z(i,1)=i;
if(condition)%%condition is met then break out of the loop
break;
end;
end;
Please note first that this is bad practise, and you should preallocate where possible.
That being said, using the end keyword is the best option for extending arrays by a single element:
z = [];
for ii = 1:x
z(end+1, 1) = ii; % Index to the (end+1)th position, extending the array
end
You can also concatenate results from previous iterations, this tends to be slower since you have the assignment variable on both sides of the equals operator
z = [];
for ii = 1:x
z = [z; ii];
end
Sadar commented that directly indexing out of bounds (as other answers are suggesting) is depreciated by MathWorks, I'm not sure on a source for this.
If your condition computation is separate from the output computation, you could get the required size first
k = 0;
while ~condition
condition = true; % evaluate the condition here
k = k + 1;
end
z = zeros( k, 1 ); % now we can pre-allocate
for ii = 1:k
z(ii) = ii; % assign values
end
Depending on your use case you might not know the actual number of iterations and therefore vector elements, but you might know the maximum possible number of iterations. As said before, resizing a vector in each loop iteration could be a real performance bottleneck, you might consider something like this:
maxNumIterations = 12345;
myVector = zeros(maxNumIterations, 1);
for n = 1:maxNumIterations
myVector(n) = someFunctionReturningTheDesiredValue(n);
if(condition)
vecLength = n;
break;
end
end
% Resize the vector to the length that has actually been filled
myVector = myVector(1:vecLength);
By the way, I'd give you the advice to NOT getting used to use i as an index in Matlab programs as this will mask the imaginary unit i. I ran into some nasty bugs in complex calculations inside loops by doing so, so I would advise to just take n or any other letter of your choice as your go-to loop index variable name even if you are not dealing with complex values in your functions ;)
You can just declare an empty matrix with
z = []
This will create a 0x0 matrix which will resize when you write data to it.
In your case it will grow to a vector ix1.
Keep in mind that this is much slower than initializing your vector beforehand with the zeros(dim,dim) function.
So if there is any way to figure out the max value of i you should initialize it withz = zeros(i,1)
cheers,
Simon
You can initialize z to be an empty array, it'll expand automatically during looping ...something like:
z = [];
for i = 1:Inf
z(i) = i;
if (condition)
break;
end
end
However this looks nasty (and throws a warning: Warning: FOR loop index is too large. Truncating to 9223372036854775807), I would do here a while (true) or the condition itself and increment manually.
z = [];
i = 0;
while !condition
i=i+1;
z[i]=i;
end
And/or if your example is really what you need at the end, replace the re-creation of the array with something like:
while !condition
i=i+1;
end
z = 1:i;
As mentioned in various times in this thread the resizing of an array is very processing intensive, and could take a lot of time.
If processing time is not an issue:
Then something like #Wolfie mentioned would be good enough. In each iteration the array length will be increased and that is that:
z = [];
for ii = 1:x
%z = [z; ii];
z(end+1) = ii % Best way
end
If processing time is an issue:
If the processing time is a large factor, and you want it to run as smooth as possible, then you need to preallocating.If you have a rough idea of the maximum number of iterations that will run then you can use #PluginPenguin's suggestion. But there could still be a change of hitting that preset limit, which will break (or severely slow down) the program.
My suggestion:
If your loop is running infinitely until you stop it, you could do occasional resizing. Essentially extending the size as you go, but only doing it once in a while. For example every 100 loops:
z = zeros(100,1);
for i=1:inf
z(i,1)=i;
fprintf("%d,\t%d\n",i,length(z)); % See it working
if i+1 >= length(z) %The array as run out of space
%z = [z; zeros(100,1)]; % Extend this array (note the semi-colon)
z((length(z)+100),1) = 0; % Seems twice as fast as the commented method
end
if(condition)%%condition is met then break out of the loop
break;
end;
end
This means that the loop can run forever, the array will increase with it, but only every once in a while. This means that the processing time hit will be minimal.
Edit:
As #Cris kindly mentioned MATLAB already does what I proposed internally. This makes two of my comments completely wrong. So the best will be to follow what #Wolfie and #Cris said with:
z(end+1) = i
Hope this helps!
I have a series of nested loops that works to store data in a cell array. I am trying to find ways to speed up the loop and also help to simplify the readability. I have already optimized the loop a fair bit, but would like to see if I could vectorize it further. My original code looked like this:
%% ORIGINAL LOOP
for iA = 1:length(arrA)
for iB = 1:length(arrB)
for iC = 1:length(arrC)
a = arrA(iA); % depends only on iA
a_x = AData.x(AData.a==a);
a_y = AData.y(AData.a==a);
b = arrB(iB); % depends only on iB
b_x = BData.x(BData.b==b);
b_y = BData.y(BData.b==b);
c = arrC(iC); % depends only on iC
FinalData{iA,iB,iC} = computedata(a_x, a_y, b_x, b_y, c);
end
end
end
Since the calculations for a, a_x, a_y depended only on iA I pulled them out of the inner loops, and did similarly for the other variables, which increased performance significantly:
%% FASTER LOOP
for iA = 1:length(arrA)
a = arrA(iA);
a_x = AData.x(AData.a==a);
a_y = AData.y(AData.a==a);
for iB = 1:length(arrB)
b = arrB(iB);
b_x = BData.x(BData.b==b);
b_y = BData.y(BData.b==b);
for iC = 1:length(arrC)
c = arrC(iC);
FinalData{iA,iB,iC} = computedata(a_x, a_y, b_x, b_y, c);
end
end
end
I am wondering if there yet a better way to speed up this process, perhaps by MATLAB vectorization (elimination of loops altogether).
I also wanted to make it more compact and easier to rearrange the order of the loops if need be, for other functions I plan to design for plotting things in various orders. Any tips would be greatly appreciated.
I'm looping in parallel and changing a variable if a condition is met. Super idiomatic code that I'm sure everyone has written a hundred times:
trials = 100;
greatest_so_far = 0;
best_result = 0;
for trial_i = 1:trials
[amount, result] = do_work();
if amount > greatest_so_far
greatest_so_far = amount;
best_result = result;
end
end
If I wanted to replace for by parfor, how can I ensure that there aren't race conditions when checking whether we should replace greatest_so_far? Is there a way to lock this variable outside of the check? Perhaps like:
trials = 100;
greatest_so_far = 0;
best_result = 0;
parfor trial_i = 1:trials
[amount, result] = do_work();
somehow_lock(greatest_so_far);
if amount > greatest_so_far
greatest_so_far = amount;
best_result = result;
end
somehow_unlock(greatest_so_far);
end
Skewed answer. It does not exactly solve your problem, but it might help you avoiding it.
If you can afford the memory to store the outputs of your do_work() in some vectors, then you could simply run your parfor on this function only, store the result, then do your scoring at the end (outside of the loop):
amount = zeros( trials , 1 ) ;
result = zeros( trials , 1 ) ;
parfor trial_i = 1:trials
[amount(i), result(i)] = do_work();
end
[ greatest_of_all , greatest_index ] = max(amount) ;
best_result = result(greatest_index) ;
Edit/comment : (wanted to put that in comment of your question but it was too long, sorry).
I am familiar with .net and understand completely your lock/unlock request. I myself tried many attempts to implement a kind of progress indicator for very long parfor loop ... to no avail.
If I understand Matlab classification of variable correctly, the mere fact that you assign greatest_so_far (in greatest_so_far=amount) make Matlab treat it as a temporary variable, which will be cleared and reinitialized at the beginning of every loop iteration (hence unusable for your purpose).
So an easy locked variable may not be a concept we can implement simply at the moment. Some convoluted class event or file writing/checking may do the trick but I am afraid the timing would suffer greatly. If each iteration takes a long time to execute, the overhead might be worth it, but if you use parfoor to accelerate a high number of short execution iterations, then the convoluted solutions would slow you down more than help ...
You can have a look at this stack exchange question, you may find something of interest for your case: Semaphores and locks in MATLAB
The solution from Hoki is the right way to solve the problem as stated. However, as you asked about race conditions and preventing them when loop iterations depend on each other you might want to investigate spmd and the various lab* functions.
You need to use SPMD to do this - SPMD allows communication between the workers. Something like this:
bestResult = -Inf;
bestIndex = NaN;
N = 97;
spmd
% we need to round up the loop range to ensure that each
% worker executes the same number of iterations
loopRange = numlabs * ceil(N / numlabs);
for idx = 1:numlabs:loopRange
if idx <= N
local_result = rand(); % obviously replace this with your actual function
else
local_result = -Inf;
end
% Work out which index has the best result - use a really simple approach
% by concatenating all the results this time from each worker
% allResultsThisTime will be 2-by-numlabs where the first row is all the
% the results this time, and the second row is all the values of idx from this time
allResultsThisTime = gcat([local_result; idx]);
% The best result this time - consider the first row
[bestResultThisTime, labOfBestResult] = max(allResultsThisTime(1, :));
if bestResultThisTime > bestResult
bestResult = bestResultThisTime;
bestIndex = allResultsThisTime(2, labOfBestResult);
end
end
end
disp(bestResult{1})
disp(bestIndex{1})