I have a variable train_set and a cell array, R = cell(1,K) with K = 1000;
So I want to do something like this
new_array = cell(1,K);
parfor j = 1:K
new_array{j} = train_set * R{j};
end
But the thing is the variable R is considered a broadcast variable and therefore the entire array is loaded into each worker, instead of each R{j}. Of course, I can do something like making a new variable R_new where R_new(:,:,j) = R{j} but I'm afraid that would consume too much memory. Is there anything I can do to make R{j} sliced variable? Thank you very much :)
Related
I'm trying to speed up the simulation of some panel data in Matlab. I have to simulate first over individuals (loop index ii from 1 to N) and then for each individual over age (loop index jj from 1 to JJ). The code is slow because inside the two loops there is a bilinear interpolation to do.
Since the iterations in the outer loop are independent, I tried to use parfor in the outer loop (the loop indexed by ii), but I get the error message "the parfor cannot run due to the way the variable hsim is used". Could someone explain why and how to solve the problem if possible? Any help is greatly appreciated!
a_sim = zeros(Nsim,JJ);
h_sim = zeros(Nsim,JJ);
% Find point on a_grid corresponding to zero assets
aa0 = find_loc(a_grid,0.0);
% Zero housing
hh0 = 1;
a_sim(:,1) = a_grid(aa0);
h_sim(:,1) = h_grid(hh0);
parfor ii=1:Nsim !illegal
for jj=1:JJ-1
z_c = z_sim_ind(ii,jj);
apol_interp = griddedInterpolant({a_grid,h_grid},apol(:,:,z_c,jj));
hpol_interp = griddedInterpolant({a_grid,h_grid},hpol(:,:,z_c,jj));
a_sim(ii,jj+1) = apol_interp(a_sim(ii,jj),h_sim(ii,jj));
h_sim(ii,jj+1) = hpol_interp(a_sim(ii,jj),h_sim(ii,jj));
end
end
I think #Ben Voigt's suggestion was correct. To spell it out, do something like this:
parfor ii=1:Nsim
a_sim_row = a_sim(ii,:);
h_sim_row = h_sim(ii,:);
for jj=1:JJ-1
z_c = z_sim_ind(ii,jj);
apol_interp = griddedInterpolant({a_grid,h_grid},apol(:,:,z_c,jj));
hpol_interp = griddedInterpolant({a_grid,h_grid},hpol(:,:,z_c,jj));
a_sim_row(jj+1) = apol_interp(a_sim_row(jj),h_sim_row(jj));
h_sim_row(jj+1) = hpol_interp(a_sim_row(jj),h_sim_row(jj));
end
a_sim(ii,:) = a_sim_row;
h_sim(ii,:) = h_sim_row;
end
This is a fairly standard parfor pattern to work around the limitation (in this case, parfor cannot spot that what you're doing is not order-independent as far as the outer loop is concerned) - extract a whole slice, do whatever is needed, then put the whole slice back.
Suppose we are running an infinite for loop in MATLAB, and we want to store the iterative values in a vector. How can we declare the vector without knowing the size of it?
z=??
for i=1:inf
z(i,1)=i;
if(condition)%%condition is met then break out of the loop
break;
end;
end;
Please note first that this is bad practise, and you should preallocate where possible.
That being said, using the end keyword is the best option for extending arrays by a single element:
z = [];
for ii = 1:x
z(end+1, 1) = ii; % Index to the (end+1)th position, extending the array
end
You can also concatenate results from previous iterations, this tends to be slower since you have the assignment variable on both sides of the equals operator
z = [];
for ii = 1:x
z = [z; ii];
end
Sadar commented that directly indexing out of bounds (as other answers are suggesting) is depreciated by MathWorks, I'm not sure on a source for this.
If your condition computation is separate from the output computation, you could get the required size first
k = 0;
while ~condition
condition = true; % evaluate the condition here
k = k + 1;
end
z = zeros( k, 1 ); % now we can pre-allocate
for ii = 1:k
z(ii) = ii; % assign values
end
Depending on your use case you might not know the actual number of iterations and therefore vector elements, but you might know the maximum possible number of iterations. As said before, resizing a vector in each loop iteration could be a real performance bottleneck, you might consider something like this:
maxNumIterations = 12345;
myVector = zeros(maxNumIterations, 1);
for n = 1:maxNumIterations
myVector(n) = someFunctionReturningTheDesiredValue(n);
if(condition)
vecLength = n;
break;
end
end
% Resize the vector to the length that has actually been filled
myVector = myVector(1:vecLength);
By the way, I'd give you the advice to NOT getting used to use i as an index in Matlab programs as this will mask the imaginary unit i. I ran into some nasty bugs in complex calculations inside loops by doing so, so I would advise to just take n or any other letter of your choice as your go-to loop index variable name even if you are not dealing with complex values in your functions ;)
You can just declare an empty matrix with
z = []
This will create a 0x0 matrix which will resize when you write data to it.
In your case it will grow to a vector ix1.
Keep in mind that this is much slower than initializing your vector beforehand with the zeros(dim,dim) function.
So if there is any way to figure out the max value of i you should initialize it withz = zeros(i,1)
cheers,
Simon
You can initialize z to be an empty array, it'll expand automatically during looping ...something like:
z = [];
for i = 1:Inf
z(i) = i;
if (condition)
break;
end
end
However this looks nasty (and throws a warning: Warning: FOR loop index is too large. Truncating to 9223372036854775807), I would do here a while (true) or the condition itself and increment manually.
z = [];
i = 0;
while !condition
i=i+1;
z[i]=i;
end
And/or if your example is really what you need at the end, replace the re-creation of the array with something like:
while !condition
i=i+1;
end
z = 1:i;
As mentioned in various times in this thread the resizing of an array is very processing intensive, and could take a lot of time.
If processing time is not an issue:
Then something like #Wolfie mentioned would be good enough. In each iteration the array length will be increased and that is that:
z = [];
for ii = 1:x
%z = [z; ii];
z(end+1) = ii % Best way
end
If processing time is an issue:
If the processing time is a large factor, and you want it to run as smooth as possible, then you need to preallocating.If you have a rough idea of the maximum number of iterations that will run then you can use #PluginPenguin's suggestion. But there could still be a change of hitting that preset limit, which will break (or severely slow down) the program.
My suggestion:
If your loop is running infinitely until you stop it, you could do occasional resizing. Essentially extending the size as you go, but only doing it once in a while. For example every 100 loops:
z = zeros(100,1);
for i=1:inf
z(i,1)=i;
fprintf("%d,\t%d\n",i,length(z)); % See it working
if i+1 >= length(z) %The array as run out of space
%z = [z; zeros(100,1)]; % Extend this array (note the semi-colon)
z((length(z)+100),1) = 0; % Seems twice as fast as the commented method
end
if(condition)%%condition is met then break out of the loop
break;
end;
end
This means that the loop can run forever, the array will increase with it, but only every once in a while. This means that the processing time hit will be minimal.
Edit:
As #Cris kindly mentioned MATLAB already does what I proposed internally. This makes two of my comments completely wrong. So the best will be to follow what #Wolfie and #Cris said with:
z(end+1) = i
Hope this helps!
Suppose I have n .mat files and each are named as follows: a1, a2, ..., an
And within each of these mat files there is a variable called: var (nxn matrix)
I would like to create a matrix: A = [a1.var a2.var, ..., an.var] without writing it all out because there are many .mat files
A for-loop comes to mind, something like this:
A = []
for i = 1:n
[B] = ['a',num2str(i),'.mat',var];
A = [A B]
end
but this doesn't seem to work or even for the most simple case where I have variables that aren't stored as a(i) but rather 'a1', 'a2' etc.
Thank you very much!
load and concatenate 'var' from each of 'a(#).mat':
n = 10;
for i = n:-1:1 % 1
file_i = sprintf('a%d.mat', i); % 2
t = load(file_i, 'var');
varsCell{i} = t.var; % 3
end
A = [varsCell{:}]; % concatenate each 'var' in one step.
Here are some comment on the above code. All the memory-related stuff isn't very important here, but it's good to keep in mind during larger projects.
1)
In MATLAB, it is rarely a good idea or necessary to grow variables during a for loop. Each time an element is added, MATLAB must find and allocate a new block of RAM. This can really slow things down, especially for long loops or large variables. When possible, pre-allocate your variables (A = zeros(n,n*n)). Alternatively, it sometimes works to count backwards in the loop. MATLAB pre-allocates the whole array, since you're effectively telling it the final size.
2)
Equivalent to file_i = ['a',num2str(i),'.mat'] in this case, sprintf can be clearer and more powerful.
3)
Store each 'var' in a cell array. This is a balance between allocating all the needed memory and the complication of indexing into the correct places of a preallocated array. Internally, the cell array is a list of pointers to the location of each loaded 'var' matrix.
to create a test set...
generate 'n' matrices of n*n random doubles
save each as 'a(#).mat' in current directory
for i = 1:n
var = rand(n);
save(sprintf('a%d.mat',i), 'var');
end
Code
%%// The final result, A would have size nX(nXn)
A = zeros(n,n*n); %%// Pre-allocation for better performance
for k =1:n
load(strcat('a',num2str(k),'.mat'))
A(1:n,(k-1)*n+1:(k-1)*n+n) = var;
end
I have a loop which iterates for 97 times and there are two arrrays
frequency[1024]
strength[1024]
these arrays change values after each iteration of the loop. so before its value changes i need to put them in a structure.For instance the structure would be something like
s(1).frame=1 %this will show the iteration no.
s(1).str=strength
s(1).freq=frequency
now i need 97 such structures say s(1) to s(97) in an array.
my question is: how can I create an array of structures within my loop. Please help me.
I like to iterate backward in cases like this, as this forces a full memory allocation the first time the loop is executed. Then the code would look something like this:
%Reset the structure
s = struct;
for ix = 97:-1:1
%Do stuff
%Store the data
s(ix).frame = ix;
s(ix).str = strength;
s(ix).freq = frequency;
end
If one frame depends on the next, or you don't know how many total frame there will be, you can scan forwards. 97 frames is not a lot of data, so you probably don;t need to worry too much about optimizing the pre-allocation portion of the problem.
%Reset the structure
s = struct;
for ix = 1:97
%Do stuff
%Store the data
s(ix).frame = ix;
s(ix).str = strength;
s(ix).freq = frequency;
end
Or, if you really need to performance of a pre-allocated array of structures, but you don't know how large it will be at the onset, you can do something like this:
%Reset the structure
s = struct;
for ix = 1:97
%Do stuff
%Extend if needed
if length(s)<ix
s(ix*2).frame = nan; %Double allocation every time you reach the end.
end
%Store the data
s(ix).frame = ix;
s(ix).str = strength;
s(ix).freq = frequency;
end
%Clip extra allocation
s = s(1:ix);
I am refering to an example like this
I have a function to analize the elements of a vector, 'input'. If these elements have a special property I store their values in a vector, 'output'.
The problem is that at the begging I don´t know the number of elements it will need to store in 'output'so I don´t know its size.
I have a loop, inside I go around the vector, 'input' through an index. When I consider special some element of this vector capture the values of 'input' and It be stored in a vector 'ouput' through a sentence like this:
For i=1:N %Where N denotes the number of elements of 'input'
...
output(j) = input(i);
...
end
The problem is that I get an Error if I don´t previously "declare" 'output'. I don´t like to "declare" 'output' before reach the loop as output = input, because it store values from input in which I am not interested and I should think some way to remove all values I stored it that don´t are relevant to me.
Does anyone illuminate me about this issue?
Thank you.
How complicated is the logic in the for loop?
If it's simple, something like this would work:
output = input ( logic==true )
Alternatively, if the logic is complicated and you're dealing with big vectors, I would preallocate a vector that stores whether to save an element or not. Here is some example code:
N = length(input); %Where N denotes the number of elements of 'input'
saveInput = zeros(1,N); % create a vector of 0s
for i=1:N
...
if (input meets criteria)
saveInput(i) = 1;
end
end
output = input( saveInput==1 ); %only save elements worth saving
The trivial solution is:
% if input(i) meets your conditions
output = [output; input(i)]
Though I don't know if this has good performance or not
If N is not too big so that it would cause you memory problems, you can pre-assign output to a vector of the same size as input, and remove all useless elements at the end of the loop.
output = NaN(N,1);
for i=1:N
...
output(i) = input(i);
...
end
output(isnan(output)) = [];
There are two alternatives
If output would be too big if it was assigned the size of N, or if you didn't know the upper limit of the size of output, you can do the following
lengthOutput = 100;
output = NaN(lengthOutput,1);
counter = 1;
for i=1:N
...
output(counter) = input(i);
counter = counter + 1;
if counter > lengthOutput
%# append output if necessary by doubling its size
output = [output;NaN(lengthOutput,1)];
lengthOutput = length(output);
end
end
%# remove unused entries
output(counter:end) = [];
Finally, if N is small, it is perfectly fine to call
output = [];
for i=1:N
...
output = [output;input(i)];
...
end
Note that performance degrades dramatically if N becomes large (say >1000).