I am trying to make a for-loop for the Matlab code below. I have named each column with JAN90, FEB90, etc. all the way up to AUG19, which can be found in a matrix named "data". At this point I need to change the month and year manually to obtain the result I want. Is there a way to iterate over the columns by the column name? Would it be easier to name the columns Var1, Var2 etc.?
clear;
clc;
data = readtable('Data.xlsx','ReadVariableNames',false);
data(1,:) = [];
data.Var2 = str2double(data.Var2);
data.Var3 = str2double(data.Var3);
data.Var4 = str2double(data.Var4);
data.Var5 = str2double(data.Var5);
data.Var6 = str2double(data.Var6);
data.Var7 = str2double(data.Var7);
data.Var8 = str2double(data.Var8);
data.Var9 = str2double(data.Var9);
data.Var10 = str2double(data.Var10);
data.Var11 = str2double(data.Var11);
data.Var12 = str2double(data.Var12);
data.Var13 = str2double(data.Var13);
data(:,1) = [];
data = table2array(data);
data = array2table(data.');
data = table2cell(data)
data = cell2table(data, 'VariableNames',{'JAN90','FEB90','MAR90','APR90','MAY90','JUN90','JUL90','AUG90'...
,'SEP90','OCT90','NOV90','DEC90','JAN91','FEB91','MAR91','APR91','MAY91','JUN91','JUL91','AUG91'...
,'SEP91','OCT91','NOV91','DEC91','JAN92','FEB92','MAR92','APR92','MAY92','JUN92','JUL92','AUG92'...
,'SEP92','OCT92','NOV92','DEC92','JAN93','FEB93','MAR93','APR93','MAY93','JUN93','JUL93','AUG93'...
,'SEP93','OCT93','NOV93','DEC93','JAN94','FEB94','MAR94','APR94','MAY94','JUN94','JUL94','AUG94'...
,'SEP94','OCT94','NOV94','DEC94','JAN95','FEB95','MAR95','APR95','MAY95','JUN95','JUL95','AUG95'...
,'SEP95','OCT95','NOV95','DEC95','JAN96','FEB96','MAR96','APR96','MAY96','JUN96','JUL96','AUG96'...
,'SEP96','OCT96','NOV96','DEC96','JAN97','FEB97','MAR97','APR97','MAY97','JUN97','JUL97','AUG97'...
,'SEP97','OCT97','NOV97','DEC97','JAN98','FEB98','MAR98','APR98','MAY98','JUN98','JUL98','AUG98'...
,'SEP98','OCT98','NOV98','DEC98','JAN99','FEB99','MAR99','APR99','MAY99','JUN99','JUL99','AUG99'...
,'SEP99','OCT99','NOV99','DEC99','JAN00','FEB00','MAR00','APR00','MAY00','JUN00','JUL00','AUG00'...
,'SEP00','OCT00','NOV00','DEC00','JAN01','FEB01','MAR01','APR01','MAY01','JUN01','JUL01','AUG01'...
,'SEP01','OCT01','NOV01','DEC01','JAN02','FEB02','MAR02','APR02','MAY02','JUN02','JUL02','AUG02'...
,'SEP02','OCT02','NOV02','DEC02','JAN03','FEB03','MAR03','APR03','MAY03','JUN03','JUL03','AUG03'...
,'SEP03','OCT03','NOV03','DEC03','JAN04','FEB04','MAR04','APR04','MAY04','JUN04','JUL04','AUG04'...
,'SEP04','OCT04','NOV04','DEC04','JAN05','FEB05','MAR05','APR05','MAY05','JUN05','JUL05','AUG05'...
,'SEP05','OCT05','NOV05','DEC05','JAN06','FEB06','MAR06','APR06','MAY06','JUN06','JUL06','AUG06'...
,'SEP06','OCT06','NOV06','DEC06','JAN07','FEB07','MAR07','APR07','MAY07','JUN07','JUL07','AUG07'...
,'SEP07','OCT07','NOV07','DEC07','JAN08','FEB08','MAR08','APR08','MAY08','JUN08','JUL08','AUG08'...
,'SEP08','OCT08','NOV08','DEC08','JAN09','FEB09','MAR09','APR09','MAY09','JUN09','JUL09','AUG09'...
,'SEP09','OCT09','NOV09','DEC09','JAN10','FEB10','MAR10','APR10','MAY10','JUN10','JUL10','AUG10'...
,'SEP10','OCT10','NOV10','DEC10','JAN11','FEB11','MAR11','APR11','MAY11','JUN11','JUL11','AUG11'...
,'SEP11','OCT11','NOV11','DEC11','JAN12','FEB12','MAR12','APR12','MAY12','JUN12','JUL12','AUG12'...
,'SEP12','OCT12','NOV12','DEC12','JAN13','FEB13','MAR13','APR13','MAY13','JUN13','JUL13','AUG13'...
,'SEP13','OCT13','NOV13','DEC13','JAN14','FEB14','MAR14','APR14','MAY14','JUN14','JUL14','AUG14'...
,'SEP14','OCT14','NOV14','DEC14','JAN15','FEB15','MAR15','APR15','MAY15','JUN15','JUL15','AUG15'...
,'SEP15','OCT15','NOV15','DEC15','JAN16','FEB16','MAR16','APR16','MAY16','JUN16','JUL16','AUG16'...
,'SEP16','OCT16','NOV16','DEC16','JAN17','FEB17','MAR17','APR17','MAY17','JUN17','JUL17','AUG17'...
,'SEP17','OCT17','NOV17','DEC17','JAN18','FEB18','MAR18','APR18','MAY18','JUN18','JUL18','AUG18'...
,'SEP18','OCT18','NOV18','DEC18','JAN19','FEB19','MAR19','APR19','MAY19','JUN19','JUL19','AUG19'});
m = [1 2 3 6 12 24 36 60 84 120 240 360]';
for i=1:100
t = i;
data.X_1 = (1-exp(-m./t))./(m./t);
data.X_2 = ((1-exp(-m./t))./(m./t))-exp(-m./t);
model_1 = fitlm(data, 'FEB95 ~ X_1 + X_2');
RSS(100,:) = zeros ;
res = model_1.Residuals.Raw;
res(any(isnan(res), 2), :) = [];
RSS(i) = sum(res.^2);
end
RSS(:,2) = [1:1:100];
min = min(RSS(:,1));
t = find(RSS(:,1) == min)
data.X_1 = (1-exp(-m./t))./(m./t);
data.X_2 = ((1-exp(-m./t))./(m./t))-exp(-m./t);
model_1 = fitlm(data, 'FEB95 ~ X_1 + X_2')
res = model_1.Residuals.Raw;
res(any(isnan(res), 2), :) = [];
RSS = sum(res.^2)
intercept = model_1.Coefficients.Estimate(1,1);
beta_1 = model_1.Coefficients.Estimate(2,1);
beta_2 = model_1.Coefficients.Estimate(3,1);
Yhat = intercept + beta_1.*data.X_1 + beta_2.*data.X_2;
plot(m, Yhat)
hold on
scatter(m, data.FEB95)
I.e "FEB95" should be dynamic? Any suggestions?
Her is how I would approach your problem. First realize that VarNames=data.Properties.VariableNames will get a list of all column names in the table. You could then loop over this list. For example
for v=VarNames
current_column = v{1};
% ....
% Define the model spec for the current column
model_spec = [current_column ' ~ X_1 + X_2'];
% and create the model
model_1 = fitlm(data, model_spec);
% ... Continue computation ... and collect results in a table or array
end
Given that I have a model that can be expressed as:
y = a + b*st + c*d2
where st is a smoothed version of some data, and a, b and c are model coffieicients that are unknown. An iterative process should be used to find the best values for a, b, and c and also an additional value alpha, shown below.
Here, I show an example using some data that I have. I'll only show a small fraction of the data here to get an idea of what I have:
17.1003710350253 16.7250000000000 681.521316544969
17.0325989276234 18.0540000000000 676.656460644882
17.0113862864815 16.2460000000000 671.738125420192
16.8744356336601 15.1580000000000 666.767363772145
16.5537077980594 12.8830000000000 661.739644621949
16.0646524243248 10.4710000000000 656.656219934146
15.5904357723302 9.35000000000000 651.523986525985
15.2894427136087 12.4580000000000 646.344231349275
15.1181450512182 9.68700000000000 641.118300709434
15.0074128442766 10.4080000000000 635.847600747838
14.9330905954828 11.5330000000000 630.533597865332
14.8201069920058 10.6830000000000 625.177819082427
16.3126863409751 15.9610000000000 619.781852331734
16.2700386755872 16.3580000000000 614.347346678083
15.8072873786912 10.8300000000000 608.876012461843
15.3788908036751 7.55000000000000 603.369621360944
15.0694302370038 13.1960000000000 597.830006367160
14.6313314652840 8.36200000000000 592.259061672302
14.2479738025295 9.03000000000000 586.658742460043
13.8147156115234 5.29100000000000 581.031064599264
13.5384821473624 7.22100000000000 575.378104234926
13.3603543306796 8.22900000000000 569.701997272687
13.2469020140965 9.07300000000000 564.004938753678
13.2064193251406 12.0920000000000 558.289182116093
13.1513460035983 12.2040000000000 552.557038340513
12.8747853506079 4.46200000000000 546.810874976187
12.5948999131388 4.61200000000000 541.053115045791
12.3969691298003 6.83300000000000 535.286235826545
12.1145822760120 2.43800000000000 529.512767505944
11.9541188991626 2.46700000000000 523.735291710730
11.7457790927936 4.15000000000000 517.956439908176
11.5202981254529 4.47000000000000 512.178891679167
11.2824263926694 2.62100000000000 506.405372863054
11.0981930749608 2.50000000000000 500.638653574697
10.8686514170776 1.66300000000000 494.881546094641
10.7122053911554 1.68800000000000 489.136902633882
10.6255883267131 2.48800000000000 483.407612975178
10.4979083986908 4.65800000000000 477.696601993434
10.3598092538338 4.81700000000000 472.006827058220
10.1929490084608 2.46700000000000 466.341275322034
10.1367069580204 2.36700000000000 460.702960898512
10.0194072271384 4.87800000000000 455.094921935306
9.88627023967911 3.53700000000000 449.520217586971
9.69091601129389 0.417000000000000 443.981924893704
9.48684595125235 -0.567000000000000 438.483135572389
9.30742664359900 0.892000000000000 433.026952726910
9.18283037670750 1.50000000000000 427.616487485241
9.02385722622626 1.75800000000000 422.254855571341
8.90355705229410 2.46700000000000 416.945173820367
8.76138912769045 1.99200000000000 411.690556646207
8.61299614111510 0.463000000000000 406.494112470755
8.56293606861698 6.55000000000000 401.358940124780
8.47831879772002 4.65000000000000 396.288125230599
8.42736865902327 6.45000000000000 391.284736577104
8.26325535934842 -1.37900000000000 386.351822497948
8.14547793724500 1.37900000000000 381.492407263967
8.00075641792910 -1.03700000000000 376.709487501030
7.83932517791044 -1.66700000000000 372.006028644665
7.68389447250257 -4.12900000000000 367.384961442799
7.63402151555169 -2.57900000000000 362.849178517935
The results that follow probably won't be meaningful as the full data would be needed (but this is an example). Using this data I have tried to solve iteratively by
y = d(:,1);
d1 = d(:,2);
d2 = d(:,3);
alpha_o = linspace(0.01,1,10);
a = linspace(0.01,1,10);
b = linspace(0.01,1,10);
c = linspace(0.01,1,10);
defining different values for a, b, and c as well as another term alpha, which is used in the model, and am now going to find every possible combination of these parameters and see which combination provides the best fit to the data:
% every possible combination of values
xx = combvec(alpha_o,a,b,c);
% loop through each possible combination of values
for j = 1:size(xx,2);
alpha_o = xx(1,j);
a_o = xx(2,j);
b_o = xx(3,j);
c_o = xx(4,j);
st = d1(1);
for i = 2:length(d1);
st(i) = alpha_o.*d1(i) + (1-alpha_o).*st(i-1);
end
st = st(:);
y_pred = a_o + (b_o*st) + (c_o*d2);
mae(j) = nanmean(abs(y - y_pred));
end
I can then re-run the model using these optimum values:
[id1,id2] = min(mae);
alpha_opt = xx(:,id2);
st = d1(1);
for i = 2:length(d1);
st(i) = alpha_opt(1).*d1(i) + (1-alpha_opt(1)).*st(i-1);
end
st = st(:);
y_pred = alpha_opt(2) + (alpha_opt(3)*st) + (alpha_opt(4)*d2);
mae_final = nanmean(abs(y - y_pred));
However, to reach a final answer I would need to increase the number of initial guesses to more than 10 for each variable. This will take a long time to run. Thereofre, I am wondering if there is a better method for what I am trying to do here? Any advice is appreciated.
Here's some thoughts: If you could decrease the amount of computation within each for loop, you could possibly speed it up. One possible way is to look for common factors between each loop and move it outside for loop:
If you look at the iteration, you'll see
st(1) = d1(1)
st(2) = a * d1(2) + (1-a) * st(1) = a *d1(2) + (1-a)*d1(1)
st(3) = a * d1(3) + (1-a) * st(2) = a * d1(3) + a *(1-a)*d1(2) +(1-a)^2 * d1(1)
st(n) = a * d1(n) + a *(1-a)*d1(n-1) + a *(1-a)^2 * d1(n-2) + ... +(1-a)^(n-1)*d1(1)
Which means st can be calculated by multiplying these two matrices (here I use n=4 for example to illustrate the concept) and sum along the first dimension:
temp1 = [ 0 0 0 a ;
0 0 a a(1-a) ;
0 a a(1-a) a(1-a)^2 ;
1 (1-a) (1-a)^2 (1-a)^3 ;]
temp2 = [ 0 0 0 d1(4) ;
0 0 d1(3) d1(3) ;
0 d1(2) d1(2) d1(2) ;
d1(1) d1(1) d1(1) d1(1) ;]
st = sum(temp1.*temp2,1)
Here's codes that utilize this concept: Computation has been moved out of the inner for loop and only assignment is left.
alpha_o = linspace(0.01,1,10);
xx = nchoosek(alpha_o, 4);
n = size(d1,1);
matrix_d1 = zeros(n, n);
d2 = d2'; % To make the dimension of d2 and st the same.
for ii = 1:n
matrix_d1(n-ii+1:n, ii) = d1(1:ii);
end
st = zeros(size(d1)'); % Pre-allocation of matrix will improve speed.
mae = zeros(1,size(xx,1));
matrix_alpha = zeros(n, n);
for j = 1 : size(xx,1)
alpha_o = xx(j,1);
temp = (power(1-alpha_o, [0:n-1])*alpha_o)';
matrix_alpha(n,:) = power(1-alpha_o, [0:n-1]);
for ii = 2:n
matrix_alpha(n-ii+1:n-1, ii) = temp(1:ii-1);
end
st = sum(matrix_d1.*matrix_alpha, 1);
y_pred = xx(j,2) + xx(j,3)*st + xx(j,4)*d2;
mae(j) = nanmean(abs(y - y_pred));
end
Then :
idx = find(min(mae));
alpha_opt = xx(idx,:);
st = zeros(size(d1)');
temp = (power(1-alpha_opt(1), [0:n-1])*alpha_opt(1))';
matrix_alpha = zeros(n, n);
matrix_alpha(n,:) = power(1-alpha_opt(1), [0:n-1]);;
for ii = 2:n
matrix_alpha(n-ii+1:n-1, ii) = temp(1:ii-1);
end
st = sum(matrix_d1.*matrix_alpha, 1);
y_pred = alpha_opt(2) + (alpha_opt(3)*st) + (alpha_opt(4)*d2);
mae_final = nanmean(abs(y - y_pred));
Let me know if this helps !
I would like to vectorize these two lines of code. I just recently learned about vectorization. I know how to vectorize the sumsurface line but I am not sure how to include the if statement, I would really like to vectorize the whole for loop and get rid of it. I want to vectorize to improve runtime the code I have right now runs very slow. I preallocated the arrays which helps improve runtime. I had forgotten to do that previously. If I could get any help that would be much appreciated.
pH = linspace(2,12, 6000);
for j = 1:300
nAsp = randi([10, 30],[1,1]);%865
nGlu = randi([12, 38],[1,1]);%1074
nLys = randi([11, 33],[1,1]);%930
nArg = randi([10, 30],[1,1]);%879
nCys = randi([2, 8],[1,1]); %214
nTyr = randi([5, 17],[1,1]); %462
nHis = randi([4, 12],[1,1]); %360
for i = 1: len;
sumsurface(i) = (nAsp).*(-(10.^((pH(i)-asp) )./(10.^((pH(i)-asp) )+1)) )+ (nGlu).*(-(10.^((pH(i)-glu) )./(10.^((pH(i)-glu) )+1)))+(nCys).*(-(10.^((pH(i)-cys) )./(10.^((pH(i)-cys) )+1)))+ (nTyr).* (-(10.^((pH(i)-tyr) )./(10.^((pH(i)-tyr) )+1)))+ (nHis).*(1./(10.^((pH(i)-his) )+1))+ (nLys).*(1./(10.^((pH(i)-lys) )+1))+ (nArg).*(1/(10.^((pH(i)-arg) )+1));
if sumsurface(i) < .01 && sumsurface(i) > -.01
%disp(sumsurface(i));
disp(pH(i));
x(1+end) = pH(i);
aspl(1+end) = nAsp;
glul(1+end) = nGlu;
cysl(1+end) = nCys;
tyrl(1+end) = nTyr;
hisl(1+end) = nHis;
lysl(1+end) = nLys;
argl(1+end) = nArg;
end
end
end
You can vectorize the whole algorithm. I'm not going to code it all out for you but here are some pointers to get you started:
Use REPMAT to create an array that contains as many copies of pH as there are iterations, i.e. len.
Change all those variables beginning with n from scalars to vectors. For example, nAsp = randi([10, 30], [len, 1])
Use FIND to determine the indices of sumsurface that match your criteria, i.e. index = find(sumsurface < 0.01 & sumsurface > -0.01);.
Create your desired vectors using index, e.g. aspl = nAsp(index);
Rinse. Repeat.
Here is one possible vectorization:
%# data
len = 6000;
pH = linspace(2,12, len);
%# some constants (fill your values here)
asp = 0; glu = 0; cys = 0; tyr = 0; his = 0; lys = 0; arg = 0;
%# random parameters for each iteration
num = 300;
nAsp = randi([10 30], [num 1]);
nGlu = randi([12 38], [num 1]);
nLys = randi([11 33], [num 1]);
nArg = randi([10 30], [num 1]);
nCys = randi([2 8] , [num 1]);
nTyr = randi([5 17] , [num 1]);
nHis = randi([4 12] , [num 1]);
params = [nAsp nGlu nCys nTyr nHis nLys nArg];
M = [
- 10.^(pH-asp) ./ (1 + 10.^(pH-asp))
- 10.^(pH-glu) ./ (1 + 10.^(pH-glu))
- 10.^(pH-cys) ./ (1 + 10.^(pH-cys))
- 10.^(pH-tyr) ./ (1 + 10.^(pH-tyr))
1 ./ (1 + 10.^(pH-his))
1 ./ (1 + 10.^(pH-lys))
1 ./ (1 + 10.^(pH-arg))
];
%# iterations
sumsurface = zeros(num,len);
x = cell(num,1); p = cell(num,1);
for j=1:num
sumsurface(j,:) = params(j,:)*M;
idx = abs(sumsurface(j,:)) < 0.01;
if any(idx)
x{j} = pH(idx);
p{j} = params(j,:); %# [aspl glul cysl tyrl hisl lysl argl]
end
end
After running the code, the cell-arrays x and p will contain, for each iteration, the pH and params respectively that satisfy your equation: -0.01<sumsurface<0.01 (if they exist).