Related
I have a training function, in which inside there are two vectors:
d_labels_a = torch.zeros(128)
d_labels_b = torch.ones(128)
Then I have these features:
# Compute output
features_a = nets[0](input_a)
features_b = nets[1](input_b)
features_c = nets[2](inputs)
And then a domain classifier (nets[4]) makes predictions:
d_pred_a = torch.squeeze(nets[4](features_a))
d_pred_b = torch.squeeze(nets[4](features_b))
d_pred_a = d_pred_a.float()
d_pred_b = d_pred_b.float()
print(d_pred_a.shape)
The error raises in the loss function: ` pred_a = torch.squeeze(nets3)
pred_b = torch.squeeze(nets3)
pred_c = torch.squeeze(nets3)
loss = criterion(pred_a, labels_a) + criterion(pred_b, labels_b) + criterion(pred_c, labels) + d_criterion(d_pred_a, d_labels_a) + d_criterion(d_pred_b, d_labels_b)
The problem is that d_pred_a/b is different from d_labels_a/b, but only after a certain point. Indeed, when I print the shape of d_pred_a/b it istorch.Size([128])but then it changes totorch.Size([112])` independently.
It comes from here:
# Compute output
features_a = nets[0](input_a)
features_b = nets[1](input_b)
features_c = nets[2](inputs)
because if I print the shape of features_a is torch.Size([128, 2048]) but it changes into torch.Size([112, 2048])
nets[0] is a VGG, like this:
class VGG16(nn.Module):
def __init__(self, input_size, batch_norm=False):
super(VGG16, self).__init__()
self.in_channels,self.in_width,self.in_height = input_size
self.block_1 = VGGBlock(self.in_channels,64,batch_norm=batch_norm)
self.block_2 = VGGBlock(64, 128,batch_norm=batch_norm)
self.block_3 = VGGBlock(128, 256,batch_norm=batch_norm)
self.block_4 = VGGBlock(256,512,batch_norm=batch_norm)
#property
def input_size(self):
return self.in_channels,self.in_width,self.in_height
def forward(self, x):
x = self.block_1(x)
x = self.block_2(x)
x = self.block_3(x)
x = self.block_4(x)
# x = self.avgpool(x)
x = torch.flatten(x,1)
return x
I solved. The problem was the last batch. I used drop_last=True in the dataloader and It worked.
I have two symbolic expressions a and b, each consists of polynomials with basic arithmetic and small, positive, integer powers.
simplify(a - b) doesn't go up to 0, and my only alternative is to subs some random numbers into the variables and compare.
I would have expected something like expanding the expressions until there are no parentheses. Then, add all fractions into a single fraction.
I converted the data into a function which can be called as:
x = sym('x', [1 8], 'real')';
err = func( x ) % should be simplified to zeros
x0 = rand( size(x) )
double( subs(err, x, x0) )
simplify(err)
The function
function err_Dpsi_Dpsi2 = func(in1)
%FUNC
% ERR_DPSI_DPSI2 = FUNC(IN1)
% This function was generated by the Symbolic Math Toolbox version 8.4.
% 29-Dec-2020 20:03:34
x1 = in1(1,:);
x2 = in1(2,:);
x3 = in1(3,:);
x4 = in1(4,:);
x5 = in1(5,:);
x6 = in1(6,:);
x7 = in1(7,:);
x8 = in1(8,:);
t2 = x1.*x6;
t3 = x2.*x5;
t4 = x1.*x7;
t5 = x3.*x5;
t6 = x2.*x7;
t7 = x3.*x6;
t8 = -x2;
t9 = -x3;
t10 = -x6;
t11 = -x7;
t15 = x1./2.0;
t16 = x2./2.0;
t17 = x1./4.0;
t18 = x3./2.0;
t19 = x2./4.0;
t20 = x3./4.0;
t21 = x5./2.0;
t22 = x6./2.0;
t23 = x5./4.0;
t24 = x7./2.0;
t25 = x6./4.0;
t26 = x7./4.0;
t43 = x2.*7.072e+3;
t44 = x3.*7.072e+3;
t45 = x4.*7.071e+3;
t46 = x6.*7.072e+3;
t47 = x7.*7.072e+3;
t48 = x8.*7.071e+3;
t60 = x2.*x8.*-7.071e+3;
t62 = x4.*x7.*-7.071e+3;
t69 = x1.*9.999907193999999e-1;
t70 = x5.*9.999907193999999e-1;
t71 = x1.*1.0000660704;
t72 = x5.*1.0000660704;
t74 = x2.*1.0001321408;
t75 = x3.*1.0001321408;
t76 = x6.*1.0001321408;
t77 = x7.*1.0001321408;
t78 = x1.*1.0000660704;
t79 = x2.*5.000660704e-1;
t80 = x2.*1.0001321408;
t81 = x3.*5.000660704e-1;
t82 = x3.*1.0001321408;
t83 = x5.*1.0000660704;
t84 = x6.*5.000660704e-1;
t85 = x6.*1.0001321408;
t86 = x7.*5.000660704e-1;
t87 = x7.*1.0001321408;
t102 = x1.*9.999907194000001e-1;
t103 = x5.*9.999907194000001e-1;
t104 = x4.*4.999953597e-1;
t105 = x8.*4.999953597e-1;
t108 = x2.*1.000132149530596;
t109 = x3.*1.000132149530596;
t110 = x6.*1.000132149530596;
t111 = x7.*1.000132149530596;
t112 = x2.*1.000056789186827;
t113 = x3.*1.000056789186827;
t114 = x6.*1.000056789186827;
t115 = x7.*1.000056789186827;
t124 = x4.*1.000056789186827;
t125 = x8.*1.000056789186827;
t126 = x4.*9.999814388861295e-1;
t127 = x8.*9.999814388861295e-1;
t128 = x2.*1.000132149530596;
t129 = x3.*1.000132149530596;
t130 = x6.*1.000132149530596;
t131 = x7.*1.000132149530596;
t139 = x4.*2.500307147434136e-1;
t140 = x8.*2.500307147434136e-1;
t141 = x2.*1.000056789186827;
t142 = x3.*1.000056789186827;
t144 = x4.*1.000056789186827;
t145 = x6.*1.000056789186827;
t146 = x7.*1.000056789186827;
t148 = x8.*1.000056789186827;
t157 = x2.*x8.*(-2.500307147434136e-1);
t158 = x4.*x7.*(-2.500307147434136e-1);
t159 = x4.*9.999814388861297e-1;
t160 = x8.*9.999814388861297e-1;
t12 = -t3;
t13 = -t4;
t14 = -t7;
t27 = t2./4.0;
t28 = t3./4.0;
t29 = t4./4.0;
t30 = t5./4.0;
t31 = t6./4.0;
t32 = t7./4.0;
t33 = t8+x1;
t34 = t9+x1;
t35 = t10+x5;
t36 = t11+x5;
t37 = -t16;
t38 = -t18;
t39 = -t20;
t40 = -t22;
t41 = -t24;
t42 = -t26;
t52 = t6.*7.072e+3;
t53 = t48.*x2;
t54 = t7.*7.072e+3;
t55 = t45.*x6;
t56 = t48.*x3;
t57 = t45.*x7;
t58 = -t45;
t59 = -t48;
t88 = -t74;
t89 = -t75;
t90 = -t76;
t91 = -t77;
t92 = -t80;
t93 = -t79;
t94 = -t82;
t95 = -t81;
t96 = -t85;
t97 = -t84;
t98 = -t87;
t99 = -t86;
t116 = -t108;
t117 = -t109;
t118 = -t110;
t119 = -t111;
t120 = -t112;
t121 = -t113;
t122 = -t114;
t123 = -t115;
t132 = -t128;
t133 = -t129;
t134 = -t130;
t135 = -t131;
t136 = t6.*2.500660747652978e-1;
t137 = t7.*2.500660747652978e-1;
t143 = -t139;
t147 = -t140;
t149 = t140.*x2;
t150 = t139.*x6;
t151 = t140.*x3;
t152 = t139.*x7;
t153 = -t141;
t154 = -t142;
t155 = -t145;
t156 = -t146;
t49 = -t28;
t50 = -t29;
t51 = -t32;
t61 = -t54;
t63 = t43+t58;
t64 = t44+t58;
t65 = t46+t59;
t66 = t47+t59;
t67 = t2+t5+t6+t12+t13+t14;
t138 = -t137;
t161 = t15+t38+t93+t104;
t162 = t15+t37+t95+t104;
t163 = t21+t41+t97+t105;
t164 = t21+t40+t99+t105;
t169 = t71+t89+t116+t124;
t170 = t71+t88+t117+t124;
t171 = t72+t91+t118+t125;
t172 = t72+t90+t119+t125;
t173 = t78+t92+t133+t144;
t174 = t78+t94+t132+t144;
t175 = t83+t96+t135+t148;
t176 = t83+t98+t134+t148;
t177 = t69+t120+t121+t126;
t178 = t70+t122+t123+t127;
t179 = t102+t153+t154+t159;
t180 = t103+t155+t156+t160;
t68 = 1.0./t67;
t73 = t27+t30+t31+t49+t50+t51;
t106 = t52+t55+t56+t60+t61+t62;
t165 = t161.^2;
t166 = t162.^2;
t167 = t163.^2;
t168 = t164.^2;
t182 = t136+t138+t150+t151+t157+t158;
t100 = 1.0./t73;
t107 = 1.0./t106;
t181 = t165+t166+t167+t168;
t183 = 1.0./t182;
t101 = t100.^2;
t184 = t183.^2;
err_Dpsi_Dpsi2 = [t181.*(t35.*t68.*t100-t36.*t68.*t100)+t101.*t181.*(t25+t42),-t100.*t169+t100.*t174+t169.*t183-t174.*t183-t101.*t181.*(t23+t42)+t181.*t184.*(t140-x7.*2.500660747652978e-1)+t36.*t68.*t100.*t181+t66.*t107.*t181.*t183.*1.0,-t100.*t170+t100.*t173+t170.*t183-t173.*t183-t181.*t184.*(t140-x6.*2.500660747652978e-1)+t101.*t181.*(t23-t25)-t35.*t68.*t100.*t181-t65.*t107.*t181.*t183.*1.0,t100.*t177-t100.*t179-t177.*t183+t179.*t183+t181.*(t65.*t107.*t183.*9.998585972850678e-1-t66.*t107.*t183.*9.998585972850678e-1)-t181.*t184.*(x6.*2.500307147434136e-1-x7.*2.500307147434136e-1),-t181.*(t33.*t68.*t100-t34.*t68.*t100)-t101.*t181.*(t19+t39),-t100.*t171+t100.*t176+t171.*t183-t176.*t183+t101.*t181.*(t17+t39)-t181.*t184.*(t139-x3.*2.500660747652978e-1)-t34.*t68.*t100.*t181-t64.*t107.*t181.*t183.*1.0,-t100.*t172+t100.*t175+t172.*t183-t175.*t183+t181.*t184.*(t139-x2.*2.500660747652978e-1)-t101.*t181.*(t17-t19)+t33.*t68.*t100.*t181+t63.*t107.*t181.*t183.*1.0,t100.*t178-t100.*t180-t178.*t183+t180.*t183-t181.*(t63.*t107.*t183.*9.998585972850678e-1-t64.*t107.*t183.*9.998585972850678e-1)+t181.*t184.*(x2.*2.500307147434136e-1-x3.*2.500307147434136e-1)];
I found what I was looking for
num = numden( err ) % convert to rational polynomial, and we care only about the numerator
collect( num ) % cancel terms--not needed, numden does some version of simplify
My example, though, wasn't good. For some reason, there are precision issues. I thought that symbolic used exact arithmetic, but I didn't look into that. However, if I use variables instead of finite precision coefficients, then it outputs zeros.
i want to optimize my code, and I would like to get a loop for my netcdf files. These are the files I want to open and the variables I want to use:
hydrofile2 = 'C:\modelana\netcdf_2019\westcoms2_20190402_0002.nc';
hydrofile3 = 'C:\modelana\netcdf_2019\westcoms2_20190403_0003.nc';
hydrofile4 = 'C:\modelana\netcdf_2019\westcoms2_20190404_0004.nc';
hydrofile5 = 'C:\modelana\netcdf_2019\westcoms2_20190405_0005.nc';
hydrofile6 = 'C:\modelana\netcdf_2019\westcoms2_20190406_0006.nc';
hydrofile7 = 'C:\modelana\netcdf_2019\westcoms2_20190407_0007.nc';
hydrofile8 = 'C:\modelana\netcdf_2019\westcoms2_20190408_0008.nc';
hydrofile9 = 'C:\modelana\netcdf_2019\westcoms2_20190409_0009.nc';
hydrofile10 = 'C:\modelana\netcdf_2019\westcoms2_20190410_0010.nc';
hydrofile11 = 'C:\modelana\netcdf_2019\westcoms2_20190411_0011.nc';
hydrofile12 = 'C:\modelana\netcdf_2019\westcoms2_20190412_0012.nc';
hydrofile13 = 'C:\modelana\netcdf_2019\westcoms2_20190413_0013.nc';
hydrofile14 = 'C:\modelana\netcdf_2019\westcoms2_20190414_0014.nc';
hydrofile15 = 'C:\modelana\netcdf_2019\westcoms2_20190415_0015.nc';
%velocity
u2 = ncread(hydrofile2,'u');
u3 = ncread(hydrofile3,'u');
u4 = ncread(hydrofile4,'u');
u5 = ncread(hydrofile5,'u');
u6 = ncread(hydrofile6,'u');
u7 = ncread(hydrofile7,'u');
u8 = ncread(hydrofile8,'u');
u9 = ncread(hydrofile9,'u');
u10 = ncread(hydrofile10,'u');
u11 = ncread(hydrofile11,'u');
u12 = ncread(hydrofile12,'u');
u13 = ncread(hydrofile13,'u');
u14 = ncread(hydrofile14,'u');
u15 = ncread(hydrofile15,'u');
%salinity
s2 = ncread(hydrofile2,'salinity');
s3 = ncread(hydrofile3,'salinity');
s4 = ncread(hydrofile4,'salinity');
s5 = ncread(hydrofile5,'salinity');
s6 = ncread(hydrofile6,'salinity');
s7 = ncread(hydrofile7,'salinity');
s8 = ncread(hydrofile8,'salinity');
s9 = ncread(hydrofile9,'salinity');
s10 = ncread(hydrofile10,'salinity');
s11 = ncread(hydrofile11,'salinity');
s12 = ncread(hydrofile12,'salinity');
s13 = ncread(hydrofile13,'salinity');
s14 = ncread(hydrofile14,'salinity');
s15 = ncread(hydrofile15,'salinity');
%temperature
t2 = ncread(hydrofile2,'temp');
t3 = ncread(hydrofile3,'temp');
t4 = ncread(hydrofile4,'temp');
t5 = ncread(hydrofile5,'temp');
t6 = ncread(hydrofile6,'temp');
t7 = ncread(hydrofile7,'temp');
t8 = ncread(hydrofile8,'temp');
t9 = ncread(hydrofile9,'temp');
t10 = ncread(hydrofile10,'temp');
t11 = ncread(hydrofile11,'temp');
t12 = ncread(hydrofile12,'temp');
t13 = ncread(hydrofile13,'temp');
t14 = ncread(hydrofile14,'temp');
t15 = ncread(hydrofile15,'temp');
So far, I've just been able to do this:
%% add hydrofiles
clear
myFolder = ('C:\modelana\netcdf_2019\');
if ~isfolder(myFolder)
errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
uiwait(warndlg(errorMessage));
return;
end
for k = 1:14
ncFilename = sprintf('westcoms2_20190402_0002.nc',k);
if isempty(ncFilename == 1)
continue;
else
ncfile=([myFolder ncFilename]);
s = ncread(ncfile,'salinity') ;
t = ncread(ncfile,'temp') ;
u=ncread(ncfile,'u');
end
end
So I just get one .nc file open with its corresponding variables. But what I want is to be able to get all the files and the variables by using the loop.
Thanks and apologize for this basic question
You almost have it, you are missing basic string concatenation/file reading.
You can concatenate strings as [str1, str2, 'random charaters']. e.g. you can do
['westcoms2_2019040', num2str(k), '_000', num2str(k) ,'.nc']
This you can do if you have obvious file formats and some constrains. If you want to read all of them, you can do
fileList = dir([myFolder '*.nc']);
To just get all files with that extension.
Also, remember to store variables independently. i.e.
t(k,:) = ncread(ncfile,'temp') ; % if it is an array
t{k} = ncread(ncfile,'temp') ; % if it is some complex structure
I have a nested cell of cells like the one below:
CellArray={1,1,1,{1,1,1,{1,1,{1,{1 1 1 1 1 1 1 1}, 1,1},1,1},1,1,1},1,1,1,{1,1,1,1}};
I need to randomly pick a location in CellArray. All members' locations of CellArray must have same chances to be chosen in the random selection process. Thanks.
You can capture the output of the celldisp function. Then use regex to extrcat indices:
s=evalc('celldisp(CellArray,'''')');
m = regexp(s, '\{[^\=]*\}', 'match');
Thanks to #excaza that suggested a clearer use of regexp
Result:
m =
{
[1,1] = {1}
[1,2] = {2}
[1,3] = {3}
[1,4] = {4}{1}
[1,5] = {4}{2}
[1,6] = {4}{3}
[1,7] = {4}{4}{1}
[1,8] = {4}{4}{2}
[1,9] = {4}{4}{3}{1}
[1,10] = {4}{4}{3}{2}{1}
[1,11] = {4}{4}{3}{2}{2}
[1,12] = {4}{4}{3}{2}{3}
[1,13] = {4}{4}{3}{2}{4}
[1,14] = {4}{4}{3}{2}{5}
[1,15] = {4}{4}{3}{2}{6}
[1,16] = {4}{4}{3}{2}{7}
[1,17] = {4}{4}{3}{2}{8}
[1,18] = {4}{4}{3}{3}
[1,19] = {4}{4}{3}{4}
[1,20] = {4}{4}{4}
[1,21] = {4}{4}{5}
[1,22] = {4}{5}
[1,23] = {4}{6}
[1,24] = {4}{7}
[1,25] = {5}
[1,26] = {6}
[1,27] = {7}
[1,28] = {8}{1}
[1,29] = {8}{2}
[1,30] = {8}{3}
[1,31] = {8}{4}
}
Use randi to select an index:
m{randi(numel(m))}
How could I construct automatically a dataset like the one below, assuming that the number of columns of matrix summary_whts is approx. 400???
lrwghts = dataset(...
{summary_whts(:,01),'w00'},...
{summary_whts(:,02),'w01'},...
{summary_whts(:,03),'w02'},...
{summary_whts(:,04),'w03'},...
{summary_whts(:,05),'w04'},...
{summary_whts(:,06),'w05'},...
{summary_whts(:,07),'w06'},...
{summary_whts(:,08),'w07'},...
{summary_whts(:,09),'w08'},...
{summary_whts(:,10),'w09'},...
{summary_whts(:,11),'w10'},...
{summary_whts(:,12),'w11'},...
'ObsNames',summary_mthd);
Why not use a simple loop to populate dataset?
nCols = size(summary_whts,1);
dataset = cell(nCols, 2);
for i = 1:nCols
dataset{i,1} = summary_whts(:,i);
dataset{i,2} = sprintf('w%04d', i);
end
dataset{end+1,1} = 'ObsNames';
dataset(end, 2} = summary_mthd;
At last, I found it! This is what I was looking for:
cat = [];
for i = 0:(size(X,2)),
cat = [cat;sprintf('w%03d',i)];
end
cat = cellstr(cat);
lrwghts = dataset({summary_whts,cat{:}},'ObsNames',cellstr(summary_mthd));