In the following code from Openturns FORM example:
import openturns as ot
model = ot.SymbolicFunction(['x1', 'x2'], ['x1^2+x2'])
R = ot.CorrelationMatrix(2)
R[0,1] = -0.6
inputDist = ot.Normal([0.,0.], R)
inputDist.setDescription(['X1', 'X2'])
inputVector = ot.RandomVector(inputDist)
Create the output random vector Y=model(X)
Y = ot.CompositeRandomVector(model, inputVector)
Create the event Y > 4
threshold = 4.0
event = ot.ThresholdEvent(Y, ot.Greater(), threshold)
Create a FORM algorithm
solver = ot.Cobyla()
startingPoint = inputDist.getMean()
algo = ot.FORM(solver, event, startingPoint)
Run the algorithm and retrieve the result
algo.run()
result_form = algo.getResult()
print(result_form)
Create the post analytical importance sampling simulation algorithm
algo = ot.PostAnalyticalImportanceSampling(result_form)
algo.run()
print(algo.getResult())
result = algo.getResult()
Create the post analytical controlled importance sampling simulation algorithm
algo = ot.PostAnalyticalControlledImportanceSampling(result_form)
algo.run()
print(algo.getResult())
Is it possible to see the values of X1, X2 and Y during the optimisation?
I am hoping to implement this on a simulation that takes a few minutes to run- so would be good to watch the steps of the optimisation process.
Thanks :-)
You just have to set the verbose flag in the solver:
solver.setVerbose(True)
then you will see how Cobyla steps during the finding of the design point:
cobyla: the initial value of RHO is 1.000000E-01 and PARMU is set to zero.
cobyla: NFVALS = 1, F = 0.000000E+00, MAXCV = 3.999990E+00
cobyla: X = 0.000000E+00 0.000000E+00
cobyla: NFVALS = 2, F = 5.000000E-03, MAXCV = 4.049990E+00
cobyla: X = 1.000000E-01 0.000000E+00
cobyla: NFVALS = 3, F = 5.000000E-03, MAXCV = 3.919990E+00
cobyla: X = 0.000000E+00 1.000000E-01
cobyla: increase in PARMU to 3.370787E-02
cobyla: NFVALS = 4, F = 5.000000E-03, MAXCV = 3.897541E+00
cobyla: X =-5.299989E-02 8.479983E-02
...
Note that the optimization is done in the standard space. You will have to implement logs in your code wrapper if you want to get the same information but in the physical space.
Cheers
Régis
I use fmincon function of MATLAB optimization toolbox for a minimization problem. The basic part of the code is as follows:
clc;
clear;
load('coord_points.mat');
load('coord_centroids.mat');
numberOfPoints=200;
numberOfCentroids=40;
Q=5;
tau=0.1;
d = zeros(numberOfPoints,numberOfCentroids);
for j=1:numberOfCentroids
centroid = coord_centroids(j,:);
for i=1:numberOfPoints
point = coord_points(i,:);
d(i,j)=norm(centroid-point);
end
end
sectorization = optimproblem;
x = optimvar('x',numberOfPoints,numberOfCentroids,'LowerBound',0,'UpperBound',1);
y = optimvar('y',1,'LowerBound',0,'UpperBound',1);
qjExpression=optimexpr(numberOfCentroids);
for j=1:numberOfCentroids
qjExpression(j) = sum(x(:,j));
end
qExpression = mean(qjExpression);
objExpression = y;
%%%
sectorization.Objective = objExpression;
cons1 = optimconstr(numberOfPoints,1);
for i=1:numberOfPoints
cons1(i) = sum(x(i,:)) == 1;
end
cons2 = optimconstr(numberOfCentroids,1);
for j=1:numberOfCentroids
cons2(j) = sum(x(:,j)) >= 1;
end
cons3 = optimconstr(numberOfCentroids,1);
for j=1:numberOfCentroids
cons3(j) = sum(x(:,j)) <= Q*(1+tau);
end
cons4 = optimconstr(1);
varianceOfQj = optimexpr(1);
for j=1:numberOfCentroids
varianceOfQj = varianceOfQj + (qjExpression(j)-qExpression)^2;
end
varianceOfQj=varianceOfQj/numberOfCentroids;
cons4 = varianceOfQj - y <= 0;
sectorization.Constraints.cons1=cons1;
sectorization.Constraints.cons2=cons2;
sectorization.Constraints.cons3=cons3;
sectorization.Constraints.cons4=cons4;
%load('sol.mat');
sol.x=zeros(numberOfPoints,numberOfCentroids);
sol.y=1;
problem = prob2struct(sectorization,sol);
options = optimoptions(#fmincon,'MaxFunctionEvaluations',5000,'Algorithm','active-set','Display','iter')
[x,fval] = fmincon(problem);
But I get the following result:
Solver stopped prematurely.
fmincon stopped because it exceeded the function evaluation limit, options.MaxFunctionEvaluations = 3.000000e+03.
However, as seen in the code, in the options section, max function evaluations is made equal to 5000.
Even, while the program is running, at first outputs as follows are appeared:
*options =
fmincon options:
Options used by current Algorithm ('active-set'):
(Other available algorithms: 'interior-point', 'sqp', 'sqp-legacy', 'trust-region-reflective')
Set properties:
Algorithm: 'active-set'
Display: 'iter'
MaxFunctionEvaluations: 5000
Default properties:
CheckGradients: 0
ConstraintTolerance: 1.0000e-06
FiniteDifferenceStepSize: 'sqrt(eps)'
FiniteDifferenceType: 'forward'
FunctionTolerance: 1.0000e-06
MaxIterations: 400
OptimalityTolerance: 1.0000e-06
OutputFcn: []
PlotFcn: []
SpecifyConstraintGradient: 0
SpecifyObjectiveGradient: 0
StepTolerance: 1.0000e-06
TypicalX: 'ones(numberOfVariables,1)'
UseParallel: 0
Show options not used by current Algorithm ('active-set')*
What can I do to make the Solver work properly? Why does it stop even though I have changed the setting for the maximum evaluation?
Please give your feedback.
I have the following script performing a nonlinear optimization (NLP), which works in Matlab and hits MaxFunctionEvaluations after about 5 minutes on my machine:
% Generate sample consumption data (4 weeks)
x = 0:pi/8:21*pi-1e-1; %figure; plot(x, 120+5*sin(0.2*x).*exp(-2e-2*x) + 10*exp(-x))
y = 120 + 5*sin(0.2*x).*exp(-2e-2*x) + 10*exp(-x);
consumptionPerWeek = (y + [0; 11; -30; 4.5]).'; % in 168x4 format
consumptionPerHour = reshape(consumptionPerWeek, [], 1);
hoursPerWeek = 168;
hoursTotal = numel(consumptionPerHour);
daysTotal = hoursTotal/24;
weeksTotal = ceil(daysTotal/7);
%% Perform some simple calculations
q_M_mean = mean(consumptionPerHour);
dvsScalingPerWeek = mean(consumptionPerWeek)/q_M_mean;
%% Assumptions about reactor, hard-coded
V_liq = 5701.0; % m^3, main reactor; from other script
initialValue = 4.9298; % kg/m^3; from other script
substrates_FM_year = [676.5362; 451.0241];
total_DVS_year = [179.9586; 20.8867];
mean_DVS_conc = 178.1238; %kg/m^3
% Product yields (m^3 per ton DVS)
Y_M = 420;
Y_N = 389;
%% Test DVS model
DVS_hour = sum(total_DVS_year)/hoursTotal; % t/h
k_1 = 0.25; % 1/d
parameters = [k_1; Y_M; Y_N; V_liq];
%% Build reference and initial values for optimization
% Distribute feed according to demand (-24%/+26% around mean)
feedInitialMatrix = DVS_hour*ones(hoursPerWeek, 1)*dvsScalingPerWeek;
% Calculate states with reference feed (improved initials)
feedInitialVector = reshape(feedInitialMatrix, [], 1);
feedInitialVector = feedInitialVector(1:hoursTotal);
resultsRef = reactorModel1(feedInitialVector, initialValue, parameters, ...
mean_DVS_conc);
V_M_PS = 0 + cumsum(resultsRef(:,2)/24 - consumptionPerHour);
neededMStorage0 = max(V_M_PS) - min(V_M_PS);
%% Setup optimization problem (NLP): feed optimization with virtual product storage
% Objective function 1: Standard deviation of theoretical product storage volume
objFun1 = #(feedVector) objFunScalar(feedVector, initialValue, parameters, ...
mean_DVS_conc, consumptionPerHour);
% Bounds (lb <= x <= ub), i.e., decision variables can only range between 0 and 0.9*dailyDvsAmount
upperfeedLimitSlot = 0.90; % Limit DVS feed amount per *slot*
upperfeedLimitDay = 1.80; % Limit DVS feed amount per *day*
upperfeedLimitWeek = 1.37; % Limit DVS feed amount per *week*
lowerBound_nlp = zeros(1, hoursTotal);
upperBound_nlp = upperfeedLimitSlot*24*DVS_hour.*ones(1, hoursTotal);
% Equality Constraint 1: feed amount mean = constant
A_eq1_nlp = ones(1, hoursTotal);
b_eq1_nlp = DVS_hour*hoursTotal;
% Inequality Constraint 1: Limit max. daily amount
A_nlp1 = zeros(daysTotal, hoursTotal);
for dI = 1:daysTotal
A_nlp1(dI, (24*dI)-(24-1):(24*dI)) = 1;
end
b_nlp1 = upperfeedLimitDay*24*DVS_hour*ones(daysTotal, 1);
% Inequality Constraint 2: Limit max. weekly amount
A_nlp2 = zeros(weeksTotal, hoursTotal);
for wIi = 1:weeksTotal
A_nlp2(wIi, (168*wIi)-(168-1):(168*wIi)) = 1;
end
b_nlp2 = upperfeedLimitWeek*168*DVS_hour*ones(weeksTotal, 1);
% Summarize all inequality constraints
A_nlp = [A_nlp1; A_nlp2]; %sparse([A_nlp1; A_nlp2]);
b_nlp = [b_nlp1; b_nlp2]; %sparse([b_nlp1; b_nlp2]);
try
% Solver: fmincon (Matlab Optimization Toolbox) --> SQP-algorithm = best
optionen_GB = optimoptions('fmincon', 'Display', 'iter', 'FunctionTolerance', 1e-5, ...
'StepTolerance', 1e-4, 'MaxIterations', 2*hoursTotal, ...
'MaxFunctionEvaluations', 100*hoursTotal, 'HonorBounds', true, 'Algorithm', 'sqp');
catch
optionen_GB = optimset('Display', 'iter', 'TolFun', 1e-5, 'TolX', 1e-4, ...
'MaxIter', 2*hoursTotal, 'MaxFunEvals', 100*hoursTotal, 'Algorithm', 'sqp');
end
%% Solve gradient-based NLP
tic; [feedOpt, fval] = fmincon(#(feedVector) objFun1(feedVector), ...
feedInitialVector, A_nlp, b_nlp, A_eq1_nlp, b_eq1_nlp, lowerBound_nlp, upperBound_nlp, ...
[], optionen_GB); toc
%% Rerun model and calculate virtual storage volume with optimized input
resultsOpt = reactorModel1(feedOpt, initialValue, parameters, mean_DVS_conc);
q_M_Opt = resultsOpt(:,2)/24;
V_M_PS_opt = 0 + cumsum(q_M_Opt - consumptionPerHour);
neededMStorageOpt = max(V_M_PS_opt) - min(V_M_PS_opt);
sprintf('Needed product storage before optimization: %.2f m^3, \nafterwards: %.2f m^3. Reduction = %.1f %%', ...
neededMStorage0, neededMStorageOpt, (1 - neededMStorageOpt/neededMStorage0)*100)
%% Objective as separate function
function prodStorageStd = objFunScalar(dvs_feed, initialValues, parameters, mean_DVS_conc, ...
MConsumptionPerHour)
resultsAlgb = reactorModel1(dvs_feed(:, 1), initialValues, parameters, mean_DVS_conc);
q_M_prod = resultsAlgb(:,2)/24;
V_M_PS1 = 0 + cumsum(q_M_prod - MConsumptionPerHour);
prodStorageStd = std(V_M_PS1);
end
The external function reads like this:
function resultsArray = reactorModel1(D_feed, initialValue, parameters, D_in)
% Simulate production per hour with algebraic reactor model
% Feed is solved via a for-loop
hoursTotal = length(D_feed);
k_1 = parameters(1);
Y_M = parameters(2);
Y_N = parameters(3);
V_liq = parameters(4);
resultsArray = zeros(hoursTotal, 3);
t = 1/24;
liquid_feed = D_feed/(D_in*1e-3); % m^3/h
initialValue4Model0 = (initialValue*(V_liq - liquid_feed(1))*1e-3 ...
+ D_feed(1))*1e3/V_liq; % kg/m^3
resultsArray(1, 1) = initialValue4Model0*exp(-k_1*t);
% Simple for-loop with feed as vector per hour
for pHour = 2:hoursTotal
initialValue4Model = (resultsArray(pHour-1, 1)*(V_liq - liquid_feed(pHour))*1e-3 ...
+ D_feed(pHour))*1e3/V_liq; % kg/m^3
resultsArray(pHour, 1) = initialValue4Model*exp(-k_1*t);
end
resultsArray(:, 2) = V_liq*Y_M*k_1*resultsArray(:, 1)*1e-3; % m^3/d
resultsArray(:, 3) = V_liq*Y_N*k_1*resultsArray(:, 1)*1e-3; % m^3/d
end
When I execute the very same script in Octave (ver 5.1.0 with optim 1.6.0), I get:
error: linear inequality constraints: wrong dimensions
When in fact, the following line (executed from the command prompt)
sum(A_nlp*feedInitialVector <= b_nlp)
gives 32 on both Octave and Matlab, thus showing that dimensions are correct.
Is this a bug? Or is Octave treating linear (in)equality constraints somehow different than Matlab?
(Also, if you have tips on how to speed up this script, they would come in handy.)
I've debugged this a bit for you to get you started.
First enable debugging on error:
debug_on_error(1)
Then find the installation folder of optim, and have a look at file /private/__linear_constraint_dimensions__.m within.
*(I found this by doing a grep operation for the exact error you were getting, and found the relevant file. There is another one outside the private folder, you may want to look at that too.)
If you look at the lines trigerring the errors, you will notice, e.g. that an error is triggered if rm != o.np, where [rm, cm] = size(f.imc)
Now run your script and let it enter debug mode on error. You will see that:
debug> [rm, cm] = size(f.imc)
rm = 32
cm = 672
debug> o.np
ans = 672
debug> rm != o.np
ans = 1 % I.e. boolean test succeeds and triggers error
I have no idea what these are, presumably r and c reflect rows and columns, but in any case, you will see that it appears you are trying to match rows with columns and vice versa.
In other words, it looks like you may have passed your inputs in a transposed fashion at some point.
In any case, if this isn't exactly what's happening, this should be a decent starting point for you to figure the exact bug out.
I don't know why matlab "works". Maybe there's a bug in your code and matlab works despite it (for better or worse).
Or there might be a bug in optim transposing inputs by accident (or, at least, in a manner that is incompatible to matlab).
If you feel after your debugging adventures that it's a bug in the optim package, feel free to file a bug report :)
I am using glmnet in MATLAB 2019a on my Macbook to do logistic regression.
Algorithm:
log(pi/(1-pi))=b0+X*b
pi=P(Y=2|X_i)=1-P(Y=1|X_i)
Code:
Y = [2;1;2;1;2;1;2;1;1;2];
X = [0.1451 0.1176 0.0872 0.0544 0.0197 -0.0164 -0.0533 -0.0907;
0.5096 0.7240 0.9038 1.0515 1.1694 1.2599 1.3253 1.3681;
-0.0593 -0.1683 -0.2738 -0.3754 -0.4730 -0.5660 -0.6543 -0.7376;
-1.0128 -0.9539 -0.9004 -0.8522 -0.8089 -0.7701 -0.7355 -0.7047;
0.7533 0.5640 0.4054 0.2752 0.1709 0.0900 0.0302 -0.0109;
0.2014 0.2595 0.3070 0.3444 0.3724 0.3918 0.4032 0.4074;
0.9174 0.8706 0.8260 0.7834 0.7423 0.7025 0.6636 0.6253;
0.7643 0.6115 0.4789 0.3653 0.2693 0.1897 0.1252 0.0744;
-0.3299 -0.5078 -0.6507 -0.7615 -0.8430 -0.8981 -0.9294 -0.9399;
-0.2141 -0.1472 -0.0818 -0.0179 0.0443 0.1045 0.1626 0.2183];
lambda=0.5;
family='binomial';
options.weights = [];
options.alpha = 1;
options.nlambda = 100;
options.lambda_min = 0;
options.lambda = lambda;
options.standardize = false;%true
options.thresh = 1E-4;
options.dfmax = 0;
options.pmax = 0;
options.exclude = [];
options.penalty_factor = [];
options.maxit = 100;
options.HessianExact = false;
options.type = 'naive';
fit = glmnet(X,Y,family,options);
Outcome:
a0 0
label [1;2]
beta [0;0;0;0;0;0;0;0]
dev 0
nulldev 13.8629
df 0
lambda 0.5000
npasses 1
jerr 0
dim [8,1]
class 'lognet'
No matter how I changed my input and output, the function always returns coefficients with all 0. I picked these options carefully so I am really confused how it comes to this outcome.
Is it because the data I picked are special or the parameters are wrong?
I have just found where the problem is. The data here is small, so lambda=0.5 implements an overly strong penalty to it.
Changing lambda to less than 0.01 solves the problem. Plus, options.pmax = 0 should be deleted, otherwise it asks for all coefficients to be 0.
How can I change "Validation Checks" value from 6 to higher or lower values using code?
I have following code:
% Create a Pattern Recognition Network
hiddenLayerSize = ns;
net = patternnet(hiddenLayerSize);
net.divideParam.trainRatio = trRa/100;
net.divideParam.valRatio = vaRa/100;
net.divideParam.testRatio = teRa/100;
% Train the Network
[net,tr] = train(net,inputs,targets);
% Test the Network
outputs = net(inputs);
errors = gsubtract(targets,outputs);
performance = perform(net,targets,outputs);
% Recalculate Training, Validation and Test Performance
trainTargets = targets .* tr.trainMask{1};
valTargets = targets .* tr.valMask{1};
testTargets = targets .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,outputs);
valPerformance = perform(net,valTargets,outputs);
testPerformance = perform(net,testTargets,outputs);
I could not find a clue at http://www.mathworks.com/help/nnet/ug/train-and-apply-multilayer-neural-networks.html
TL;DL: net.trainParam.max_fail = 8;
I've used the example provided in the page you linked to get a working instance of nntraintool.
When you open nntraintool.m you see a small piece of documentation that says (among else):
% net.trainParam.showWindow = false;
This hinted that some properties are stored within net.trainParam. When querying it to see what it holds you get:
ans =
Function Parameters for 'trainlm'
Show Training Window Feedback showWindow: true
Show Command Line Feedback showCommandLine: false
Command Line Frequency show: 25
Maximum Epochs epochs: 1000
Maximum Training Time time: Inf
Performance Goal goal: 0
Minimum Gradient min_grad: 1e-07
Maximum Validation Checks max_fail: 6
Mu mu: 0.001
Mu Decrease Ratio mu_dec: 0.1
Mu Increase Ratio mu_inc: 10
Maximum mu mu_max: 10000000000
Here you can see how the Maximum Validation Checks is stored: in a field named max_fail. Now it was only a case of testing if it was a read-only field or not, which could easily be tested using net.trainParam.max_fail = 8; train(net,...); - which correctly changed the default value from 6 to 8.