Related
I'm solving a Dynamic Optimization problem on gekko but the solution doesn't converge if I vary the number of time points or initial/lb/ub values for manipulated variable or change the collocation nodes. What could be the issue?
Just changed the values mentioned above, and on some combinations, the solution doesn't converge at all, and the following error message appears:
raise Exception(response)
Exception: #error: Solution Not Found
Code:
from gekko import GEKKO
import numpy as np
import matplotlib.pyplot as plt
m = GEKKO()
nt = 101 # no. of time steps
m.time = np.linspace(0,147,nt)
k17 = 0.0508
k26 = 1
k27 = 0.0577
k28 = 0.0104
k32 = 2
k33 = 2
k37 = 0.0016
k38 = 0.0107
k47 = 0.006
k48 = 0.072
k53 = 2
k57 = 0.0201
k58 = 0.082
k61 = 2
k62 = 2
k63 = 2
k72 = 2
k77 = 0.0133
k78 = 0.011
k87 = 0.081
k88 = 0.0148
kd = 0.06
w = 0.043
s1 = m.Var(value=0,lb=0)
s2 = m.Var(value=3,lb=0)
s3 = m.Var(value=0.01,lb=0)
s4 = m.Var(value=0.46,lb=0)
s5 = m.Var(value=0.27,lb=0)
p1 = m.Var(value=0.51,lb=0)
p2 = m.Var(value=0.33,lb=0)
p3 = m.Var(value=0.3,lb=0)
p4 = m.Var(value=0.34e-4,lb=0)
p5 = m.Var(value=2.01,lb=0)
p6 = m.Var(value=0.05,lb=0)
X = m.Var(value=0.09,lb=0)
Xd = m.Var(value=0.02,lb=0)
V = m.Var(value=5,lb=0,ub=10)
u = m.MV(value=0.05,lb=0,ub=0.1) # manipulated variable
u.STATUS = 1
u.DCOST = 0
f1 = (8.443e-4)*X*s1/(8.989e5 + s1)
f2 = (2.481e6)*X*s1*s3/((6.495e4 + s1)*(7.076e2 + s3))
f3 = (3.968e5)*X*s1*s3/((3.723e4 + s1)*(2.782e3 + s3))
f4 = (1.09e2)*X*s3/(0.019+s3)
f5 = 7.283*X*s4/(1.92e3 + s4)
f6 = (3.337e5)*X*s2*s5/((2.719e4 + s2)*(4.488e4 + s5))
f7 = (3.977e3)*X*s2/(9.324e3 + s2)
f8 = (6.697e-6)*X*s2/(0.537+s2)
f9 = (3.261e4)*X*s2/(6.683e5 + s2)
P = np.zeros(nt)
P[-1] = 1.0
final = m.Param(value=P)
# Equations
m.Equation(V.dt()==u)
m.Equation(s1.dt()==-f1-f2-f3-k17*f7+(2-s1)*u/V)
m.Equation(s2.dt()==-f6-k27*f7-k28*f8-f9-s2*u/V)
m.Equation(s3.dt()==-k32*f2-k33*f3-f4+f6-k37*f7-k38*f8+f9-s3*u/V)
m.Equation(s4.dt()==-f5+f6-k47*f7-k48*f8-s4*u/V)
m.Equation(s5.dt()==k53*f3+f5-f6-k57*f7-k58*f8-s5*u/V)
m.Equation(p1.dt()==k61*f1+k62*f2+k63*f3-p1*u/V)
m.Equation(p2.dt()==k72*f2-k77*f7-k78*f8-p2*u/V)
m.Equation(p3.dt()==f4-k87*f7-k88*f8-p3*u/V)
m.Equation(p4.dt()==f8-p4*u/V)
m.Equation(p5.dt()==f7-p5*u/V)
m.Equation(p6.dt()==f5+f9-p6*u/V)
m.Equation(X.dt()==w*X-kd*X*Xd-X*u/V)
m.Equation(Xd.dt()==kd*X*Xd-Xd*u/V)
m.Obj(-final*V*p4)
m.options.IMODE = 6
m.options.NODES = 4
#m.options.COLDSTART=2
#m.options.MAX_ITER=1000
m.solve(disp=True)
p4_ = np.multiply(p4.value,1000)
plt.figure(1)
plt.subplot(2,1,1)
plt.plot(m.time,u.value,'r-')
plt.subplot(2,1,2)
plt.plot(m.time,p4_,'b--')
There are several reasons that the solver fails to reach a solution with Solution Not Found. Here are some of the reasons and things to try:
If the solver reports No Feasible Solution then the bounds are too restrictive. Widen the variable bounds until a solution is obtained.
If the solver reaches the maximum iterations, try changing the solver with m.options.SOLVER=1 or letting the solver continue for more iterations with m.options.MAX_ITER=500.
If the solver stops with the message Unbounded Solution then try setting bounds on the MV values.
Try initialization with no degrees of freedom by setting {FV}.STATUS=0 and {MV}.STATUS=0). The simulation should have the same number of equations and variables as a square problem. Switching to m.options.IMODE=7 is a sequential simulation and can also help with the initialization. Setting m.options.COLDSTART=1 or m.options.COLDSTART=2 can help find an initial solution that helps the optimization. Additional initialization resources are available in the Dynamic Optimization course.
Please post a copy of your code for more specific suggestions. There are several Dynamic Optimization Benchmark problems posted to the Dynamic Optimization Course.
Response to Edit
Try setting a smaller time scale until the problem converges. Here is a successful solution with a final time of 0.08.
m.time = np.linspace(0,0.08,11)
The solver does not find a solution after this time point. It reports an infeasible solution. One cause of this may be variable p4 that makes it infeasible because of the lower bound of zero.
Here is the diagnostic code with the plots:
from gekko import GEKKO
import numpy as np
import matplotlib.pyplot as plt
m = GEKKO()
m.time = np.linspace(0,0.08,11)
nt = len(m.time)
k17 = 0.0508
k26 = 1
k27 = 0.0577
k28 = 0.0104
k32 = 2
k33 = 2
k37 = 0.0016
k38 = 0.0107
k47 = 0.006
k48 = 0.072
k53 = 2
k57 = 0.0201
k58 = 0.082
k61 = 2
k62 = 2
k63 = 2
k72 = 2
k77 = 0.0133
k78 = 0.011
k87 = 0.081
k88 = 0.0148
kd = 0.06
w = 0.043
s1 = m.Var(value=0,lb=0)
s2 = m.Var(value=3,lb=0)
s3 = m.Var(value=0.01,lb=0)
s4 = m.Var(value=0.46,lb=0)
s5 = m.Var(value=0.27,lb=0)
p1 = m.Var(value=0.51,lb=0)
p2 = m.Var(value=0.33,lb=0)
p3 = m.Var(value=0.3,lb=0)
p4 = m.Var(value=0.34e-4,lb=0)
p5 = m.Var(value=2.01,lb=0)
p6 = m.Var(value=0.05,lb=0)
X = m.Var(value=0.09,lb=0)
Xd = m.Var(value=0.02,lb=0)
V = m.Var(value=5,lb=0.01)
u = m.MV(value=0.05,lb=0,ub=0.1) # manipulated variable
u.STATUS = 1
u.DCOST = 0
f1 = m.Intermediate((8.443e-4)*X*s1/(8.989e5 + s1))
f2 = m.Intermediate((2.481e6)*X*s1*s3/((6.495e4 + s1)*(7.076e2 + s3)))
f3 = m.Intermediate((3.968e5)*X*s1*s3/((3.723e4 + s1)*(2.782e3 + s3)))
f4 = m.Intermediate((1.09e2)*X*s3/(0.019+s3))
f5 = m.Intermediate(7.283*X*s4/(1.92e3 + s4))
f6 = m.Intermediate((3.337e5)*X*s2*s5/((2.719e4 + s2)*(4.488e4 + s5)))
f7 = m.Intermediate((3.977e3)*X*s2/(9.324e3 + s2))
f8 = m.Intermediate((6.697e-6)*X*s2/(0.537+s2))
f9 = m.Intermediate((3.261e4)*X*s2/(6.683e5 + s2))
P = np.zeros(nt)
P[-1] = 1.0
final = m.Param(value=P)
# Equations
m.Equation(V.dt()==u)
m.Equation(s1.dt()==-f1-f2-f3-k17*f7+(2-s1)*u/V)
m.Equation(s2.dt()==-f6-k27*f7-k28*f8-f9-s2*u/V)
m.Equation(s3.dt()==-k32*f2-k33*f3-f4+f6-k37*f7-k38*f8+f9-s3*u/V)
m.Equation(s4.dt()==-f5+f6-k47*f7-k48*f8-s4*u/V)
m.Equation(s5.dt()==k53*f3+f5-f6-k57*f7-k58*f8-s5*u/V)
m.Equation(p1.dt()==k61*f1+k62*f2+k63*f3-p1*u/V)
m.Equation(p2.dt()==k72*f2-k77*f7-k78*f8-p2*u/V)
m.Equation(p3.dt()==f4-k87*f7-k88*f8-p3*u/V)
m.Equation(p4.dt()==f8-p4*u/V)
m.Equation(p5.dt()==f7-p5*u/V)
m.Equation(p6.dt()==f5+f9-p6*u/V)
m.Equation(X.dt()==w*X-kd*X*Xd-X*u/V)
m.Equation(Xd.dt()==kd*X*Xd-Xd*u/V)
m.Maximize(final*V*p4)
m.options.IMODE = 6
m.options.NODES = 3
m.options.SOLVER = 1
m.options.COLDSTART=0
#m.options.MAX_ITER=1000
m.solve(disp=True)
p4_ = np.multiply(p4.value,1000)
plt.figure(1)
plt.subplot(2,1,1)
plt.plot(m.time,u.value,'r-')
plt.subplot(2,1,2)
plt.plot(m.time,p4_,'b--')
plt.figure(2)
plt.plot(m.time,s1,label='s1')
plt.plot(m.time,s2,label='s2')
plt.plot(m.time,s3,label='s3')
plt.plot(m.time,s4,label='s4')
plt.plot(m.time,s5,label='s5')
plt.legend()
plt.figure(3)
plt.plot(m.time,p1,label='p1')
plt.plot(m.time,p2,label='p2')
plt.plot(m.time,p3,label='p3')
plt.plot(m.time,p4,label='p4')
plt.plot(m.time,p5,label='p5')
plt.plot(m.time,p6,label='p6')
plt.legend()
plt.figure(4)
plt.subplot(3,1,1)
plt.plot(m.time,s1,label='s1')
plt.legend()
plt.subplot(3,1,2)
plt.plot(m.time,s3,label='s3')
plt.legend()
plt.subplot(3,1,3)
plt.plot(m.time,p4,label='p4')
plt.legend()
plt.show()
I am attempting to write a Multi-Layer Perceptron Network inside MATLAB to help me better understand the calculus required for backpropagation.
The aim is so provide the network with XOR data (where upper-right and lower-left quadrant data is class 1 and the remaining quadrants class 0), train the network on this data, and then test it on new data.
My problem is that my loss curve looks very very strange:
It appears to bounce between very low error very high error and converge in the middle to a pretty poor error.
I was wondering if someone could check that I have correctly implemented the chain rule in MATLAB syntax.
The MLP network is structured as follows: Input-layer has 2 neurons, 1 hidden-layer with 2 neurons, and 1 output neuron.
Here is the MATLAB code:
%Create XOR Dataset
x1pos = rand(500,1);
x1neg = -rand(500,1);
x1 = [x1pos; x1neg];
p = randperm(length(x1));
x1 = x1(p);
x2pos = rand(500,1);
x2neg = -rand(500,1);
x2 = [x1pos; x1neg];
p = randperm(length(x2));
x2 = x2(p);
Data = [x1 x2];
TrainingData = Data(1:800,:);
TestData = Data(801:length(Data),:);
T = gt((Data(:,1).*Data(:,2)),0); %Create class label for data and assign to matrix T
%Neural Net
%Training
W1 = rand(2,2); %Initialize random weights
W2 = rand(1,2); %Initialize random weights
B1 = rand(2,1); %Initialize random biases
B2 = rand(1,1); %Initialize random biases
n = 0.05; %Set Learning Rate
for i = 1:800
%Fwd Pass
x1 = Data(i,1);
x2 = Data(i,2);
X = [x1; x2];
A1 = W1*X + B1;
H1 = sigmoid(A1);
A2 = W2*H1 + B2;
Y = sigmoid(A2);
%Loss
Loss = (Y-T(i))*(Y-T(i));
scatter(i, Loss)
hold on;
%Backpropagation
dEdY = 2*(Y-T(i)); %The partial derivative of the loss with respect to the output
dYdA2 = Y*(1-Y); %The partial derivative of the output with respect to the hidden layer output
dA2dH1 = W2.'; %The partial derivative of the hidden layer output with respect to the first layer activations
dH1dA1 = H1.*(1-H1); %The partial derivative of the first layer activations with respect to the first layer output
%Chain Rule
dEdW2 = dEdY.*dYdA2.*W2.';
dEdW1 = dEdY.*dYdA2.*dA2dH1.*dH1dA1.*W1.';
dEdB2 = dEdY.*dYdA2;
dEdB1 = dEdY.*dYdA2.*dA2dH1.*dH1dA1;
%Update Weights
W2 = (W2.' - n.*dEdW2).';
W1 = (W1.' - n.*dEdW1).';
%Update Biases
B2 = B2 - n.*dEdB2;
B1 = B1 - n.*dEdB1;
%Next training loop
end
%Testing
for i = 801:1000
x1 = Data(i,1);
x2 = Data(i,2);
X = [x1; x2];
A1 = W1*X + B1;
H1 = sigmoid(A1);
A2 = W2*H1 + B2;
Y = sigmoid(A2);
end
function o = sigmoid(input)
o = [];
for i = 1:length(input)
o = [o; 1/(1+exp(-input(i)))];
end
end
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
input = [[0,0,1],[0,1,1],[1,0,1],[1,1,1]]
output = [0,1,1,0]
N = np.size(input,0) # number of samples
Ni = np.size(input,1) # dimension of the samples of input
No = 1 # dimension of the sample of output
Nh = 10 # number of hidden units
Ws = 1/4*np.random.rand(Nh,Ni+1)
print(Ws)
Wo = 1/4*np.random.rand(No,Nh)
print(Wo)
alpha = 0.05 # Learning rate
t_ = []
loss_ = []
def ReLU(x):
return np.maximum(0,x)
def sigmoid(x):
return 1/(1+np.exp(-x))
## train the model ====================================================================
for epoch in range(0,3000):
loss = 0
for id_ in range(0,N):
dWs = 0*Ws
dWo = 0*Wo
x = np.append(input[id_],1)
Z_1 = np.dot(Ws,x)
Z_2 = np.dot(Wo,ReLU(Z_1))
y = sigmoid(Z_2)
d = output[id_]
for j in range(0,Nh):
for i in range(0,No):
if Z_1[j] >= 0:
dWo[i,j] = dWo[i,j] + (y[i]-d)*Z_1[j]
#dWo[i,j] = dWo[i,j] + sigmoid(Z_1[j])*(y[i]-d)
else:
dWo[i,j] += 0
Wo = Wo - alpha*dWo
for k in range(0,Ni+1):
for j in range(0,Nh):
for i in range(0,No):
if Z_1[j] >= 0:
dWs[j,k] = dWs[j,k] + x[k]*Wo[i,j]*(y[i]-d)
#dWs[j,k] = dWs[j,k] + x[k]*Wo[i,j]*sigmoid(Z_1[j])*(1-sigmoid(Z_1[j]))*(y[i]-d)
else:
dWs[j,k] += 0
Ws = Ws - alpha*dWs
loss = loss + 1/2*np.linalg.norm(y-d)
if np.mod(epoch,50) == 0:
print(epoch,"-th epoch trained")
t_ = np.append(t_,epoch)
loss_ = np.append(loss_,loss)
fig = plt.figure(num=0,figsize=[10,5])
plt.plot(t_,loss_,marker="")
plt.title('Loss decay')
plt.xlabel('epoch',FontSize=20)
plt.ylabel('Loss',FontSize=20)
plt.show()
## figure out the function shape the model==========================================
xn = np.linspace(0,1,20)
yn = np.linspace(0,1,20)
xm, ym = np.meshgrid(xn, yn)
xx = np.reshape(xm,np.size(xm,0)*np.size(xm,1))
yy = np.reshape(ym,np.size(xm,0)*np.size(xm,1))
Z = []
for id__ in range(0,np.size(xm)):
x = np.append([xx[id__],yy[id__]],[1,1])
Z_1 = np.dot(Ws,x)
y_ = sigmoid(np.dot(Wo,ReLU(Z_1)))
Z = np.append(Z,y_)
fig = plt.figure(num=1,figsize=[10,5])
ax = fig.gca(projection='3d')
surf = ax.plot_surface(xm,ym,np.reshape(Z,(np.size(xm,0),np.size(xm,1))),cmap='coolwarm',linewidth=0,antialiased=False)
print("====================================================================")
plt.show()
## test the trained model ====================================================================
for id_ in range(0,N):
x = np.append(input[id_],1)
Z_1 = np.dot(Ws,x)
y = sigmoid(np.dot(Wo,ReLU(Z_1)))
print(y)
If I try this with sigmoid function, it works fine but when the ReLU activation function is implemented, the the program doesn't learning anything.
The NN consist of 3 input, hidden, output layers and sigmoid activation fuction is implemented for output function. Hand calculation seems fine but can't find the flaw.
The code below with sigmoid activation function works just fine.
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
input = [[0,0,1],[0,1,1],[1,0,1],[1,1,1]]
output = [0,1,1,0]
N = np.size(input,0) # number of samples
Ni = np.size(input,1) # dimension of the samples of input
No = 1 # dimension of the sample of output
Nh = 5 # number of hidden units
Ws = 1/4*np.random.rand(Nh,Ni+1)
#print(Ws)
Wo = 1/4*np.random.rand(No,Nh)
#print(Wo)
alpha = 0.1 # Learning rate
t_ = []
loss_ = []
def sigmoid(x):
return 1/(1+np.exp(-x))
## train the model ====================================================================
for epoch in range(0,5000):
loss = 0
for id_ in range(0,N):
dWs = 0*Ws
dWo = 0*Wo
x = np.append(input[id_],1)
Z_1 = np.dot(Ws,x)
A_1 = sigmoid(Z_1)
Z_2 = np.dot(Wo,A_1)
y = sigmoid(Z_2)
d = output[id_]
for j in range(0,Nh):
for i in range(0,No):
dWo[i,j] = dWo[i,j] + sigmoid(Z_1[j])*(y[i]-d)
Wo = Wo - alpha*dWo
for k in range(0,Ni+1):
for j in range(0,Nh):
for i in range(0,No):
dWs[j,k] = dWs[j,k] + x[k]*Wo[i,j]*sigmoid(Z_1[j])*(1-sigmoid(Z_1[j]))*(y[i]-d)
Ws = Ws - alpha*dWs
loss = loss + 1/2*np.linalg.norm(y-d)
if np.mod(epoch,50) == 0:
print(epoch,"-th epoch trained")
t_ = np.append(t_,epoch)
loss_ = np.append(loss_,loss)
fig = plt.figure(num=0,figsize=[10,5])
plt.plot(t_,loss_,marker="")
plt.title('Loss decay')
plt.xlabel('epoch',FontSize=20)
plt.ylabel('Loss',FontSize=20)
plt.show()
## figure out the function shape the model==========================================
xn = np.linspace(0,1,20)
yn = np.linspace(0,1,20)
xm, ym = np.meshgrid(xn, yn)
xx = np.reshape(xm,np.size(xm,0)*np.size(xm,1))
yy = np.reshape(ym,np.size(xm,0)*np.size(xm,1))
Z = []
for id__ in range(0,np.size(xm)):
x = np.append([xx[id__],yy[id__]],[1,1])
Z_1 = np.dot(Ws,x)
y_ = sigmoid(np.dot(Wo,sigmoid(Z_1)))
Z = np.append(Z,y_)
fig = plt.figure(num=1,figsize=[10,5])
ax = fig.gca(projection='3d')
surf = ax.plot_surface(xm,ym,np.reshape(Z,(np.size(xm,0),np.size(xm,1))),cmap='coolwarm',linewidth=0,antialiased=False)
print("====================================================================")
plt.show()
## test the trained model ====================================================================
for id_ in range(0,N):
x = np.append(input[id_],1)
Z_1 = np.dot(Ws,x)
y = sigmoid(np.dot(Wo,sigmoid(Z_1)))
print(y)
I found similar case in Quora.
And have tested it in my networks that involves modelling logics to resolve some noisy cost function.
I found that ReLu outputs are usually blasted all over, by the 3rd layer of MLP, the values before the output have accumulated to thousands if not millions.
And with that, I prefer sigmoid with MLPs. Don't forget, sigmoid limits output to 1, but ReLu does not.
The intuition behind ReLu is that it filters out unneeded info by means of MAX(0,X) function, before forwarded to the next layer of processing. For the same reason you see it being used in Convolution problems. Note: Normalization Layer is used in these cases so that the output values of the nodes will not blast all over.
But in the case of an MLP, you didn't implement any Norm Layer after ReLu, for that reason, it is difficult to model a simple function such as XOR. In short, without Norm Layer, I don't recommend the use of ReLu, although in some cases, it still can function properly.
I am trying to implement Asynchronous Methods for Deep Reinforcement Learning and one of the steps requires to accumulate the gradient over different steps and then apply it.
What is the best way to achieve this in tensorflow?
I got so far as to accumulate the gradient and I don't think is the fastest way to achieve it (lots of transfers from tensorflow to python and back).
Any suggestions are welcome.
This is my code of a toy NN. It does not model or compute anything it just exercise the operations that I want to use.
import tensorflow as tf
from model import *
graph = tf.Graph()
with graph.as_default():
state = tf.placeholder(tf.float32, shape=[None, 80,80,1])
with tf.variable_scope('layer1'):
W = weight_variable([8, 8, 1, 32])
variable_summaries(W, "layer1/W")
b = bias_variable([32])
variable_summaries(b, "layer1/b")
h = conv2d(state, W, 4) + b
activation = tf.nn.relu(h)
pool1 = max_pool_2x2(activation)
print(pool1.get_shape())
pool1 = tf.reshape(pool1, [-1, 3200])
with tf.variable_scope('readout'):
W = weight_variable([3200, 3])
b = bias_variable([3])
logits = tf.matmul(pool1, W) + b
variable_summaries(h, "y")
action_indexes = tf.placeholder(tf.int32, shape=[None], name="action_indexes")
loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits, action_indexes)
starter_learning_rate = 1e-6
global_step = tf.Variable(0, trainable=False)
# decay every 1000 steps with a base of 0.96:
learning_rate = tf.train.exponential_decay(starter_learning_rate,
global_step,
10000, 0.96, staircase=True)
optimizer = tf.train.RMSPropOptimizer(learning_rate)
gradients_and_variables = optimizer.compute_gradients(loss, tf.trainable_variables())
discounted_values = tf.placeholder(tf.float32, shape=[None, 1])
with tf.Session(graph=graph) as s:
for v in tf.trainable_variables():
print(v.name, v.dtype, v.get_shape())
s.run(tf.initialize_all_variables())
feed_dict= {
state : np.zeros([1, 80, 80, 1]),
action_indexes: [1],
}
var_to_grad = dict((var.name, grad) for grad, var in gradients_and_variables)
keys = sorted(var_to_grad.keys())
print(keys)
name_to_var = dict((var.name, var) for _, var in gradients_and_variables)
for i in range(10):
gradients = s.run([ var_to_grad[k] for k in keys], feed_dict=feed_dict)
for k,v in zip(keys, gradients):
var_to_grad[k] += v
for k in keys:
print(var_to_grad[k])
s.run( optimizer.apply_gradients( (g, name_to_var[v]) for v,g in var_to_grad.iteritems()), feed_dict=feed_dict)
Updated code after #yaroslave suggestion:
import tensorflow as tf
from model import *
graph = tf.Graph()
with graph.as_default():
minibatch = 32
state = tf.placeholder(tf.float32, shape=[minibatch, 80,80,1], name="input")
with tf.variable_scope('layer1'):
W = weight_variable([8, 8, 1, 32])
variable_summaries(W, "layer1/W")
b = bias_variable([32])
variable_summaries(b, "layer1/b")
h = conv2d(state, W, 4) + b
activation = tf.nn.relu(h)
pool1 = max_pool_2x2(activation)
print(pool1.get_shape())
pool1 = tf.reshape(pool1, [-1, 3200])
with tf.variable_scope('readout'):
W = weight_variable([3200, 3])
b = bias_variable([3])
logits = tf.matmul(pool1, W) + b
variable_summaries(h, "y")
action_indexes = tf.placeholder(tf.int32, shape=[minibatch], name="action_indexes")
loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits, action_indexes)
starter_learning_rate = 1e-6
global_step = tf.Variable(0, trainable=False)
# decay every 1000 steps with a base of 0.96:
learning_rate = tf.train.exponential_decay(starter_learning_rate,
global_step,
10000, 0.96, staircase=True)
optimizer = tf.train.RMSPropOptimizer(learning_rate)
trainable_variables = tf.trainable_variables()
varname_to_var = dict( (v.name, v) for v in trainable_variables )
keys = sorted(varname_to_var.keys())
gradients_and_variables = optimizer.compute_gradients(loss, [ varname_to_var[k] for k in keys])
var_to_grad = dict((var.name, grad) for grad, var in gradients_and_variables)
name_to_var = dict((var.name, var) for _, var in gradients_and_variables)
# save the gradients in memory
var_to_ref_grad = {}
for k in keys:
grad = var_to_grad[k]
print(k, grad.get_shape())
ref = tf.Variable(tf.zeros_like(grad))
ref = ref.assign_add(grad)
var_to_ref_grad[k] = ref
discounted_values = tf.placeholder(tf.float32, shape=[None, 1], name='discounted_values')
# control when to apply gradients
compute_gradients_flag = tf.placeholder(tf.int32, name="compute_gradients")
def fn1():
var_grad_list = []
for k in keys:
grad = var_to_ref_grad[k]
var = varname_to_var[k]
var_grad_list.append((grad,var))
optimizer.apply_gradients(var_grad_list)
return tf.no_op()
fn2 = lambda : tf.no_op()
last_op = tf.cond(tf.equal(compute_gradients_flag, 1), fn1, fn2)
with tf.Session(graph=graph) as s:
feed_dict= {
state : np.zeros([minibatch, 80, 80, 1]),
action_indexes: [1],
compute_gradients_flag: False,
}
s.run(tf.initialize_all_variables())
for i in range(10):
# accumulate gradients
s.run(last_op, feed_dict=feed_dict)
You don't really have to manually accumulate gradients. You can have Tensorflow accumulate them for you by applying the rollout update as a batch.
s_list = list_of_states_visited
a_list = list_of_actions_taken
R_list = list_of_value_targets
sess.run(local_net.update, feed_dict={
local_net.input: s_list,
local_net.a: a_list,
local_net.R: R_list
})
Something like this might work to create ops for accumulating gradients, resetting the accumulated gradients, and applying the accumulated gradients (untested!):
def build_gradient_accumulators(optimizer, gradients_and_variables):
accum_grads_and_vars = []
accumulators = []
resetters = []
for grad, var in gradients_and_variables:
accum = tf.Variable(tf.zeros_like(grad))
accum = accum.assign_add(grad)
accumulators.append(accum)
accum_grads_and_vars.append((accum, var))
resetters.append(tf.assign(accum, tf.zeros_like(accum)))
reset_op = tf.group(*resetters)
accum_op = tf.group(*accumulators)
apply_op = optimizer.apply_gradients(accum_grads_and_vars)
return reset_op, accum_op, apply_op
Given the following function to solve the two-dimensional advection equation in a rectangle:
function qnew = SemiLagrAdvect(u,v,q,qS,qN,qW,qE)
global N M
global dx dy
global dt Re
u = reshape(u,N,M);
v = reshape(v,N,M);
q = reshape(q,N,M);
%...embedding
qq = zeros(N+2,M+2);
qq(2:N+1,2:M+1) = q;
%...set the ghost values (four edges)
qq(1,2:M+1) = 2*qW-qq(2,2:M+1);
qq(N+2,2:M+1) = 2*qE-qq(N+1,2:M+1);
qq(2:N+1,1) = 2*qS-qq(2:N+1,2);
qq(2:N+1,M+2) = 2*qN-qq(2:N+1,M+1);
%...set the ghost values (four corners)
qq(1,1) = -qq(2,2);
qq(N+2,1) = -qq(N+1,2);
qq(N+2,M+2) = -qq(N+1,M+1);
qq(1,M+2) = -qq(2,M+1);
q1 = qq(2:N+1,2:M+1);
q2p = qq(3:N+2,2:M+1);
q2m = qq(1:N,2:M+1);
q3p = qq(2:N+1,3:M+2);
q3m = qq(2:N+1,1:M);
q4pp = qq(3:N+2,3:M+2);
q4mm = qq(1:N,1:M);
q4pm = qq(3:N+2,1:M);
q4mp = qq(1:N,3:M+2);
xi = -u*dt/dx;
eta = -v*dt/dy;
Q2 = q2p.*(xi>0) + q2m.*(xi<0);
Q3 = q3p.*(eta>0) + q3m.*(eta<0);
Q4 = q4pp.*((xi>0) & (eta>0)) + q4mm.*((xi<0) & (eta<0)) + ...
q4pm.*((xi>0) & (eta<0)) + q4mp.*((xi<0) & (eta>0));
qnew = (1-abs(xi)).*(1-abs(eta)).*q1 + ...
abs(xi).*(1-abs(eta)).*Q2 + ...
abs(eta).*(1-abs(xi)).*Q3 + ...
abs(xi).*abs(eta).*Q4;
qnew = qnew(:);
Having elementary knowledge in MATLAB, how can I modify it to solve the equation in a composite domain?
you need to use continuity on the interface.Depending on your method.
In the method of characteristics the interface is part of the initial curve.
S
I'm