If I have a model with several different layers, is there a way for me to modify the values in between those layers and see what the result would be if that modified value was passed through the rest of the network? I know you can use hooks to obtain the intermediate values during a forward pass but I would also like to modify them and send them through the rest of the network. How would I go about doing this?
In the forward function you can do what you want, for example:
class Model(nn.Module):
def __init__(self):
super(Model, self).__init__()
# example layers
self.dense1 = nn.Linear(1024, 512)
self.dense2 = nn.Linear(512, 256)
self.dense3 = nn.Linear(256, 128)
self.dense4 = nn.Linear(128)
self.dense5 = nn.Linear(128, 3)
def forward(self, x):
x = self.dense1(x)
# do what you want with x, for example...
# ... print the shape
print(x.shape)
# ... do any modification to x, ie. adding random numbers
x = x + torch.tensor(512, 1)
x = dense2(x)
x = dense3(x)
# loop through x for whatevery reason
for e in x:
# do smth with e
pass
# apply activation
x = F.relu(x)
# and so on and so on
(The model above makes not really sense but its for demonstration)
As you can see, in forward you can do (almost) whatevery you want between the layers
Related
I have an SGPR model:
import numpy as np
import gpflow
X, Y = np.random.randn(50, 2), np.random.randn(50, 1)
Z1 = np.random.randn(13, 2)
k = gpflow.kernels.SquaredExponential()
m = gpflow.models.SGPR(data=(X, Y), kernel=k, inducing_variable=Z1)
And I would like to assign inducing variable but with different shape, like:
Z2 = np.random.randn(29, 2)
m.inducing_variable.Z.assign(Z2)
But if I do it, I got:
ValueError: Shapes (13, 2) and (29, 2) are incompatible
is there a way to reassign the inducing variables without redefining the model?
Context: Instead of optimizing the model with the inducing variables, I would like to optimize the model without optimizing the inducing variables, manually reassigning the inducing variables at each step of the optimization.
UPDATE: This issue is resolved by https://github.com/GPflow/GPflow/pull/1594, which will become part of the next GPflow patch release (2.1.4).
With that fix, you don't need a custom class. All you need to do is explicitly set the static shape with None along the first dimension:
inducing_variable = gpflow.inducing_variables.InducingPoints(
tf.Variable(
Z1, # initial value
trainable=False, # True does not work - see Note below
shape=(None, Z1.shape[1]), # or even tf.TensorShape(None)
dtype=gpflow.default_float(), # required due to tf's 32bit default
)
)
m = gpflow.models.SGPR(data=(X, Y), kernel=k, inducing_variable=inducing_variable)
Then m.inducing_variable.Z.assign(Z2) should work just fine.
Note that in this case Z cannot be trainable, as the TensorFlow optimizers need to know the shape at construction time and don't support dynamic shapes.
Right now (as of GPflow 2.1.2) there is no built-in way to change the shape of inducing variables for SGPR, though it is in principle possible. You can get what you want with your own inducing variable class though:
class VariableInducingPoints(gpflow.inducing_variables.InducingPoints):
def __init__(self, Z, name=None):
super().__init__(Z, name=name)
# overwrite with Variable with None as first element in shape so
# we can assign arrays with arbitrary length along this dimension:
self.Z = tf.Variable(Z, dtype=gpflow.default_float(),
shape=(None, Z.shape[1])
)
def __len__(self):
return tf.shape(self.Z)[0] # dynamic shape
# instead of the static shape returned by the InducingPoints parent class
and then do
m = gpflow.models.SGPR(
data=(X, Y), kernel=k, inducing_variable=VariableInducingPoints(Z1)
)
instead. Then your m.inducing_variable.Z.assign() should work as you like it.
(For SVGP, the size of the inducing variable and the distribution defined by q_mu and q_sqrt has to match, as well as be known at construction time, so in this case changing the number of inducing variables is less trivial.)
I am trying to add two layers each of size (None, 24, 24, 8) but getting the class error as below:
Code:
x = add([layers[i-1],layers[i-9]])
or
x = Add()([layers[i-1],layers[i-9]])
Error:
/keras_222/local/lib/python2.7/site-packages/keras/engine/base_layer.py", line 285, in assert_input_compatibility
str(inputs) + '. All inputs to the layer '
ValueError: Layer add_1 was called with an input that isn't a symbolic tensor. **Received type: <class** 'keras.layers.normalization.BatchNormalization'>. Full input: [<keras.layers.normalization.BatchNormalization object at 0x7f04e4085850>, <keras.layers.normalization.BatchNormalization object at 0x7f050013cd10>]. All inputs to the **layer should be tensors**.
Please advise how to move forward. I also tried putting axis=1 or axis=-1 but it didn't work.
x = Add()([layers[i-1],layers[i-9]],axis=1)
or
x = Add()([layers[i-1],layers[i-9]], axis=-1)
The problem is that you are passing layers instead of tensors to your Add() layer. I suppose you have an Input() layer somewhere in your code. You need to pass this input through your other layers. Your code should instead look something like this:
input = Input(shape)
# pass input through other intermediate layers first if needed
output_1 = layers[i-1](input)
output_2 = layers[i-9](input)
x = Add()([output_1, output_2])
Is there a way to update a subset of parameters in dynet? For instance in the following toy example, first update h1, then h2:
model = ParameterCollection()
h1 = model.add_parameters((hidden_units, dims))
h2 = model.add_parameters((hidden_units, dims))
...
for x in trainset:
...
loss.scalar_value()
loss.backward()
trainer.update(h1)
renew_cg()
for x in trainset:
...
loss.scalar_value()
loss.backward()
trainer.update(h2)
renew_cg()
I know that update_subset interface exists for this and works based on the given parameter indexes. But then it is not documented anywhere how we can get the parameter indexes in dynet Python.
A solution is to use the flag update = False when creating expressions for parameters (including lookup parameters):
import dynet as dy
import numpy as np
model = dy.Model()
pW = model.add_parameters((2, 4))
pb = model.add_parameters(2)
trainer = dy.SimpleSGDTrainer(model)
def step(update_b):
dy.renew_cg()
x = dy.inputTensor(np.ones(4))
W = pW.expr()
# update b?
b = pb.expr(update = update_b)
loss = dy.pickneglogsoftmax(W * x + b, 0)
loss.backward()
trainer.update()
# dy.renew_cg()
print(pb.as_array())
print(pW.as_array())
step(True)
print(pb.as_array()) # b updated
print(pW.as_array())
step(False)
print(pb.as_array()) # b not updated
print(pW.as_array())
For update_subset, I would guess that the indices are the integers suffixed at the end of parameter names (.name()).
In the doc, we are supposed to use a get_index function.
Another option is: dy.nobackprop() which prevents the gradient to propagate beyond a certain node in the graph.
And yet another option is to zero the gradient of the parameter that do not need to be updated (.scale_gradient(0)).
These methods are equivalent to zeroing the gradient before the update. So, the parameter will still be updated if the optimizer uses its momentum from previous training steps (MomentumSGDTrainer, AdamTrainer, ...).
Can I convert a symbol that is a product of products into an array of products?
I tried to do something like this:
syms A B C D;
D = A*B*C;
factor(D);
but it doesn't factor it out (mostly because that isn't what factor is designed to do).
ans =
A*B*C
I need it to work if A B or C is replaced with any arbitrarily complicated parenthesized function, and it would be nice to do it without knowing what variables are in the function.
For example (all variables are symbolic):
D = x*(x-1)*(cos(z) + n);
factoring_function(D);
should be:
[x, x-1, (cos(z) + n)]
It seems like a string parsing problem, but I'm not confident that I can convert back to symbolic variables afterwards (also, string parsing in matlab sounds really tedious).
Thank you!
Use regexp on the string to split based on *:
>> str = 'x*(x-1)*(cos(z) + n)';
>> factors_str = regexp(str, '\*', 'split')
factors_str =
'x' '(x-1)' '(cos(z) + n)'
The result factor_str is a cell array of strings. To convert to a cell array of sym objects, use
N = numel(factors_str);
factors = cell(1,N); %// each cell will hold a sym factor
for n = 1:N
factors{n} = sym(factors_str{n});
end
I ended up writing the code to do this in python using sympy. I think I'm going to port the matlab code over to python because it is a more preferred language for me. I'm not claiming this is fast, but it serves my purposes.
# Factors a sum of products function that is first order with respect to all symbolic variables
# into a reduced form using products of sums whenever possible.
# #params orig_exp A symbolic expression to be simplified
# #params depth Used to control indenting for printing
# #params verbose Whether to print or not
def factored(orig_exp, depth = 0, verbose = False):
# Prevents sympy from doing any additional factoring
exp = expand(orig_exp)
if verbose: tabs = '\t'*depth
terms = []
# Break up the added terms
while(exp != 0):
my_atoms = symvar(exp)
if verbose:
print tabs,"The expression is",exp
print tabs,my_atoms, len(my_atoms)
# There is nothing to sort out, only one term left
if len(my_atoms) <= 1:
terms.append((exp, 1))
break
(c,v) = collect_terms(exp, my_atoms[0])
# Makes sure it doesn't factor anything extra out
exp = expand(c[1])
if verbose:
print tabs, "Collecting", my_atoms[0], "terms."
print tabs,'Seperated terms with ',v[0], ', (',c[0],')'
# Factor the leftovers and recombine
c[0] = factored(c[0], depth + 1)
terms.append((v[0], c[0]))
# Combines trivial terms whenever possible
i=0
def termParser(thing): return str(thing[1])
terms = sorted(terms, key = termParser)
while i<len(terms)-1:
if equals(terms[i][1], terms[i+1][1]):
terms[i] = (terms[i][0]+terms[i+1][0], terms[i][1])
del terms[i+1]
else:
i += 1
recombine = sum([terms[i][0]*terms[i][1] for i in range(len(terms))])
return simplify(recombine, ratio = 1)
I have a structure, from which I want to access repeatedly the fields to I load them in the current space like this (where M is type with fields X and Y):
X = M.X
Y = M.Y
in R, I often use the with command to do that. For now I would just like to be able to have a macro that expends that code, something along the lines of
#attach(M,[:X,:Y])
I am just not sure how exactly to do this.
I've included in this answer a macro that does pretty much what you describe. Comments explaining what's going on are inline.
macro attach(struct, fields...)
# we want to build up a block of expressions.
block = Expr(:block)
for f in fields
# each expression in the block consists of
# the fieldname = struct.fieldname
e = :($f = $struct.$f)
# add this new expression to our block
push!(block.args, e)
end
# now escape the evaled block so that the
# new variable declarations get declared in the surrounding scope.
return esc(:($block))
end
You use it like this: #attach M, X, Y
you can see the generated code like so: macroexpand(:(#attach M, X, Y)) which will show something like this:
quote
X = M.X
Y = M.Y
end