I am trying to use NumbaLSODA, see here, to solve many systems iteratively.
However, it seems that I can't put the function in the numba cache and so I have a big overhead at each iteration.
I was wondering if anybody could help.
This is my code.
First I put all numba related function inside a file
numba_func.py
NOPYTHON = True
CACHE = True
PARALLEL = True
#nb.cfunc(lsoda_sig, nopython=NOPYTHON, cache=CACHE)
def rhs(x, i, di, p):
di[0] = ...
di[1] = ...
di[2] = ...
funcptr = rhs.address
#nb.njit('(float64)(float64, float64, float64, int16)', nopython=NOPYTHON, parallel=PARALLEL, cache=CACHE)
def solve(a, b, c, funcptr):
w0 = np.array([a, b, ...], dtype=np.float64)
p = np.array([c], dtype=np.float64)
x = np.linspace(0, 100, 500)
usol, success = lsoda(funcptr, w0, x, data=p)
return usol[-1][1]
Then I use another file to solve the systems one after the others
from numba_func import solve, funcptr
gs = []
for a, b, c in zip(as, bs, cs):
gs = np.append(gs, solve(a, b, c, funcptr))
I got the following warning:
NumbaWarning: Cannot cache compiled function "solve" as it uses dynamic globals (such as ctypes pointers and large global arrays)
I guess the idea is to pass the variable funcptr correctly so that numba is happy but so far I failed...
Related
I am new with Numba. I am trying to accelerate a pretty complicated solver. However, I keep getting an error such as
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend) Untyped global name 'isinstance': Cannot determine Numba type of <class 'builtin_function_or_method'>
I wrote a small example to reproduce the same error:
import numba
import numpy as np
from numba import types
from numpy import zeros_like, isfinite
from numpy.linalg import solve
from numpy.random import uniform
#numba.njit(parallel=True)
def foo(A_, b_, M1=None, M2=None):
x_ = zeros_like(b_)
r = b_ - A_.dot(x_)
flag = 1
if isinstance(M1, types.NoneType): # Error here
y = r
else:
y = solve(M1, r)
if not isfinite(y).any():
flag = 2
if isinstance(M2, types.NoneType):
z = y
else:
z = solve(M2, y)
if not isfinite(z).any():
flag = 2
return z, flag
N = 10
tmp = np.random.rand(N, N)
A = np.dot(tmp, tmp.T)
x = np.zeros((N, 1), dtype=np.float64)
b = np.vstack([uniform(0.0, 1.0) for i in range(N)])
X_1, info = foo(A, b)
Also if I change the decorator to generated_jit() I get the following error:
r = b_ - A_.dot(x_)
AttributeError: 'Array' object has no attribute 'dot'
Numba compiles the function and requires every variables to be statically typed. This means that each variable has only one unique type: one variable cannot be of both the type NoneType and something else as opposed to with CPython based on dynamic typing. Dynamic typing is also a major source of the slowdown of CPython. Thus, using isinstance in nopython JITed Numba functions does not make much sense. In fact, this built-in function is not supported.
That being said, Numba supports optional arguments by specifying optional(ArgumentType) in the signature (note that the resulting type of the variable is optional(ArgumentType) and not ArgumentType nor NoneType. You can then test if the argument is set using if yourArgument is None:. I do not know what is the type of M1 and M2 in your code but they need to be explicitly defined in the signature with optional argument.
I am trying to replicate the results posted here (How to use scipy.optimize minimize_scalar when objective function has multiple arguments?) using a different structure. The goal is exactly the same but I just want to code it in a different way. Here is my code:
def mini(g,a,b,args):
object=lambda w1: g(w1, *args)
result=minimize_scalar(object, bounds=(a, b))
minzer, minval=result.x, result.fun
return minzer,minval
def errorr(w0,w1,x,y):
y_pred = w0 + w1*x
mse = ((y-y_pred)**2).mean()
return mse
x = np.array([1,2,3])
y = np.array([52,54,56])
w0=50
mini(errorr, -5, 5, (w0,x,y))
However, the results obtained using my code is quite different from the one in the original posts. I am wondering where did I make the mistake in my code that caused the different results. Thanks!
Since you use lambda w1: g(w1, *args), you are minimizing with respect to the first function argument w0. To minimize with respect to w1, you can write lambda w1: g(args[0], w1, *args[1:]) instead.
However, please avoid python keywords as variable names (e.g. object). In addition, a lambda function is an anonymous function, so assigning it to a variable contradicts its purpose. Consequently, I'd propose either
def mini(g,a,b,args):
def obj_fun(w1): return g(args[0], w1, *args[1:])
result = minimize_scalar(obj_fun, bounds=(a, b))
return result.x, result.fun
or
def mini(g,a,b,args):
result = minimize_scalar(lambda w1: g(args[0], w1, *args[1:]), bounds=(a, b))
return result.x, result.fun
Is there a way to update a subset of parameters in dynet? For instance in the following toy example, first update h1, then h2:
model = ParameterCollection()
h1 = model.add_parameters((hidden_units, dims))
h2 = model.add_parameters((hidden_units, dims))
...
for x in trainset:
...
loss.scalar_value()
loss.backward()
trainer.update(h1)
renew_cg()
for x in trainset:
...
loss.scalar_value()
loss.backward()
trainer.update(h2)
renew_cg()
I know that update_subset interface exists for this and works based on the given parameter indexes. But then it is not documented anywhere how we can get the parameter indexes in dynet Python.
A solution is to use the flag update = False when creating expressions for parameters (including lookup parameters):
import dynet as dy
import numpy as np
model = dy.Model()
pW = model.add_parameters((2, 4))
pb = model.add_parameters(2)
trainer = dy.SimpleSGDTrainer(model)
def step(update_b):
dy.renew_cg()
x = dy.inputTensor(np.ones(4))
W = pW.expr()
# update b?
b = pb.expr(update = update_b)
loss = dy.pickneglogsoftmax(W * x + b, 0)
loss.backward()
trainer.update()
# dy.renew_cg()
print(pb.as_array())
print(pW.as_array())
step(True)
print(pb.as_array()) # b updated
print(pW.as_array())
step(False)
print(pb.as_array()) # b not updated
print(pW.as_array())
For update_subset, I would guess that the indices are the integers suffixed at the end of parameter names (.name()).
In the doc, we are supposed to use a get_index function.
Another option is: dy.nobackprop() which prevents the gradient to propagate beyond a certain node in the graph.
And yet another option is to zero the gradient of the parameter that do not need to be updated (.scale_gradient(0)).
These methods are equivalent to zeroing the gradient before the update. So, the parameter will still be updated if the optimizer uses its momentum from previous training steps (MomentumSGDTrainer, AdamTrainer, ...).
I have various structs with fields W, P, E, which contain numerical values. I want to develop a clean way to add and subtract these structs without unpacking and repacking in subfunctions each time (which has been my solution thus far)
For example, given:
S.W = 2
S.P = 3
S.E = 4
M.W = 20
M.P = 30
M.E = 40
I want to be able to do X = S + M and end up with:
X.W = 22
X.P = 33
X.E = 44
My current attempt to do this, is by means of a new class, which looks as follows:
classdef CV
properties
W
P
E
end
methods
function r = plus(o1,o2)
r = CV;
r.E = o1.E + o2.E;
r.P = o1.P + o2.P;
r.W = o1.W + o2.W;
end
end
end
This allows for doing S + M and returns a new variable in the same form as the inputs. I'm generally unfamiliar with classes, and wanted to know if this is proper form. If so, I might go ahead and add functions for minus and times in the methods section. However, this seems like it requires a lot of repetitive code and I feel there must be a simpler solution. Any advice is much appreciated.
The following code directly works on structs without nesting them in a class. It is assumed that the two input structs have the same field names (in this example W, P, and E), however, the order of those may be arbitrary (you mentioned in a comment that this is important for your application).
function X = structplus(S, M)
fn = fieldnames(S);
for i = 1 : numel(fn)
X.(fn{i}) = M.(fn{i}) + S.(fn{i});
end
end
So defining
S.W = 2
S.P = 3
S.E = 4
M.E = 40
M.P = 30
M.W = 20
(note the reverse order of M) and calling
X = structplus(S, M)
yields a struct with field names that are ordered like the first argument:
X =
struct with fields:
W: 22
P: 33
E: 44
To expand on Le Phlaux's answer, you can provide a function handle to the required binary operator (e.g. #plus, #minus) and work on sub-structures recursively
function out = structBinaryFunc(in1, in2, func)
fn = fieldnames(in1);
for ii = 1:numel(fn)
if isstruct(in1.(fn{ii}))
out.(fn{ii}) = structBinaryFunc(in1.(fn{ii}), in2.(fn{ii}), func)
else
out.(fn{ii}) = func(in1.(fn{ii}), in2.(fn{ii}));
end
end
For your example you would call X = structBinaryFunc(S, M, #plus);.
I have an optimization problem (1d) coded in 2 ways - one using a for loop and an other using numpy arrays. The for loop version works fine but the numpy one fails.
Actually it is a bit more complicated, it can work with different starting points (!!) or if I choose an other optimization algo like CG.
The 2 versions (functions and gradients) are giving the same results and the returned types are also the same as far as I can tell.
Here is my example, what am I missing?
import numpy as np
from scipy.optimize import minimize
# local params
v1 = np.array([1., 1.])
v2 = np.array([1., 2.])
# local functions
def f1(x):
s = 0
for i in range(len(v1)):
s += (v1[i]*x-v2[i])**2
return 0.5*s/len(v1)
def df1(x):
g = 0
for i in range(len(v1)):
g += v1[i]*(v1[i]*x-v2[i])
return g/len(v1)
def f2(x):
return 0.5*np.sum((v1*x-v2)**2)/len(v1)
def df2(x):
return np.sum(v1*(v1*x-v2))/len(v1)
x0 = 10. # x0 = 2 works
# tests...
assert np.abs(f1(x0)-f2(x0)) < 1.e-6 and np.abs(df1(x0)-df2(x0)) < 1.e-6 \
and np.abs((f1(x0+1.e-6)-f1(x0))/(1.e-6)-df1(x0)) < 1.e-4
# BFGS for f1: OK
o = minimize(f1, x0, method='BFGS', jac=df1)
if not o.success:
print('FAILURE', o)
else:
print('SUCCESS min = %f reached at %f' % (f1(o.x[0]), o.x[0]))
# BFGS for f2: failure
o = minimize(f2, x0, method='BFGS', jac=df2)
if not o.success:
print('FAILURE', o)
else:
print('SUCCESS min = %f reached at %f' % (f2(o.x[0]), o.x[0]))
The error I get is
A1 = I - sk[:, numpy.newaxis] * yk[numpy.newaxis, :] * rhok
IndexError: invalid index to scalar variable.
but I doesn't really helps me since it can work with some other starting values.
I am using an all new fresh python install (python 3.5.2, scipy 0.18.1 and numpy 1.11.3).
The solver expects the return value of jacobian df2 to be the same shape as its input x. Even though you passed in a scalar here, it's actually converted into a single element ndarray. Since you used np.sum, your result became scalar and that causes strange things to happen.
Enclose the scalar result of df2 with np.array, and your code should work.