Constraints in scipy.optimize throwing x0 error - scipy

Looking to take a list of stocks and adjust their weight in a portfolio until the overall portfolio beta is 1.0 the output of "stonkBetas" is static and is:
[3.19292010501853,
0.7472001935364129,
1.0889157697158605,
0.8944059912707691,
0.04192080860817828,
1.0011520737327186,
0.9155119223385676]
I then create two functions. One to define how the betas are weighted. The second just as con that will make the constraint that the sum of the minimized weighted portfolio will have an overall summed beta of 1.0.
def betaOpp(weights):
a,b,c,d,e,f,g=weights
f=a*stonkBetas[0]+b*stonkBetas[1]+c*stonkBetas[2]+d*stonkBetas[3]+e*stonkBetas[4]+f*stonkBetas[5]+g*stonkBetas[6]
return f
initial_guess = [.1,.1,.1,.1,.2,.2,.2]
print('hi')
print(sum(initial_guess))
print('bye')
def con(t):
print('this should be zero:')
print(sum(t)-1)
return sum(t) - 1.0
cons = {'type':'eq', 'fun': con}
bnds = ((.02,.8),(.02,.8),(.02,.8),(.02,.8),(.02,.8),(.02,.8),(.02,.8))
res = optimize.minimize(betaOpp,initial_guess, bounds=bnds, constraints=cons)
print(res)
This gives me this output
hi
1.0
bye
this should be zero:
0.0
this should be zero:
0.0
this should be zero:
0.0
this should be zero:
1.4901161193847656e-08
this should be zero:
1.4901161193847656e-08
this should be zero:
1.4901161193847656e-08
this should be zero:
1.4901161193847656e-08
this should be zero:
1.4901161193847656e-08
this should be zero:
1.4901161193847656e-08
this should be zero:
1.4901161193847656e-08
this should be zero:
6.661338147750939e-16
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-30-6567109e94a4> in <module>
16 cons = {'type':'eq', 'fun': con}
17 bnds = ((.02,.8),(.02,.8),(.02,.8),(.02,.8),(.02,.8),(.02,.8),(.02,.8))
---> 18 res = optimize.minimize(betaOpp,x0=initial_guess, bounds=bnds, constraints=cons)
19 print(res)
/opt/miniconda3/lib/python3.6/site-packages/scipy/optimize/_minimize.py in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
624 elif meth == 'slsqp':
625 return _minimize_slsqp(fun, x0, args, jac, bounds,
--> 626 constraints, callback=callback, **options)
627 elif meth == 'trust-constr':
628 return _minimize_trustregion_constr(fun, x0, args, jac, hess, hessp,
/opt/miniconda3/lib/python3.6/site-packages/scipy/optimize/slsqp.py in _minimize_slsqp(func, x0, args, jac, bounds, constraints, maxiter, ftol, iprint, disp, eps, callback, finite_diff_rel_step, **unknown_options)
424
425 if mode == -1: # gradient evaluation required
--> 426 g = append(sf.grad(x), 0.0)
427 a = _eval_con_normals(x, cons, la, n, m, meq, mieq)
428
/opt/miniconda3/lib/python3.6/site-packages/scipy/optimize/_differentiable_functions.py in grad(self, x)
186 if not np.array_equal(x, self.x):
187 self._update_x_impl(x)
--> 188 self._update_grad()
189 return self.g
190
/opt/miniconda3/lib/python3.6/site-packages/scipy/optimize/_differentiable_functions.py in _update_grad(self)
169 def _update_grad(self):
170 if not self.g_updated:
--> 171 self._update_grad_impl()
172 self.g_updated = True
173
/opt/miniconda3/lib/python3.6/site-packages/scipy/optimize/_differentiable_functions.py in update_grad()
90 self.ngev += 1
91 self.g = approx_derivative(fun_wrapped, self.x, f0=self.f,
---> 92 **finite_diff_options)
93
94 self._update_grad_impl = update_grad
/opt/miniconda3/lib/python3.6/site-packages/scipy/optimize/_numdiff.py in approx_derivative(fun, x0, method, rel_step, abs_step, f0, bounds, sparsity, as_linear_operator, args, kwargs)
389
390 if np.any((x0 < lb) | (x0 > ub)):
--> 391 raise ValueError("`x0` violates bound constraints.")
392
393 if as_linear_operator:
ValueError: `x0` violates bound constraints.
And I just don't understand where I'm going wrong. The x0 is perfectly 1.0 - I can see it! Hopefully I'm just doing something stupid here. Please help!

Related

networkx maximum_flow crashes on some pairs of nodes

I have a graph composed of 742 edges, and 360 nodes.
I want to compute max flow between some pairs of nodes and it happens, for some of them the nx.maximum_flow ends with the pasted error, despite the fact that a path exists between the two concerned nodes.
Any idea what causes that?
Thanks.
ValueError Traceback (most recent call last)
<ipython-input-186-6dae3501e3d0> in <module>()
1 #print(nx.shortest_path(G,source="Sink_0",target="node_32"))
----> 2 nx.maximum_flow(G, "Sink_0", "Aircraft2_32")
/Library/Python/2.7/site-packages/networkx/algorithms/flow/maxflow.pyc in maximum_flow(G, s, t, capacity, flow_func, **kwargs)
156 raise nx.NetworkXError("flow_func has to be callable.")
157
--> 158 R = flow_func(G, s, t, capacity=capacity, value_only=False, **kwargs)
159 flow_dict = build_flow_dict(G, R)
160
/Library/Python/2.7/site-packages/networkx/algorithms/flow/preflowpush.pyc in preflow_push(G, s, t, capacity, residual, global_relabel_freq, value_only)
420 """
421 R = preflow_push_impl(G, s, t, capacity, residual, global_relabel_freq,
--> 422 value_only)
423 R.graph['algorithm'] = 'preflow_push'
424 return R
/Library/Python/2.7/site-packages/networkx/algorithms/flow/preflowpush.pyc in preflow_push_impl(G, s, t, capacity, residual, global_relabel_freq, value_only)
279 break
280 u = next(iter(level.active))
--> 281 height = discharge(u, False)
282 if grt.is_reached():
283 # Global relabeling heuristic.
/Library/Python/2.7/site-packages/networkx/algorithms/flow/preflowpush.pyc in discharge(u, is_phase1)
156 # We have run off the end of the adjacency list, and there can
157 # be no more admissible edges. Relabel the node to create one.
--> 158 height = relabel(u)
159 if is_phase1 and height >= n - 1:
160 # Although the node is still active, with a height at least
/Library/Python/2.7/site-packages/networkx/algorithms/flow/preflowpush.pyc in relabel(u)
125 """
126 grt.add_work(len(R_succ[u]))
--> 127 return min(R_node[v]['height'] for v, attr in R_succ[u].items()
128 if attr['flow'] < attr['capacity']) + 1
129
ValueError: min() arg is an empty sequence

Bicubic interpolation beyond grid values in Matlab

Is it possible to achieve bi-cubic interpolation beyond grid values?
For example:
L = [5,10,20,25,40];
W= 1:3;
S= [50 99 787
779 795 850
803 779 388
886 753 486
849 780 598];
size1 = griddata(W,L,S,2,40,'cubic')
sizeBeyond = griddata(W,L,S,2,41,'cubic')
sizeV4 = griddata(W,L,S,2,41,'v4')
returns:
size1 = 780
sizeBeyond = NaN
sizeV4 = 721.57
What I was suggesting is, you can input the values which are extrapolated. Check the below code. But note that, as suggested by flawr, the extrapolation behave really bad.
l = [5,10,20,25,40];
w = 1:3;
li = [l 41] ;
S = [50 99 787
779 795 850
803 779 388
886 753 486
849 780 598];
[W,L] = meshgrid(w,l) ;
[Wi,Li] = meshgrid(w,li) ;
Si = interp2(W,L,S,Wi,Li,'spline') ;
size1 = griddata(W,L,S,2,40,'cubic')
sizeBeyond = griddata(Wi,Li,Si,2,41,'cubic')
sizeV4 = griddata(W,L,S,2,41,'v4')
Note: Don't use inbuilt commands like length,size etc as variables in the code, even for demonstration, it is trouble some.
Though, this is not answer, I have to post it here as for discussion.

Kalman Filter implementation for one dimensional vector

I am having an hard time figuring out something really stupid I think.
I want to refresh my kalman filter knowledge but I am not able to make it work.
In a simple case with
real states x:
x = [ 1000 750 563 422 316 237 178 133 100 75]';
and observations
z_array = [ 1056 943 469 235 433 230 116 136 -75 267]';
I get something really nice (estimated stands for FILTERED):
When instead I use another signal, more complex. I don't know why but it doesn't work (estimated stands for FILTERED).
The problem, I am quite sure is in how I choose the a and r in my code for the second case. But how can I change them?
Here is the short code I have written in MATLAB and I am getting lost (I am really ashamed because I should know how to do this!!).
close all
z_array = noisyMeas.signals.values;
x = meas.signals.values;
a = 0.75;
r= 200;
p=1;
% x = [ 1000 750 563 422 316 237 178 133 100 75]';
% z_array = [ 1056 943 469 235 433 230 116 136 -75 267]';
xhat_array = zeros(length(z_array),1);
xhat = z_array(1);
xhat_array(1) = xhat;
% Core of the Kalman Filter
for k=2:length(x)
z = z_array(k);
% Predict
xhat = a * xhat;
p = a * p * a;
% Update
g = p / (p + r);
xhat = xhat + g * (z - xhat);
p = (1 - g) * p;
xhat_array(k) = xhat;
end
% plotting
figure
hold on
plot(xhat_array,'r')
plot(x,'b');
plot(z_array,'g')
grid on
legend('estimated','notNoisy','Noisy')
Thanks in advance for your help.

Applying scipy.sparse.linalg.svds throws a Memory Error?

I try to decompose a sparse matrix(40,000×1,400,000) with scipy.sparse.linalg.svds on my 64-bit machine with 140GB RAM. as following:
k = 5000
tfidf_mtx = tfidf_m.tocsr()
u_45,s_45,vT_45 = scipy.sparse.linalg.svds(tfidf_mtx, k=k)
When the K ranges from 1000 to 4500, it works. But the K is 5000, it throws an MemoryError.The precise error is given below:
---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
<ipython-input-6-31a69ce54e2c> in <module>()
4 k = 4000
5 tfidf_mtx = tfidf_m.tocsr()
----> 6 get_ipython().magic(u'time u_50,s_50,vT_50 =linalg.svds(tfidf_mtx, k=k))
7 # print len(s),s
8
/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.pyc in magic(self, arg_s)
2163 magic_name, _, magic_arg_s = arg_s.partition(' ')
2164 magic_name = magic_name.lstrip(prefilter.ESC_MAGIC)
-> 2165 return self.run_line_magic(magic_name, magic_arg_s)
2166
2167 #-------------------------------------------------------------------------
/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.pyc in run_line_magic(self, magic_name, line)
2084 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
2085 with self.builtin_trap:
-> 2086 result = fn(*args,**kwargs)
2087 return result
2088
/usr/lib/python2.7/dist-packages/IPython/core/magics/execution.pyc in time(self, line, cell, local_ns)
/usr/lib/python2.7/dist-packages/IPython/core/magic.pyc in <lambda>(f, *a, **k)
189 # but it's overkill for just that one bit of state.
190 def magic_deco(arg):
--> 191 call = lambda f, *a, **k: f(*a, **k)
192
193 if callable(arg):
/usr/lib/python2.7/dist-packages/IPython/core/magics/execution.pyc in time(self, line, cell, local_ns)
1043 else:
1044 st = clock2()
-> 1045 exec code in glob, local_ns
1046 end = clock2()
1047 out = None
<timed exec> in <module>()
/usr/local/lib/python2.7/dist-packages/scipy/sparse/linalg/eigen/arpack/arpack.pyc in svds(A, k, ncv, tol, which, v0, maxiter, return_singular_vectors)
1751 else:
1752 ularge = eigvec[:, above_cutoff]
-> 1753 vhlarge = _herm(X_matmat(ularge) / slarge)
1754
1755 u = _augmented_orthonormal_cols(ularge, nsmall)
/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.pyc in dot(self, other)
244
245 """
--> 246 return self * other
247
248 def __eq__(self, other):
/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.pyc in __mul__(self, other)
298 return self._mul_vector(other.ravel()).reshape(M, 1)
299 elif other.ndim == 2 and other.shape[0] == N:
--> 300 return self._mul_multivector(other)
301
302 if isscalarlike(other):
/usr/local/lib/python2.7/dist-packages/scipy/sparse/compressed.pyc in _mul_multivector(self, other)
463
464 result = np.zeros((M,n_vecs), dtype=upcast_char(self.dtype.char,
--> 465 other.dtype.char))
466
467 # csr_matvecs or csc_matvecs
MemoryError:
The when the k is 3000 and 4500, the ratio of the sum of the square of singular values to the sum of the square of all matrix entities is respectively 0.7033 and 0.8230. I am searching for a long time on net. But no use. Please help or try to give some ideas how to achieve this.
So the return is an (M,k) array. On an ordinary older machine:
In [368]: np.ones((40000,1000))
....
In [369]: np.ones((40000,4000))
...
In [370]: np.ones((40000,5000))
...
--> 190 a = empty(shape, dtype, order)
191 multiarray.copyto(a, 1, casting='unsafe')
192 return a
MemoryError:
Now may just be a coincidence that I hit the memory error at the same size are your code. But if you make the problem big enough you will hit memory errors at some point.
Your stacktrace shows the error occurs while multiplying a sparse matrix and a dense 2d array (other), and the result will be dense as well.

Fast way to find the neighboor of pixel

I am programming for task that finds the neighbor of a given pixel x in image Dthat can formula as:
The formula shown pixels y which satisfy the distance to pixel x is 1, then they are neighbor of pixel x. This is my matlab code. However, it still takes long time to find. Could you suggest a faster way to do it. Thank you so much
%-- Find the neighborhood of one pixel
% x is pixel coordinate
% nrow, ncol is size of image
function N = find_neighbor(x,nrow,ncol)
i = x(1);
j = x(2);
I1 = i+1;
if (I1 > nrow)
I1 = nrow;
end
I2 = i-1;
if (I2 < 1)
I2 = 1;
end
J1 = j+1;
if (J1 > ncol)
J1 = ncol;
end
J2 = j-1;
if (J2 < 1)
J2 = 1;
end
N = [I1, I2, i, i; j, j, J1, J2];
For example: ncol=128; nrow=128; x =[30;110] then output
N =31 29 30 30; 110 110 111 109]
For calling the function in loop
x=[30 31 32 33; 110 123 122 124]
for i=1:length(x)
N = find_neighbor(x(:,i),nrow,ncol);
end
Here's a vectorized approach using bsxfun:
% define four neighbors as coordinate differences
d = [-1 0 ; 1 0 ; 0 -1 ; 0 1]';
% add to pixel coordinates
N = bsxfun(#plus, x, permute(d, [1 3 2]));
% make one long list for the neighbors of all pixels together
N = reshape(N, 2, []);
% identify out-of-bounds coordinates
ind = (N(1, :) < 1) | (N(1, :) > nrow) | (N(2, :) < 1) | (N(2, :) > ncol);
% and remove those "neighbors"
N(:, ind) = [];
The permute is there to move the "dimension" of four different neighbors into the 3rd array index. This way, using bsxfun, we get the combination of every pair of original pixel coordinates with every pair of relative neighbor coordinates. The out-of-bounds check assumes that nrow belongs to the first coordinate and ncol to the second coordinate.
With
ncol=128;
nrow=128;
x = [30 31 32 33; 110 123 122 124];
the result is
N =
29 30 31 32 31 32 33 34 30 31 32 33 30 31 32 33
110 123 122 124 110 123 122 124 109 122 121 123 111 124 123 125
Different neighbors of different pixels can end up to be the same pixel, so there can be duplicates in the list. If you only want each resulting pixel once, use
% remove duplicates?
N = unique(N', 'rows')';
to get
N =
29 30 30 30 31 31 31 32 32 32 33 33 33 34
110 109 111 123 110 122 124 121 123 124 122 123 125 124
Matlab's performance is horrible when calling small functions many time. The Matlab approach is to do vectorize as much as possible. A vectorized version of your code:
function N = find_neighbor(x,nrow,ncol)
N = [min(x(1,:)+1,nrow), max(x(1,:)-1,1), x(1,:), x(1,:); x(2,:), x(2,:),min(x(2,:)+1,ncol), max(x(2,:)-1,1)];
end
and usage
x=[30 31 32 33; 110 123 122 124]
N = find_neighbor(x,nrow,ncol);
BTW, for pixels on the border , your solution always gives 4 neighbors. This is wrong. the neighbors of (1,1) for examples should be only (2,1) and (1,2), while you add two extra (1,1).
The solution to this is quite simple - delete all neighbors that are outside the image
function N = find_neighbor(x,nrow,ncol)
N = [x(1,:)+1, x(1,:)-1, x(1,:), x(1,:); x(2,:), x(2,:),x(2,:)+1, x(2,:)-1];
N(:,N(1,:)<1 | N(1,:)> nrow | N(2,:)<1 | N(2,:)>ncol)=[];
end