I am looking to minimize an objective function subject to certain constraints.
The function that I am looking to minimize is:
def distance_function(choice_matrix, distance_matrix, factory_distance):
hub_to_demand_distance = distance_matrix.dot(choice_matrix)
hub_factory_distance = pd.concat([hub_to_demand_distance, factory_distance],axis=1)
min_dist_to_demand = pd.DataFrame(hub_factory_distance.min(axis=1))
transposed_choice = choice_matrix.T
factory_to_hub = transposed_choice.dot(factory_distance)
total_distance = min_dist_to_demand.sum(axis=0)+factory_to_hub.sum(axis=0)
return total_distance
These are the constraints that I have defined:
cons = (
{'type':'ineq','fun': lambda f: 1-choice_matrix[0][0]-choice_matrix[1][0]},
{'type':'ineq','fun': lambda f: 1-choice_matrix[0][1]-choice_matrix[1][1]},
{'type':'ineq','fun': lambda f: 1-choice_matrix[0][2]-choice_matrix[1][2]},
{'type':'ineq','fun': lambda f: 1-choice_matrix[0][3]-choice_matrix[1][3]},
{'type':'eq','fun': lambda f: choice_matrix[0][0]+choice_matrix[0][1]+choice_matrix[0][2]+choice_matrix[0][3]-1},
{'type':'eq','fun': lambda f: choice_matrix[1][0]+choice_matrix[1][1]+choice_matrix[1][2]+choice_matrix[1][3]-1}
)
I have tried using Scipy Optimize to minimize the function as shown:
optimize.minimize(distance_function, choice_matrix, args=(distance_matrix, factory_distance),method='SLSQP',jac=None,constraints=cons)
When I run this, I get the following error:
ValueError: Dot product shape mismatch, (4, 4) vs (8,)
Could you please tell me:
Why this is happening and what needs to be done?
In the code that I shown, I have taken Choice Matrix to have 4 rows and 2 columns and hence I have manually defined 6 constraints (The constraint is the sum of the elements in each row should be lesser than or equal to 1. The other constraint is the sum of the elements in each column should be equal to 1)
My question is if my Choice Matrix has 40 rows and 5 columns, is there a better way to define the constraints than manually entering 45 lines?
Thank you in advance for your help!
Related
A little bit of background. I'm trying to make a model for clustering a Design Structure Matrix(DSM). I made a draft model and have a couple of questions. Most of them are not directly related to DSM per se.
include "globals.mzn";
int: dsmSize = 7;
int: maxClusterSize = 7;
int: maxClusters = 4;
int: powcc = 2;
enum dsmElements = {A, B, C, D, E, F,G};
array[dsmElements, dsmElements] of int: dsm =
[|1,1,0,0,1,1,0
|0,1,0,1,0,0,1
|0,1,1,1,0,0,1
|0,1,1,1,1,0,1
|0,0,0,1,1,1,0
|1,0,0,0,1,1,0
|0,1,1,1,0,0,1|];
array[1..maxClusters] of var set of dsmElements: clusters;
array[1..maxClusters] of var int: clusterCard;
constraint forall(i in 1..maxClusters)(
clusterCard[i] = pow(card(clusters[i]), powcc)
);
% #1
% constraint forall(i, j in clusters where i != j)(card(i intersect j) == 0);
% #2
constraint forall(i, j in 1..maxClusters where i != j)(
card(clusters[i] intersect clusters[j]) == 0
);
% #3
% constraint all_different([i | i in clusters]);
constraint (clusters[1] union clusters[2] union clusters[3] union clusters[4]) = dsmElements;
var int: intraCost = sum(i in 1..maxClusters, j, k in clusters[i] where k != j)(
(dsm[j,k] + dsm[k,j]) * clusterCard[i]
) ;
var int: extraCost = sum(el in dsmElements,
c in clusters where card(c intersect {el}) = 0,
k,j in c)(
(dsm[j,k] + dsm[k,j]) * pow(card(dsmElements), powcc)
);
var int: TCC = trace("\(intraCost), \(extraCost)\n", intraCost+extraCost);
solve maximize TCC;
Question 1
I was under the impression, that constraints #1 and #2 are the same. However, seems like they are not. The question here is why? What is the difference?
Question 2
How can I replace constraint #2 with all_different? Does it make sense?
Question 3
Why the trace("\(intraCost), \(extraCost)\n", intraCost+extraCost); shows nothing in the output? The output I see using gecode is:
Running dsm.mzn
intraCost, extraCost
clusters = array1d(1..4, [{A, B, C, D, E, F, G}, {}, {}, {}]);
clusterCard = array1d(1..4, [49, 0, 0, 0]);
----------
<sipped to save space>
----------
clusters = array1d(1..4, [{B, C, D, G}, {A, E, F}, {}, {}]);
clusterCard = array1d(1..4, [16, 9, 0, 0]);
----------
==========
Finished in 5s 419msec
Question 4
The expression constraint (clusters[1] union clusters[2] union clusters[3] union clusters[4]) = dsmElements;, here I wanted to say that the union of all clusters should match the set of all nodes. Unfortunately, I did not find a way to make this big union more dynamic, so for now I just manually provide all clusters. Is there a way to make this expression return union of all sets from the array of sets?
Question 5
Basically, if I understand it correctly, for example from here, the Intra-cluster cost is the sum of all interactions within a cluster multiplied by the size of the cluster in some power, basically the cardinality of the set of nodes, that represents the cluster.
The Extra-cluster cost is a sum of interactions between some random element that does not belong to a cluster and all elements of that cluster multiplied by the cardinality of the whole space of nodes to some power.
The main question here is are the intraCost and extraCost I the model correct (they seem to be but still), and is there a better way to express these sums?
Thanks!
(Perhaps you would get more answers if you separate this into multiple questions.)
Question 3:
Here's an answer on the trace question:
When running the model, the trace actually shows this:
intraCost, extraCost
which is not what you expect, of course. Trace is in effect when creating the model, but at that stage there is no value of these two decision values and MiniZinc shows only the variable names. They got some values to show after the (first) solution is reached, and can then be shown in the output section.
trace is mostly used to see what's happening in loops where one can trace the (fixed) loop variables etc.
If you trace an array of decision variables then they will be represented in a different fashion, the array x will be shown as X_INTRODUCED_0_ etc.
And you can also use trace for domain reflection, e.g. using lb and ub to get the lower/upper value of the domain of a variable ("safe approximation of the bounds" as the documentation states it: https://www.minizinc.org/doc-2.5.5/en/predicates.html?highlight=ub_array). Here's an example which shows the domain of the intraCost variable:
constraint
trace("intraCost: \(lb(intraCost))..\(ub(intraCost))\n")
;
which shows
intraCost: -infinity..infinity
You can read a little more about trace here https://www.minizinc.org/doc-2.5.5/en/efficient.html?highlight=trace .
Update Answer to question 1, 2 and 4.
The constraint #1 and #2 means the same thing, i.e. that the elements in clusters should be disjoint. The #1 constraint is a little different in that it loops over decision variables while the #2 constraint use plain indices. One can guess that #2 is faster since #1 use the where i != j which must be translated to some extra constraints. (And using i < j instead should be a little faster.)
The all_different constraint states about the same and depending on the underlying solver it might be faster if it's translated to an efficient algorithm in the solver.
In the model there is also the following constraint which states that all elements must be used:
constraint (clusters[1] union clusters[2] union clusters[3] union clusters[4]) = dsmElements;
Apart from efficiency, all these constraints above can be replaced with one single constraint: partition_set which ensure that all elements in dsmElements must be used in clusters.
constraint partition_set(clusters,dsmElements);
It might be faster to also combine with the all_different constraint, but that has to be tested.
I don't understand how this could be fixed, I went through on some of the questions here already, but didn't find a perfectly fitting answer.
I have a dataframe which has the following important columns: building_id, area, height.
The UDF I tried to write calculates a difference between the square root of the area and the height. It returns a value, which should be added to the dataframe.
def calculate_difference(area, height):
# calculate the square root of the area
import numpy as np
nr = np.sqrt(area)
# calculate the difference between the square root of the area and the height
dif = nr - height
return dif
And then I register this UDF:
calculate_differenceUDF = udf(calculate_difference)
The function works when I pass two numbers, it returns the value I expect. I want to add a new column to my dataframe, where we have a calculated value, based on the function.
display(df.withColumn("diff", calculate_differenceUDF(col("area"), col("height"))))
Then I receive this error:
PickleException: expected zero arguments for construction of ClassDict
(for numpy.dtype)
I understand that I don't return maybe a correct type, but I don't see how to fix it! :)
I think you should first convert the returned value of numpy.sqrt() to python's float type.
def calculate_difference(area, height):
nr = float(np.sqrt(area))
dif = nr - height
return dif
then register the UDF
calculate_differenceUDF = udf(calculate_difference, FloatType())
Other answer correct on ensuring the appropriate datatype is returned (in this case, float). If others are still facing the same error, I also had to ensure my inputs were of the appropriate type. For example:
def calculate_difference(area, height):
# specify input datatype
area = float(area)
height = float(height)
# calculate the square root of the area
import numpy as np
nr = np.sqrt(area)
# calculate the difference between the square root of the area and the height
dif = nr - height
return dif
I have been trying to understand how to increment the reference to some value.
In C I would simply increment the pointer to retrieve a value in the next array location.
How does this mechanism work in Postgres? is it possible?
For an example, I have created a table with some data in:
create table mathtest (
x int, y int, val int)
insert into mathtest (x,y,val)
values (1,1,10),(2,2,20),(3,3,30),(4,4,40),(5,5,50),(6,6,60),(7,7,70),(8,8,80),(9,9,90),(10,10,100),(11,11,110)
What I want to do is add the val value from the current row and then the val value when the x value in the row equals the current x value plus 2, and then plus 4. I realise that I can't assume the next row that is retrieved will be in a set order so I can't use 'lead'
If it was C I would simply increment the pointer.
The data output needs to be when the modulo of x and y = 0 for certain divisors. (this bit works)
select
x base,
(x+2) plus1x,
(x+4) plus2x,
y,
val
from mathtest
where x%2 =0 and y%3 = 0
This outputs the following:
base plus1x plus2x y val
1 6 8 10 6 60
The output I would like is:
60 + 80 +100 = 240
I can't conceptualise how to do it. My mind seems to be stuck in procedural C mode!
Whatever I type and try is an error.
Can any body help me to get over this hurdle?
Welcome to the world of window functions.
You need an explicit ordering, otherwise it makes no sense to speak of the "previous row".
As a simple example, to get the difference to the previous value, you can query like
SELECT val -
lag(val) OVER (ORDER BY x)
FROM mathtest;
I was looking for Scala's equivalent code or underlying theory for pythons np.random.choice (Numpy as np). I have a similar implementation that uses Python's np.random.choice method to select the random moves from the probability distribution.
Python's code
Input list: ['pooh', 'rabbit', 'piglet', 'Christopher'] and probabilies: [0.5, 0.1, 0.1, 0.3]
I want to select one of the value from the input list given the associated probability of each input element.
The Scala standard library has no equivalent to np.random.choice but it shouldn't be too difficult to build your own, depending on which options/features you want to emulate.
Here, for example, is a way to get an infinite Stream of submitted items, with the probability of any one item weighted relative to the others.
def weightedSelect[T](input :(T,Int)*): Stream[T] = {
val items :Seq[T] = input.flatMap{x => Seq.fill(x._2)(x._1)}
def output :Stream[T] = util.Random.shuffle(items).toStream #::: output
output
}
With this each input item is given with a multiplier. So to get an infinite pseudorandom selection of the characters c and v, with c coming up 3/5ths of the time and v coming up 2/5ths of the time:
val cvs = weightedSelect(('c',3),('v',2))
Thus the rough equivalent of the np.random.choice(aa_milne_arr,5,p=[0.5,0.1,0.1,0.3]) example would be:
weightedSelect("pooh"-> 5
,"rabbit" -> 1
,"piglet" -> 1
,"Christopher" -> 3).take(5).toArray
Or perhaps you want a better (less pseudo) random distribution that might be heavily lopsided.
def weightedSelect[T](items :Seq[T], distribution :Seq[Double]) :Stream[T] = {
assert(items.length == distribution.length)
assert(math.abs(1.0 - distribution.sum) < 0.001) // must be at least close
val dsums :Seq[Double] = distribution.scanLeft(0.0)(_+_).tail
val distro :Seq[Double] = dsums.init :+ 1.1 // close a possible gap
Stream.continually(items(distro.indexWhere(_ > util.Random.nextDouble())))
}
The result is still an infinite Stream of the specified elements but the passed-in arguments are a bit different.
val choices :Stream[String] = weightedSelect( List("this" , "that")
, Array(4998/5000.0, 2/5000.0))
// let's test the distribution
val (choiceA, choiceB) = choices.take(10000).partition(_ == "this")
choiceA.length //res0: Int = 9995
choiceB.length //res1: Int = 5 (not bad)
Here's the thing: I want to modify (and then return) a matrix of integers that is given in the parameters of the function. The funcion average (of the class MatrixMotionBlur) gives the average between the own pixel, upper, down and left pixels. Follows the following formula:
result(x, y) = (M1(x, y)+M1(x-1, y)+M1(x, y-1)+M1(x, y+1)) / 4
This is the code i've implemented so far
MatrixMotionBlur - Average function
MotionBlurSingleThread - run
The objetive here is to apply "average" method to alter the matrix value and return that matrix. The thing is the program gives me error when I to insert the value on the matrix.
Any ideas how to do this ?
The functional way
val updatedData = data.map{ outter =>
outter(i).map{ inner =>
mx.average(i.j)
}
}
Pay attention that Seq is immutable collection type and you can't just modify it, you can create new, modified collection only.
By the way, why you iterate starting 1, but not 0. Are you sure you want it?