I'm new to scala (from python) and I'm trying to create a Json object that has dynamic keys. I would like to use some starting number as the top-level key and then combinations involving that number as second-level keys.
From reading the play-json docs/examples, I've seen how to build these nested structures. While that will work for the top-level keys (there are only 17 of them), this is a combinatorial problem and the power set contains ~130k combinations that would be the second-level keys so it isn't feasible to list that structure out. I also saw the use of a case class for structures, however the parameter name becomes the key in those instances which is not what I'm looking for.
Currently, I'm considering using HashMaps with the MultiMap trait so that I can map multiple combinations to the same original starting number and then second-level keys would be the combinations themselves.
I have python code that does this, but it takes 3-4 days to work through up-to-9-number combinations for all 17 starting numbers. The ideal final format would look something like below.
Perhaps it isn't possible to do in scala given the goal of using immutable structures. I suppose using regex on a string of the output might be an option as well. I'm open to any solutions regarding data structures to hold the info and how to approach the problem. Thanks!
{
"2": {
"(2, 3, 4, 5, 6)": {
"best_permutation": "(2, 4, 3, 5, 6)",
"amount": 26.0
},
"(2, 4, 5, 6)": {
"best_permutation": "(2, 5, 4, 6)",
"amount": 21.0
}
},
"3": {
"(3, 2, 4, 5, 6)": {
"best_permutation": "(3, 4, 2, 5, 6)",
"amount": 26.0
},
"(3, 4, 5, 6)": {
"best_permutation": "(3, 5, 4, 6)",
"amount": 21.0
}
}
}
EDIT:
There is no real data source other than the matrix I'm using as my lookup table. I've posted the links to the lookup table I'm using and the program if it might help, but essentially, I'm generating the content myself within the code.
For a given combination, I have a function that basically takes the first value of the combination (which is to be the starting point) and then uses the tail of that combination to generate a permutation.
After that I prepend the starting location to the front of each permutation and then use sliding(2) to work my way through the permutation looking up the amount which is in a breeze.linalg.DenseMatrix by using the two values to index the matrix I've provided below and summing the amounts gathered by indexing the matrix with the two sliding values (subtracting 1 from each value to account for the 0-based indexing).
At this point, it is just a matter of gathering the information (starting_location, combination, best_permutation and the amount) and constructing the nested HashMap. I'm using scala 2.11.8 if it makes any difference.
MATRIX: see here.
PROGRAM:see here.
Related
I'm trying to migrate a stream processing project to Spark Structured Streaming.
Within this project, there is a correlation logic like this:
A dict with init values
{
1: [2, 3],
4: [5, 6],
}
Then a new input comes, saying that 2 and 5 should be correlated together.
We know the key for 2 is 1, and for 5 is 4, so we merge all values in entry 4 to entry 1.
Finally, the dict becomes { 1: [2, 3, 4, 5, 6] }.
Currently, we use a distributed database to save the dict. But with Spark, I want to retire the database and only rely on Spark's memory state.
According to this tutorial, I created a mapping function:
def mappingFunction(
key: String,
values: Iterator[Input],
state: GroupState[State]
): Iterator[...] = {
}
But it seems I can only access the state of the specific key (first param in this func).
My questions are:
If I receive <2, 5>, how can I update the group state of 1 and delete the group state of 4?
Can we rely on Spark for maintaining a complicated state like this? Or is a distributed global state store always needed for this case?
Thanks!
I'm new to Scala and I'm having a mental block on a seemingly easy problem. I'm using the Scala library breeze and need to take an array buffer (mutable) and put the results into a matrix. This... should be simple but? Scala is so insanely type casted breeze seems really picky about what data types it will take when making a DenseVector. This is just some prototype code, but can anyone help me come up with a solution?
Right now I have something like...
//9 elements that need to go into a 3x3 matrix, 1-3 as top row, 4-6 as middle row, etc)
val numbersForMatrix: ArrayBuffer[Double] = (1, 2, 3, 4, 5, 6, 7, 8, 9)
//the empty 3x3 matrix
var M: breeze.linalg.DenseMatrix[Double] = DenseMatrix.zeros(3,3)
In breeze you can do stuff like
M(0,0) = 100 and set the first value to 100 this way,
You can also do stuff like:
M(0, 0 to 2) := DenseVector(1, 2, 3)
which sets the first row to 1, 2, 3
But I cannot get it to do something like...
var dummyList: List[Double] = List(1, 2, 3) //this works
var dummyVec = DenseVector[Double](dummyList) //this works
M(0, 0 to 2) := dummyVec //this does not work
and successfully change the first row to the 1, 2,3.
And that's with a List, not even an ArrayBuffer.
Am willing to change datatypes from ArrayBuffer but just not sure how to approach this at all... could try updating the matrix values one by one but that seems like it would be VERY hacky to code up(?).
Note: I'm a Python programmer who is used to using numpy and just giving it arrays. The breeze documentation doesn't provide enough examples with other datatypes for me to have been able to figure this out yet.
Thanks!
Breeze is, in addition to pickiness over types, pretty picky about vector shape: DenseVectors are column vectors, but you are trying to assign to a subset of a row, which expects a transposed DenseVector:
M(0, 0 to 2) := dummyVec.t
For example, I need a valuable which is an Int, but I limited it can only set from 1-10. Is this features build in in swift? Except from override the setter. Is this possible to do so?
btw, what is this feature named? I remember I come across some languages got this features, but I don't recall the name of that languages.
You might be looking for an enumeration. It allows you to set a range of values (not strictly numbers though), which are allowed as input. Do something like this:
enum NumsTill10 {
case 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
}
And to set it as a variable:
let number:NumsTill10 = NumsTill10.1
Or even
let number:NumsTill10 = .1
And then you can do:
if number == .7 {
//It's 7!
}
My question is probably really easy, but I am a mathematica beginner.
I have a dataset, lets say:
Column: Numbers from 1 to 10
Column Signs
Column Other signs.
{{1,2,3,4,5,6,7,8,9,10},{d,t,4,/,g,t,w,o,p,m},{g,h,j,k,l,s,d,e,w,q}}
Now I want to extract all rows for which column 1 provides an odd number. In other words I want to create a new dataset.
I tried to work with Select and OddQ as well as with the IF function, but I have absolutely no clue how to put this orders in the right way!
Taking a stab at what you might be asking..
(table = {{1, 2, 3, 4, 5, 6, 7, 8, 9, 10} ,
Characters["abcdefghij"],
Characters["ABCDEFGHIJ"]}) // MatrixForm
table[[All, 1 ;; -1 ;; 2]] // MatrixForm
or perhaps this:
Select[table, OddQ[#[[1]]] &]
{{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}}
The convention in Mathematica is the reverse of what you use in your description.
Rows are first level sublists.
Let's take your original data
mytable = {{1,2,3,4,5,6,7,8,9,10},{d,t,4,"/",g,t,w,o,p,m},{g,h,j,k,l,s,d,e,w,q}}
Just as you suggested, Select and OddQ can do what you want, but on your table, transposed. So we transpose first and back:
Transpose[Select[Transpose[mytable], OddQ[First[#]]& ]]
Another way:
Mathematica functional command MapThread can work on synchronous lists.
DeleteCases[MapThread[If[OddQ[#1], {##}] &, mytable], Null]
The inner function of MapThread gets all elements of what you call a 'row' as variables (#1, #2, etc.). So it test the first column and outputs all columns or a Null if the test fails. The enclosing DeleteCases suppresses the unmatching "rows".
I was given this question on programming in java and was wondering what would be the best way of doing it.
The question was on the lines of:
From the numbers provided, how would you in java display the most frequent number. The numbers was: 0, 3, 4, 1, 1, 3, 7, 9, 1
At first I am thinking well they should be in an array and sorted first then maybe have to go through a for loop. Am I on the right lines. Some examples will help greatly
If the numbers are all fairly small, you can quickly get the most frequent value by creating an array to keep track of the count for each number. The algorithm would be:
Find the maximum value in your list
Create an integer array of size max + 1 (assuming all non-negative values) to store the counts for each value in your list
Loop through your list and increment the count at the index of each value
Scan through the count array and find the index with the highest value
The run-time of this algorithm should be faster than sorting the list and finding the longest string of duplicate values. The tradeoff is that it takes up more memory if the values in your list are very large.
With Java 8, this can be implemented rather smoothly. If you're willing to use a third-party library like jOOλ, it could be done like this:
List<Integer> list = Arrays.asList(0, 3, 4, 1, 1, 3, 7, 9, 1);
System.out.println(
Seq.seq(list)
.grouped(i -> i, Agg.count())
.sorted(Comparator.comparing(t -> -t.v2))
.map(t -> t.v1)
.toList());
(disclaimer, I work for the company behind jOOλ)
If you want to stick with the JDK 8 dependency, the following code would be equivalent to the above:
System.out.println(
list.stream()
.collect(Collectors.groupingBy(i -> i, Collectors.counting()))
.entrySet()
.stream()
.sorted(Comparator.comparing(e -> -e.getValue()))
.map(e -> e.getKey())
.collect(Collectors.toList()));
Both solutions yield:
[1, 3, 0, 4, 7, 9]