Related
I have N maps (Map[String, Double]) each having the same set of keys. Let's say something like the following:
map1 = ("elem1": 2.0, "elem2": 4.0, "elem3": 3.0)
map2 = ("elem1": 4.0, "elem2": 1.0, "elem3": 1.0)
map3 = ("elem1": 3.0, "elem2": 10.0, "elem3": 2.0)
I need to return a new map with element-wise average of those input maps:
resultMap = ("elem1": 3.0, "elem2": 5.0, "elem3": 2.0)
What's the cleanest way to do that in scala? Preferrably without using extra external libraries.
This all happens in Spark*. Thus any answers suggesting spark-specific usage could be helpful.
One option is to convert all Maps to Seqs, union them to a single Seq, group by key and take the average of values:
val maps = Seq(map1, map2, map3)
maps.map(_.toSeq).reduce(_++_).groupBy(_._1).mapValues(x => x.map(_._2).sum/x.length)
// res6: scala.collection.immutable.Map[String,Double] = Map(elem1 -> 3.0, elem3 -> 2.0, elem2 -> 5.0)
Since your question is tagged with apache-spark you can get your desired output by combining the maps into RDD[Map[String, Double]] as
scala> val rdd = sc.parallelize(Seq(Map("elem1"-> 2.0, "elem2"-> 4.0, "elem3"-> 3.0),Map("elem1"-> 4.0, "elem2"-> 1.0, "elem3"-> 1.0),Map("elem1"-> 3.0, "elem2"-> 10.0, "elem3"-> 2.0)))
rdd: org.apache.spark.rdd.RDD[scala.collection.immutable.Map[String,Double]] = ParallelCollectionRDD[1] at parallelize at <console>:24
Then you can use flatMap to flatten the entries of maps into individual rows and use groupBy function with key and sum the grouped values and devide it with the size of the grouped maps. You should get Your desired output as
scala> rdd.flatMap(row => row).groupBy(kv => kv._1).mapValues(values => values.map(value => value._2).sum/values.size)
res0: org.apache.spark.rdd.RDD[(String, Double)] = MapPartitionsRDD[5] at mapValues at <console>:27
scala> res0.foreach(println)
[Stage 0:> (0 + 0) / 4](elem2,5.0)
(elem3,2.0)
(elem1,3.0)
Hope the answer is helpful
I have a Vector and want to remove elements from the Vector. How can I do it in Scala?My input is a Vector[2.0, 3.0, 0.3, 1.0, 4.0] -->Vector[Double] and I want a Vector[2.0, 0.3, 4.0] as output, so I want to remove the element with the index 1 and 3 from my input Vector...
def removeElementFromVector(input: Vector) = {
val inputAsArray = input.toArray
inputAsArray
// ...
val reducedInputAsVector = inputAsArray.toVector
}
Yes, you can use filter to achieve it, but we need to add index to remove element at an index:
Ex: Your vector (scala.collection.immutable.Vector[Double]):
scala> val v1 = val v1 = Vector(2.2, 3.3, 4.4, 5.5, 6.6, 4.4)
Output: Vector(2.2, 3.3, 4.4, 5.5, 6.6, 4.4)
Now, we will remove element at index 2:
scala> var indexRemove=2
scala> val v2 = v1.zipWithIndex.filter(x => x._2!=indexRemove).map(x=>x._1).toVector
Output: Vector(2.2, 3.3, 5.5, 6.6, 4.4)
Now, we will remove element at index 3
scala> var indexRemove=3
scala> val v2 = v1.zipWithIndex.filter(x => x._2!=indexRemove).map(x=>x._1).toVector
Output: Vector(2.2, 3.3, 4.4, 6.6, 4.4)
Hope this helps.
My input is: Vector[2.0, 3.0, 0.3, 1.0, 4.0] -->Vector[Double] and I want a Vector[2.0, 3.0, 0.3, 4.0] as output, so I want to remove the element with the index 1 and 3 from my input Vector. Sry for my question was not clear enough..
You can use filter method to remove them.
> val reducedInputVector = input.filter(x => !(Array(1,3) contains input.indexOf(x)))
reducedInputVector: scala.collection.immutable.Vector[Double] = Vector(2.0, 0.3, 4.0)
I solved it with apply:
val vec1 = Vector(2.0,3.0,0.3,1.0, 4.0)
val vec2 = Vectors.dense(vec1.apply(0), vec1.apply(1),vec1.apply(2), vec1.apply(4))
the output is
vec1: scala.collection.immutable.Vector[Double] = Vector(2.0, 3.0, 0.3, 1.0, 4.0)
vec2: org.apache.spark.mllib.linalg.Vector = [2.0,3.0,0.3,4.0]
def deleteItem[A](row: Vector[A], item: Int): Vector[A] = {
val (a,b) = row.splitAt(item)
if (b!=Nil) a ++ b.tail else a
}
I try to use (1,), but doesn't work, what's the syntax to define Tuple1 in scala ?
scala> val a=(1,)
<console>:1: error: illegal start of simple expression
val a=(1,)
For tuple with cardinality 2 or more, you can use parentheses, however for with cardinality 1, you need to use Tuple1:
scala> val tuple1 = Tuple1(1)
tuple1: (Int,) = (1,)
scala> val tuple2 = ('a', 1)
tuple2: (Char, Int) = (a,1)
scala> val tuple3 = ('a', 1, "name")
tuple3: (Char, Int, java.lang.String) = (a,1,name)
scala> tuple1._1
res0: Int = 1
scala> tuple2._2
res1: Int = 1
scala> tuple3._1
res2: Char = a
scala> tuple3._3
res3: String = name
To declare the type, use Tuple1[T], for example val t : Tuple1[Int] = Tuple1(22)
A tuple is, by definition, an ordered list of elements. While Tuple1 exists, I haven't seen it used explicitly given you'd normally use a single element. Nevertheless, there is no sugar, you need to use Tuple1(1).
There is a valid use case in Spark that requires Tuple1: create a dataframe with one column.
import org.apache.spark.ml.linalg.Vectors
val data = Seq(
Vectors.sparse(5, Seq((1, 1.0), (3, 7.0))),
Vectors.dense(2.0, 0.0, 3.0, 4.0, 5.0),
Vectors.dense(4.0, 0.0, 0.0, 6.0, 7.0)
)
data.toDF("features").show()
It will throw an error:
"value toDF is not a member of Seq[org.apache.spark.ml.linalg.Vector]"
To make it work, we have to convert each row to Tuple1:
val data = Seq(
Tuple1(Vectors.sparse(5, Seq((1, 1.0), (3, 7.0)))),
Tuple1(Vectors.dense(2.0, 0.0, 3.0, 4.0, 5.0)),
Tuple1(Vectors.dense(4.0, 0.0, 0.0, 6.0, 7.0))
)
or a better way:
val data = Seq(
Vectors.sparse(5, Seq((1, 1.0), (3, 7.0))),
Vectors.dense(2.0, 0.0, 3.0, 4.0, 5.0),
Vectors.dense(4.0, 0.0, 0.0, 6.0, 7.0)
).map(Tuple1.apply)
I can do this quite easily, and cleanly, using a for loop. For instance, if I wanted to traverse a Seq from every element back to itself I would do the following:
val seq = Seq(1,2,3,4,5)
for (i <- seq.indices) {
for (j <- seq.indices) {
print(seq(i + j % seq.length))
}
}
But as I'm looking to fold over the collection, I'm wondering if there is a more idiomatic approach. A recursive approach would allow me to avoid any vars. But basically, I'm wondering if something like the following is possible:
seq.rotatedView(i)
Which would create a rotated view, like rotating bits (or circular shift).
Is it like below:
scala> def rotatedView(i:Int)=Seq(1,2,3,4,5).drop(i)++Seq(1,2,3,4,5).take(i)
rotatedView: (i: Int)Seq[Int]
scala> rotatedView(1)
res48: Seq[Int] = List(2, 3, 4, 5, 1)
scala> rotatedView(2)
res49: Seq[Int] = List(3, 4, 5, 1, 2)
This ought to do it in a fairly generic way, and allow for arbitrary rotations:
def rotateLeft[A](seq: Seq[A], i: Int): Seq[A] = {
val size = seq.size
seq.drop(i % size) ++ seq.take(i % size)
}
def rotateRight[A](seq: Seq[A], i: Int): Seq[A] = {
val size = seq.size
seq.drop(size - (i % size)) ++ seq.take(size - (i % size))
}
The idea is simple enough, to rotate left, drop the first i elements from the left, and take them again from the left to concatenate them in the opposite order. If you don't mind calculating the size of the collection, you can do your operations modulo the size, to allow i to be arbitrary.
scala> rotateRight(seq, 1)
res34: Seq[Int] = List(5, 1, 2, 3, 4)
scala> rotateRight(seq, 7)
res35: Seq[Int] = List(4, 5, 1, 2, 3)
scala> rotateRight(seq, 70)
res36: Seq[Int] = List(1, 2, 3, 4, 5)
Similarly, you can use splitAt:
def rotateLeft[A](seq: Seq[A], i: Int): Seq[A] = {
val size = seq.size
val (first, last) = seq.splitAt(i % size)
last ++ first
}
def rotateRight[A](seq: Seq[A], i: Int): Seq[A] = {
val size = seq.size
val (first, last) = seq.splitAt(size - (i % size))
last ++ first
}
To make it even more generic, using the enrich my library pattern:
import scala.collection.TraversableLike
import scala.collection.generic.CanBuildFrom
implicit class TraversableExt[A, Repr <: TraversableLike[A, Repr]](xs: TraversableLike[A, Repr]) {
def rotateLeft(i: Int)(implicit cbf: CanBuildFrom[Repr, A, Repr]): Repr = {
val size = xs.size
val (first, last) = xs.splitAt(i % size)
last ++ first
}
def rotateRight(i: Int)(implicit cbf: CanBuildFrom[Repr, A, Repr]): Repr = {
val size = xs.size
val (first, last) = xs.splitAt(size - (i % size))
last ++ first
}
}
scala> Seq(1, 2, 3, 4, 5).rotateRight(2)
res0: Seq[Int] = List(4, 5, 1, 2, 3)
scala> List(1, 2, 3, 4, 5).rotateLeft(2)
res1: List[Int] = List(3, 4, 5, 1, 2)
scala> Stream(1, 2, 3, 4, 5).rotateRight(1)
res2: scala.collection.immutable.Stream[Int] = Stream(5, ?)
Keep in mind these are not all necessarily the most tuned for performance, and they also can't work with infinite collections (none can).
Following the OP's comment that they want to fold over it, here's a slightly different take on it that avoids calculating the length of the sequence first.
Define an iterator that will iterate over the rotated sequence
class RotatedIterator[A](seq: Seq[A], start: Int) extends Iterator[A] {
var (before, after) = seq.splitAt(start)
def next = after match {
case Seq() =>
val (h :: t) = before; before = t; h
case h :: t => after = t; h
}
def hasNext = after.nonEmpty || before.nonEmpty
}
And use it like this:
val seq = List(1, 2, 3, 4, 5)
val xs = new RotatedIterator(seq, 2)
println(xs.toList) //> List(3, 4, 5, 1, 2)
A simple method is to concatenate the sequence with itself and then take the slice that is required:
(seq ++ seq).slice(start, start + seq.length)
This is just a variant of the drop/take version but perhaps a little clearer.
Given:
val seq = Seq(1,2,3,4,5)
Solution:
seq.zipWithIndex.groupBy(_._2<3).values.flatMap(_.map(_._1))
or
seq.zipWithIndex.groupBy(_._2<3).values.flatten.map(_._1)
Result:
List(4, 5, 1, 2, 3)
If rotation is more than length of collection - we need to use rotation%length, if negative than formula (rotation+1)%length and take absolute value.
It's not efficient
Another tail-recursive approach. When I benchmarked it with JMH it was about 2 times faster than solution based on drop/take:
def rotate[A](list: List[A], by: Int): List[A] = {
#tailrec
def go(list: List[A], n: Int, acc: List[A]): List[A] = {
if(n > 0) {
list match {
case x :: xs => go(xs, n-1, x :: acc)
}
} else {
list ++ acc.reverse
}
}
if (by < 0) {
go(list, -by % list.length, Nil)
} else {
go(list, list.length - by % list.length, Nil)
}
}
//rotate right
rotate(List(1,2,3,4,5,6,7,8,9,10), 3) // List(8, 9, 10, 1, 2, 3, 4, 5, 6, 7)
//use negative number to rotate left
rotate(List(1,2,3,4,5,6,7,8,9,10), -3) // List(4, 5, 6, 7, 8, 9, 10, 1, 2, 3)
Here is one liner solution
def rotateRight(A: Array[Int], K: Int): Array[Int] = {
if (null == A || A.size == 0) A else (A drop A.size - (K % A.size)) ++ (A take A.size - (K % A.size))
}
rotateRight(Array(1,2,3,4,5), 3)
Here's a fairly simple and idiomatic Scala collections way to write it:
def rotateSeq[A](seq: Seq[A], isLeft: Boolean = false, count: Int = 1): Seq[A] =
if (isLeft)
seq.drop(count) ++ seq.take(count)
else
seq.takeRight(count) ++ seq.dropRight(count)
We can simply use foldLeft to reverse a list as below.
val input = List(1,2,3,4,5)
val res = input.foldLeft(List[Int]())((s, a) => { List(a) ++: s})
println(res) // List(5, 4, 3, 2, 1)
Another one line solution if you don't need to validate the "offset":
def rotate[T](seq: Seq[T], offset: Int): Seq[T] = Seq(seq, seq).flatten.slice(offset, offset + seq.size)
This is a simple piece of code
object tesing_it extends App
{
val one = ArrayBuffer(1,2,3,4,5,6)
val i = 2 //the number of index you want to move
for(z<-0 to i){
val y = 0
var x = one += one(y)
x = x -= x(y)
println("for seq after process " +z +" " + x)
}
println(one)
}
Result:
for seq after process 0 ArrayBuffer(2, 3, 4, 5, 6, 1)
for seq after process 1 ArrayBuffer(3, 4, 5, 6, 1, 2)
for seq after process 2 ArrayBuffer(4, 5, 6, 1, 2, 3)
ArrayBuffer(4, 5, 6, 1, 2, 3)
It's easy to initialize a 2D array (or, in fact, any multidimensional array) in Java by putting something like that:
int[][] x = new int[][] {
{ 3, 5, 7, },
{ 0, 4, 9, },
{ 1, 8, 6, },
};
It's easy to read, it resembles a 2D matrix, etc, etc.
But how do I do that in Scala?
The best I could come up with looks, well, much less concise:
val x = Array(
Array(3, 5, 7),
Array(0, 4, 9),
Array(1, 8, 6)
)
The problems I see here:
It repeats "Array" over and over again (like there could be anything else besides Array)
It requires to omit trailing , in every Array invocation
If I screw up and insert something besides Array() in the middle of array, it will go okay with compiler, but type of x would silently become Array[Any] instead of Array[Array[Int]]:
val x = Array(
Array(3, 5, 7),
Array(0, 4), 9, // <= OK with compiler, silently ruins x
Array(1, 8, 6)
)
There is a guard against it, to specify the type directly, but it looks even more overkill than in Java:
val x: Array[Array[Int]] = Array(
Array(3, 5, 7),
Array(0, 4), 9, // <= this one would trigger a compiler error
Array(1, 8, 6)
)
This last example needs Array even 3 times more than I have to say int[][] in Java.
Is there any clear way around this?
Personally I'd suck it up and type out (or cut and paste) "Array" a few times for clarity's sake. Include the type annotation for safety, of course. But if you're really running out of e-ink, a quick easy hack would be simply to provide an alias for Array, for example:
val > = Array
val x: Array[Array[Int]] = >(
>(3, 5, 7),
>(0, 4, 9),
>(1, 8, 6)
)
You could also provide a type alias for Array if you want to shorten the annotation:
type >[T] = Array[T]
val x: >[>[Int]] = ...
I suggest to use Scala 2.10 and macros:
object MatrixMacro {
import language.experimental.macros
import scala.reflect.macros.Context
import scala.util.Try
implicit class MatrixContext(sc: StringContext) {
def matrix(): Array[Array[Int]] = macro matrixImpl
}
def matrixImpl(c: Context)(): c.Expr[Array[Array[Int]]] = {
import c.universe.{ Try => _, _ }
val matrix = Try {
c.prefix.tree match {
case Apply(_, List(Apply(_, List(Literal(Constant(raw: String)))))) =>
def toArrayAST(c: List[TermTree]) =
Apply(Select(Select(Ident("scala"), newTermName("Array")), newTermName("apply")), c)
val matrix = raw split "\n" map (_.trim) filter (_.nonEmpty) map {
_ split "," map (_.trim.toInt)
}
if (matrix.map(_.length).distinct.size != 1)
c.abort(c.enclosingPosition, "rows of matrix do not have the same length")
val matrixAST = matrix map (_ map (i => Literal(Constant(i)))) map (i => toArrayAST(i.toList))
toArrayAST(matrixAST.toList)
}
}
c.Expr(matrix getOrElse c.abort(c.enclosingPosition, "not a matrix of Int"))
}
}
Usage with:
scala> import MatrixMacro._
import MatrixMacro._
scala> matrix"1"
res86: Array[Array[Int]] = Array(Array(1))
scala> matrix"1,2,3"
res87: Array[Array[Int]] = Array(Array(1, 2, 3))
scala> matrix"""
| 1, 2, 3
| 4, 5, 6
| 7, 8, 9
| """
res88: Array[Array[Int]] = Array(Array(1, 2, 3), Array(4, 5, 6), Array(7, 8, 9))
scala> matrix"""
| 1, 2
| 1
| """
<console>:57: error: rows of matrix do not have the same length
matrix"""
^
scala> matrix"a"
<console>:57: error: not a matrix of Int
matrix"a"
^
I don't think you will get it shorter. ;)
If using a mere List of List (which in itself cannot guarantee that every sub list is of the same size) is not a problem for you, and you are only concerned with easy syntax and avoiding errors at creation-time, scala has many ways to create nice syntax constructs.
One such possibility would be a simple helper:
object Matrix {
def apply[X]( elements: Tuple3[X, X, X]* ): List[List[X]] = {
elements.toList.map(_.productIterator.toList.asInstanceOf[List[X]] )
}
// Here you might add other overloads for Tuple4, Tuple5 etc if you need "matrixes" of those sizes
}
val x = Matrix(
(3, 5, 7),
(0, 4, 9),
(1, 8, 6)
)
About your concerns:
It repeats "List" over and over again (like there could be anything else besides List)
Not the case here.
It requires to omit trailing , in every List invocation
Unfortunately that is still true here, not much you can do given scala's syntactic rules.
If I screw up and insert something besides List() in the middle of array, it will go okay with compiler, but type of x would silently become List[Any] instead of List[List[Int]]:
val x = List(
List(3, 5, 7),
List(0, 4), 9, // <= OK with compiler, silently ruins x
List(1, 8, 6)
)
The equivalent code now faile to compile:
scala> val x = Matrix(
| (3, 5, 7),
| (0, 4), 9,
| (1, 8, 6)
| )
<console>:10: error: type mismatch;
found : (Int, Int)
required: (?, ?, ?)
(0, 4), 9,
And finally if you want to explicitly specify the type of elements (say that you want to protect against the possibility of inadvertently mixing Ints and Doubles), you only have to specify Matrix[Int] instead of the ugly List[List[Int]]:
val x = Matrix[Int](
(3, 5, 7),
(0, 4, 9),
(1, 8, 6)
)
EDIT: I see that you replaced List with Array in your question. To use arrays all you have to use is to replace List with Array and toList with toArray in my code above.
Since I'm also in disgust with this trailing comma issue (i.e. I cannot simply exchange the last line with any other) I sometimes use either a fluent API or the constructor syntax trick to get the syntax I like. An example using the constructor syntax would be:
trait Matrix {
// ... and the beast
private val buffer = ArrayBuffer[Array[Int]]()
def >(vals: Int*) = buffer += vals.toArray
def build: Array[Array[Int]] = buffer.toArray
}
Which allows:
// beauty ...
val m = new Matrix {
>(1, 2, 3)
>(4, 5, 6)
>(7, 8, 9)
} build
Unfortunately, this relies on mutable data although it is only used temporarily during the construction. In cases where I want maximal beauty for the construction syntax I would prefer this solution.
In case build is too long/verbose you might want to replace it by an empty apply function.
I don't know if this is the easy way, but I've included some code below for converting nested tuples into '2D' arrays.
First, you need some boiler plate for getting the size of the tuples as well as converting the tuples into [Array[Array[Double]]. The series of steps I used were:
Figure out the number of rows and columns in the tuple
Turn the nested tuple into a one row Array
Reshape the array based on the size of the original tuple.
The code for that is:
object Matrix {
/**
* Returns the size of a series of nested tuples.
*/
def productSize(t: Product): (Int, Int) = {
val a = t.productArity
val one = t.productElement(0)
if (one.isInstanceOf[Product]) {
val b = one.asInstanceOf[Product].productArity
(a, b)
}
else {
(1, a)
}
}
/**
* Flattens out a nested tuple and returns the contents as an iterator.
*/
def flattenProduct(t: Product): Iterator[Any] = t.productIterator.flatMap {
case p: Product => flattenProduct(p)
case x => Iterator(x)
}
/**
* Convert a nested tuple to a flattened row-oriented array.
* Usage is:
* {{{
* val t = ((1, 2, 3), (4, 5, 6))
* val a = Matrix.toArray(t)
* // a: Array[Double] = Array(1, 2, 3, 4, 5, 6)
* }}}
*
* #param t The tuple to convert to an array
*/
def toArray(t: Product): Array[Double] = flattenProduct(t).map(v =>
v match {
case c: Char => c.toDouble
case b: Byte => b.toDouble
case sh: Short => sh.toDouble
case i: Int => i.toDouble
case l: Long => l.toDouble
case f: Float => f.toDouble
case d: Double => d
case s: String => s.toDouble
case _ => Double.NaN
}
).toArray[Double]
def rowArrayTo2DArray[#specialized(Int, Long, Float, Double) A: Numeric](m: Int, n: Int,
rowArray: Array[A]) = {
require(rowArray.size == m * n)
val numeric = implicitly[Numeric[A]]
val newArray = Array.ofDim[Double](m, n)
for (i <- 0 until m; j <- 0 until n) {
val idx = i * n + j
newArray(i)(j) = numeric.toDouble(rowArray(idx))
}
newArray
}
/**
* Factory method for turning tuples into 2D arrays
*/
def apply(data: Product): Array[Array[Double]] = {
def size = productSize(data)
def array = toArray(data)
rowArrayTo2DArray(size._1, size._2, array)
}
}
Now to use this, you could just do the following:
val a = Matrix((1, 2, 3))
// a: Array[Array[Double]] = Array(Array(1.0, 2.0, 3.0))
val b = Matrix(((1, 2, 3), (4, 5, 6), (7, 8, 9)))
// b: Array[Array[Double]] = Array(Array(1.0, 2.0, 3.0),
// Array(4.0, 5.0, 6.0),
// Array(7.0, 8.0, 9.0))
val c = Matrix((1L, 2F, "3")) // Correctly handles mixed types
// c: Array[Array[Double]] = Array(Array(1.0, 2.0, 3.0))
val d = Matrix((1L, 2F, new java.util.Date())) // Non-numeric types convert to NaN
// d: Array[Array[Double]] = Array(Array(1.0, 2.0, NaN))
Alternatively, if you could just call the rowArrayTo2DArray directly using the size of the array you want and a 1D array of values:
val e = Matrix.rowArrayTo2DArray(1, 3, Array(1, 2, 3))
// e: Array[Array[Double]] = Array(Array(1.0, 2.0, 3.0))
val f = Matrix.rowArrayTo2DArray(3, 1, Array(1, 2, 3))
// f: Array[Array[Double]] = Array(Array(1.0), Array(2.0), Array(3.0))
val g = Matrix.rowArrayTo2DArray(3, 3, Array(1, 2, 3, 4, 5, 6, 7, 8, 9))
// g: Array[Array[Double]] = Array(Array(1.0, 2.0, 3.0),
// Array(4.0, 5.0, 6.0),
// Array(7.0, 8.0, 9.0))
glancing through the answers, i did not find what to me seems the most obvious & simple way to do it. instead of Array, you can use a tuple.
would look something like that:
scala> val x = {(
| (3,5,7),
| (0,4,9),
| (1,8,6)
| )}
x: ((Int, Int, Int), (Int, Int, Int), (Int, Int, Int)) = ((3,5,7),(0,4,9),(1,8,6))
seems clean & elegant?
i think so :)