class P(val a: Int, val b: Int)
val listp = List[P](new P(2,1))
listp.sortWith( (p1, p2) => p1.b < p2.b )
listp.sortBy(p => (p.b))
listp is easy to sort
val list = List((2, 1))
list.sortWith( (a1, b1), (a2, b2) => b1 < b2) // Too many arguements
list.sortWith( ((a1, b1), (a2, b2)) => b1 < b2) // Not a legal formal parameter / <error> is already defined as <error>
list.sortBy((a, b) => (b)) // wrong number of parameters expected
How do I sort list? All the method calls for list result in compile errors (I know I'm ignoring the return value, I just care about syntax).
I'm just looking for the syntax for the anonymous function. I know that it is possible to subclass Ordered like Comparable in Java.
Edit - This has it done for me, thanks for all answers:
list.sortWith( _._2 < _._2)
list.sortBy(_._2)
list.sortWith( (t1, t2) => t1._2 < t2._2 )
this is it explicitly
list.sortWith( (t1: Tuple2[Int, Int], t2: Tuple2[Int, Int]) => t1._2 < t2._2 )
Given
val list = List((2,1), (3,4))
to sort by second element:
list.sortBy(_._2)
or much verbose:
list.sortBy { case(a,b) => b }
Try this:
scala> val list = List((2, 1),(5,-10))
list: List[(Int, Int)] = List((2,1), (5,-10))
scala> list.sortWith{ case ((a1, b1), (a2, b2)) => b1 < b2}
res1: List[(Int, Int)] = List((5,-10), (2,1))
val list = List((2,1), (3,4))
list.sort(_._1 < _._2)
res0: List[(Int, Int)] = List((2,1), (3,4))
Related
I am cogrouping two RDDs and I want to process its values. That is,
rdd1.cogroup(rdd2)
as a result of this cogrouping I get results as below:
(ion,(CompactBuffer(100772C121, 100772C111, 6666666666),CompactBuffer(100772C121)))
Considering this result I would like to obtain all distinct pairs. e.g.
For the key 'ion'
100772C121 - 100772C111
100772C121 - 666666666
100772C111 - 666666666
How can I do this in scala?
You could try something like the following:
(l1 ++ l2).distinct.combinations(2).map { case Seq(x, y) => (x, y) }.toList
You would need to update l1 and l2 for your CompactBuffer fields. When I tried this locally, I get this (which is what I believe you want):
scala> val l1 = List("100772C121", "100772C111", "6666666666")
l1: List[String] = List(100772C121, 100772C111, 6666666666)
scala> val l2 = List("100772C121")
l2: List[String] = List(100772C121)
scala> val combine = (l1 ++ l2).distinct.combinations(2).map { case Seq(x, y) => (x, y) }.toList
combine: List[(String, String)] = List((100772C121,100772C111), (100772C121,6666666666), (100772C111,6666666666))
If you would like all of these pairs on separate rows, you can enclose this logic within a flatMap.
EDIT: Added steps per your example above.
scala> val rdd1 = sc.parallelize(Array(("ion", "100772C121"), ("ion", "100772C111"), ("ion", "6666666666")))
rdd1: org.apache.spark.rdd.RDD[(String, String)] = ParallelCollectionRDD[0] at parallelize at <console>:12
scala> val rdd2 = sc.parallelize(Array(("ion", "100772C121")))
rdd2: org.apache.spark.rdd.RDD[(String, String)] = ParallelCollectionRDD[1] at parallelize at <console>:12
scala> val cgrp = rdd1.cogroup(rdd2).flatMap {
| case (key: String, (l1: Iterable[String], l2: Iterable[String])) =>
| (l1.toSeq ++ l2.toSeq).distinct.combinations(2).map { case Seq(x, y) => (x, y) }.toList
| }
cgrp: org.apache.spark.rdd.RDD[(String, String)] = FlatMappedRDD[4] at flatMap at <console>:16
scala> cgrp.foreach(println)
...
(100772C121,100772C111)
(100772C121,6666666666)
(100772C111,6666666666)
EDIT 2: Updated again per your use case.
scala> val cgrp = rdd1.cogroup(rdd2).flatMap {
| case (key: String, (l1: Iterable[String], l2: Iterable[String])) =>
| for { e1 <- l1.toSeq; e2 <- l2.toSeq; if (e1 != e2) }
| yield if (e1 > e2) ((e1, e2), 1) else ((e2, e1), 1)
| }.reduceByKey(_ + _)
...
((6666666666,100772C121),2)
((6666666666,100772C111),1)
((100772C121,100772C111),1)
I need to implement a "zipLongest" function in Scala; that is, combine two sequences together as pairs, and if one is longer than the other, use a default value. (Unlike the standard zip method, which will just truncate to the shortest sequence.)
I've implemented it directly as follows:
def zipLongest[T](xs: Seq[T], ys: Seq[T], default: T): Seq[(T, T)] = (xs, ys) match {
case (Seq(), Seq()) => Seq()
case (Seq(), y +: rest) => (default, y) +: zipLongest(Seq(), rest, default)
case (x +: rest, Seq()) => (x, default) +: zipLongest(rest, Seq(), default)
case (x +: restX, y +: restY) => (x, y) +: zipLongest(restX, restY, default)
}
Is there a better way to do it?
Use zipAll :
scala> val l1 = List(1,2,3)
l1: List[Int] = List(1, 2, 3)
scala> val l2 = List("a","b")
l2: List[String] = List(a, b)
scala> l1.zipAll(l2,0,".")
res0: List[(Int, String)] = List((1,a), (2,b), (3,.))
If you want to use the same default value for the first and second seq :
scala> def zipLongest[T](xs:Seq[T], ys:Seq[T], default:T) = xs.zipAll(ys, default, default)
zipLongest: [T](xs: Seq[T], ys: Seq[T], default: T)Seq[(T, T)]
scala> val l3 = List(4,5,6,7)
l3: List[Int] = List(4, 5, 6, 7)
scala> zipLongest(l1,l3,0)
res1: Seq[(Int, Int)] = List((1,4), (2,5), (3,6), (0,7))
You can do this as a oneliner:
xs.padTo(ys.length, x).zip(ys.padTo(xs.length, y))
I'm working on the following problem:
Pack consecutive duplicates of list elements into sublists. If a list
contains repeated elements they should be placed in separate sublists.
Example:
scala> pack(List('a, 'a, 'a, 'a, 'b, 'c, 'c, 'a, 'a, 'd, 'e, 'e, 'e,
'e)) res0: List[List[Symbol]] = List(List('a, 'a, 'a, 'a), List('b),
List('c, 'c), List('a, 'a), List('d), List('e, 'e, 'e, 'e))
I am wondering whether it can be implemented using foldRight. So far I can only make a recursive solution like below work:
def pack(list: List[Char]) = {
def checkNext(a: List[List[Char]], prev: Char, l: List[Char]): List[List[Char]] = l match {
case Nil => a
case h::tail if h == prev => checkNext((h::a.head)::a.tail,h,tail)
case h::tail => checkNext(List(h)::a,h,tail)
}
checkNext(List(List[Char](list.last)), list.last, list.init.reverse)
}
Absolutely! I find it to be very natural to use folds to accumulate a complex result from iterating a sequence. Essentially, it works the same as what you're doing now, except the matching on the list is provided to you by fold, and you just provide the processing for the cases. I'm not sure if you wanted an actual answer, so I'll try to give you a couple hints.
Think of the type of your final result. Now think of what value of that type would be the result of applying your process to an empty sequence. That's your first argument to foldRight/foldLeft.
Now you have to define what to do to extend your accumulator for each item you process. It seems to me you have two cases: either you've encountered a new letter that you haven't seen before or you're adding another instance to an existing list. You can use some fancy matching to detect which case you're in.
Here's how I'd do it:
def pack(list: List[Char]) = list.foldLeft(List.empty[List[Char]]) { case (acc, next) =>
acc.headOption.flatMap(_.headOption) match {
case Some(x) if x == next => (acc.head :+ next) +: acc.tail
case _ => List(next) +: acc
}
}.reverse
I used flatMap to join the two checks for whether there's a list at all yet and whether the a list for the current character exists. I find foldLeft to be more intuitive and it also has the added benefit of being tail recursive on List.
The result:
scala> pack(List('a', 'a', 'a', 'a', 'b', 'c', 'c', 'a', 'a', 'd',
'e', 'e', 'e', 'e'))
res1: List[List[Char]] = List(List(a, a, a, a),
List(b), List(c, c), List(a, a), List(d), List(e, e, e, e))
Here is my version of fold:
def pack[A](xs: List[A]): List[List[A]] =
xs.foldRight(List[List[A]]()){
case (x, (ys#(y::_)) :: rs) if x == y => (x::ys) :: rs
case (x, ys) => List(x) :: ys
}
However, I prefer the recursive one:
def pack2[A](xs: List[A]): List[List[A]] = xs match {
case Nil => Nil
case x::_ => val (hs, ts) = xs.span(x==); hs::pack2(ts)
}
The recursive one is clearer and shorter than the fold version, in addition it is faster!
scala> def time(n: Int)(call : => Unit): Long = {
| var cnt = 0
| val start = System.currentTimeMillis
| while(cnt < n) {
| cnt += 1
| call
| }
| System.currentTimeMillis - start
| }
time: (n: Int)(call: => Unit)Long
scala> val xs = ("A"*100 + "B"*1000 + "C"*10 + "DEFGH"*1000).toList
xs: List[Char] = List(A, A, A...)
scala> time(10000){ pack(xs) }
res3: Long = 19961
scala> time(10000){ pack2(xs) }
res4: Long = 4382
And named #acjay's version as pack3:
scala> def pack3(list: List[Char]) = list.foldLeft(List.empty[List[Char]]) { case (acc, next) =>
| acc.headOption.flatMap(_.headOption) match {
| case Some(x) if x == next => (acc.head :+ next) +: acc.tail
| case _ => List(next) +: acc
| }
| }.reverse
pack3: (list: List[Char])List[List[Char]]
scala> time(10000){ pack3(xs) }
res5: Long = 420946
scala> pack3(xs) == pack2(xs)
res6: Boolean = true
scala> pack3(xs) == pack(xs)
res7: Boolean = true
Implementation by Martin Odersky
def pack[T](xs: List[T]): List[List[T]] = xs match{
case Nil => Nil
case x :: xs1 =>
val (first, rest) = xs span (y => y == x)
first :: pack(rest)
}
In ML, one can assign names for each element of a matched pattern:
fun findPair n nil = NONE
| findPair n (head as (n1, _))::rest =
if n = n1 then (SOME head) else (findPair n rest)
In this code, I defined an alias for the first pair of the list and matched the contents of the pair. Is there an equivalent construct in Scala?
You can do variable binding with the # symbol, e.g.:
scala> val wholeList # List(x, _*) = List(1,2,3)
wholeList: List[Int] = List(1, 2, 3)
x: Int = 1
I'm sure you'll get a more complete answer later as I'm not sure how to write it recursively like your example, but maybe this variation would work for you:
scala> val pairs = List((1, "a"), (2, "b"), (3, "c"))
pairs: List[(Int, String)] = List((1,a), (2,b), (3,c))
scala> val n = 2
n: Int = 2
scala> pairs find {e => e._1 == n}
res0: Option[(Int, String)] = Some((2,b))
OK, next attempt at direct translation. How about this?
scala> def findPair[A, B](n: A, p: List[Tuple2[A, B]]): Option[Tuple2[A, B]] = p match {
| case Nil => None
| case head::rest if head._1 == n => Some(head)
| case _::rest => findPair(n, rest)
| }
findPair: [A, B](n: A, p: List[(A, B)])Option[(A, B)]
I have a Set of items of some type and want to generate its power set.
I searched the web and couldn't find any Scala code that adresses this specific task.
This is what I came up with. It allows you to restrict the cardinality of the sets produced by the length parameter.
def power[T](set: Set[T], length: Int) = {
var res = Set[Set[T]]()
res ++= set.map(Set(_))
for (i <- 1 until length)
res = res.map(x => set.map(x + _)).flatten
res
}
This will not include the empty set. To accomplish this you would have to change the last line of the method simply to res + Set()
Any suggestions how this can be accomplished in a more functional style?
Looks like no-one knew about it back in July, but there's a built-in method: subsets.
scala> Set(1,2,3).subsets foreach println
Set()
Set(1)
Set(2)
Set(3)
Set(1, 2)
Set(1, 3)
Set(2, 3)
Set(1, 2, 3)
Notice that if you have a set S and another set T where T = S ∪ {x} (i.e. T is S with one element added) then the powerset of T - P(T) - can be expressed in terms of P(S) and x as follows:
P(T) = P(S) ∪ { p ∪ {x} | p ∈ P(S) }
That is, you can define the powerset recursively (notice how this gives you the size of the powerset for free - i.e. adding 1-element doubles the size of the powerset). So, you can do this tail-recursively in scala as follows:
scala> def power[A](t: Set[A]): Set[Set[A]] = {
| #annotation.tailrec
| def pwr(t: Set[A], ps: Set[Set[A]]): Set[Set[A]] =
| if (t.isEmpty) ps
| else pwr(t.tail, ps ++ (ps map (_ + t.head)))
|
| pwr(t, Set(Set.empty[A])) //Powerset of ∅ is {∅}
| }
power: [A](t: Set[A])Set[Set[A]]
Then:
scala> power(Set(1, 2, 3))
res2: Set[Set[Int]] = Set(Set(1, 2, 3), Set(2, 3), Set(), Set(3), Set(2), Set(1), Set(1, 3), Set(1, 2))
It actually looks much nicer doing the same with a List (i.e. a recursive ADT):
scala> def power[A](s: List[A]): List[List[A]] = {
| #annotation.tailrec
| def pwr(s: List[A], acc: List[List[A]]): List[List[A]] = s match {
| case Nil => acc
| case a :: as => pwr(as, acc ::: (acc map (a :: _)))
| }
| pwr(s, Nil :: Nil)
| }
power: [A](s: List[A])List[List[A]]
Here's one of the more interesting ways to write it:
import scalaz._, Scalaz._
def powerSet[A](xs: List[A]) = xs filterM (_ => true :: false :: Nil)
Which works as expected:
scala> powerSet(List(1, 2, 3)) foreach println
List(1, 2, 3)
List(1, 2)
List(1, 3)
List(1)
List(2, 3)
List(2)
List(3)
List()
See for example this discussion thread for an explanation of how it works.
(And as debilski notes in the comments, ListW also pimps powerset onto List, but that's no fun.)
Use the built-in combinations function:
val xs = Seq(1,2,3)
(0 to xs.size) flatMap xs.combinations
// Vector(List(), List(1), List(2), List(3), List(1, 2), List(1, 3), List(2, 3),
// List(1, 2, 3))
Note, I cheated and used a Seq, because for reasons unknown, combinations is defined on SeqLike. So with a set, you need to convert to/from a Seq:
val xs = Set(1,2,3)
(0 to xs.size).flatMap(xs.toSeq.combinations).map(_.toSet).toSet
//Set(Set(1, 2, 3), Set(2, 3), Set(), Set(3), Set(2), Set(1), Set(1, 3),
//Set(1, 2))
Can be as simple as:
def powerSet[A](xs: Seq[A]): Seq[Seq[A]] =
xs.foldLeft(Seq(Seq[A]())) {(sets, set) => sets ++ sets.map(_ :+ set)}
Recursive implementation:
def powerSet[A](xs: Seq[A]): Seq[Seq[A]] = {
def go(xsRemaining: Seq[A], sets: Seq[Seq[A]]): Seq[Seq[A]] = xsRemaining match {
case Nil => sets
case y :: ys => go(ys, sets ++ sets.map(_ :+ y))
}
go(xs, Seq[Seq[A]](Seq[A]()))
}
All the other answers seemed a bit complicated, here is a simple function:
def powerSet (l:List[_]) : List[List[Any]] =
l match {
case Nil => List(List())
case x::xs =>
var a = powerSet(xs)
a.map(n => n:::List(x)):::a
}
so
powerSet(List('a','b','c'))
will produce the following result
res0: List[List[Any]] = List(List(c, b, a), List(b, a), List(c, a), List(a), List(c, b), List(b), List(c), List())
Here's another (lazy) version... since we're collecting ways of computing the power set, I thought I'd add it:
def powerset[A](s: Seq[A]) =
Iterator.range(0, 1 << s.length).map(i =>
Iterator.range(0, s.length).withFilter(j =>
(i >> j) % 2 == 1
).map(s)
)
Here's a simple, recursive solution using a helper function:
def concatElemToList[A](a: A, list: List[A]): List[Any] = (a,list) match {
case (x, Nil) => List(List(x))
case (x, ((h:List[_]) :: t)) => (x :: h) :: concatElemToList(x, t)
case (x, (h::t)) => List(x, h) :: concatElemToList(x, t)
}
def powerSetRec[A] (a: List[A]): List[Any] = a match {
case Nil => List()
case (h::t) => powerSetRec(t) ++ concatElemToList(h, powerSetRec (t))
}
so the call of
powerSetRec(List("a", "b", "c"))
will give the result
List(List(c), List(b, c), List(b), List(a, c), List(a, b, c), List(a, b), List(a))