Scala elegant list comprehension as in F# - scala

Just using the basic JDBC interface to read some data using Scala.
In F# (using System.Data.SqlClient namespace) we could do something like this to return an immutable list from the database.
let rs = cmd.ExecuteReader()
[while rs.Read() do yield rs.GetInt32(1)]
In Scala this proves more difficult, as far as I know there is no "while" comprehension like F#. Effectively I'd like to do something close to F# in Scala without having to use mutable vars - if only because they look ugly and add to Lines of Code.
Something like this seems to be commonplace in my Scala code right now:
var result = Seq.empty[Int]
val rs = stmt.executeQuery()
while (rs.next()) {
result = result :+ rs.getInt(1) }

I would create a custom subclass of Iterator that wraps a query result. It's really easy; senia showed how.
But you could also
val rs = stmt.executeQuery
val it = Iterator.continually(if (rs.next()) Some(rs.getInt(1)) else None)
val result = it.takeWhile(_.isDefined).toList.flatten

You could use same way in scala, but I think it's ugly:
class Reader(var i: Int){
def read = { i-=1; i > 0 }
def getInt32 = i
}
val r = new Reader(10)
Stream.from(0).takeWhile{ _ => r.read}.map{ _ => r.getInt32}.toList
// List(9, 8, 7, 6, 5, 4, 3, 2, 1)
Idiomatic scala way is to convert your Reader to an Iterator:
implicit class ReaderIterator(r: Reader) extends Iterator[Int] {
def hasNext = r.read
def next = r.getInt32
}
scala> new Reader(10).toList
res0: List[Int] = List(9, 8, 7, 6, 5, 4, 3, 2, 1)
But if you are really missing this syntax you could add it:
import scala.collection.immutable.VectorBuilder
class FWhile(c: => Boolean){
def apply[T](e: => T): Seq[T] = {
val b = new VectorBuilder[T]
while (c) b += e
b.result
}
}
object FWhile{
def apply(c: => Boolean) = new FWhile(c)
}
scala> FWhile(r.read){r.getInt32}
res0: Seq[Int] = Vector(9, 8, 7, 6, 5, 4, 3, 2, 1)

You could use an implicit class together with an implicit CanBuildFrom. This does use a mutable builder, but not at the caller's side:
object MyResultSetContainer {
implicit class MyResultSet(rs: ResultSet) {
def map[T, C <: Iterable[T]](f: (ResultSet) => T)
(implicit cbf: CanBuildFrom[Nothing, T, C]): C = {
val builder = cbf()
while (rs.next()) {
builder += f(rs)
}
builder.result()
}
}
}
to be used like this:
import MyResultSetContainer._
val rs = stmnt.executeQuery("select * from pg_user")
val names = for (row <- rs) yield (row.getString(1))
println(names)
rs.close()
The for comprehension uses map under the hood, so if you prefer map directly:
val names = rs.map(row => row.getString(1))
which produces a sequence. Thanks to CanBuildFrom you can produce other collections as well by providing a type explicitly:
val names: List[String] = rs.map(row => row.getString(1))
How does CanBuildFrom work? The Scala compiler looks at the types involved in this expression: There is the resulting type, and the type returned by the function called by map. Based on this information, the Scala compiler provides a factory implicitly which can be used to create a suitable builder. So you need only one method to produce different types of collections.
If you want to return multiple values, just return a tuple:
val columns = rs.map(row => (row.getInt(2), row.getString(1)))
and the tuple can be used to create a Map directly:
val keyNamesMap: Map[Int, String] =
rs.map(row => (row.getInt(2), row.getString(1)))
This is based on the idea that a result set is a list of rows, and so the map function should be available on top of it. The implicit class is used to add the map method to the underlying result set implicitly.

Related

map on a TreeMap returns a Map and not a TreeMap in Scala

I am new to Scala, and I am implementing a TreeMap with a multidimensional key like this:
class dimSet (val d:Vector[Int]) extends IndexedSeq[Int] {
def apply(idx:Int) = d(idx)
def length: Int = d.length
}
…
var vals : TreeMap[dimSet, A] = TreeMap[dimSet, A]()(orddimSet)
I have this method
def appOp0(t:TreeMap[dimSet,A], t1:TreeMap[dimSet,A], op:(A,A) => A, unop : (A) => A):TreeMap[dimSet,A] = {
if (t.isEmpty) t1.map((e:Tuple2[dimSet, A]) => (e._1, unop(e._2)))
else if (t1.isEmpty) t.map((t:Tuple2[dimSet, A]) => (t._1, unop(t._2)))
else {
val h = t.head
val h1 = t1.head
if ((h._1) == (h1._1)) appOp0(t.tail, t1.tail, op, unop) + ((h._1, op(h._2, h1._2)))
else if (orddimSet.compare(h._1,h1._1) == 1) appOp0(t, t1.tail, op, unop) + ((h1._1, unop(h1._2)))
else appOp0(t.tail, t1, op, unop) + ((h._1, unop(h._2)))
}
}
But the map method on the TreeMaps (second and third lines) returns a Map, not a TreeMap
I tried on repl with a simplier example and I got this:
scala> val t = TreeMap[dimSet, Double]( (new dimSet(Vector(1,1)), 5.1), (new dimSet(Vector(1,2)), 6.3), (new dimSet(Vector(3,1)), 7.1), (new dimSet(Vector(2,2)), 8.4)) (orddimSet)
scala> val tsq = t.map[(dimSet,Double), TreeMap[dimSet,Double]]((v:Tuple2[dimSet, Double]) => ((v._1, v._2 * v._2)))
<console>:41: error: Cannot construct a collection of type scala.collection.immutable.TreeMap[dimSet,Double] with elements of type (dimSet, Double) based on a collection of type scala.collection.immutable.TreeMap[dimSet,Double].
val tsq = t.map[(dimSet,Double), TreeMap[dimSet,Double]]((v:Tuple2[dimSet, Double]) => ((v._1, v._2 * v._2)))
^
scala> val tsq = t.map((v:Tuple2[dimSet, Double]) => ((v._1, v._2 * v._2)))
tsq: scala.collection.immutable.Map[dimSet,Double] = Map((1, 1) -> 26.009999999999998, (1, 2) -> 39.69, (2, 2) -> 70.56, (3, 1) -> 50.41)
I think CanBuildFrom cannot build my TreeMap as it can do with other TreeMaps, but I couldn't find why, ¿What can I do to return a TreeMap?
Thanks
The problem probably is that there is no implicit Ordering[dimSet] available when you call map. That call requires a CanBuildFrom, which in turn requires an implicit Ordering for TreeMap keys: see in docs.
So make orddimSet implicitly available before calling map:
implicit val ev = orddimSet
if (t.isEmpty) t1.map((e:Tuple2[dimSet, A]) => (e._1, unop(e._2)))
Or you can make an Ordering[dimSet] always automatically implicitly available, if you define an implicit Ordering in dimSet's companion object:
object dimSet {
implicit val orddimSet: Ordering[dimSet] = ??? // you code here
}

Scala - Iterate over an Iterator of type Product[K,V]

I am a newbie to Scala and I am trying to understand collectives. I have a sample Scala code in which a method is defined as follows:
override def write(records: Iterator[Product2[K, V]]): Unit = {...}
From what I understand, this function is passed an argument record which is an Iterator of type Product2[K,V]. Now what I don't understand is this Product2 a user defined class or is it a built in data structure. Moreover how do explore the key-value pair contents of Product2 and how do I iterate over them.
Chances are Product2 is a built-in class and you can easily check it if you're in modern IDE (just hover over it with ctrl pressed), or, by inspecting file header -- if there is no related imports, like some.custom.package.Product2, it's built-in.
What is Product2 and where it's defined? You can easily found out such things by utilizing Scala's ScalaDoc:
In case of build-in class you can treat it like tuple of 2 elements (in fact Tuple2 extends Product2, as you may see below), which has ._1 and ._2 accessor methods.
scala> val x: Product2[String, Int] = ("foo", 1)
// x: Product2[String,Int] = (foo,1)
scala> x._1
// res0: String = foo
scala> x._2
// res1: Int = 1
See How should I think about Scala's Product classes? for more.
Iteration is also hassle free, for example here is the map operation:
scala> val xs: Iterator[Product2[String, Int]] = List("foo" -> 1, "bar" -> 2, "baz" -> 3).iterator
xs: Iterator[Product2[String,Int]] = non-empty iterator
scala> val keys = xs.map(kv => kv._1)
keys: Iterator[String] = non-empty iterator
scala> val keys = xs.map(kv => kv._1).toList
keys: List[String] = List(foo, bar, baz)
scala> xs
res2: Iterator[Product2[String,Int]] = empty iterator
Keep in mind though, that once iterator was consumed, it transitions to empty state and can't be re-used again.
Product2 is just two values of type K and V.
use it like this:
write(List((1, "one"), (2, "two")))
the prototype can also be written like: override def write(records: Iterator[(K, V)]): Unit = {...}
To access values k of type K and v of type V.
override def write(records: Iterator[(K, V)]): Unit = {
records.map{case (k, v) => w(k, v)}
}

Map any collection to its own type, not just Seq? [duplicate]

One of the most powerful patterns available in Scala is the enrich-my-library* pattern, which uses implicit conversions to appear to add methods to existing classes without requiring dynamic method resolution. For example, if we wished that all strings had the method spaces that counted how many whitespace characters they had, we could:
class SpaceCounter(s: String) {
def spaces = s.count(_.isWhitespace)
}
implicit def string_counts_spaces(s: String) = new SpaceCounter(s)
scala> "How many spaces do I have?".spaces
res1: Int = 5
Unfortunately, this pattern runs into trouble when dealing with generic collections. For example, a number of questions have been asked about grouping items sequentially with collections. There is nothing built in that works in one shot, so this seems an ideal candidate for the enrich-my-library pattern using a generic collection C and a generic element type A:
class SequentiallyGroupingCollection[A, C[A] <: Seq[A]](ca: C[A]) {
def groupIdentical: C[C[A]] = {
if (ca.isEmpty) C.empty[C[A]]
else {
val first = ca.head
val (same,rest) = ca.span(_ == first)
same +: (new SequentiallyGroupingCollection(rest)).groupIdentical
}
}
}
except, of course, it doesn't work. The REPL tells us:
<console>:12: error: not found: value C
if (ca.isEmpty) C.empty[C[A]]
^
<console>:16: error: type mismatch;
found : Seq[Seq[A]]
required: C[C[A]]
same +: (new SequentiallyGroupingCollection(rest)).groupIdentical
^
There are two problems: how do we get a C[C[A]] from an empty C[A] list (or from thin air)? And how do we get a C[C[A]] back from the same +: line instead of a Seq[Seq[A]]?
* Formerly known as pimp-my-library.
The key to understanding this problem is to realize that there are two different ways to build and work with collections in the collections library. One is the public collections interface with all its nice methods. The other, which is used extensively in creating the collections library, but which are almost never used outside of it, is the builders.
Our problem in enriching is exactly the same one that the collections library itself faces when trying to return collections of the same type. That is, we want to build collections, but when working generically, we don't have a way to refer to "the same type that the collection already is". So we need builders.
Now the question is: where do we get our builders from? The obvious place is from the collection itself. This doesn't work. We already decided, in moving to a generic collection, that we were going to forget the type of the collection. So even though the collection could return a builder that would generate more collections of the type we want, it wouldn't know what the type was.
Instead, we get our builders from CanBuildFrom implicits that are floating around. These exist specifically for the purpose of matching input and output types and giving you an appropriately typed builder.
So, we have two conceptual leaps to make:
We aren't using standard collections operations, we're using builders.
We get these builders from implicit CanBuildFroms, not from our collection directly.
Let's look at an example.
class GroupingCollection[A, C[A] <: Iterable[A]](ca: C[A]) {
import collection.generic.CanBuildFrom
def groupedWhile(p: (A,A) => Boolean)(
implicit cbfcc: CanBuildFrom[C[A],C[A],C[C[A]]], cbfc: CanBuildFrom[C[A],A,C[A]]
): C[C[A]] = {
val it = ca.iterator
val cca = cbfcc()
if (!it.hasNext) cca.result
else {
val as = cbfc()
var olda = it.next
as += olda
while (it.hasNext) {
val a = it.next
if (p(olda,a)) as += a
else { cca += as.result; as.clear; as += a }
olda = a
}
cca += as.result
}
cca.result
}
}
implicit def iterable_has_grouping[A, C[A] <: Iterable[A]](ca: C[A]) = {
new GroupingCollection[A,C](ca)
}
Let's take this apart. First, in order to build the collection-of-collections, we know we'll need to build two types of collections: C[A] for each group, and C[C[A]] that gathers all the groups together. Thus, we need two builders, one that takes As and builds C[A]s, and one that takes C[A]s and builds C[C[A]]s. Looking at the type signature of CanBuildFrom, we see
CanBuildFrom[-From, -Elem, +To]
which means that CanBuildFrom wants to know the type of collection we're starting with--in our case, it's C[A], and then the elements of the generated collection and the type of that collection. So we fill those in as implicit parameters cbfcc and cbfc.
Having realized this, that's most of the work. We can use our CanBuildFroms to give us builders (all you need to do is apply them). And one builder can build up a collection with +=, convert it to the collection it is supposed to ultimately be with result, and empty itself and be ready to start again with clear. The builders start off empty, which solves our first compile error, and since we're using builders instead of recursion, the second error also goes away.
One last little detail--other than the algorithm that actually does the work--is in the implicit conversion. Note that we use new GroupingCollection[A,C] not [A,C[A]]. This is because the class declaration was for C with one parameter, which it fills it itself with the A passed to it. So we just hand it the type C, and let it create C[A] out of it. Minor detail, but you'll get compile-time errors if you try another way.
Here, I've made the method a little bit more generic than the "equal elements" collection--rather, the method cuts the original collection apart whenever its test of sequential elements fails.
Let's see our method in action:
scala> List(1,2,2,2,3,4,4,4,5,5,1,1,1,2).groupedWhile(_ == _)
res0: List[List[Int]] = List(List(1), List(2, 2, 2), List(3), List(4, 4, 4),
List(5, 5), List(1, 1, 1), List(2))
scala> Vector(1,2,3,4,1,2,3,1,2,1).groupedWhile(_ < _)
res1: scala.collection.immutable.Vector[scala.collection.immutable.Vector[Int]] =
Vector(Vector(1, 2, 3, 4), Vector(1, 2, 3), Vector(1, 2), Vector(1))
It works!
The only problem is that we don't in general have these methods available for arrays, since that would require two implicit conversions in a row. There are several ways to get around this, including writing a separate implicit conversion for arrays, casting to WrappedArray, and so on.
Edit: My favored approach for dealing with arrays and strings and such is to make the code even more generic and then use appropriate implicit conversions to make them more specific again in such a way that arrays work also. In this particular case:
class GroupingCollection[A, C, D[C]](ca: C)(
implicit c2i: C => Iterable[A],
cbf: CanBuildFrom[C,C,D[C]],
cbfi: CanBuildFrom[C,A,C]
) {
def groupedWhile(p: (A,A) => Boolean): D[C] = {
val it = c2i(ca).iterator
val cca = cbf()
if (!it.hasNext) cca.result
else {
val as = cbfi()
var olda = it.next
as += olda
while (it.hasNext) {
val a = it.next
if (p(olda,a)) as += a
else { cca += as.result; as.clear; as += a }
olda = a
}
cca += as.result
}
cca.result
}
}
Here we've added an implicit that gives us an Iterable[A] from C--for most collections this will just be the identity (e.g. List[A] already is an Iterable[A]), but for arrays it will be a real implicit conversion. And, consequently, we've dropped the requirement that C[A] <: Iterable[A]--we've basically just made the requirement for <% explicit, so we can use it explicitly at will instead of having the compiler fill it in for us. Also, we have relaxed the restriction that our collection-of-collections is C[C[A]]--instead, it's any D[C], which we will fill in later to be what we want. Because we're going to fill this in later, we've pushed it up to the class level instead of the method level. Otherwise, it's basically the same.
Now the question is how to use this. For regular collections, we can:
implicit def collections_have_grouping[A, C[A]](ca: C[A])(
implicit c2i: C[A] => Iterable[A],
cbf: CanBuildFrom[C[A],C[A],C[C[A]]],
cbfi: CanBuildFrom[C[A],A,C[A]]
) = {
new GroupingCollection[A,C[A],C](ca)(c2i, cbf, cbfi)
}
where now we plug in C[A] for C and C[C[A]] for D[C]. Note that we do need the explicit generic types on the call to new GroupingCollection so it can keep straight which types correspond to what. Thanks to the implicit c2i: C[A] => Iterable[A], this automatically handles arrays.
But wait, what if we want to use strings? Now we're in trouble, because you can't have a "string of strings". This is where the extra abstraction helps: we can call D something that's suitable to hold strings. Let's pick Vector, and do the following:
val vector_string_builder = (
new CanBuildFrom[String, String, Vector[String]] {
def apply() = Vector.newBuilder[String]
def apply(from: String) = this.apply()
}
)
implicit def strings_have_grouping(s: String)(
implicit c2i: String => Iterable[Char],
cbfi: CanBuildFrom[String,Char,String]
) = {
new GroupingCollection[Char,String,Vector](s)(
c2i, vector_string_builder, cbfi
)
}
We need a new CanBuildFrom to handle the building of a vector of strings (but this is really easy, since we just need to call Vector.newBuilder[String]), and then we need to fill in all the types so that the GroupingCollection is typed sensibly. Note that we already have floating around a [String,Char,String] CanBuildFrom, so strings can be made from collections of chars.
Let's try it out:
scala> List(true,false,true,true,true).groupedWhile(_ == _)
res1: List[List[Boolean]] = List(List(true), List(false), List(true, true, true))
scala> Array(1,2,5,3,5,6,7,4,1).groupedWhile(_ <= _)
res2: Array[Array[Int]] = Array(Array(1, 2, 5), Array(3, 5, 6, 7), Array(4), Array(1))
scala> "Hello there!!".groupedWhile(_.isLetter == _.isLetter)
res3: Vector[String] = Vector(Hello, , there, !!)
As of this commit it's a lot easier to "enrich" Scala collections than it was when Rex gave his excellent answer. For simple cases it might look like this,
import scala.collection.generic.{ CanBuildFrom, FromRepr, HasElem }
import language.implicitConversions
class FilterMapImpl[A, Repr](val r : Repr)(implicit hasElem : HasElem[Repr, A]) {
def filterMap[B, That](f : A => Option[B])
(implicit cbf : CanBuildFrom[Repr, B, That]) : That = r.flatMap(f(_).toSeq)
}
implicit def filterMap[Repr : FromRepr](r : Repr) = new FilterMapImpl(r)
which adds a "same result type" respecting filterMap operation to all GenTraversableLikes,
scala> val l = List(1, 2, 3, 4, 5)
l: List[Int] = List(1, 2, 3, 4, 5)
scala> l.filterMap(i => if(i % 2 == 0) Some(i) else None)
res0: List[Int] = List(2, 4)
scala> val a = Array(1, 2, 3, 4, 5)
a: Array[Int] = Array(1, 2, 3, 4, 5)
scala> a.filterMap(i => if(i % 2 == 0) Some(i) else None)
res1: Array[Int] = Array(2, 4)
scala> val s = "Hello World"
s: String = Hello World
scala> s.filterMap(c => if(c >= 'A' && c <= 'Z') Some(c) else None)
res2: String = HW
And for the example from the question, the solution now looks like,
class GroupIdenticalImpl[A, Repr : FromRepr](val r: Repr)
(implicit hasElem : HasElem[Repr, A]) {
def groupIdentical[That](implicit cbf: CanBuildFrom[Repr,Repr,That]): That = {
val builder = cbf(r)
def group(r: Repr) : Unit = {
val first = r.head
val (same, rest) = r.span(_ == first)
builder += same
if(!rest.isEmpty)
group(rest)
}
if(!r.isEmpty) group(r)
builder.result
}
}
implicit def groupIdentical[Repr : FromRepr](r: Repr) = new GroupIdenticalImpl(r)
Sample REPL session,
scala> val l = List(1, 1, 2, 2, 3, 3, 1, 1)
l: List[Int] = List(1, 1, 2, 2, 3, 3, 1, 1)
scala> l.groupIdentical
res0: List[List[Int]] = List(List(1, 1),List(2, 2),List(3, 3),List(1, 1))
scala> val a = Array(1, 1, 2, 2, 3, 3, 1, 1)
a: Array[Int] = Array(1, 1, 2, 2, 3, 3, 1, 1)
scala> a.groupIdentical
res1: Array[Array[Int]] = Array(Array(1, 1),Array(2, 2),Array(3, 3),Array(1, 1))
scala> val s = "11223311"
s: String = 11223311
scala> s.groupIdentical
res2: scala.collection.immutable.IndexedSeq[String] = Vector(11, 22, 33, 11)
Again, note that the same result type principle has been observed in exactly the same way that it would have been had groupIdentical been directly defined on GenTraversableLike.
As of this commit the magic incantation is slightly changed from what it was when Miles gave his excellent answer.
The following works, but is it canonical? I hope one of the canons will correct it. (Or rather, cannons, one of the big guns.) If the view bound is an upper bound, you lose application to Array and String. It doesn't seem to matter if the bound is GenTraversableLike or TraversableLike; but IsTraversableLike gives you a GenTraversableLike.
import language.implicitConversions
import scala.collection.{ GenTraversable=>GT, GenTraversableLike=>GTL, TraversableLike=>TL }
import scala.collection.generic.{ CanBuildFrom=>CBF, IsTraversableLike=>ITL }
class GroupIdenticalImpl[A, R <% GTL[_,R]](val r: GTL[A,R]) {
def groupIdentical[That](implicit cbf: CBF[R, R, That]): That = {
val builder = cbf(r.repr)
def group(r: GTL[_,R]) {
val first = r.head
val (same, rest) = r.span(_ == first)
builder += same
if (!rest.isEmpty) group(rest)
}
if (!r.isEmpty) group(r)
builder.result
}
}
implicit def groupIdentical[A, R <% GTL[_,R]](r: R)(implicit fr: ITL[R]):
GroupIdenticalImpl[fr.A, R] =
new GroupIdenticalImpl(fr conversion r)
There's more than one way to skin a cat with nine lives. This version says that once my source is converted to a GenTraversableLike, as long as I can build the result from GenTraversable, just do that. I'm not interested in my old Repr.
class GroupIdenticalImpl[A, R](val r: GTL[A,R]) {
def groupIdentical[That](implicit cbf: CBF[GT[A], GT[A], That]): That = {
val builder = cbf(r.toTraversable)
def group(r: GT[A]) {
val first = r.head
val (same, rest) = r.span(_ == first)
builder += same
if (!rest.isEmpty) group(rest)
}
if (!r.isEmpty) group(r.toTraversable)
builder.result
}
}
implicit def groupIdentical[A, R](r: R)(implicit fr: ITL[R]):
GroupIdenticalImpl[fr.A, R] =
new GroupIdenticalImpl(fr conversion r)
This first attempt includes an ugly conversion of Repr to GenTraversableLike.
import language.implicitConversions
import scala.collection.{ GenTraversableLike }
import scala.collection.generic.{ CanBuildFrom, IsTraversableLike }
type GT[A, B] = GenTraversableLike[A, B]
type CBF[A, B, C] = CanBuildFrom[A, B, C]
type ITL[A] = IsTraversableLike[A]
class FilterMapImpl[A, Repr](val r: GenTraversableLike[A, Repr]) {
def filterMap[B, That](f: A => Option[B])(implicit cbf : CanBuildFrom[Repr, B, That]): That =
r.flatMap(f(_).toSeq)
}
implicit def filterMap[A, Repr](r: Repr)(implicit fr: ITL[Repr]): FilterMapImpl[fr.A, Repr] =
new FilterMapImpl(fr conversion r)
class GroupIdenticalImpl[A, R](val r: GT[A,R])(implicit fr: ITL[R]) {
def groupIdentical[That](implicit cbf: CBF[R, R, That]): That = {
val builder = cbf(r.repr)
def group(r0: R) {
val r = fr conversion r0
val first = r.head
val (same, other) = r.span(_ == first)
builder += same
val rest = fr conversion other
if (!rest.isEmpty) group(rest.repr)
}
if (!r.isEmpty) group(r.repr)
builder.result
}
}
implicit def groupIdentical[A, R](r: R)(implicit fr: ITL[R]):
GroupIdenticalImpl[fr.A, R] =
new GroupIdenticalImpl(fr conversion r)

How do I apply the enrich-my-library pattern to Scala collections?

One of the most powerful patterns available in Scala is the enrich-my-library* pattern, which uses implicit conversions to appear to add methods to existing classes without requiring dynamic method resolution. For example, if we wished that all strings had the method spaces that counted how many whitespace characters they had, we could:
class SpaceCounter(s: String) {
def spaces = s.count(_.isWhitespace)
}
implicit def string_counts_spaces(s: String) = new SpaceCounter(s)
scala> "How many spaces do I have?".spaces
res1: Int = 5
Unfortunately, this pattern runs into trouble when dealing with generic collections. For example, a number of questions have been asked about grouping items sequentially with collections. There is nothing built in that works in one shot, so this seems an ideal candidate for the enrich-my-library pattern using a generic collection C and a generic element type A:
class SequentiallyGroupingCollection[A, C[A] <: Seq[A]](ca: C[A]) {
def groupIdentical: C[C[A]] = {
if (ca.isEmpty) C.empty[C[A]]
else {
val first = ca.head
val (same,rest) = ca.span(_ == first)
same +: (new SequentiallyGroupingCollection(rest)).groupIdentical
}
}
}
except, of course, it doesn't work. The REPL tells us:
<console>:12: error: not found: value C
if (ca.isEmpty) C.empty[C[A]]
^
<console>:16: error: type mismatch;
found : Seq[Seq[A]]
required: C[C[A]]
same +: (new SequentiallyGroupingCollection(rest)).groupIdentical
^
There are two problems: how do we get a C[C[A]] from an empty C[A] list (or from thin air)? And how do we get a C[C[A]] back from the same +: line instead of a Seq[Seq[A]]?
* Formerly known as pimp-my-library.
The key to understanding this problem is to realize that there are two different ways to build and work with collections in the collections library. One is the public collections interface with all its nice methods. The other, which is used extensively in creating the collections library, but which are almost never used outside of it, is the builders.
Our problem in enriching is exactly the same one that the collections library itself faces when trying to return collections of the same type. That is, we want to build collections, but when working generically, we don't have a way to refer to "the same type that the collection already is". So we need builders.
Now the question is: where do we get our builders from? The obvious place is from the collection itself. This doesn't work. We already decided, in moving to a generic collection, that we were going to forget the type of the collection. So even though the collection could return a builder that would generate more collections of the type we want, it wouldn't know what the type was.
Instead, we get our builders from CanBuildFrom implicits that are floating around. These exist specifically for the purpose of matching input and output types and giving you an appropriately typed builder.
So, we have two conceptual leaps to make:
We aren't using standard collections operations, we're using builders.
We get these builders from implicit CanBuildFroms, not from our collection directly.
Let's look at an example.
class GroupingCollection[A, C[A] <: Iterable[A]](ca: C[A]) {
import collection.generic.CanBuildFrom
def groupedWhile(p: (A,A) => Boolean)(
implicit cbfcc: CanBuildFrom[C[A],C[A],C[C[A]]], cbfc: CanBuildFrom[C[A],A,C[A]]
): C[C[A]] = {
val it = ca.iterator
val cca = cbfcc()
if (!it.hasNext) cca.result
else {
val as = cbfc()
var olda = it.next
as += olda
while (it.hasNext) {
val a = it.next
if (p(olda,a)) as += a
else { cca += as.result; as.clear; as += a }
olda = a
}
cca += as.result
}
cca.result
}
}
implicit def iterable_has_grouping[A, C[A] <: Iterable[A]](ca: C[A]) = {
new GroupingCollection[A,C](ca)
}
Let's take this apart. First, in order to build the collection-of-collections, we know we'll need to build two types of collections: C[A] for each group, and C[C[A]] that gathers all the groups together. Thus, we need two builders, one that takes As and builds C[A]s, and one that takes C[A]s and builds C[C[A]]s. Looking at the type signature of CanBuildFrom, we see
CanBuildFrom[-From, -Elem, +To]
which means that CanBuildFrom wants to know the type of collection we're starting with--in our case, it's C[A], and then the elements of the generated collection and the type of that collection. So we fill those in as implicit parameters cbfcc and cbfc.
Having realized this, that's most of the work. We can use our CanBuildFroms to give us builders (all you need to do is apply them). And one builder can build up a collection with +=, convert it to the collection it is supposed to ultimately be with result, and empty itself and be ready to start again with clear. The builders start off empty, which solves our first compile error, and since we're using builders instead of recursion, the second error also goes away.
One last little detail--other than the algorithm that actually does the work--is in the implicit conversion. Note that we use new GroupingCollection[A,C] not [A,C[A]]. This is because the class declaration was for C with one parameter, which it fills it itself with the A passed to it. So we just hand it the type C, and let it create C[A] out of it. Minor detail, but you'll get compile-time errors if you try another way.
Here, I've made the method a little bit more generic than the "equal elements" collection--rather, the method cuts the original collection apart whenever its test of sequential elements fails.
Let's see our method in action:
scala> List(1,2,2,2,3,4,4,4,5,5,1,1,1,2).groupedWhile(_ == _)
res0: List[List[Int]] = List(List(1), List(2, 2, 2), List(3), List(4, 4, 4),
List(5, 5), List(1, 1, 1), List(2))
scala> Vector(1,2,3,4,1,2,3,1,2,1).groupedWhile(_ < _)
res1: scala.collection.immutable.Vector[scala.collection.immutable.Vector[Int]] =
Vector(Vector(1, 2, 3, 4), Vector(1, 2, 3), Vector(1, 2), Vector(1))
It works!
The only problem is that we don't in general have these methods available for arrays, since that would require two implicit conversions in a row. There are several ways to get around this, including writing a separate implicit conversion for arrays, casting to WrappedArray, and so on.
Edit: My favored approach for dealing with arrays and strings and such is to make the code even more generic and then use appropriate implicit conversions to make them more specific again in such a way that arrays work also. In this particular case:
class GroupingCollection[A, C, D[C]](ca: C)(
implicit c2i: C => Iterable[A],
cbf: CanBuildFrom[C,C,D[C]],
cbfi: CanBuildFrom[C,A,C]
) {
def groupedWhile(p: (A,A) => Boolean): D[C] = {
val it = c2i(ca).iterator
val cca = cbf()
if (!it.hasNext) cca.result
else {
val as = cbfi()
var olda = it.next
as += olda
while (it.hasNext) {
val a = it.next
if (p(olda,a)) as += a
else { cca += as.result; as.clear; as += a }
olda = a
}
cca += as.result
}
cca.result
}
}
Here we've added an implicit that gives us an Iterable[A] from C--for most collections this will just be the identity (e.g. List[A] already is an Iterable[A]), but for arrays it will be a real implicit conversion. And, consequently, we've dropped the requirement that C[A] <: Iterable[A]--we've basically just made the requirement for <% explicit, so we can use it explicitly at will instead of having the compiler fill it in for us. Also, we have relaxed the restriction that our collection-of-collections is C[C[A]]--instead, it's any D[C], which we will fill in later to be what we want. Because we're going to fill this in later, we've pushed it up to the class level instead of the method level. Otherwise, it's basically the same.
Now the question is how to use this. For regular collections, we can:
implicit def collections_have_grouping[A, C[A]](ca: C[A])(
implicit c2i: C[A] => Iterable[A],
cbf: CanBuildFrom[C[A],C[A],C[C[A]]],
cbfi: CanBuildFrom[C[A],A,C[A]]
) = {
new GroupingCollection[A,C[A],C](ca)(c2i, cbf, cbfi)
}
where now we plug in C[A] for C and C[C[A]] for D[C]. Note that we do need the explicit generic types on the call to new GroupingCollection so it can keep straight which types correspond to what. Thanks to the implicit c2i: C[A] => Iterable[A], this automatically handles arrays.
But wait, what if we want to use strings? Now we're in trouble, because you can't have a "string of strings". This is where the extra abstraction helps: we can call D something that's suitable to hold strings. Let's pick Vector, and do the following:
val vector_string_builder = (
new CanBuildFrom[String, String, Vector[String]] {
def apply() = Vector.newBuilder[String]
def apply(from: String) = this.apply()
}
)
implicit def strings_have_grouping(s: String)(
implicit c2i: String => Iterable[Char],
cbfi: CanBuildFrom[String,Char,String]
) = {
new GroupingCollection[Char,String,Vector](s)(
c2i, vector_string_builder, cbfi
)
}
We need a new CanBuildFrom to handle the building of a vector of strings (but this is really easy, since we just need to call Vector.newBuilder[String]), and then we need to fill in all the types so that the GroupingCollection is typed sensibly. Note that we already have floating around a [String,Char,String] CanBuildFrom, so strings can be made from collections of chars.
Let's try it out:
scala> List(true,false,true,true,true).groupedWhile(_ == _)
res1: List[List[Boolean]] = List(List(true), List(false), List(true, true, true))
scala> Array(1,2,5,3,5,6,7,4,1).groupedWhile(_ <= _)
res2: Array[Array[Int]] = Array(Array(1, 2, 5), Array(3, 5, 6, 7), Array(4), Array(1))
scala> "Hello there!!".groupedWhile(_.isLetter == _.isLetter)
res3: Vector[String] = Vector(Hello, , there, !!)
As of this commit it's a lot easier to "enrich" Scala collections than it was when Rex gave his excellent answer. For simple cases it might look like this,
import scala.collection.generic.{ CanBuildFrom, FromRepr, HasElem }
import language.implicitConversions
class FilterMapImpl[A, Repr](val r : Repr)(implicit hasElem : HasElem[Repr, A]) {
def filterMap[B, That](f : A => Option[B])
(implicit cbf : CanBuildFrom[Repr, B, That]) : That = r.flatMap(f(_).toSeq)
}
implicit def filterMap[Repr : FromRepr](r : Repr) = new FilterMapImpl(r)
which adds a "same result type" respecting filterMap operation to all GenTraversableLikes,
scala> val l = List(1, 2, 3, 4, 5)
l: List[Int] = List(1, 2, 3, 4, 5)
scala> l.filterMap(i => if(i % 2 == 0) Some(i) else None)
res0: List[Int] = List(2, 4)
scala> val a = Array(1, 2, 3, 4, 5)
a: Array[Int] = Array(1, 2, 3, 4, 5)
scala> a.filterMap(i => if(i % 2 == 0) Some(i) else None)
res1: Array[Int] = Array(2, 4)
scala> val s = "Hello World"
s: String = Hello World
scala> s.filterMap(c => if(c >= 'A' && c <= 'Z') Some(c) else None)
res2: String = HW
And for the example from the question, the solution now looks like,
class GroupIdenticalImpl[A, Repr : FromRepr](val r: Repr)
(implicit hasElem : HasElem[Repr, A]) {
def groupIdentical[That](implicit cbf: CanBuildFrom[Repr,Repr,That]): That = {
val builder = cbf(r)
def group(r: Repr) : Unit = {
val first = r.head
val (same, rest) = r.span(_ == first)
builder += same
if(!rest.isEmpty)
group(rest)
}
if(!r.isEmpty) group(r)
builder.result
}
}
implicit def groupIdentical[Repr : FromRepr](r: Repr) = new GroupIdenticalImpl(r)
Sample REPL session,
scala> val l = List(1, 1, 2, 2, 3, 3, 1, 1)
l: List[Int] = List(1, 1, 2, 2, 3, 3, 1, 1)
scala> l.groupIdentical
res0: List[List[Int]] = List(List(1, 1),List(2, 2),List(3, 3),List(1, 1))
scala> val a = Array(1, 1, 2, 2, 3, 3, 1, 1)
a: Array[Int] = Array(1, 1, 2, 2, 3, 3, 1, 1)
scala> a.groupIdentical
res1: Array[Array[Int]] = Array(Array(1, 1),Array(2, 2),Array(3, 3),Array(1, 1))
scala> val s = "11223311"
s: String = 11223311
scala> s.groupIdentical
res2: scala.collection.immutable.IndexedSeq[String] = Vector(11, 22, 33, 11)
Again, note that the same result type principle has been observed in exactly the same way that it would have been had groupIdentical been directly defined on GenTraversableLike.
As of this commit the magic incantation is slightly changed from what it was when Miles gave his excellent answer.
The following works, but is it canonical? I hope one of the canons will correct it. (Or rather, cannons, one of the big guns.) If the view bound is an upper bound, you lose application to Array and String. It doesn't seem to matter if the bound is GenTraversableLike or TraversableLike; but IsTraversableLike gives you a GenTraversableLike.
import language.implicitConversions
import scala.collection.{ GenTraversable=>GT, GenTraversableLike=>GTL, TraversableLike=>TL }
import scala.collection.generic.{ CanBuildFrom=>CBF, IsTraversableLike=>ITL }
class GroupIdenticalImpl[A, R <% GTL[_,R]](val r: GTL[A,R]) {
def groupIdentical[That](implicit cbf: CBF[R, R, That]): That = {
val builder = cbf(r.repr)
def group(r: GTL[_,R]) {
val first = r.head
val (same, rest) = r.span(_ == first)
builder += same
if (!rest.isEmpty) group(rest)
}
if (!r.isEmpty) group(r)
builder.result
}
}
implicit def groupIdentical[A, R <% GTL[_,R]](r: R)(implicit fr: ITL[R]):
GroupIdenticalImpl[fr.A, R] =
new GroupIdenticalImpl(fr conversion r)
There's more than one way to skin a cat with nine lives. This version says that once my source is converted to a GenTraversableLike, as long as I can build the result from GenTraversable, just do that. I'm not interested in my old Repr.
class GroupIdenticalImpl[A, R](val r: GTL[A,R]) {
def groupIdentical[That](implicit cbf: CBF[GT[A], GT[A], That]): That = {
val builder = cbf(r.toTraversable)
def group(r: GT[A]) {
val first = r.head
val (same, rest) = r.span(_ == first)
builder += same
if (!rest.isEmpty) group(rest)
}
if (!r.isEmpty) group(r.toTraversable)
builder.result
}
}
implicit def groupIdentical[A, R](r: R)(implicit fr: ITL[R]):
GroupIdenticalImpl[fr.A, R] =
new GroupIdenticalImpl(fr conversion r)
This first attempt includes an ugly conversion of Repr to GenTraversableLike.
import language.implicitConversions
import scala.collection.{ GenTraversableLike }
import scala.collection.generic.{ CanBuildFrom, IsTraversableLike }
type GT[A, B] = GenTraversableLike[A, B]
type CBF[A, B, C] = CanBuildFrom[A, B, C]
type ITL[A] = IsTraversableLike[A]
class FilterMapImpl[A, Repr](val r: GenTraversableLike[A, Repr]) {
def filterMap[B, That](f: A => Option[B])(implicit cbf : CanBuildFrom[Repr, B, That]): That =
r.flatMap(f(_).toSeq)
}
implicit def filterMap[A, Repr](r: Repr)(implicit fr: ITL[Repr]): FilterMapImpl[fr.A, Repr] =
new FilterMapImpl(fr conversion r)
class GroupIdenticalImpl[A, R](val r: GT[A,R])(implicit fr: ITL[R]) {
def groupIdentical[That](implicit cbf: CBF[R, R, That]): That = {
val builder = cbf(r.repr)
def group(r0: R) {
val r = fr conversion r0
val first = r.head
val (same, other) = r.span(_ == first)
builder += same
val rest = fr conversion other
if (!rest.isEmpty) group(rest.repr)
}
if (!r.isEmpty) group(r.repr)
builder.result
}
}
implicit def groupIdentical[A, R](r: R)(implicit fr: ITL[R]):
GroupIdenticalImpl[fr.A, R] =
new GroupIdenticalImpl(fr conversion r)

How can I extend Scala collections with an argmax method?

I would like to add to all collections where it makes sense, an argMax method.
How to do it? Use implicits?
On Scala 2.8, this works:
val list = List(1, 2, 3)
def f(x: Int) = -x
val argMax = list max (Ordering by f)
As pointed by mkneissl, this does not return the set of maximum points. Here's an alternate implementation that does, and tries to reduce the number of calls to f. If calls to f don't matter that much, see mkneissl's answer. Also, note that his answer is curried, which provides superior type inference.
def argMax[A, B: Ordering](input: Iterable[A], f: A => B) = {
val fList = input map f
val maxFList = fList.max
input.view zip fList filter (_._2 == maxFList) map (_._1) toSet
}
scala> argMax(-2 to 2, (x: Int) => x * x)
res15: scala.collection.immutable.Set[Int] = Set(-2, 2)
The argmax function (as I understand it from Wikipedia)
def argMax[A,B](c: Traversable[A])(f: A=>B)(implicit o: Ordering[B]): Traversable[A] = {
val max = (c map f).max(o)
c filter { f(_) == max }
}
If you really want, you can pimp it onto the collections
implicit def enhanceWithArgMax[A](c: Traversable[A]) = new {
def argMax[B](f: A=>B)(implicit o: Ordering[B]): Traversable[A] = ArgMax.argMax(c)(f)(o)
}
and use it like this
val l = -2 to 2
assert (argMax(l)(x => x*x) == List(-2,2))
assert (l.argMax(x => x*x) == List(-2,2))
(Scala 2.8)
Yes, the usual way would be to use the 'pimp my library' pattern to decorate your collection. For example (N.B. just as illustration, not meant to be a correct or working example):
trait PimpedList[A] {
val l: List[A]
//example argMax, not meant to be correct
def argMax[T <% Ordered[T]](f:T => T) = {error("your definition here")}
}
implicit def toPimpedList[A](xs: List[A]) = new PimpedList[A] {
val l = xs
}
scala> def f(i:Int):Int = 10
f: (i: Int) Int
scala> val l = List(1,2,3)
l: List[Int] = List(1, 2, 3)
scala> l.argMax(f)
java.lang.RuntimeException: your definition here
at scala.Predef$.error(Predef.scala:60)
at PimpedList$class.argMax(:12)
//etc etc...
Nice and easy ? :
val l = List(1,0,10,2)
l.zipWithIndex.maxBy(x => x._1)._2
You can add functions to an existing API in Scala by using the Pimp my Library pattern. You do this by defining an implicit conversion function. For example, I have a class Vector3 to represent 3D vectors:
class Vector3 (val x: Float, val y: Float, val z: Float)
Suppose I want to be able to scale a vector by writing something like: 2.5f * v. I can't directly add a * method to class Float ofcourse, but I can supply an implicit conversion function like this:
implicit def scaleVector3WithFloat(f: Float) = new {
def *(v: Vector3) = new Vector3(f * v.x, f * v.y, f * v.z)
}
Note that this returns an object of a structural type (the new { ... } construct) that contains the * method.
I haven't tested it, but I guess you could do something like this:
implicit def argMaxImplicit[A](t: Traversable[A]) = new {
def argMax() = ...
}
Here's a way of doing so with the implicit builder pattern. It has the advantage over the previous solutions that it works with any Traversable, and returns a similar Traversable. Sadly, it's pretty imperative. If anyone wants to, it could probably be turned into a fairly ugly fold instead.
object RichTraversable {
implicit def traversable2RichTraversable[A](t: Traversable[A]) = new RichTraversable[A](t)
}
class RichTraversable[A](t: Traversable[A]) {
def argMax[That, C](g: A => C)(implicit bf : scala.collection.generic.CanBuildFrom[Traversable[A], A, That], ord:Ordering[C]): That = {
var minimum:C = null.asInstanceOf[C]
val repr = t.repr
val builder = bf(repr)
for(a<-t){
val test: C = g(a)
if(test == minimum || minimum == null){
builder += a
minimum = test
}else if (ord.gt(test, minimum)){
builder.clear
builder += a
minimum = test
}
}
builder.result
}
}
Set(-2, -1, 0, 1, 2).argmax(x=>x*x) == Set(-2, 2)
List(-2, -1, 0, 1, 2).argmax(x=>x*x) == List(-2, 2)
Here's a variant loosely based on #Daniel's accepted answer that also works for Sets.
def argMax[A, B: Ordering](input: GenIterable[A], f: A => B) : GenSet[A] = argMaxZip(input, f) map (_._1) toSet
def argMaxZip[A, B: Ordering](input: GenIterable[A], f: A => B): GenIterable[(A, B)] = {
if (input.isEmpty) Nil
else {
val fPairs = input map (x => (x, f(x)))
val maxF = fPairs.map(_._2).max
fPairs filter (_._2 == maxF)
}
}
One could also do a variant that produces (B, Iterable[A]), of course.
Based on other answers, you can pretty easily combine the strengths of each (minimal calls to f(), etc.). Here we have an implicit conversion for all Iterables (so they can just call .argmax() transparently), and a stand-alone method if for some reason that is preferred. ScalaTest tests to boot.
class Argmax[A](col: Iterable[A]) {
def argmax[B](f: A => B)(implicit ord: Ordering[B]): Iterable[A] = {
val mapped = col map f
val max = mapped max ord
(mapped zip col) filter (_._1 == max) map (_._2)
}
}
object MathOps {
implicit def addArgmax[A](col: Iterable[A]) = new Argmax(col)
def argmax[A, B](col: Iterable[A])(f: A => B)(implicit ord: Ordering[B]) = {
new Argmax(col) argmax f
}
}
class MathUtilsTests extends FunSuite {
import MathOps._
test("Can argmax with unique") {
assert((-10 to 0).argmax(_ * -1).toSet === Set(-10))
// or alternate calling syntax
assert(argmax(-10 to 0)(_ * -1).toSet === Set(-10))
}
test("Can argmax with multiple") {
assert((-10 to 10).argmax(math.pow(_, 2)).toSet === Set(-10, 10))
}
}