Understand Scala Implicit classes - scala

I am reading a Spark in Action book, I came across a construct that even after a while working with Scala I still can't understand. Here is the code (complete code) :
First we have a case class where RDD will be converted to DataFrame using this case class:
import java.sql.Timestamp
case class Post(
commentCount:Option[Int],
lastActivityDate:Option[java.sql.Timestamp],
ownerUserId:Option[Long],
body:String,
score:Option[Int],
... )
Then we define an object with implicit class
object StringImplicits {
implicit class StringImprovements(val s: String) {
import scala.util.control.Exception.catching
def toIntSafe = catching(classOf[NumberFormatException]) opt s.toInt
def toLongSafe = catching(classOf[NumberFormatException]) opt s.toLong
def toTimestampSafe = catching(classOf[IllegalArgumentException]) opt
Timestamp.valueOf(s)
}
}
Then we import the object which is already in scope for some reason and now looks like all String object has the three methods defined in StringImprovements
import StringImplicits._
def stringToPost(row: String): Post = {
val r = row.split("~")
Post(r(0).toIntSafe,
r(1).toTimestampSafe,
r(2).toLongSafe,
r(3),
...
}
Questions:
Why do we need to need to import an already in scope object?
Which statement exactly assigned these methods to the String object, this is so confusing?
Reading implicit classes doc it makes some sense but not everything is clear. For example how come this is a valid expression
scala> 5 times println("HI")
I understand if it was
IntWithTimes(5).time(println("HI"))

Implicits are sought among all methods and values that are visible in the current scope AND within companion objects of types that appear in the implicit type you are looking for.
implicit val a: A = ???
implicit val bc: B[C] = ???
// code here sees a and bc in implicit scope
def helper = {
implicit val c: C = ???
// code here sees a and bc and c in implicit scope
}
def anotherHelper = {
val a: A = ??? // overshades a
// code here sees b in implicit scope
}
object Helper {
implicit val e: E = ???
// code here sees a, bc and e in implicit scope
}
// implicits within helper, anotherHelper and Helper are not seen here
// though Helper one's can be imported with import Helper._
class D[T]
object {
implicit def provide[T](implicit a: T): D[T] = ???
}
class E
object E {
implicit val e: E = ???
}
class F
object F
// sees no implicits that are always available but
// if you require implicitly[E], E companion will be searched
// and implicit e will be found
// if you require implicitly[D[F]], both D and F companions
// will be searched, D.provide[F] will require implicit F,
// and no such implicit will be found - failure
If you asked for implicit A[B, C, (D, E), F[G]], then Scala would look into companions of A, B, C, D, E, Tuple2, F, G (if they exist). If this rule doesn't apply (your implicits aren't in companion, your type doesn't appear in sought implicit) you have to import them manually.
When Scala sees that you try to call a method on object that doesn't exists, it will try to convert it to some object which has this method. If you have some X type, it will look for implicits your X and returning some Y (implicit defs(x: X): Y and implicit val f: X => Y) and checking if that Y has this method.
Implicit class is basically a combination of a class and implicit method:
// implicit class Y(x: X) is the same as
class Y(x: X)
implicit def convert(x: X): Y = new Y(x)
which means that you can add some implicit conversions/classes from type X into X's companion and they will be always available.
In docs:
implicit class IntWithTimes(x: Int)
created implicit conversion from Int => IntWithTimes. String doesn't have times method, but there is a conversion Int => IntWithTimes in implicit scope imperted with import Helper._ (without it compilation would fail). Its result, IntWithTimes has the times method with the right signature, so the conversion is used.

Related

What exactly does 'is more specific than' mean in the context of implicits?

So, implicit precedence is based on two factors: one is about the scope of declaration itself having precedence (scope A extending scope/trait B, or scope A being a companion object of a type extended from a type with scope B as its companion object). The other simply mentions that declaration A is more specific than declaration B. Now, when reading it for the first time I had several possible interpretations in mind, especially considering potential parameters (implicit and not) of implicit method and type parameters. Experience seemed to teach me that it means that the type of the returned value by declaration A, after all type inference/tapply, is a subtype of the return type of declaration B. So, this is fine:
class A
class B extends A
implicit val A = new A
implicit val B = new B
implicitly[A]
Why this doesn't compile, then?
implicit class A(val toInt :Int) {
def ! = (1 /: (2 to toInt))(_ * _)
}
implicit class B(i :Int) extends A(i)
1.!
When this does?
class A(val toInt :Int) {
def ! = (1 /: (2 to toInt))(_ * _)
}
class B(i :Int) extends A(i)
implicit val A = { i :Int => new A(i) }
implicit val B = { i :Int => new B(i) }
1.!
Is it another case of 'the compiler works in mysterious ways'?
Based on SIP-13 - IMPLICIT CLASSES proposal your declaration:
implicit class A(val toInt :Int) {
def ! = (1 /: (2 to toInt))(_ * _)
}
Will be transformed to something like:
class A(toInt: Int) {
...
}
implicit final def A(toInt: Int): A = new A(toInt);
And your B in its turn will be transformed to something like:
class B(i: Int) extends A(i) {
...
}
implicit final def B(i: Int): B = new B(i);
So you are basically declaring 2 implicit conversion methods with the same parameters which are ambiguous. While the last one is not ambiguous due to already mentioned "most specific" rule for implicit parameters resolution.

What is meant by a type that denotes an implicit conversion

The scala language specification, section 7.2 is about implicit scope:
It explains that implicit scope are the modules associated in some way with the parts of type T. What the parts of T are is listed below. One of those points is
if T denotes an implicit conversion to a type with a method with
argument types T1,…,Tn and result type U, the union of the parts of
T1,…,Tn and U;
I can't make head or tails from this. I don't understand how a type T can denote an implicit conversion.
What is meant by this part of the specification of implicit scope?
I believe this is referring to the following situation
case class Foo(v: Int)
object Foo {
implicit def stringToFoo(s: String) = Foo(42)
}
def f[A](v: A)(implicit ev: A => Foo) = ev(v)
f("woohoo")
where the implicit conversion type T = A => Foo, and Foo is the part associated with type parameter A, therefore object Foo becomes part of the implicit scope and stringToFoo implicit conversion is resolved without needing an import.
Here are examples:
trait A
object A {
implicit def f(a: A): B = null
}
trait B
val b: B = new A {} // compiles
or
trait A
trait B
object B {
implicit def f(a: A): B = null
}
val b: B = new A {} // compiles
Let T denote an implicit conversion to a type, from A to B (val b: B = new A {}), it denotes the conversion with a method with argument types T1,…,Tn = A and result type U = B. So the parts of T are the union of the parts of A and parts of B. So an implicit can be defined as a member of the companion object of a base class of the parts of T (i.e. A or B).

why does scala's implicit lookup ignore companion object of nested class

I was playing around with the following piece of code:
class A
class B
class C
trait Codecs[L] {
case class Codec[R](val code: L => R, val decode: R => L)
object Codec
def code[R](foo: L)(implicit codec: Codec[R]): R = codec.code(foo)
def decode[R](bar: R)(implicit codec: Codec[R]): L = codec.decode(bar)
}
object Codecs {
implicit object ACodecs extends Codecs[A] {
object Codec {
implicit val ab: Codec[B] = new Codec(_ => new B, _ => new A)
implicit val ac: Codec[C] = new Codec(_ => new C, _ => new A)
}
}
}
object test extends App {
val codecs = implicitly[Codecs[A]]
codecs.code[B](new A)
}
It won't compile, as the compiler is unable to find an implicit value of type Codecs.Codec[B].
As I understand, the two values ab and ac are of type Acodecs.Codec[_](or something like that), which isn't exactly what the compiler is looking for. I am also aware that moving the case class Codec[_] and its companion outside of the trait solves the problem (after making it take 2 type params). If an implicit value is required, the compiler should include the companion object of the required type in the implicit scope. My questions are:
How do I point the compiler at the companion of a path-dependent subtype, more specifically:
Is it possible to alter the signature of the two methods of the trait (ideally alter the type signature of the implicit param) to make this compile? How would one refer to the type Acodecs.Codec[_] from inside the trait Codecs[_]?
Like, how do you do this typeclass thing on a nested type?
Is there like a pattern or something dealing with this sort of problem?
The issue is that your type is bound to a specific instance since it's an inner class. And the compiler doesn't know that implicitly[Codecs[A]] is giving the exact same instance as what it's finding implicitly on the next line. For instance, if you pass it explicitly:
codecs.code[B](new A)(Codecs.ACodecs.Codec.ab)
You get this error message:
type mismatch;
found : Codecs.ACodecs.Codec[B]
required: codecs.Codec[B]
So it believes the enclosing instances to be possibly different, and so different types.
I've never really seen this specific kind of nesting of implicits - i.e. an implicit typeclass with path-dependent implicit typeclasses within it. So I doubt there's a pattern for dealing with it and would in fact kind of recommend against it. It seems overly complicated. Here's how I would personally treat this case:
case class Codec[L, R](val code: L => R, val decode: R => L)
trait Codecs[L] {
type LocalCodec[R] = Codec[L, R]
def code[R](foo: L)(implicit codec: LocalCodec[R]): R = codec.code(foo)
def decode[R](bar: R)(implicit codec: LocalCodec[R]): L = codec.decode(bar)
}
object Codecs {
implicit object ACodecs extends Codecs[A] {
implicit val ab: LocalCodec[B] = new LocalCodec(_ => new B, _ => new A)
implicit val ac: LocalCodec[C] = new LocalCodec(_ => new C, _ => new A)
}
}
object test extends App {
import Codecs.ACodecs._
val codecs = implicitly[Codecs[A]]
codecs.code[B](new A)
}
You still get the benefit of a "half-narrowed" type to work with but it's just a type alias so there's no path-dependency issues.

Scalaz instancing typeclass and case class

This problem is pretty hard to explain. I try to implement a kind of Arrow, call it MyArr, which extends the Arrow trait from scalaz. However, MyArr is also a kind of Algbraic Data Type, so I make a few case classes out of if, and wrote down code like this(minimal example):
package org.timeless
import scalaz._
import Scalaz._
import std.option._, std.list._
sealed trait MyArr[A,B] extends Arrow[MyArr] {
def id[A]: MyArr[A,A] = SId()
def compose[A,B,C](s1: MyArr[B,C], s2: MyArr[A,B]): MyArr[A,C] =
SRight(s2, s1)
def arr[A,B](f: A=>B): MyArr[A,B] = SPure(f)
def first[A,B,C](s: MyArr[A,B]): MyArr[(A,C),(B,C)] = SPar(s, SId())
}
case class SId[A] () extends MyArr[A,A]()
case class SRight[M[_],A,B,C](s1: MyArr[A,B], s2: MyArr[B,C]) extends MyArr[A,C]
case class SPure[A,B](f: A=>B) extends MyArr[A,B]
case class SPar[M[_],A,B,C,D](s1: MyArr[A,B], s2: MyArr[C,D]) extends MyArr[(A,C),(B,D)]
object MyArr {
val b = SId[Int]()
val c = SId[Int]()
// val a: MyArr[Int,Int] = id[Int] // Error
// val d = c compose b // Error
// val d = b >>> c // Error
val d = b >>> (b,c) // ??? Arguments
}
As easily seen, the id function is actually the constructor of SId, etc. The classes themselves compile fine, but I don't see any way to actually use it as an Arrow. I actually came from Haskell programming, so I might be doing it totally wrong, but equivalent code in Haskell should be like this(with GADTs):
data MyArr a b where
SId :: MyArr a b
SRight :: MyArr a b -> MyArr b c -> MyArr a c
SPure :: (a -> b) -> MyArr a b
SPar :: MyArr a b -> MyArr c d -> MyArr (a,c) (b,d)
instance Category MyArr where
id = SId
(.) = SRight
instance Arrow MyArr where
arr = SPure
first s = SPar s SId
Typeclasses in Scala
Typeclasses in Scala are not built into the language like in Haskell. They are just a design pattern and are usually implemented with the help of implicit arguments:
def pure[M[_], T](
implicit monad: Monad[M] // Fetching a monad instance for type M
): M[T] =
monad.pure[T] // Using methods of the monad instance
You don't have to extend the typeclass trait to provide a typeclass instance for some ADT. The typeclass instance has to be defined somewhere as an implicit val, implicit def, implicit object, etc. Then you can import the relevant instance into scope before using it.
There is also a way to supply the typeclass instance automatically, without having to import it explicitly. To do that the instance should be declared in the companion object of either the typeclass class/trait or ADT class/trait, or one of their parent classes/traits; or in a package object in which one of those is defined. You can read more on the topic of implicit resolution in this answer: Where does Scala look for implicits?
Defining a typeclass instance
So to get back to your problem, you can define your ADT like this:
sealed trait MyArr[A,B]
case class SId[A] () extends MyArr[A,A]()
case class SRight[M[_],A,B,C](s1: MyArr[A,B], s2: MyArr[B,C]) extends MyArr[A,C]
case class SPure[A,B](f: A=>B) extends MyArr[A,B]
case class SPar[M[_],A,B,C,D](s1: MyArr[A,B], s2: MyArr[C,D]) extends MyArr[(A,C),(B,D)]
And provide the Arrow instance in the companion object of MyArr:
object MyArr {
implicit val arrow: Arrow[MyArr] = new Arrow[MyArr] {
def id[A]: MyArr[A,A] = SId()
def compose[A,B,C](s1: MyArr[B,C], s2: MyArr[A,B]): MyArr[A,C] =
SRight(s2, s1)
def arr[A,B](f: A=>B): MyArr[A,B] = SPure(f)
def first[A,B,C](s: MyArr[A,B]): MyArr[(A,C),(B,C)] = SPar(s, SId())
}
def apply[T]: MyArr[T, T] = arrow.id[T]
}
Using the typeclass instance
I believe it's not possible to provide a generic id function (or return, pure, etc.) in Scala due to some limitations of Scala's type inference.
Instead you can use the MyArr.apply method defined above:
val a: MyArr[Int,Int] = MyArr[Int]
As the ADT doesn't extend the typeclass, it doesn't inherit its methods like compose or >>>. They can be provided separately using the 'Pimp My Library' pattern.
Indeed, the >>> method in scalaz comes from the ComposeOps class, which is defined like this:
final class ComposeOps[F[_, _],A, B] private[syntax]
(self: F[A, B])
(val F: Compose[F])
extends Ops[F[A, B]]
And there is an implicit conversion defined from some type F[_, _] to this class ComposeOps, if there as a Compose instance available for F:
implicit def ToComposeOps[F[_, _],A, B](v: F[A, B])(implicit F0: Compose[F])
This conversion is imported with import Scalaz._, and that's why this line works now:
val d = b >>> c
But there is no extension in scalaz that provides a compose method on ADTs with an Arrow instance, so this line still gives an error:
val e = c compose b
To make it work, you can provide an extension yourself. A simple example could look like this:
// implicit class is equivalent to a normal class
// with an implicit conversion to this class
implicit class ArrowOps[T[_, _], A, B](f: T[A, B])(implicit arrow: Arrow[T]) {
def compose[C](g: T[C, A]): T[C, B] = arrow.compose(f, g)
}

Implicit parameters won't work on unapply. How to hide ubiquitous parameters from extractors?

Apparently unapply/unapplySeq in extractor objects do not support implicit parameters. Assuming here an interesting parameter a, and a disturbingly ubiquitous parameter b that would be nice to hide away, when extracting c.
[EDIT]: It appears something was broken in my intellij/scala-plugin installation that caused this. I cannot explain. I was having numerous strange problems with my intellij lately. After reinstalling, I can no longer reprodce my problem. Confirmed that unapply/unapplySeq do allow for implicit parameters! Thanks for your help.
This does not work (**EDIT:yes, it does):**
trait A; trait C; trait B { def getC(a: A): C }
def unapply(a:A)(implicit b:B):Option[C] = Option(b.getC(a))
In my understanding of what an ideal extractor should be like, in which the intention is intuitively clear also to Java folks, this limitation basically forbids extractor objects which depend on additional parameter(s).
How do you typically handle this limitation?
So far I've got those four possible solutions:
1) The simplest solution that I want to improve on: don't hide b, provide parameter b along with a, as normal parameter of unapply in form of a tuple:
object A1{
def unapply(a:(A,B)):Option[C] = Option(a._2.getC(a._1)) }
in client code:
val c1 = (a,b) match { case A1(c) => c1 }
I don't like it because there is more noise deviating that deconstruction of a into c is important here. Also since java folks, that have to be convinced to actually use this scala code, are confronted with one additional synthactic novelty (the tuple braces). They might get anti-scala aggressions "What's all this? ... Why then not use a normal method in the first place and check with if?".
2) define extractors within a class encapsulating the dependence on a particular B, import extractors of that instance. At import site a bit unusual for java folks, but at pattern match site b is hidden nicely and it is intuitively evident what happens. My favorite. Some disadvantage I missed?
class BDependent(b:B){
object A2{
def unapply(a:A):Option[C] = Option(b.getC(a))
} }
usage in client code:
val bDeps = new BDependent(someB)
import bDeps.A2
val a:A = ...
val c2 = a match { case A2(c) => c }
}
3) declare extractor objects in scope of client code. b is hidden, since it can use a "b" in local scope. Hampers code reuse, heavily pollutes client code (additionally, it has to be stated before code using it).
4) have unapply return Option of function B => C. This allows import and usage of an ubitious-parameter-dependent extractor, without providing b directly to the extractor, but instead to the result when used. Java folks maybe confused by usage of function values, b not hidden:
object A4{
def unapply[A,C](a:A):Option[B => C] = Option((_:B).getC(a))
}
then in client code:
val b:B = ...
val soonAC: B => C = a match { case A4(x) => x }
val d = soonAC(b).getD ...
Further remarks:
As suggested in this answer, "view bounds" may help to get extractors work with implicit conversions, but this doesn't help with implicit parameters. For some reason I prefer not to workaround with implicit conversions.
looked into "context bounds", but they seem to have the same limitation, don't they?
In what sense does your first line of code not work? There's certainly no arbitrary prohibition on implicit parameter lists for extractor methods.
Consider the following setup (I'm using plain old classes instead of case classes to show that there's no extra magic happening here):
class A(val i: Int)
class C(val x: String)
class B(pre: String) { def getC(a: A) = new C(pre + a.i.toString) }
Now we define an implicit B value and create an extractor object with your unapply method:
implicit val b = new B("prefix: ")
object D {
def unapply(a: A)(implicit b: B): Option[C] = Option(b getC a)
}
Which we can use like this:
scala> val D(c) = new A(42)
c: C = C#52394fb3
scala> c.x
res0: String = prefix: 42
Exactly as we'd expect. I don't see why you need a workaround here.
The problem you have is that implicit parameters are compile time (static) constraints, whereas pattern matching is a runtime (dynamic) approach.
trait A; trait C; trait B { def getC(a: A): C }
object Extractor {
def unapply(a: A)(implicit b: B): Option[C] = Some(b.getC(a))
}
// compiles (implicit is statically provided)
def withImplicit(a: A)(implicit b: B) : Option[C] = a match {
case Extractor(c) => Some(c)
case _ => None
}
// does not compile
def withoutImplicit(a: A) : Option[C] = a match {
case Extractor(c) => Some(c)
case _ => None
}
So this is a conceptual problem, and the solution depends on what you actually want to achieve. If you want something along the lines of an optional implicit, you might use the following:
sealed trait FallbackNone {
implicit object None extends Optional[Nothing] {
def toOption = scala.None
}
}
object Optional extends FallbackNone {
implicit def some[A](implicit a: A) = Some(a)
final case class Some[A](a: A) extends Optional[A] {
def toOption = scala.Some(a)
}
}
sealed trait Optional[+A] { def toOption: Option[A]}
Then where you had implicit b: B you will have implicit b: Optional[B]:
object Extractor {
def unapply(a:A)(implicit b: Optional[B]):Option[C] =
b.toOption.map(_.getC(a))
}
def test(a: A)(implicit b: Optional[B]) : Option[C] = a match {
case Extractor(c) => Some(c)
case _ => None
}
And the following both compile:
test(new A {}) // None
{
implicit object BImpl extends B { def getC(a: A) = new C {} }
test(new A {}) // Some(...)
}