Strange NPE with io.circe.Decoder - scala

I have 2 variables declared as follows,
implicit val decodeURL: Decoder[URL] = Decoder.decodeString.emapTry(s => Try(new URL(s))) // #1
implicit val decodeCompleted = Decoder[List[URL]].prepare(_.downField("completed")) // #2
Both lines compile and run.
However, if I annotate #2 with type i.e. implicit val decodeCompleted: Decoder[List[URL]] = Decoder[List[URL]].prepare(_.downField("completed")). It compiles and #2 will throw NullPointerException (NPE) during runtime.
How could this happen? I don't know if this is Circe or just plain Scala issue. Why #2 is different from #1? Thanks

The issue is that you are supposed to use implicits with annotation always.
Now, when you use them when they are not annotated you get into some sort on undefined/invalid behavior zone. That is why with unannotated happens sth like this:
I want to use Decoder[List[URL]]
there is no (annotated) Decoder[List[URL]] implicit in the scope
let's derive it normally (no need for generic.auto._ because the definition for that is in a companion object)
once derived you call on it .prepare(_.downField("completed"))
the final result is of type Decoder[List[URL]], so that is inferred type of decodeCompleted
Now, what happens if you annotate?
I want to use Decoder[List[URL]]
there is decodeCompleted declared as something that fulfills that definition
let use decodeCompleted value
but decodeCompleted wasn't initialized! in fact we are initializing it right now!
as a result you end up with decodeCompleted = null
This is virtually equal to:
val decodeCompleted = decodeCompleted
except that the layer of indirection get's in the way of discovering the absurdity of this by compiler. (If you replaced val with def you would end up with an infinite recursion and stack overflow):
# implicit val s: String = implicitly[String]
s: String = null
# implicit def s: String = implicitly[String]
defined function s
# s
java.lang.StackOverflowError
ammonite.$sess.cmd1$.s(cmd1.sc:1)
ammonite.$sess.cmd1$.s(cmd1.sc:1)
ammonite.$sess.cmd1$.s(cmd1.sc:1)
ammonite.$sess.cmd1$.s(cmd1.sc:1)
Yup, its messed up by compiler. You did nothing wrong and in a perfect world it would work.
Scala community mitigates that by distincting:
auto derivation - when you need implicit somewhere and it is automatically derived without you defining a variable for it
semiauto derivation - when you derive into a value, and make that value implicit
In the later case, you usually have some utilities like:
import io.circe.generic.semiauto._
implicit val decodeCompleted: Decoder[List[URL]] = deriveDecoder[List[URL]]
It works because it takes DerivedDecoder[A] implicit and then it extracts Decoder[A] from it, so you never end up with implicit val a: A = implicitly[A] scenario.

Indeed the problem is that you introduce a recursive val, like #MateuszKubuszok explained.
The most straightforward—although slightly ugly—workaround is:
implicit val decodeCompleted: Decoder[List[URL]] = {
val decodeCompleted = null
Decoder[List[URL]].prepare(_.downField("completed"))
}
By shadowing decodeCompleted in the right-hand side, implicit search will no longer consider it as a candidate inside that code block, because it can no longer be referenced.

Related

Scala Implicit Parameters Projection Conflict , "Ambigious Implicit Values" Error

I have been reading Bruno's TypeClasses paper and he mentioned that implicits in the argument list are projected/propagated into the implicit scope. I followed the example with this code:
package theory
import cats.implicits.toShow
import java.io.PrintStream
import java.util.Date
object Equality extends App {
import cats.Eq
// assert(123 == "123")
println(Eq.eqv(123, 123).show)
implicit val out: PrintStream = System.out
def log(m: String)(implicit o: PrintStream ): Unit =
o.println(m)
def logTime(m: String)(implicit o: PrintStream): Unit =
log(s"${new Date().getTime} : $m")
}
The kicker is that this code will not compile, with:
ambiguous implicit values:
both value out in object Equality of type java.io.PrintStream
and value o of type java.io.PrintStream
match expected type java.io.PrintStream
log(s"${new Date().getTime} : $m")
So, I assume that the compiler sees 2 instances of the same implicit and complains. I was able to silence the compiler by explicitly adding the PrintStream passed as an argument as the second argument to log:
def logTime(m: String)(implicit o: PrintStream): Unit =
log(s"${new Date().getTime} : $m")(o)
This works, but am I missing something? Why is there confusion inside the body of logTime()? I thought Bruno was implying that the implicit from the caller would be projected into the scope of the method. His example does not add the extra parameter to the log() call. Why does scalac see these as 2? I suppose I assumed that the implicit from the outer method would "hide" the val out. Not so.
If anyone can explain why I see this, it would be appreciated.
Recall that the value of an implicit parameter is determined at the call site. That's why...
Equality.log("log this")
...won't compile unless an implicit value of the appropriate type is brought into scope.
implicit val ps: PrintStream = ...
Equality.log("log this")
The logTime() definition code is a call site for the log() method and, since it is defined within the Equality object, it has the implicit val out value available to it. But it is also the recipient of the implicit o value of the same type that was passed from its call site. Thus the ambiguity. Should the compiler send the implicit out value or the implicit o value to the log() method?
Now, it might seem a bit odd that the received implicit value (from the call site) is both assigned to a local identifier, o, and inserted into the local implicit namespace as well. It turns out that Scala-3 has modified that behavior and your code compiles without error, even without the new given/using syntax. (I assume the implicit out value is passed to the log() method and not the received o value.)

Scala return implicit in the context

I wonder if it's possible to modify an implicit in a context with a function?
With a syntax like this
def modifyImplicit(implicit myImplicit: ImplicitType) : implicit ImplicitType {
myImplicit.setSomthing(something)
myImplicit
}
Because now I must return a type and after the function transform this in a new implicit
if I need to use the function more than once it's became quickly painful.
That's would introduce side-effect (automagically alter the environment without much notice), with it's not "very good".
Instead you can allow some operation to be executed within a managed context, in which you explicitly provide a replacement for the implicit.
implicit def TheDefaultTypeClass: ImplicitType
def withMyContext[T](f: (ImplicitType) => T): T = f(anotherTypeClass)
Then it can be used as following:
val s: String = withMyContext { i =>
val x: ImplicitType = i // Dumb statement just to check type of `i`
// some operations ...
"OK" // result
}
No, it isn't possible. You could write
implicit def modifyImplicit(implicit myImplicit: ImplicitType): ImplicitType = ...
but this won't work the way you want (because for it to ever be called, an implicit of this type must already be available, so either the compiler won't continue looking for an implicit or it will and report conflicting implicits).
Also, having a mutable implicit value seems very likely to lead to bugs.
One possible workaround (in addition to the method proposed by applicius): extract your code into a method and call it with a modified implicit value.
def myMethod(args: ...)(implicit i: ImplicitType) = ...
myMethod(args)(modifyImplicit(implicitly[ImplicitType]))
Yes I now but implicit are mutable because :
```
def modifyImplicit(implicit myImplicit: ImplicitType) {
implicit val myNewImplicit = myImplicit.setSomthing(something)
imASweetMethodWitchUseImplicit
....
}
```
imASweetMethodWitchUseImplicit will use the last implicit setted in the context so we can't "stuck the imutability of the implicit"
I's actually the way i use to made what I whan but I thinks it's a little bit ugly.
I do that for "preparing" the context for other's function so I'm confident because it's just the variable whitch are hide not the call of my function ( witch modify the variables ) you know?
so Alexey I use the same option than you,but I take directly un implicit.
If I call more than one function it's become ugly
```
val result = modifyImplicit()
val result2 = modifyImplicit(result)
implicit val result3 = modifyImplicit(result2)
```
So maybe the solution of applicius can be more beautiful ?

Reify a ValDef from compile to runtime

I want to reify a ValDef into runtime, but i does not work directly. If i encapsulate the ValDef into a Block, everything works perfectly, like in the following example:
case class Container(expr: Expr[Any])
def lift(expr: Any): Container = macro reifyValDef
def reifyValDef(c: Context)(expr: c.Expr[Any]): c.Expr[Container] = {
import c.universe._
expr.tree match {
case Block(List(v: ValDef), _) =>
val asBlock = q"{$v}"
val toRuntime = q"scala.reflect.runtime.universe.reify($asBlock)"
c.Expr[Container](q"Container($toRuntime)")
}
}
lift {
val x: Int = 10
}
If i would use v directly, instead of wrapping it into a block, I get the error:
Error:(10, 11) type mismatch;
found :
required: Any
Note that extends Any, not AnyRef.
Such types can participate in value classes, but instances
cannot appear in singleton types or in reference comparisons.
val x: Int = 10
^
Is it just not working directly with ValDefs or is something wrong with my code?
That's one of the known issues in the reflection API. Definitions are technically not expressions, so you can't e.g. pass them directly as arguments to functions. Wrapping the definition in a block is a correct way of addressing the block.
The error message is of course confusing, but it does make some twisted sense. To signify the fact that a definition by itself doesn't have a type, the tpe field of the corresponding Tree is set to NoType. Then the type of the argument of a macro is checked against Any and the check fails (because NoType is a special type, which isn't compatible with anything), so a standard error message is printed. The awkward printout is an artifact of how the prettyprinter behaves in this weird situation.

Scala - how to go resolve "Value is not a member of Nothing" error

This example code is based on Atmosphere classes, but if someone could give me some insights into what the error means in general, I think I can figure out any Atmosphere-specific solution...
val bc = BroadcasterFactory.getDefault().lookup(_broadcasterId)
bc.broadcast(message)
After the first line, bc should contain a handle to an object whose class definition includes the method broadcast() -- in fact, it contains several overloaded variations. However, the compiler chokes on the second line of code with the following: "value broadcast is not a member of Nothing"
Any ideas/suggestions on what would be causing this?
Thanks.
EDIT: signature for [BroadcasterFactor].lookup :
abstract Broadcaster lookup(Object id)
Note: 1) that is the signature version that I've used in the example, 2) it is the java Inteface signature - whereas the getDefault() hands back an instantiated object that implements that interface.
Solution: force type cast on value:
val bc: Broadcaster = BroadcasterFactory.getDefault().lookup(_broadcasterId)
Nothing is the type name. It's the subtype of all other types. You can't call methods from Nothing itself, you have to specify exact type ((bc: ExactType).broadcast(message)). Nothing has no instances. Method, that returns Nothing will, actually, never return value. It will throw an exception eventually.
Type inference
Definition of lookup:
abstract public <T extends Broadcaster> T lookup(Object id);
in scala this definition looks this way:
def lookup[T <: Broadcaster](Object id): T
There is not specified type parameter in lookup method. In this case compiler will infer this type parameter as the most specific type - Nothing:
scala> def test[T](i: Int): T = ???
test: [T](i: Int)T
scala> lazy val x = test(1)
x: Nothing = <lazy>
scala> lazy val x = test[String](1)
x: String = <lazy>
You could specify type parameter like this:
val bc = BroadcasterFactory.getDefault().lookup[Broadcaster](_broadcasterId)
Draft implementation
In development process lookup can be "implemented" like this:
def lookup(...) = ???
??? returns Nothing.
You should specify either result type of lookup method like this: def lookup(...): <TypeHere> = ... or type of bc: val bc: <TypeHere> =.

Understanding implicit in Scala

I was making my way through the Scala playframework tutorial and I came across this snippet of code which had me puzzled:
def newTask = Action { implicit request =>
taskForm.bindFromRequest.fold(
errors => BadRequest(views.html.index(Task.all(), errors)),
label => {
Task.create(label)
Redirect(routes.Application.tasks())
}
)
}
So I decided to investigate and came across this post.
I still don't get it.
What is the difference between this:
implicit def double2Int(d : Double) : Int = d.toInt
and
def double2IntNonImplicit(d : Double) : Int = d.toInt
other than the obvious fact they have different method names.
When should I use implicit and why?
I'll explain the main use cases of implicits below, but for more detail see the relevant chapter of Programming in Scala.
Implicit parameters
The final parameter list on a method can be marked implicit, which means the values will be taken from the context in which they are called. If there is no implicit value of the right type in scope, it will not compile. Since the implicit value must resolve to a single value and to avoid clashes, it's a good idea to make the type specific to its purpose, e.g. don't require your methods to find an implicit Int!
example:
// probably in a library
class Prefixer(val prefix: String)
def addPrefix(s: String)(implicit p: Prefixer) = p.prefix + s
// then probably in your application
implicit val myImplicitPrefixer = new Prefixer("***")
addPrefix("abc") // returns "***abc"
Implicit conversions
When the compiler finds an expression of the wrong type for the context, it will look for an implicit Function value of a type that will allow it to typecheck. So if an A is required and it finds a B, it will look for an implicit value of type B => A in scope (it also checks some other places like in the B and A companion objects, if they exist). Since defs can be "eta-expanded" into Function objects, an implicit def xyz(arg: B): A will do as well.
So the difference between your methods is that the one marked implicit will be inserted for you by the compiler when a Double is found but an Int is required.
implicit def doubleToInt(d: Double) = d.toInt
val x: Int = 42.0
will work the same as
def doubleToInt(d: Double) = d.toInt
val x: Int = doubleToInt(42.0)
In the second we've inserted the conversion manually; in the first the compiler did the same automatically. The conversion is required because of the type annotation on the left hand side.
Regarding your first snippet from Play:
Actions are explained on this page from the Play documentation (see also API docs). You are using
apply(block: (Request[AnyContent]) ⇒ Result): Action[AnyContent]
on the Action object (which is the companion to the trait of the same name).
So we need to supply a Function as the argument, which can be written as a literal in the form
request => ...
In a function literal, the part before the => is a value declaration, and can be marked implicit if you want, just like in any other val declaration. Here, request doesn't have to be marked implicit for this to type check, but by doing so it will be available as an implicit value for any methods that might need it within the function (and of course, it can be used explicitly as well). In this particular case, this has been done because the bindFromRequest method on the Form class requires an implicit Request argument.
WARNING: contains sarcasm judiciously! YMMV...
Luigi's answer is complete and correct. This one is only to extend it a bit with an example of how you can gloriously overuse implicits, as it happens quite often in Scala projects. Actually so often, you can probably even find it in one of the "Best Practice" guides.
object HelloWorld {
case class Text(content: String)
case class Prefix(text: String)
implicit def String2Text(content: String)(implicit prefix: Prefix) = {
Text(prefix.text + " " + content)
}
def printText(text: Text): Unit = {
println(text.content)
}
def main(args: Array[String]): Unit = {
printText("World!")
}
// Best to hide this line somewhere below a pile of completely unrelated code.
// Better yet, import its package from another distant place.
implicit val prefixLOL = Prefix("Hello")
}
In scala implicit works as:
Converter
Parameter value injector
Extension method
There are some uses of Implicit
Implicitly type conversion : It converts the error producing assignment into intended type
val x :String = "1"
val y:Int = x
String is not the sub type of Int , so error happens in line 2. To resolve the error the compiler will look for such a method in the scope which has implicit keyword and takes a String as argument and returns an Int .
so
implicit def z(a:String):Int = 2
val x :String = "1"
val y:Int = x // compiler will use z here like val y:Int=z(x)
println(y) // result 2 & no error!
Implicitly receiver conversion: We generally by receiver call object's properties, eg. methods or variables . So to call any property by a receiver the property must be the member of that receiver's class/object.
class Mahadi{
val haveCar:String ="BMW"
}
class Johnny{
val haveTv:String = "Sony"
}
val mahadi = new Mahadi
mahadi.haveTv // Error happening
Here mahadi.haveTv will produce an error. Because scala compiler will first look for the haveTv property to mahadi receiver. It will not find. Second it will look for a method in scope having implicit keyword which take Mahadi object as argument and returns Johnny object. But it does not have here. So it will create error. But the following is okay.
class Mahadi{
val haveCar:String ="BMW"
}
class Johnny{
val haveTv:String = "Sony"
}
val mahadi = new Mahadi
implicit def z(a:Mahadi):Johnny = new Johnny
mahadi.haveTv // compiler will use z here like new Johnny().haveTv
println(mahadi.haveTv)// result Sony & no error
Implicitly parameter injection: If we call a method and do not pass its parameter value, it will cause an error. The scala compiler works like this - first will try to pass value, but it will get no direct value for the parameter.
def x(a:Int)= a
x // ERROR happening
Second if the parameter has any implicit keyword it will look for any val in the scope which have the same type of value. If not get it will cause error.
def x(implicit a:Int)= a
x // error happening here
To slove this problem compiler will look for a implicit val having the type of Int because the parameter a has implicit keyword.
def x(implicit a:Int)=a
implicit val z:Int =10
x // compiler will use implicit like this x(z)
println(x) // will result 10 & no error.
Another example:
def l(implicit b:Int)
def x(implicit a:Int)= l(a)
we can also write it like-
def x(implicit a:Int)= l
Because l has a implicit parameter and in scope of method x's body, there is an implicit local variable(parameters are local variables) a which is the parameter of x, so in the body of x method the method-signature l's implicit argument value is filed by the x method's local implicit variable(parameter) a implicitly.
So
def x(implicit a:Int)= l
will be in compiler like this
def x(implicit a:Int)= l(a)
Another example:
def c(implicit k:Int):String = k.toString
def x(a:Int => String):String =a
x{
x => c
}
it will cause error, because c in x{x=>c} needs explicitly-value-passing in argument or implicit val in scope.
So we can make the function literal's parameter explicitly implicit when we call the method x
x{
implicit x => c // the compiler will set the parameter of c like this c(x)
}
This has been used in action method of Play-Framework
in view folder of app the template is declared like
#()(implicit requestHreader:RequestHeader)
in controller action is like
def index = Action{
implicit request =>
Ok(views.html.formpage())
}
if you do not mention request parameter as implicit explicitly then you must have been written-
def index = Action{
request =>
Ok(views.html.formpage()(request))
}
Extension Method
Think, we want to add new method with Integer object. The name of the method will be meterToCm,
> 1 .meterToCm
res0 100
to do this we need to create an implicit class within a object/class/trait . This class can not be a case class.
object Extensions{
implicit class MeterToCm(meter:Int){
def meterToCm={
meter*100
}
}
}
Note the implicit class will only take one constructor parameter.
Now import the implicit class in the scope you are wanting to use
import Extensions._
2.meterToCm // result 200
Why and when you should mark the request parameter as implicit:
Some methods that you will make use of in the body of your action have an implicit parameter list like, for example, Form.scala defines a method:
def bindFromRequest()(implicit request: play.api.mvc.Request[_]): Form[T] = { ... }
You don't necessarily notice this as you would just call myForm.bindFromRequest() You don't have to provide the implicit arguments explicitly. No, you leave the compiler to look for any valid candidate object to pass in every time it comes across a method call that requires an instance of the request. Since you do have a request available, all you need to do is to mark it as implicit.
You explicitly mark it as available for implicit use.
You hint the compiler that it's "OK" to use the request object sent in by the Play framework (that we gave the name "request" but could have used just "r" or "req") wherever required, "on the sly".
myForm.bindFromRequest()
see it? it's not there, but it is there!
It just happens without your having to slot it in manually in every place it's needed (but you can pass it explicitly, if you so wish, no matter if it's marked implicit or not):
myForm.bindFromRequest()(request)
Without marking it as implicit, you would have to do the above. Marking it as implicit you don't have to.
When should you mark the request as implicit? You only really need to if you are making use of methods that declare an implicit parameter list expecting an instance of the Request. But to keep it simple, you could just get into the habit of marking the request implicit always. That way you can just write beautiful terse code.
Also, in the above case there should be only one implicit function whose type is double => Int. Otherwise, the compiler gets confused and won't compile properly.
//this won't compile
implicit def doubleToInt(d: Double) = d.toInt
implicit def doubleToIntSecond(d: Double) = d.toInt
val x: Int = 42.0
I had the exact same question as you had and I think I should share how I started to understand it by a few really simple examples (note that it only covers the common use cases).
There are two common use cases in Scala using implicit.
Using it on a variable
Using it on a function
Examples are as follows
Using it on a variable. As you can see, if the implicit keyword is used in the last parameter list, then the closest variable will be used.
// Here I define a class and initiated an instance of this class
case class Person(val name: String)
val charles: Person = Person("Charles")
// Here I define a function
def greeting(words: String)(implicit person: Person) = person match {
case Person(name: String) if name != "" => s"$name, $words"
case _ => "$words"
}
greeting("Good morning") // Charles, Good moring
val charles: Person = Person("")
greeting("Good morning") // Good moring
Using it on a function. As you can see, if the implicit is used on the function, then the closest type conversion method will be used.
val num = 10 // num: Int (of course)
// Here I define a implicit function
implicit def intToString(num: Int) = s"$num -- I am a String now!"
val num = 10 // num: Int (of course). Nothing happens yet.. Compiler believes you want 10 to be an Int
// Util...
val num: String = 10 // Compiler trust you first, and it thinks you have `implicitly` told it that you had a way to covert the type from Int to String, which the function `intToString` can do!
// So num is now actually "10 -- I am a String now!"
// console will print this -> val num: String = 10 -- I am a String now!
Hope this can help.
A very basic example of Implicits in scala.
Implicit parameters:
val value = 10
implicit val multiplier = 3
def multiply(implicit by: Int) = value * by
val result = multiply // implicit parameter wiil be passed here
println(result) // It will print 30 as a result
Note: Here multiplier will be implicitly passed into the function multiply. Missing parameters to the function call are looked up by type in the current scope meaning that code will not compile if there is no implicit variable of type Int in the scope.
Implicit conversions:
implicit def convert(a: Double): Int = a.toInt
val res = multiply(2.0) // Type conversions with implicit functions
println(res) // It will print 20 as a result
Note: When we call multiply function passing a double value, the compiler will try to find the conversion implicit function in the current scope, which converts Int to Double (As function multiply accept Int parameter). If there is no implicit convert function then the compiler will not compile the code.