I'm just in the process of upgrading from Scala 2.10.x to 2.11.2 and I'm receiving the following warning with the following code:
override def validateKey(key: String): Either[InvalidKeyError, Unit] =
keys.contains(key) match {
case true => Right()
case false => Left(InvalidKeyError(context, key))
}
Adaptation of argument list by inserting () has been deprecated: this
is unlikely to be what you want. signature: Right.apply[A, B](b: B):
scala.util.Right[A,B] given arguments: after adaptation:
Right((): Unit)
I am able to solve this by changing the "true" case statement to:
case true => Right(()) //() is a shortcut to a Unit instance
Is this the proper way to address this warning?
Edit: perhaps a "why we have to do this now" type answer would be appropriate, my cursory investigation seems to indicate that Scala inserting "Unit" when it thinks it needs to causes other problems
Automatic Unit inference has been deprecated in scala 2.11, and the reason behind this is that it can lead to confusing behavior, especially for people learning the language.
Here's an example
class Foo[T](value: T)
val x = new Foo
This should not compile, right? You are calling the constructor with no arguments, where one is required. Surprisingly, until scala 2.10.4 this compiles just fine, with no errors or warnings.
And that's because the compiler inferred a Unit argument, so it actually replaced your code with
val x = new Foo[Unit](()) // Foo[Unit]
As the newly introduced warning message says, this is unlikely to be what you want.
Another famous example is this
scala> List(1,2,3).toSet()
// res1: Boolean = false
calling toSet() should be a compile-time error, since toSet does not take arguments, but the compiler desperately tries to make it compile, ultimately interpreting the code as
scala> List(1,2,3).toSet.apply(())
which means: test whether () belongs to the set. Since it's not the case, you get a false!
So, starting from scala 2.11, you have to be explicit if you want to pass () (aka Unit) as an argument. That's why you have to write:
Right(())
instead of
Right()
examples taken from Simplifying Scala — The Past, Present and Future by Simon Ochsenreither.
Perhaps it should be Right(()). Have you tried that?
My explanation is that since the Right.apply is polymorphic it can take all kind of parameters, doing Right() means passing in a Unit and the compiler simply advise you that maybe that's not what you want, he doesn't know that this is what you actually want.
If you see your deprecate message it states:
... after adaptation: Right((): Unit)
Which means that the compiler has automatically decided that you are passing in a Unit, since this is kinda like void he doesn't really likes it, specifically passing in a Unit like () explicitly tells the compiler that you do want a Unit there. Anyway seems a new deprecation form scala 2.11, I can't reproduce this on 2.10.4.
val finalRDD = joinedRDD.map(x => {
val d1 = x._2._1
val d2 = x._2._2
(x._1, d1 + d2)
})
In the above code, joinedRDD has type RDD[(Row, (Double, Double))] (according to IntelliJ) while Scala compiler says d1 & d2 are AnyVal.
For time being, I cast d1 & d2 as Double using asInstanceOf but next time it says
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Double
Is it Scala compiler issue or IntelliJ issue which shows me wrong inferred types. Any insights?
Seems good to me :-S
Type inference is far from omniscient. Sometimes you need to specify the types explicitly. In my experience, this is especially true when the result type can be anything. Some things to try:
My preferred option since you are not touching the key: joinedRDD.mapValues(x => x._1 + x._2)
Add some type information: val d1: Double = x._2._1. With some luck, at least the compiler might be more explicit.
Define your function separately, assigning types to the parameters, and use if inside: map(myFunc)
Also, I've seen some differences between IntelliJ Scala Plugin and the actual Scala compiler. Given the errors you are getting and the fact that AnyVal is the common parent class for both Int and Double, there is a good chance you don't have doubles to begin with (and the compiler is trying to find a shared parent). Do double check that you are getting the type you mention by putting it explicitly. It is very possible that your type confusion occurs before this line.
Good luck!
Well, I tried in IntelliJ IDEA 14 and the type inference is correct, recognizing d1 and d2 as Double (this was expected). Nonetheless, I usually avoid the type-aware highlighting feature of IDEA since many times it goes crazy and reports fake results.
As a side note, since you are not changing the key of your RDD, consider using mapValues instead of map (this provides clarity, as well as performance since it would take advantage of the partitioner of the input RDD and reuse it in the output RDD).
I've been learning Scala and decided to play with JSON parsing using json4s. I decided to use XPath syntax for deserializing and came across this strange bit of syntax I've never seen before.
val json = JsonMethods.parse("""{"meaningOfLife": 42}""")
val JInt(x) = json\"meaningOfLife"
The part confusing me is this bit right here
val JInt(x) = ...
I can't wrap my mind around what's happening there, I don't even know how to search this syntax or what it's called. Can anyone help me out? Scala is an amazing language with a lot of neat features that I'm not used to in other languages like C++ and Java.
Edit
To clarify, I'm confused because x isn't defined, but it's somehow being passed into a function or constructor then being assigned to the result of json\"meaningOfLife" which returns a JValue.
Edit 2
After some research and playing around, I figured out that this has something to do with case classes. I was able to run the following code.
case class MyCaseClass (x: Int)
val MyCaseClass(x) = new MyCaseClass(5)
println(x, x.getClass) // prints (5,int)
Which, after looking at some of the code, gives me a good understanding at what's happening.
val MyCaseClass(x) = MyCaseClass(5)
Is extracting (for lack of a better term) the Int value 5 from the instantiated MyCaseClass and storing that into x, meaning x will be of type Int.
In the code for json4s a JInt is a JValue which the \ operator returns. So the JInt(x) is taking out a BigInt (stored in the class JInt) and putting that into the value x from what I gather.
But I still have a question. What is this process called? Is there any documentation on it?
It's called "irrefutable pattern matching" and it's essentially equivalent to this bit of code:
val json = JsonMethods.parse("""{"meaningOfLife": 42}""")
val x = json match {
case JInt(xMatched) => xMatched
}
In other words, any case class or any extractor that fits the pattern of the declaration's left-hand-side can be used in this way.
Addendum:
The "irrefutable" means that a MatchError will be thrown if the pattern cannot be satisfied.
I've been working with Scala for a while now and have written a 10,000+ line program with it, but I'm still confused by some of the inner workings. I came to Scala from Python after already having intimate familiarity with Java, C and Lisp, but even so it's been slow going, and a huge problem is the frustrating difficulty I've often found when trying to investigate the inner workings of objects/types/classes/etc. using the Scala REPL as compared with Python. In Python you can investigate any object foo (type, object in a global variable, built-in function, etc.) using foo to see what the thing evaluates to, type(foo) to show its type, dir(foo) to tell you the methods you can call on it, and help(foo) to get the built-in documentation. You can even do things like help("re") to find out the documentation on the package named re (which holds regular-expression objects and methods), even though there isn't an object associated with it.
In Scala, you can try and read the documentation online, go look up the source code to the library, etc., but this can often be very difficult for things where you don't know where or even what they are (and it's often a big chunk to bite off, given the voluminous type hierarchy) -- stuff is floating around in various places (package scala, Predef, various implicit conversions, symbols like :: that are nearly impossible to Google). The REPL should be the way to explore directly, but in reality, things are far more mysterious. Say that I've seen a reference to foo somewhere, but I have no idea what it is. There's apparently no such thing as a "guide to systematically investigating Scala thingies with the REPL", but the following is what I've pieced together after a great deal of trial and error:
If foo is a value (which presumably includes things stored in variables plus companion objects and other Scala objects), you can evaluate foo directly. This ought to tell you the type and value of the result. Sometimes the result is helpful, sometimes not.
If foo is a value, you can use :type foo to get its type. (Not necessarily enlightening.) If you use this on a function call, you get the type of the return value, without calling the function.
If foo is a value, you can use foo.getClass to get its class. (Often more enlightening than the previous, but how does an object's class differ from its type?)
For a class foo, you can use classOf[foo], although it's not obvious what the result means.
Theoretically, you can use :javap foo to disassemble a class -- which should be the most useful of all, but fails entirely and uniformly for me.
Sometimes you have to piece things together from error messages.
Example of failure using :javap:
scala> :javap List
Failed: Could not find class bytes for 'List'
Example of enlightening error message:
scala> assert
<console>:8: error: ambiguous reference to overloaded definition,
both method assert in object Predef of type (assertion: Boolean, message: => Any)Unit
and method assert in object Predef of type (assertion: Boolean)Unit
match expected type ?
assert
^
OK, now let's try a simple example.
scala> 5
res63: Int = 5
scala> :type 5
Int
scala> 5.getClass
res64: java.lang.Class[Int] = int
Simple enough ...
Now, let's try some real cases, where it's not so obvious:
scala> Predef
res65: type = scala.Predef$#3cd41115
scala> :type Predef
type
scala> Predef.getClass
res66: java.lang.Class[_ <: object Predef] = class scala.Predef$
What does this mean? Why is the type of Predef simply type, whereas the class is scala.Predef$? I gather that the $ is the way that companion objects are shoehorned into Java ... but Scala docs on Google tell me that Predef is object Predef extends LowPriorityImplicits -- how can I deduce this from the REPL? And how can I look into what's in it?
OK, let's try another confusing thing:
scala> `::`
res77: collection.immutable.::.type = ::
scala> :type `::`
collection.immutable.::.type
scala> `::`.getClass
res79: java.lang.Class[_ <: object scala.collection.immutable.::] = class scala.collection.immutable.$colon$colon$
scala> classOf[`::`]
<console>:8: error: type :: takes type parameters
classOf[`::`]
^
scala> classOf[`::`[Int]]
res81: java.lang.Class[::[Int]] = class scala.collection.immutable.$colon$colon
OK, this left me hopelessly confused, and eventually I had to go read the source code to make sense of this all.
So, my questions are:
What's the recommended best way from the true Scala experts of using the REPL to make sense of Scala objects, classes, methods, etc., or at least investigate them as best as can be done from the REPL?
How do I get :javap working from the REPL for built-in stuff? (Shouldn't it work by default?)
Thanks for any enlightenment.
You mentioned an important point which Scala lacks a bit: the documentation.
The REPL is a fantastic tool, but it is not as fantastic at it can be. There are too much missing features and features which can be improved - some of them are mentioned in your post. Scaladoc is a nice tool, too, but is far away to be perfect. Furthermore lots of code in the API is not yet or too less documented and code examples are often missing. The IDEs are full ob bugs and compared to the possibilities Java IDEs show us they look like some kindergarten toys.
Nevertheless there is a gigantic difference of Scalas current tools compared to the tools available as I started to learn Scala 2-3 years ago. At that time IDEs compiled permanently some trash in the background, the compiler crashed every few minutes and some documentation was absolutely nonexistent. Frequently I got rage attacks and wished death and corruption to Scala authors.
And now? I do not have any of these rage attacks any more. Because the tools we currently have are great although the are not perfect!
There is docs.scala-lang.org, which summarizes a lot of great documentation. There are Tutorials, Cheat-sheets, Glossaries, Guides and a lot of more great stuff. Another great tools is Scalex, which can find even the weirdest operator one can think of. It is Scalas Hoogle and even though it is not yet as good as his great ideal, it is very useful.
Great improvements are coming with Scala2.10 in form of Scalas own Reflection library:
// needs Scala2.10M4
scala> import scala.reflect.runtime.{universe => u}
import scala.reflect.runtime.{universe=>u}
scala> val t = u.typeOf[List[_]]
t: reflect.runtime.universe.Type = List[Any]
scala> t.declarations
res10: Iterable[reflect.runtime.universe.Symbol] = SynchronizedOps(constructor List, method companion, method isEmpty, method head, method tail, method ::, method :::, method reverse_:::, method mapConserve, method ++, method +:, method toList, method take, method drop, method slice, method takeRight, method splitAt, method takeWhile, method dropWhile, method span, method reverse, method stringPrefix, method toStream, method removeDuplicates)
Documentation for the new Reflection library is still missing, but in progress. It allows one to use scalac in an easy way inside of the REPL:
scala> u reify { List(1,2,3) map (_+1) }
res14: reflect.runtime.universe.Expr[List[Int]] = Expr[List[Int]](immutable.this.List.apply(1, 2, 3).map(((x$1) => x$1.$plus(1)))(immutable.this.List.canBuildFrom))
scala> import scala.tools.reflect.ToolBox
import scala.tools.reflect.ToolBox
scala> import scala.reflect.runtime.{currentMirror => m}
import scala.reflect.runtime.{currentMirror=>m}
scala> val tb = m.mkToolBox()
tb: scala.tools.reflect.ToolBox[reflect.runtime.universe.type] = scala.tools.reflect.ToolBoxFactory$ToolBoxImpl#32f7fa37
scala> tb.parseExpr("List(1,2,3) map (_+1)")
res16: tb.u.Tree = List(1, 2, 3).map(((x$1) => x$1.$plus(1)))
scala> tb.runExpr(res16)
res18: Any = List(2, 3, 4)
This is even greater when we want to know how Scala code is translated internally. Formerly wen need to type scala -Xprint:typer -e "List(1,2,3) map (_+1)"
to get the internally representation. Furthermore some small improvements found there way to the new release, for example:
scala> :type Predef
scala.Predef.type
Scaladoc will gain some type-hierarchy graph (click on type-hierarchy).
With Macros it is possible now, to improve error messages in a great way. There is a library called expecty, which does this:
// copied from GitHub page
import org.expecty.Expecty
case class Person(name: String = "Fred", age: Int = 42) {
def say(words: String*) = words.mkString(" ")
}
val person = Person()
val expect = new Expecty()
// Passing expectations
expect {
person.name == "Fred"
person.age * 2 == 84
person.say("Hi", "from", "Expecty!") == "Hi from Expecty!"
}
// Failing expectation
val word1 = "ping"
val word2 = "pong"
expect {
person.say(word1, word2) == "pong pong"
}
/*
Output:
java.lang.AssertionError:
person.say(word1, word2) == "pong pong"
| | | | |
| | ping pong false
| ping pong
Person(Fred,42)
*/
There is a tool which allows one to find libraries hosted on GitHub, called ls.implicit.ly.
The IDEs now have some semantic highlighting, to show if a member is a object/type/method/whatever. The semantic highlighting feature of ScalaIDE.
The javap feature of the REPL is only a call to the native javap, therefore it is not a very featue-rich tool. You have to fully qualify the name of a module:
scala> :javap scala.collection.immutable.List
Compiled from "List.scala"
public abstract class scala.collection.immutable.List extends scala.collection.AbstractSeq implements scala.collection.immutable.LinearSeq,scala.Product,scala.collection.LinearSeqOptimized{
...
Some time ago I have written a summary of how Scala code is compiled to Bytecode, which offers a lot of things to know.
And the best: This is all done in the last few months!
So, how to use all of these things inside of the REPL? Well, it is not possible ... not yet. ;)
But I can tell you that one day we will have such a REPL. A REPL which shows us documentation if we want to see it. A REPL which let us communicate with it (maybe like lambdabot). A REPL which let us do cool things we still cannot imagine. I don't know when this will be the case, but I know that a lot of stuff was done in the last years and I know even greater stuff will be done in the next years.
Javap works, but you are pointing it to scala.Predef.List, which is a type, not a class. Point it instead to scala.collection.immutable.List.
Now, for the most part just entering a value and seeing what the result's type is is enough. Using :type can be helpful sometimes. I find that use getClass is a really bad way of going about it, though.
Also, you are sometimes mixing types and values. For example, here you refer to the object :::
scala> `::`.getClass res79: java.lang.Class[_ <: object
scala.collection.immutable.::] = class
scala.collection.immutable.$colon$colon$
And here you refer to the class :::
scala> classOf[`::`[Int]] res81: java.lang.Class[::[Int]] = class
scala.collection.immutable.$colon$colon
Objects and classes are not the same thing, and, in fact, there's a common pattern of objects and classes by the same name, with a specific name for their relationship: companions.
Instead of dir, just use tab completion:
scala> "abc".
+ asInstanceOf charAt codePointAt codePointBefore codePointCount
compareTo compareToIgnoreCase concat contains contentEquals endsWith
equalsIgnoreCase getBytes getChars indexOf intern isEmpty
isInstanceOf lastIndexOf length matches offsetByCodePoints regionMatches
replace replaceAll replaceFirst split startsWith subSequence
substring toCharArray toLowerCase toString toUpperCase trim
scala> "abc".compareTo
compareTo compareToIgnoreCase
scala> "abc".compareTo
def compareTo(String): Int
If you enter the power mode, you'll get way more information, but that's hardly for beginners. The above shows types, methods, and method signatures. Javap will decompile stuff, though that requires you to have a good handle on bytecode.
There's other stuff in there -- be sure to look up :help, and see what's available.
Docs are only available through the scaladoc API. Keep it open on the browser, and use its search capability to quickly find classes and methods. Also, note that, as opposed to Java, you don't need to navigate through the inheritance list to get the description of the method.
And they do search perfectly fine for symbols. I suspect you haven't spent much time on scaladoc because other doc tools out there just aren't up to it. Javadoc comes to mind -- it's awful browsing through packages and classes.
If you have specific questions Stack Overflow style, use Symbol Hound to search with symbols.
Use the nightly Scaladocs: they'll diverge from whatever version you are using, but they'll always be the most complete. Besides, right now they are far better in many respects: you can use TAB to alternate between frames, with auto-focus on the search boxes, you can use arrows to navigate on the left frame after filtering, and ENTER to have the selected element appear on the right frame. They have the list of implicit methods, and have class diagrams.
I've made do with a far less powerful REPL, and a far poorer Scaladoc -- they do work, together. Granted, I skipped to trunk (now HEAD) just to get my hands on tab-completion.
Note that scala 2.11.8 New tab-completion in the Scala REPL can facilitate the type exploration/discovery.
It now includes:
CamelCase completion:
try:
(l: List[Int]).rroTAB,
it expands to:
(l: List[Int]).reduceRightOption
Find members by typing any CamelCased part of the name:
try:
classOf[String].typTAB,
to get getAnnotationsByType, getComponentType and others
Complete bean getters without typing get:
try:
(d: java.util.Date).dayTAB
Press TAB twice to see the method signature:
try:
List(1,2,3).partTAB,
which completes to:
List(1,2,3).partition;
press TAB again to display:
def partition(p: Int => Boolean): (List[Int], List[Int])
You need to pass fully qualified class name to javap.
First take it using classOf:
scala> classOf[List[_]]
res2: java.lang.Class[List[_]] = class scala.collection.immutable.List
Then use javap (doesn't work from repl for me: ":javap unavailable on this platform.") so example is from a command line, in repl, I believe, you don't need to specify classpath:
d:\bin\scala\scala-2.9.1-1\lib>javap -classpath scala-library.jar "scala.collection.immutable.List"
But I doubt this will help you. Probably you're trying to use techniques you used to use in dynamic languages. I extremely rarely use repl in scala (while use it often in javascript). An IDE and sources are my all.
I want to come out a way to define a new method in some existing class in scala.
For example, I think the asInstanceOf[T] method has too long a name, I want to replace it with as[T].
A straight forward approach can be:
class WrappedAny(val a: Any) {
def as[T] = a.asInstanceOf[T]
}
implicit def wrappingAny(a: Any): WrappedAny = new WrappedAny(a)
Is there a more natural way with less code?
Also, a strange thing happens when I try this:
scala> class A
defined class A
scala> implicit def toA(x: Any): A = x
toA: (x: Any)A
scala> toA(1)
And the console hang. It seems that toA(Any) should not pass the type checking phase, and it can't when it's not implicit. And putting all the code into a external source code can produce the same problem. How did this happen? Is it a bug of the compiler(version 2.8.0)?
There's nothing technically wrong with your approach to pimping Any, although I think it's generally ill-advised. Likewise, there's a reason asInstanceOf and isInstanceOf are so verbosely named; it's to discourage you from using them! There's almost certainly a better, statically type-safe way to do whatever you're trying to do.
Regarding the example which causes your console to hang: the declared type of toA is Any => A, yet you've defined its result as x, which has type Any, not A. How can this possibly compile? Well, remember that when an apparent type error occurs, the compiler looks around for any available implicit conversions to resolve the problem. In this case, it needs an implicit conversion Any => A... and finds one: toA! So the reason toA type checks is because the compiler is implicitly redefining it as:
implicit def toA(x: Any): A = toA(x)
... which of course results in infinite recursion when you try to use it.
In your second example you are passing Any to a function that must return A. However it never returns A but the same Any you passed in. The compiler then tries to apply the implicit conversion which in turn does not return an A but Any, and so on.
If you define toA as not being implicit you get:
scala> def toA(x: Any): A = x
<console>:6: error: type mismatch;
found : Any
required: A
def toA(x: Any): A = x
^
As it happens, this has been discussed on Scala lists before. The pimp my class pattern is indeed a bit verbose for what it does, and, perhaps, there might be a way to clean the syntax without introducing new keywords.
The bit about new keywords is that one of Scala goals is to make the language scalable through libraries, instead of turning the language into a giant quilt of ideas that passed someone's criteria for "useful enough to add to the language" and, at the same time, making other ideas impossible because they weren't deemed useful and/or common enough.
Anyway, nothing so far has come up, and I haven't heard that there is any work in progress towards that goal. You are welcome to join the community through its mailing lists and contribute to its development.