I'm porting some java code across and have the following
val overnightChanges: java.util.Hashtable[String, Double] = ...
When I try
if (null != overnightChanges.get(...))
I get the following warning
warning: comparing values of types Null and Double using `!=' will always yield true
Primitive and reference types are much less different in scala than they are in java, and so the convention is that name starts with an uppercase for all of them. Double is scala.Double which is the primitive java double, not the reference java.lang.Double.
When you need "a double or no value" in scala, you would use Option[Double] most of the time. Option has strong library support, and the type system will not let you ignore that there might be no value. However, when you need to interact closely with java, as in your example, your table does contain java.lang.Double and you should say it so.
val a = new java.util.HashMap[String, java.lang.Double]
If java.lang.Double starts to appear everywhere in your code, you can alias to JDouble, either by importing
import java.lang.{Double => JDouble}
or by defining
type JDouble = java.lang.Double
There are implicit conversions between scala.Double and java.lang.Double, so interaction should be reasonably smooth. However, java.lang.Double should probably be confined to the scala/java interaction layer, it would be confusing to have it go deep into scala code.
In Scala Double are primitives and thus cannot be null. That's annoying when using directly java maps, because when a key is not defined, you get the default primitive value, (here 0.0):
scala> val a = new java.util.Hashtable[String,Double]()
a: java.util.Hashtable[String,Double] = {}
scala> a.get("Foo")
res9: Double = 0.0
If the value is a object like String or List, your code should work as expected.
So, to solve the problem, you can:
Use contains in an outer if condition.
Use one of the Scala maps (many conversions are defined in scala.collection.JavaConversions)
Use Scala "options", also known as "maybe" in Haskell:
http://blog.danielwellman.com/2008/03/using-scalas-op.html
Related
I would like to do dynamic casting for a scala variable, the casting type is stored in a different variable or in the database or provided by the user.
I am new to Scala and have mainly done coding on python. Here we are trying to take Any type input and query the type of the variable as per the type saved in DB eg.: "String/Int" and user-defined classes and cast them before any future processing.
in python:
eval("str('123')")
in scala I have tried
var a = 123.05
a.asInstanceOf['Int']
gives me error
error: identifier expected but string literal found.
And I want the code to be something as follows:
var a = 123.05
var b = "Int"
a.asInstanceOf[b]
gives me error
error: not found: type b
Ok so I have to be fairly exhaustive here because the stuff you're trying to do is tricky due to the static vs dynamic typing difference of scala and python.
First what you're doing in Python is not type casting but rather a conversion.
str(12)
in python takes the integer value 12 and converts it to a (UTF-8 ? dunno in python) string. The same thing in scala would be
12.toString
typecasting in the meantime is basically a pinky promise to the compilers typechecker that you know more than it and it should just believe you. It also should be avoided like the pest because you basically drop all the safety that static typechecking gives you. However there are certain cases where it is unavoidable
in a bit more fundamental terms. lets say you have the following scala snippet
sealed trait Foo
final case class Bar(i:Int) extends Foo
final case class Baz(s:String) extends Foo
val f:Foo = Bar(2)
//won't work because the compiler thinks f is of type Foo due to the type ascription bove
println(f.i)
//will work
println(f.asInstanceOf[Bar].i)
the Type Foo can be either Bar or Baz (because of the sealed and final, this is called an ADT) but we specifically told the compiler to treat f as a Foo which means we forgot which specific type it was. this is the case where you could use typecasting, note that we don't convert but rather tell the compiler that it is actually a Bar
Now this all happens during compile time which means you can't cast according to a runtime value like this
var a = 123.05
var b = "Int"
a.asInstanceOf[b]
Now as I understand you have some sort of stringly typed input and need to convert it according to some schema. The scalafiddle below has an example how you could do this: https://scalafiddle.io/sf/i97WZlA/0
However note that this makes use of some fairly advanced concepts in scala to ensure the types line up.
This will also work https://scalafiddle.io/sf/i97WZlA/1 but note that we lose all the type information requiring us to do a typecast if we want to do anything meaningful with out
EDIT/ADDENDUM: I thought I should also do an example on how to consume such values and make the schema dynamic.
https://scalafiddle.io/sf/i97WZlA/2
As a final note be warned that this is getting close to the limits what the compiler can do and necessitates a lot of boxing and unboxing to carry along the type information (in SchemaValue) also it's not stacksafe and has lackluster errorhandling. this solution would require some serious engineering to make viable but it should get the idea across
If you have a String value, then you can use toInt or toDouble to parse that string:
val s = "123.05"
val i = s.toInt
val d = s.toDouble
If you have an Any value then it is best to use match to convert it:
val a: Any = ...
a match {
case i: Int => // It is an Int
case d: Double => // It is a Double
case _ => // It is a different type
}
If all else fails you can explicitly pick a type, but this is not good practice:
val i = a.asInstanceOf[Int]
Any decent database framework will give you the ability to read and write values of specific types, so this should not be a problem with the right library.
Scala appears to have different semantics for regular value assignment versus assignment during an extraction. This has created some very subtle runtime bugs for me as my codebase has migrated over time.
To illustrate:
case class Foo(s: String)
def anyRef(): AnyRef = { ... }
val Foo(x) = anyRef() // compiles but throws exception if anyRef() is not a Foo
val f: Foo = anyRef() // does not compile
I don't see why the two val assignment lines would be imbalanced with regard to compile/runtime behavior.
Some I am curious: Is there a reason for this behavior? Is it an undesirable artifact of the way the extraction is implemented?
(tested in Scala 2.11.7)
Yes, there is.
In the case of an extractor you are specifying a pattern that you expect to match at this position. This is translated to a call of an extractor method from Any to Option[String] and this method can be called with the value of type AnyRef you provide. You are just asserting, that you do indeed get a result and not "None".
In the other line you are using a type annotation, i.e. you are specifying the type of "f" explicitly. But then you are assigning an incompatible value.
Of course the compiler could add implicit type casts but making type casts so easy would not really suit a language like Scala.
You should also keep in mind that extractors have nothing to do with types. In the Pattern Foo(x) the name Foo has nothing to do with the type Foo. It is just the Name of an extractor method.
Using patterns in this way is of course a quite dynamic feature, but I think it is intentional.
val finalRDD = joinedRDD.map(x => {
val d1 = x._2._1
val d2 = x._2._2
(x._1, d1 + d2)
})
In the above code, joinedRDD has type RDD[(Row, (Double, Double))] (according to IntelliJ) while Scala compiler says d1 & d2 are AnyVal.
For time being, I cast d1 & d2 as Double using asInstanceOf but next time it says
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Double
Is it Scala compiler issue or IntelliJ issue which shows me wrong inferred types. Any insights?
Seems good to me :-S
Type inference is far from omniscient. Sometimes you need to specify the types explicitly. In my experience, this is especially true when the result type can be anything. Some things to try:
My preferred option since you are not touching the key: joinedRDD.mapValues(x => x._1 + x._2)
Add some type information: val d1: Double = x._2._1. With some luck, at least the compiler might be more explicit.
Define your function separately, assigning types to the parameters, and use if inside: map(myFunc)
Also, I've seen some differences between IntelliJ Scala Plugin and the actual Scala compiler. Given the errors you are getting and the fact that AnyVal is the common parent class for both Int and Double, there is a good chance you don't have doubles to begin with (and the compiler is trying to find a shared parent). Do double check that you are getting the type you mention by putting it explicitly. It is very possible that your type confusion occurs before this line.
Good luck!
Well, I tried in IntelliJ IDEA 14 and the type inference is correct, recognizing d1 and d2 as Double (this was expected). Nonetheless, I usually avoid the type-aware highlighting feature of IDEA since many times it goes crazy and reports fake results.
As a side note, since you are not changing the key of your RDD, consider using mapValues instead of map (this provides clarity, as well as performance since it would take advantage of the partitioner of the input RDD and reuse it in the output RDD).
The following Scala code works correctly:
val str1 = "hallo"
val str2 = "huhu"
val zipped: IndexedSeq[(Char, Char)] = str1.zip(str2)
However if I import the implicit method
implicit def stringToNode(str: String): xml.Node = new xml.Text(str)
then the Scala (2.10) compiler shows an error: value zip is not a member of String
It seems that the presence of stringToNode somehow blocks the implicit conversion of str1 and str2 to WrappedString. Why? And is there a way to modify stringToNode such that zip works but stringToNode is still used when I call a function that requires a Node argument with a String?
You have ambiguous implicits here. Both StringOps and xml.Node have the zip-method, therefore the implicit conversion is ambiguous and cannot be resolved. I don't know why it doesn't give a better error message.
Here are some links to back it up:
http://www.scala-lang.org/api/current/index.html#scala.collection.immutable.StringOps
and
http://www.scala-lang.org/api/current/index.html#scala.xml.Node
edit: it was StringOps, not WrappedString, changed the links :) Have a look at Predef: http://www.scala-lang.org/api/current/index.html#scala.Predef$
to see predefined implicits in Scala.
I would avoid using implicits in this case. You want 2 different implicit conversions which both provide a method of the same name (zip). I don't think this is possible. Also, if you import xml.Text, you can convert with just Text(str) which should be concise enough for anyone. If you must have this implicit conversion to xml.Node, I would pack the implicit def into an object and then import it only in the places where you need it to make your code readable and to, possibly, avoid conflicts where you also need to zip strings. But basically, I would very much avoid using implicits just to make convenient conversions.
Like #Felix wrote, it is generally a bad idea to define implicit conversions between similar data types, like the one you used. Doing that weakens type system, leads to ambiguities like you encountered and may produce extremely unclear ("magic") code which is very hard to analyze and debug.
Implicit conversions in Scala are mostly used to define lightweight, short-lived wrappers in order to enrich API of wrapped type. Implicit conversion that converts String into WrappedString falls into that category.
Twitter's Effective Scala has a section about this issue.
Although the situation of conversion from Doubles to BigDecimals has improved a bit compared to Java
scala> new java.math.BigDecimal(0.2)
res0: java.math.BigDecimal = 0.20000000000000001110223024625156...
scala> BigDecimal(0.2)
res1: scala.math.BigDecimal = 0.2
and things like
val numbers: List[BigDecimal] = List(1.2, 3.2, 0.7, 0.8, 1.1)
work really well, wouldn't it be reasonable to have an implicit conversion like
implicit def String2BigDecimal(s: String) = BigDecimal(s)
available by default which can convert Strings to BigDecimals like this?
val numbers: List[BigDecimal] = List("1.2", "3.2", "0.7", "0.8", "1.1")
Or am I missing something and Scala resolved all "problems" of Java with using the BigDecimal constructor with a floating point value instead of a String, and BigDecimal(String) is basically not needed anymore in Scala?
This was thought of, but apparently rolled back because it created ambiguous conversions. See this thread on the scala-list.
The thread is old, but as far as I can see, string2Bigdecimal is still not defined as an implicit.
If you still want to have a local string2BigDecimal implicit for your personal use:
the rules for implicit scope can be found in the specification, §7.2,
to resolve ambiguities in favor of your string2BigDecimal, you should define it using subclassing, see this paper
(§6.5) for an example, and this one (§ Avoiding Ambiguities) for an explanation.
Because a BigDecimal can always be created from a Double or a Float, but not always from a String. In general, it is a good idea that, where something has this "property" to use an explicit implicit. For example, this would be nice:
"1.23".toBigDecimal