I was trying to create a map that goes from integers to Nd4j arrays (as in INDArrays) using the nd4s library. I'm struggling with this issue:
import org.nd4j.linalg.factory._
scala> Map(0 -> Nd4j.create(2))
<console>:17: error: type mismatch;
found : org.nd4j.linalg.api.ndarray.INDArray
required: Int
Map(0 -> Nd4j.create(2))
^
If I set the key type as string, it works:
scala> Map("0" -> Nd4j.create(2))
res28: scala.collection.immutable.Map[String,org.nd4j.linalg.api.ndarray.INDArray] = Map(0 -> [ 0.00, 0.00])
It works also by inverting key and value type.
I can't understand what is happening.
I found the answer by asking help on the Scala channel on gitter.
The problem is that nd4s does an implicit conversion for the -> operator, leading to this issue.
https://github.com/deeplearning4j/nd4s/blob/master/src/main/scala/org/nd4s/Implicits.scala#L143-L176
Related
I have 2 maps, map1 and map2, both of type Map[String, MyType].
What I wanted to do was, searching for a key in the first map, if not found, search in the second one, and if still not found, use a default value:
map1.getOrElse(name, map2.getOrElse(name, defVal))
However, what I accidently wrote was:
map1.getOrElse(name, map2.get(name, defVal)).
Surprisingly, this didn't cause a compile-time error (and returned null), although i called get with 2 parameters.
As I can see in the IDE (Eclipse), it calls get(x$1 : Any) : MyType of java.util.Map, instead of get(key : String) : Option[MyType] of scala.collection.MapLike
Why doesn't it report an error, when I call map2.get with a second parameter? As far as I can see in the documentation, map.get only takes one parameter? Does it interpret the two parameters as a tuple or something similar (and use this tuple as Any-parameter)?
If relevant, MyType is a class from a referenced Java-project.
I am quiet new in Scala, so if this is trivial or a basic concept that I missed (and everyone programming scala should already know), please tell me what to search for.
EDIT:
As I saw in the comments, that the problem is not reproduceable, I saw, that I hava import scala.collection.javaConversions._, because I get the collections from a Java-Project that I reference. Now it is also reproducable with a short code like this:
import scala.collection.JavaConversions._
object Main {
def main (args : Array[String]) : Unit = {
val map1 = Map("1" -> 2)
val map2 = Map("2" -> 1)
map1.getOrElse("1", map2.get("2", 0))
}
}
Might be argument adaptation from map2.get(name, defVal) to map2.get((name, defVal)).
Edit: Use JavaConverters. JavaConversions is deprecated.
Well... I am not sure how you made this work, it should not be possible.
It is still not doable ( at least with Scala 2.12.1)
scala> val map1 = Map("1" -> "!", "2" -> "#")
map1: scala.collection.immutable.Map[String,String] = Map(1 -> !, 2 -> #)
scala> val map2 = Map("3" -> "#", "4" -> "$")
map2: scala.collection.immutable.Map[String,String] = Map(3 -> #, 4 -> $)
scala> val s = map1.getOrElse("3", map2.get("3", "%"))
<console>:13: error: too many arguments (2) for method get: (key: String)Option[String]
val s = map1.getOrElse("3", map2.get("3", "%"))
^
I have a function as follows:
def foo(sequence: Seq): Unit = {
...
bar.select(sequence);
}
I obtained an error, overloaded method value select with alternatives. I'm new to Scala's syntax, and I don't know the exact term, so I have been searching online without any results.
In particular, Scala appears to be confused as to which of the select method I am calling. From the documentation, there are 2 select methods (Source: https://spark.apache.org/docs/1.5.2/api/scala/index.html#org.apache.spark.sql.DataFrame)
I have tried the following, but perhaps because I am not familiar with the syntax, I was not able to succeed:
bar.select(sequence.asInstanceOf[Seq[Column]]);
How do I tell Scala that I want it to use select(cols: Column*) instead of alternatives? And if it is fine, could the answer include what concept this is - just so I can tag it properly, and learn something new.
Declare your foo method like this and use _*. As shown below
def foo(sequence: Seq[Column]): Unit = {
bar.select(sequence: _*)
}
Explanation select takes var args. When you want to pass Seq to a method which takes var args you have to explicitly say _* So that compiler can accept the Seq as var args.
Scala REPL example
scala> def foo(a: Int*) = 1
foo: (a: Int*)Int
scala> foo(Seq(1, 2))
<console>:13: error: type mismatch;
found : Seq[Int]
required: Int
foo(Seq(1, 2))
^
scala> foo(Seq(1, 2): _*)
res1: Int = 1
notice calling foo(Seq(1, 2) gives a compilation error this error can be fixed if we guide the compiler by giving type explicitly as _*
I am thoroughly confused by this behavior in the Scala REPL:
scala> import java.util.Map
import java.util.Map
scala> import java.util.HashMap
import java.util.HashMap
scala> val jMap:java.util.Map[String,Int]=new HashMap[String,Int]("first"->1,"second" -> 2)
<console>:12: error: type mismatch;
found : (String, Int)
required: Float
val jMap =new HashMap[String,Int]("first"->1,"second" -> 2)
^
<console>:12: error: type mismatch;
found : (String, Int)
required: Float
val jMap=new HashMap[String,Int]("first"->1,"second" -> 2)
^
Can someone help explain what's going on here ?
java.util.HashMap does not provide capability to construct the map by passing a vararg of (K, V), but there is a two-arg constructor accepting initialCapacity: Int and loadFactor: Float, that's why you're getting the compile error about a Float being required.
(updated for scala 2.13+) The idiomatic approach in Scala would be to just use Scala immutable maps (no imports required):
val map = Map("first" -> 1, "second" -> 2).asJava
If your Scala code works with a Java library that returns a java.util.Map, you can convert it into a Scala map explicitly by using scala.jdk.CollectionConverters like so:
import scala.jdk.CollectionConverters._
val javaMap: java.util.Map[String, String] = // ...
val map = javaMap.asScala
// or vice versa if you need to pass it into the Java API
val javaMap = Map("first" -> 1, "second" -> 2).asJava
Note that asScala is just a wrapper to the underlying Java map (so if that one is mutable, the wrapped map will also be mutable) and is an instance of scala.collection.Map. In order to be fully idiomatic and benefit from Scala's immutability guarantees, you might need to add another .toMap at the end that will convert it to scala.collection.immutable.Map (which is the default Map).
If you use scala 2.12 or older, instead of scala.jdk.CollectionConverters._ import scala.collection.JavaConverters._.
In Scala I can build a Map this way:
val map = Map(1 -> "one", 2 -> "two", 3 -> "three")
But what does it do? The arguments should be evaluated, so 1 -> "one" has a value. What's it?
I also noticed this expression returns true:
scala> 1 -> "one" == (1, "one")
res1: Boolean = true
So what's the difference?
It comes from the class ArrowAssoc (http://www.scala-lang.org/api/current/scala/Predef$$ArrowAssoc.html). Look in the object Predef (http://www.scala-lang.org/api/current/scala/Predef$.html) which contains all the pre-imported methods. There you should find the method any2ArrowAssoc[A](x: A): ArrowAssoc[A] which is the implicit conversion allowing the method -> to be called on anything.
To elaborate on this answer, this means that the following code is implicit in your examples:
scala> 'a' -> 1 == (any2ArrowAssoc('a').->(1))
res0: Boolean = true
In short it's just an extension method on Pair (or Tuple2) type. If you take a look at Map type in scala, you'' see that it consist of pairs or tuples of arity of 2. As i know it was defined specially for Map construction to denote key value association key -> value, but you can easily write Map(("a", 1), ("b", 2)) which is equivalent to Map("a" -> 1, "b"-> 2), just looks better. So 1 -> "one" == (1, "one") is equals cause it's the same. In scala, -> is defined as an implicit class in Predef object
From book "Programming in Scala" (http://booksites.artima.com/programming_in_scala_2ed):
The Scala compiler transforms a binary operation expression like 1 -> "Go to island." into (1).->("Go to island.").
Thus, when you say 1 -> "Go to island.", you are actually calling a method named -> on an integer with the value 1, passing in a string with the value "Go to island."
This -> method, which you can invoke on any object in a Scala program, returns a two-element tuple containing the key and value.
The Scala mechanism that allows you to invoke -> on any object, implicit conversion.
The println in the following code works (with or without toInt)
println("retweets : ", e.getOrElse("retweets", 0).toInt)
top10Tweets(""+e.get("text").get, e.getOrElse("retweets", 0).toInt)
But when I pass it as an argument of a function (as above), it does not work. It says "value toInt is not a member of Any"
When I remove toInt, it says,
type mismatch;
[error] found : Any
[error] required: Int
e is a Map, as follows,
def tweetDetails(obj: twitter4j.Status) = {
Map(
"id" -> obj.getUser().getId(),
"screenName" -> obj.getUser().getScreenName(),
"text" -> obj.getText(),
"retweets" -> obj.getRetweetCount(),
"mentions" -> obj.getUserMentionEntities().length)
}
signature of top10Tweets,
def top10Tweets(tweets: String, retweet_c: Int, mention_c: Int) = {
}
edit:
Ok, with the new information I would suggest you to create a case class that holds the data instead of using a Map, this way you will preserve type information. I know it is common to use hashes/maps for that in dynamically typed languages, but in statically typed languages as scala data types are the preferred way.
orig:
As I neither know what e is, nor what signature top10Tweets has, I can only assume. But from your code and the error I assume that e is a Map[String, String] and you are trying to get the string representation of an integer for the key "retweets" and convert it to an Int. As a default value you pass in an Int, so the type inferencer infers type Any, because that is the most common super type of String and Int. However Any does not have a toInt method and thus you get the error.
Map("x" -> "2").getOrElse("x", 4).toInt
<console>:8: error: value toInt is not a member of Any
Map("x" -> "2").getOrElse("x", 4).toInt
Either pass in the default value as String, or convert the value of "retweets" to an Int before, if it exists:
e.get("retweets").map(_.toInt).getOrElse(0)
Anyway a little more information would help to give an accurate answer.
Yes because in Map is "string" -> "string" and You did when getOrElse ( else ) string -> int, thats why its Any.
Map("x" -> 2).getOrElse("x", 4).toInt
works fine or You can:
Map("x" -> "2").getOrElse("x", "4").toInt
The same problem bothers me before you could check below
Map("x" -> 2).getOrElse[Int](x, 4)