I know this question looks silly at first, but please look at the code.
I created a broadcasted map this way :
val rdd = sqlc
.read
.format("jdbc")
.options(Map("url" -> driver, "dbtable" -> clientsTable))
.load()
.select("client_name","client_age")
.map { data => (data.getString(0),data.getInt(1)) }
.collectAsMap()
val clients = sqlc.sparkContext.broadcast(rdd)
The I create a method with value as parameter :
def doSomething(clients: Broadcast[Map[String,Int]]) clients.toString()
But, when I call this method in my code,Scala IDE throws this error :
type mismatch; found : org.apache.spark.broadcast.Broadcast[scala.collection.Map[String,Int]] required: org.apache.spark.broadcast.Broadcast[scala.collection.immutable.Map[String,Int]] Note: scala.collection.Map[String,Int] >: Map[String,Int], but class Broadcast is invariant in type T. You may wish to define T as -T instead. (SLS 4.5)
I can't find what is wrong here, even changing the method signature with a silly scala.collection.immutable.Map doesn't work... The compiler gives me the same error.
FYI : I am using scala 2.10 and scala IDE 4.3.0
Thanks for any help.
It appears that PairRDDFunctions.collectAsMap returns a scala.collection.Map, so your function's signature should match this type, and not the more-specific scala.collection.immutable.Map, which is the default when you just write Map:
def doSomething(clients: Broadcast[scala.collection.Map[String,Int]])
Related
SBT is throwing the following error:
value split is not a member of (String, String)
[error] .filter(arg => arg.split(delimiter).length >= 2)
For the following code block:
implicit def argsToMap(args: Array[String]): Map[String, String] = {
val delimiter = "="
args
.filter(arg => arg.split(delimiter).length >= 2)
.map(arg => arg.split(delimiter)(0) -> arg.split(delimiter)(1))
.toMap
}
Can anyone explain what might be going on here?
Some details:
java version "1.8.0_191"
sbt version 1.2.7
scala version 2.11.8
I've tried both on the command line and also with intellij. I've also tried Java 11 and Scala 2.11.12 to no avail.
I'm not able to replicate this on another machine (different OS, SBT, IntelliJ, etc. though) and I can also write a minimal failing case:
value split is not a member of (String, String)
[error] Array("a", "b").map(x => x.split("y"))
The issue is that the filter method is added to arrays via an implicit.
When you call args.filter(...), args is converted to ArrayOps via the Predef.refArrayOps implicit method.
You are defining a implicit conversion from Array[String] to Map[(String, String)].
This implicit has higher priority than Predef.refArrayOps and is therefore used instead.
So args is converted into a Map[(String, String)]. The filter method of that Map would expect a function of type (String, String) => Boolean as parameter.
I believe what happened is that the implicit method is getting invoked a bit too eagerly. That is, the Tuple2 that's seemingly coming out of nowhere is the result of the implicit function converting each String into a key/value pair. The implicit function was recursively calling itself. I found this out after eventually getting a stack overflow with some other code that was manipulating a collection of Strings.
I asked a longer question, but it seems it's too much code for people to sort through so I've created this question to focus on one smaller, specific problem I'm facing regarding use of macros in Scala.
Consider the following code snippet:
val tpe = weakTypeOf[T]
val companion = tpe.typeSymbol.companionSymbol
val fields = tpe.declarations.collectFirst {
case m: MethodSymbol if m.isPrimaryConstructor => m
}.get.paramss.head
val toMapParams = fields.map { field =>
val name = field.name
val decoded = name.decoded
q"$decoded -> t.$name"
}
Note that fields is just the list of parameters for the primary constructor of a case class in this code. Where I'm confused is the result of the quasiquote q"$decoded -> t.$name". What does this mean exactly? And what type should it have? I'm getting a compile error stating the following:
Multiple markers at this line
- Implicit conversions found: q"$decoded -> t.$name" => Quasiquote(q"$decoded -> t.
$name")
- type mismatch; found : field.NameType required: c.universe.TermName
- type mismatch; found : field.NameType required: c.universe.TermName
Can anyone explain this error? Thanks.
The type of fields is List[Symbol], which means that the type of names of those fields is inconclusive (unknown whether it's a TermName or TypeName). This means that you can't insert such names essentially anywhere in a quasiquote.
A simple fix would be to do val name = field.name.toTermName, explicitly telling the compiler that it's looking at a term name, so that quasiquote knows how to process it.
What is the best way to resolve the compilation error in the example below? Assume that 'm' must be of type GenMap and I do not have control over the arguments of myFun.
import scala.collection.GenMap
object Test {
def myFun(m: Map[Int, String]) = m
val m: GenMap[Int, String] = Map(1 -> "One", 2 -> "two")
//Build error here on m.seq
// Found scala.collection.Map[Int, String]
// Required scala.collection.immutable.Map[Int, String]
val result = myFun(m.seq)
}
EDIT:
I should have been clearer. In my actual use-case I don't have control over myFun, so I have to pass it a Map. The 'm' also arises from another scala component as a GenMap. I need to convert one to another, but there appears to be a conflict between collection.Map and collection.immutable.Map
m.seq.toMap will solve your problem.
According to the signature presented in the API toMap returns a scala.collection.immutable.Map which is said to be required in your error message. scala.collection.Map returned by the seq method is a more general trait which besides being a parent to immutable map is also a parent to the mutable and concurrent map.
I've recently returned to scala after a long hiatus in python and trying to wrap my mind around the type system again. I'm trying to make a very simple web URL dispatcher to get familiar with the language again. So far I have:
trait Executable {
def execute(request: HttpRequest, response: HttpResponse): Future[HttpResponse]
}
class HelloWorldHandler extends Executable {
override def execute(request: HttpRequest, response: HttpResponse) = {
Future.value(response)
}
}
What I think I have here is the scala equivalent of an interface Executable and a class that implements that interface. Now I'd like to create a mapping of URLs to handlers like so:
val mapping: Map[String, _ <: Executable] = {
"/hello" -> new HelloWorldHandler()
}
When I compile this, I get the following error:
type mismatch;
found : (java.lang.String, pollcaster.handlers.HelloWorldHandler)
required: Map[String,pollcaster.Executable]
"/" -> new HelloWorldHandler()
^
I'm not sure where I went wrong in my understanding here but would appreciate any help in understanding how can I put a bunch of different classes that have the trait of Executable into a map object?
TIA
Scala doesn't have a map literal like that. The following should work, though:
val mapping: Map[String, _ <: Executable] = Map(
"/hello" -> new HelloWorldHandler(),
"/something" -> new SomeOtherHandler()
)
This is just using the Map object's apply method to create a new map.
You can create a Map from a generic Seq of Tuples2 but there is no automatic conversion from Tuple2 to Map. That makes perfectly sense: why would you create a map with a single key and a single value?
I have the following code:
private lazy val keys: List[String] = obj.getKeys().asScala.toList
obj.getKeys returns a java.util.Iterator<java.lang.String>
Calling asScala, via JavaConverers (which is imported) according to the docs..
java.util.Iterator <==> scala.collection.Iterator
scala.collection.Iterator defines
def toList: List[A]
So based on this I believed this should work, however here is the compilation error:
[scalac] <file>.scala:11: error: type mismatch;
[scalac] found : List[?0] where type ?0
[scalac] required: List[String]
[scalac] private lazy val keys : List[String] = obj.getKeys().asScala.toList
[scalac] one error found
I understand the type parameter or the java Iterator is a Java String, and that I am trying to create a list of Scala strings, but (perhaps naively) thought that there would be an implicit conversion.
You don't need to call asScala, it is an implicit conversion:
import scala.collection.JavaConversions._
val javaList = new java.util.LinkedList[String]() // as an example
val scalaList = javaList.iterator.toList
If you really don't have the type parameter of the iterator, just cast it to the correct type:
javaList.iterator.asInstanceOf[java.util.Iterator[String]].toList
EDIT: Some people prefer not to use the implicit conversions in JavaConversions, but use the asScala/asJava decorators in JavaConverters to make the conversions more explicit.
That would work if obj.getKeys() was a java.util.Iterator<String>. I suppose it is not.
If obj.getKeys() is just java.util.Iterator in raw form, not java.util.Iterator<String>, not even java.util.Iterator<?>, this is something scala tend to dislikes, but anyway, there is no way scala will type your expression as List[String] if it has no guarantee obj.getKeys() contains String.
If you know your iterator is on Strings, but the type does not say so, you may cast :
obj.getKeys().asInstanceOf[java.util.Iterator[String]]
(then go on with .asScala.toList)
Note that, just as in java and because of type erasure, that cast will not be checked (you will get a warning). If you want to check immediately that you have Strings, you may rather do
obj.getKeys().map(_.asInstanceOf[String])
which will check the type of each element while you iterate to build the list
I dislike the other answers. Hell, I dislike anything that suggests using asInstanceOf unless there's no alternative. In this case, there is. If you do this:
private lazy val keys : List[String] = obj.getKeys().asScala.collect {
case s: String => s
}.toList
You turn the Iterator[_] into a Iterator[String] safely and efficiently.
Note that starting Scala 2.13, package scala.jdk.CollectionConverters replaces deprecated packages scala.collection.JavaConverters/JavaConversions when it comes to implicit conversions between Java and Scala collections:
import scala.jdk.CollectionConverters._
// val javaIterator: java.util.Iterator[String] = java.util.Arrays.asList("a", "b").iterator
javaIterator.asScala
// Iterator[String] = <iterator>
As of scala 2.12.8 one could use
import scala.collection.JavaConverters._
asScalaIterator(java.util.Iterator variable).toSeq