SBT is throwing the following error:
value split is not a member of (String, String)
[error] .filter(arg => arg.split(delimiter).length >= 2)
For the following code block:
implicit def argsToMap(args: Array[String]): Map[String, String] = {
val delimiter = "="
args
.filter(arg => arg.split(delimiter).length >= 2)
.map(arg => arg.split(delimiter)(0) -> arg.split(delimiter)(1))
.toMap
}
Can anyone explain what might be going on here?
Some details:
java version "1.8.0_191"
sbt version 1.2.7
scala version 2.11.8
I've tried both on the command line and also with intellij. I've also tried Java 11 and Scala 2.11.12 to no avail.
I'm not able to replicate this on another machine (different OS, SBT, IntelliJ, etc. though) and I can also write a minimal failing case:
value split is not a member of (String, String)
[error] Array("a", "b").map(x => x.split("y"))
The issue is that the filter method is added to arrays via an implicit.
When you call args.filter(...), args is converted to ArrayOps via the Predef.refArrayOps implicit method.
You are defining a implicit conversion from Array[String] to Map[(String, String)].
This implicit has higher priority than Predef.refArrayOps and is therefore used instead.
So args is converted into a Map[(String, String)]. The filter method of that Map would expect a function of type (String, String) => Boolean as parameter.
I believe what happened is that the implicit method is getting invoked a bit too eagerly. That is, the Tuple2 that's seemingly coming out of nowhere is the result of the implicit function converting each String into a key/value pair. The implicit function was recursively calling itself. I found this out after eventually getting a stack overflow with some other code that was manipulating a collection of Strings.
Related
Scala compiler behaves weirdly with boxing/unboxing in tuples as parameters.
Consider the code:
scala> class Test { def test(p: (Int, String)) = println(p) }
defined class Test
scala> classOf[Test].getMethods()(0)
res2: java.lang.reflect.Method = public void Test.test(scala.Tuple2)
scala> classOf[Test].getMethods()(0).getGenericParameterTypes
res3: Array[java.lang.reflect.Type] = Array(scala.Tuple2<java.lang.Object, java.lang.String>)
scala> // WTF?????? ^^^^^^^^^^^^^^^^
Thus, I'm getting Object instead of Integer. I assume this is somehow related to tuple parameter being #specialized, but cannot wrap my head around how to avoid/fix this.
The problem it causes - it is impossible to reconstruct method parameter via reflection on method signature (e.g. while parsing json).
Even if there's a way to get the right type with scala-reflect it doesn't help much, cause there are a lot of Java libraries around (like Jersey) that use just Java reflection.
UPD:
OK, putting an Integer (instead of Int) into Tuple works ok. But why isn't it done automatically?
I have the snippet below from a code where rdd is RDD[(String,Vector)], but unfortunately my Scala compiler is complaining with the error Type mismatch, expected: RDD[(String,Vector)], actual: RDD[(String,Vector)] where I call flagVectorOutlier(rdd, predictedRDD)
def someFunction() {
testData.foreachRDD( rdd => {
val vectorsRDD = rdd.map( pair => pair._2 )
val predictedRDD = model.latestModel().predict( vectorsRDD )
flagVectorOutlier( rdd, predictedRDD )
} )
ssc.start()
ssc.awaitTermination()
}
def flagVectorOutlier(testVectors: RDD[(String, Vector)], predicts: RDD[Int]): Unit = {
}
Considering the actual and expected types are the same, what is the wrong point here? How could I solve this issue?
I've got this kind of error before.
It happened when I was using a java library which would use java.util.List, while my own code was using scala.collection.immutable.List and I mixed them up.
You can have two classes with the same name coexist in code easily, but the error message will not display fully-qualified names, so I would get Type mismatch, expected: List[Integer], actual: List[Integer] which seems puzzling. The solution is just to fully qualify the types of your input parameters or use typeDefs to distinguish them.
Bonus: I've also had a similar problem with tuples. For instance, when a method can either expect tuples or direct params:
def f(param: (Integer, String)) versus def f(param1: Integer, param2: String)
When calling the method with the wrong parameters (say, 2 params instead of a tuple), on a cursory inspection the error message can seem to show two identical types. This gets worse if you have several nested tuples (thus too many parentheses but the same types).
I have the following code:
private lazy val keys: List[String] = obj.getKeys().asScala.toList
obj.getKeys returns a java.util.Iterator<java.lang.String>
Calling asScala, via JavaConverers (which is imported) according to the docs..
java.util.Iterator <==> scala.collection.Iterator
scala.collection.Iterator defines
def toList: List[A]
So based on this I believed this should work, however here is the compilation error:
[scalac] <file>.scala:11: error: type mismatch;
[scalac] found : List[?0] where type ?0
[scalac] required: List[String]
[scalac] private lazy val keys : List[String] = obj.getKeys().asScala.toList
[scalac] one error found
I understand the type parameter or the java Iterator is a Java String, and that I am trying to create a list of Scala strings, but (perhaps naively) thought that there would be an implicit conversion.
You don't need to call asScala, it is an implicit conversion:
import scala.collection.JavaConversions._
val javaList = new java.util.LinkedList[String]() // as an example
val scalaList = javaList.iterator.toList
If you really don't have the type parameter of the iterator, just cast it to the correct type:
javaList.iterator.asInstanceOf[java.util.Iterator[String]].toList
EDIT: Some people prefer not to use the implicit conversions in JavaConversions, but use the asScala/asJava decorators in JavaConverters to make the conversions more explicit.
That would work if obj.getKeys() was a java.util.Iterator<String>. I suppose it is not.
If obj.getKeys() is just java.util.Iterator in raw form, not java.util.Iterator<String>, not even java.util.Iterator<?>, this is something scala tend to dislikes, but anyway, there is no way scala will type your expression as List[String] if it has no guarantee obj.getKeys() contains String.
If you know your iterator is on Strings, but the type does not say so, you may cast :
obj.getKeys().asInstanceOf[java.util.Iterator[String]]
(then go on with .asScala.toList)
Note that, just as in java and because of type erasure, that cast will not be checked (you will get a warning). If you want to check immediately that you have Strings, you may rather do
obj.getKeys().map(_.asInstanceOf[String])
which will check the type of each element while you iterate to build the list
I dislike the other answers. Hell, I dislike anything that suggests using asInstanceOf unless there's no alternative. In this case, there is. If you do this:
private lazy val keys : List[String] = obj.getKeys().asScala.collect {
case s: String => s
}.toList
You turn the Iterator[_] into a Iterator[String] safely and efficiently.
Note that starting Scala 2.13, package scala.jdk.CollectionConverters replaces deprecated packages scala.collection.JavaConverters/JavaConversions when it comes to implicit conversions between Java and Scala collections:
import scala.jdk.CollectionConverters._
// val javaIterator: java.util.Iterator[String] = java.util.Arrays.asList("a", "b").iterator
javaIterator.asScala
// Iterator[String] = <iterator>
As of scala 2.12.8 one could use
import scala.collection.JavaConverters._
asScalaIterator(java.util.Iterator variable).toSeq
How can I convert a java 1.4 Collection to a Scala Seq?
I am trying to pass a java-collection to a scala method:
import scala.collection.JavaConversions._
// list is a 1.4 java.util.ArrayList
// repository.getDir is Java legacy code
val list = repository.getDir(...)
perform(list)
def perform(entries: List[SVNDirEntry]) = ...
I always receive this error:
type mismatch; found : java.util.Collection[?0] where type ?0 required: List
[SVNDirEntry]
So I guess I have to create the parameterized Sequence myself as Scala is only able to create an unparameterized Iterable?
First you have to make sure that list has the type java.util.List[SVNDirEntry]. To do this, use a cast:
list.asInstanceOf[java.util.List[SVNDirEntry]]
After that, the implicit conversion will be resolved for you if you import the JavaConversions object. An implicit conversion to a Scala sequence exists in the JavaConversions object. See the following example with a list of strings being passed to a method that expects a Scala sequence:
scala> val jvlist: java.util.List[_] = new java.util.ArrayList[String]
jvlist: java.util.List[_] = []
scala> jvlist.asInstanceOf[java.util.List[String]]
res0: java.util.List[String] = []
scala> import collection.JavaConversions._
import collection.JavaConversions._
scala> def perform(scalaseq: Seq[String]) = println("ok")
perform: (scalaseq: scala.collection.Seq[String])Unit
scala> perform(res0)
ok
These conversions do not copy the elements - they simply construct a wrapper around a Java collection. Both versions point to the same underlying data. Thus, there is no implicit conversion in JavaConversions to immutable Scala lists from mutable Java lists, because that would enable changing the contents of a Scala collection that is guaranteed to be immutable.
In short - prefer Seq[...] to List[...] when defining parameters for methods if you can live with a less specific interface (as in perform above). Or write your own function that does the conversion by copying the elements.
You have to cast the legacy collection down to the target type. Something along the lines of:
perform(list.asInstanceOf[List[SVNDirEntry]])
The source code of map for Array is:
override def map[B](f: A => B): Array[B] = throw new Error()
But the following works:
val name : Array[String]= new Array(1)
name(0)="Oscar"
val x = name.map { ( s: String ) => s.toUpperCase }
// returns: x: Array[java.lang.String] = Array(OSCAR)
Generally, when you see throw new Error() in the source code of a library class, it represents a point where the compiler is intervening and implementing the method by bridging to a facility of the platform (remember this could be Java or .NET).
The Array SID explains how arrays used to be treated in Scala 2.7.x, and how they have changed in 2.8. The compiler used to magically convert the object to a BoxedArray if you called map.
In 2.8, integration of Arrays into the Scala collections framework is largely handled with use of normal langauges features -- implicit conversions from Array[T] to WrappedArray[T] or ArraySeq[T], depending on the context, and implicit parameters of type Manifest[T] to support creation of arrays of a generic type T. Array indexing, length and update still appear as throw new Error(). Array#map no longer exists, instead you find this on WrappedArray and ArraySeq as a regular method.
UPDATE
If you're interested to know this compiler magic is defined, take a look at Scala 2.8 incarnation of Cleanup.scala.
Looks like it's just dummy code, as Scala arrays are really Java ones.