I am using cascading 2.0 in scala, and meet a weird compilation issue
This is main parts of the code
val sojSource = createSojSource(optionMap("input"))
val sinkScheme = createSojScheme(true)
val sink = new Hfs(sinkScheme, optionMap("output"), SinkMode.REPLACE)
var pipe = new Pipe("soj")
val each = new Each(pipe, new SojSampleFilter())
val flowDef = new FlowDef().addSource(pipe, sojSource).addTailSink(each, sink)// compile error
And here is the error message for the last line of code
type mismatch; found : cascading.tap.hadoop.Hfs required: cascading.tap.Tap[_, _, _]
Note: org.apache.hadoop.mapred.JobConf <: Any (and cascading.tap.hadoop.Hfs <: cascading.tap.Tap[org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector]), but Java-defined class Tap is invariant in type Config. You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
Note: org.apache.hadoop.mapred.RecordReader <: Any (and cascading.tap.hadoop.Hfs <: cascading.tap.Tap[org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector]), but Java-defined class Tap is invariant in type Input. You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
Note: org.apache.hadoop.mapred.OutputCollector <: Any (and cascading.tap.hadoop.Hfs <: cascading.tap.Tap[org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector]), but Java-defined class Tap is invariant in type Output. You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
Could you, please, tell me how can I fix this and what is the reason?
I found this code in the scalding tree which works around this problem:
// The scala compiler has problems with the generics in Cascading
protected def castHfsTap(tap : Hfs) : Tap[JobConf, RecordReader[_,_], OutputCollector[_,_]] = {
tap.asInstanceOf[Tap[JobConf, RecordReader[_,_], OutputCollector[_,_]]]
}
Related
I worked with TypeTags and noticed very strange scala compiler behaviour
import scala.reflect.runtime.universe.TypeTag
def accept[T](arg : TypeTag[T]) = true
def pass[T](arg : TypeTag[T]) : TypeTag[T] = arg
val ntt = implicitly[TypeTag[Nothing]]
accept(ntt)
pass(ntt)
returns error:
NastyNothing.scala:12: error: type mismatch;
found : reflect.runtime.universe.TypeTag[Nothing]
required: reflect.runtime.universe.TypeTag[T]
Note: Nothing <: T, but trait TypeTag is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: T`. (SLS 3.2.10)
pass(ntt)
^
I could accept TypeTag of Nothing but could not pass it around. Moreover it fails similar with any object parametrized by the Nothing type.
final case class Hold[T]()
def accept[T](arg : Hold[T]) = true
def pass[T](arg : Hold[T]) : Hold[T] = arg
val hn = Hold[Nothing]()
accept(hn)
pass(hn)
return the same error:
NastyNothing.scala:24: error: type mismatch;
found : TestHolder.Hold[Nothing]
required: TestHolder.Hold[T]
Note: Nothing <: T, but class Hold is invariant in type T.
You may wish to define T as +T instead. (SLS 4.5)
pass(hn)
^
What is proper way to handle Nothing-parametrized objects? How should the pass method be redefined to process TypeTag[Nothing] correctly?
Is there any way to make this compile (without explicitly writing types)?
import java.util.{function => juf}
def jfun[A,B](f: A => B): juf.Function[A,B] =
new juf.Function[A,B] { def apply(a: A): B = f(a) }
val l = new java.util.ArrayList[String]
l.stream.map(jfun(_.toInt))
Result:
error: no type parameters for method map: (x$1: java.util.function.Function[_ >: String, _ <: R])java.util.stream.Stream[R] exist so that it can be applied to arguments (java.util.function.Function[String,Int])
--- because ---
argument expression's type is not compatible with formal parameter type;
found : java.util.function.Function[String,Int]
required: java.util.function.Function[_ >: String, _ <: ?0R]
Note: String <: Any, but Java-defined trait Function is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
l.stream.map(jfun(_.toInt))
^
<console>:18: error: type mismatch;
found : java.util.function.Function[String,Int]
required: java.util.function.Function[_ >: String, _ <: R]
l.stream.map(jfun(_.toInt))
^
Is there any better explanation to why this doesn't work than "scala simply can't do it"?
PS. I know about workarounds like adding extension map method on Stream - I nevertheless would like to hear some explanation for this type inference limitation.
On Scala 2.10.4, given the following trait and case classes:
scala> trait Parent
defined trait Parent
scala> case class Girl() extends Parent
defined class Girl
scala> case class Boy() extends Parent
defined class Boy
I'm trying to define a method, f, that produces a Set[Class[A]] where A's type is A <: Parent.
scala> def f[A <: Parent]: Set[Class[A]] = Set[Class[A]](classOf[Boy], classOf[Girl])
<console>:12: error: type mismatch;
found : Class[Boy](classOf[$Boy])
required: Class[A]
def f[A <: Parent]: Set[Class[A]] = Set[Class[A]](classOf[Boy], classOf[Girl])
^
<console>:12: error: type mismatch;
found : Class[Girl](classOf[$Girl])
required: Class[A]
def f[A <: Parent]: Set[Class[A]] = Set[Class[A]](classOf[Boy], classOf[Girl])
But, I can make it work if I use, what I believe is the "wildcard":
scala> def g[A <: Parent]: Set[Class[_]] = Set[Class[_]](classOf[Boy], classOf[Girl])
g: [A <: Parent]=> Set[Class[_]]
And it works:
scala> g
res5: Set[Class[_]] = Set(class Boy, class Girl)
Why did the first approach fail, but the second succeeded? Lastly, is there any risk (to type safety) using Class[_] in the above definition of g?
The problem is that Class[A] is invariant in its type parameter. So Class[Boy] is not a Class[Parent]. The compiler will warn you of this if you set an explicit return type of Set[Class[Parent]].
scala> def f[_ <: Parent]: Set[Class[Parent]] = Set(classOf[Boy], classOf[Girl])
<console>:24: error: type mismatch;
found : Class[Boy](classOf[$Boy])
required: Class[Parent]
Note: Boy <: Parent, but Java-defined class Class is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: Parent`. (SLS 3.2.10)
def f[_ <: Parent]: Set[Class[Parent]] = Set(classOf[Boy], classOf[Girl])
The following method uses existential types, which essentially means you don't care what the type is. _ is a completely unbound, and the type parameter A is superfluous.
def g[A <: Parent]: Set[Class[_]] = Set[Class[_]](classOf[Boy], classOf[Girl])
You might as well write:
def g: Set[Class[_]] = Set[Class[_]](classOf[Boy], classOf[Girl])
Lastly, is there any risk (to type safety) using Class[_] in the above definition of g?
Type safety has pretty much gone out the window at this point, because _ can be anything. Boy, Girl, Parent, Any, Toaster, ...
I think the best you can hope for is to heed the warning and get something like this:
def g: Set[Class[_ <: Parent]] = Set(classOf[Boy], classOf[Girl])
This will at least ensure that you have a Set with elements bounded above by Parent. Exactly what you intend to do with it, I don't know.
I am new to scala and am having an issue in calling a "generic" function in the java NIO library (from scala 2.10.x). Reducing the code to a simple test:
import java.net._
import java.nio.channels.{MembershipKey, DatagramChannel}
object Test {
val channel = DatagramChannel.open(StandardProtocolFamily.INET)
.setOption(StandardSocketOptions.SO_REUSEADDR, true)
...
}
This results in:
Error:(40, 48) type mismatch;
found : java.net.SocketOption[Boolean]
required: java.net.SocketOption[Any]
Note: Boolean <: Any, but Java-defined trait SocketOption is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
channel.setOption(StandardSocketOptions.SO_REUSEADDR, true)
^
I assume there is some way to resolve this without resorting to writing a wrapper in java. Have tried recasting in a variety of ways without success.
Question: how do I resolve the above?
Method setOption is polymorphic
setOption[T](name: SocketOption[T], value: T): DatagramChannel
so when calling
setOption(StandardSocketOptions.SO_REUSEADDR, true)
scala see that types of arguments are slightly different
StandardSocketOptions.SO_REUSEADDR: SocketOption[java.lang.Boolean]
true: scala.Boolean
and compiler tries to narrow T to the most common type between java.lang.Boolean <: AnyRef and scala.Boolean <: AnyVal which is Any
to fix this issue, you need to either provide explicit type for setOption
setOption[java.lang.Boolean](StandardSocketOptions.SO_REUSEADDR, true)
or use type ascription (then T will be inferred correctly)
setOption(StandardSocketOptions.SO_REUSEADDR, true: java.lang.Boolean)
I'm trying to make a converter:
scala> implicit def enumSetToArray[T : ClassTag](enumSet: EnumSet[T]): Array[T] = enumSet.toArray[T](new Array[T](enumSet.size()))
<console>:9: error: type mismatch;
found : Array[T]
required: Array[T with Object]
Note: T >: T with Object, but class Array is invariant in type T.
You may wish to investigate a wildcard type such as `_ >: T with Object`. (SLS 3.2.10)
implicit def enumSetToArray[T : ClassTag](enumSet: EnumSet[T]): Array[T] = enumSet.toArray[T](new Array[T](enumSet.size()))
^
<console>:9: error: type mismatch;
found : Array[T with Object]
required: Array[T]
Note: T with Object <: T, but class Array is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: T`. (SLS 3.2.10)
implicit def enumSetToArray[T : ClassTag](enumSet: EnumSet[T]): Array[T] = enumSet.toArray[T](new Array[T](enumSet.size()))
^
Ideas? My understanding is that I have to use the ClassTag to save the class from erasure so that reflection can instantiate the array, but apparently doing so messes with the variance.
Did you try
implicit def enumSetToArray[T <: Enum[T]](enumSet: EnumSet[T])(implicit ev: ClassTag[T]): Array[T] = enumSet.toArray[T](new Array[T](enumSet.size))