I have the following code for building a cache using google collections:
val cache = new MapMaker().softValues().expiration(30,
TimeUnit.DAYS).makeComputingMap(
new com.google.common.base.Function[String,Int] {
def apply(key:String):Int ={
1
}
})
And I am getting the following error message:
error: type mismatch;
found : java.lang.Object with
com.google.common.base.Function[java.lang.String,Int]{ ... }
required: com.google.common.base.Function[?, ?]
new com.google.common.base.Function[String,Int] {
^
I am wondering why the types don't match ?
The actual code is:
import com.google.common.collect.MapMaker
trait DataCache[V] {
private val cache = new MapMaker().softValues().makeComputingMap(
new com.google.common.base.Function[String,V] {
def apply(key:String):V = null.asInstanceOf[V]
})
def get(key:String):V = cache.get(key)
}
Kind regards,
Ali
PS - I am using google-collections v1
You need to supply type parameters to the final method call. You are going through the raw type interface and scala cannot reconstruct the type information.
val cache = new MapMaker().softValues().expiration(30,
TimeUnit.DAYS).makeComputingMap[String, Int](
new com.google.common.base.Function[String,Int] {
def apply(key:String):Int ={
1
}
})
Does the following works?
new com.google.common.base.Function[_,_] {
If that doesn't work, you may wish to keep the declaration as it is right now, and then add a : com.google.common.base.Function[_, _] after it, like this:
val cache = new MapMaker().softValues().expiration(30,
TimeUnit.DAYS).makeComputingMap(
new com.google.common.base.Function[String,Int] {
def apply(key:String):Int ={
1
}
}: com.google.common.base.Function[_, _])
I have heard that some Google stuff use raw types, which are rather hard to integrate well with Scala. And, in fact, should be banished back to hell, where they came from, but that's just imho.
Also, if you could compile that with -explaintypes, we may get a better notion of what is failing.
Related
I hope to process various generic data sources accessed by Kafka, so I developed the following code:
def accessKafkaSource[T: ClassTag](sEnv: StreamExecutionEnvironment): DataStream[T] = {
val kafkaSource: KafkaSource[T] = KafkaSource.builder()
.setBootstrapServers("")
.setGroupId("")
.setTopics("test")
.setStartingOffsets(OffsetsInitializer.committedOffsets(OffsetResetStrategy.LATEST))
.setValueOnlyDeserializer(new AbstractDeserializationSchema[T]() {
override def deserialize(msg: Array[Byte]): T = {
// JSONUtil.toBean(StrUtil.str(msg, StandardCharsets.UTF_8), classOf[T])
JSONUtil.toBean(StrUtil.str(msg, StandardCharsets.UTF_8), classTag[T].runtimeClass)
}
})
.build()
Since the commented out code will get an error: class type required but t found, I modified the code, but caused a new problem: type mismatch; found : _$ 1 where type _$ 1 required: T。 How should my needs be realized?
As AminMal notes, runtimeClass is not guaranteed to return the class object of T, just what it erases to in the runtime. AnyVals in particular will break this.
If everything you wish to deserialize is an AnyRef (this is likely the case), you can often safely cast the result of runtimeClass:
def kindaSafeClass[T <: AnyRef : ClassTag]: Class[T] = classTag[T].runtimeClass.asInstanceOf[Class[T]]
The situation this would be unsafe is when generics are involved (erasure...), as can be seen by
val clazz = kindaSafeClass[List[String]]
val lst = List(1)
val cast =
if (clazz.isInstance(lst)) {
println(s"$lst is an instance of $clazz")
clazz.cast(lst)
} else ???
println(cast)
println(cast.head.isEmpty)
which will print List(1) is an instance of class scala.collection.immutable.List, then List(1), and then blow up with a ClassCastException when we try to cast 1 to a String.
But if your T will always be an AnyRef and you can be sure that it's not generic, you can
// Note: T must not be generic (e.g. List[String])
def accessKafkaSource[T <: AnyRef : ClassTag](sEnv: StreamExecutionEnvironment): DataStream[T] =
// as before until...
JSONUtils.toBean(StrUtil.str(msg, StandardCharsets.UTF_8), kindaSafeClass[T])
// as before...
That's because runtimeClass returns a Class[_], not a Class[T]. This kind of approach would make perfect sense in Java, like:
JSONUtils.toBean(whateverString, MyClass.class); // and so on
In Scala, (of course there are unsafe approaches to make this work) but if you're using some JSON library (like Play Json, circe, etc,.), you can do this:
// method signature would look something like this
def accessKafkaSource[T : Reads](sEnv: StreamExecutionEnvironment): DataStream[T] = {
// just going to write the deserialization part:
override def deserialize(msg: Array[Byte]): T = {
Json.parse(msg).as[T] // asOpt is safer, not going to get into exception handling and other stuff
}
}
This behavior is also applicable to other json libraries in scala. Or if you have other kind of documents like xml, expect an implicit function from Array[Byte] => DocumentType (Like JsValue, String, Xml, anything), and another one DocumentType => T. Because this function accessKafkaSource should not be responsible to figure out how data should be deserialized/serialized.
Error while passing arguments to methods. I have an object builddeequ_rules and calling methods using Scala reflection.
def build(rules: List[Map[String, Any]]): Check = {
for (constraint <- rules) {
val name = constraint("name")
val args = constraint("args")
val hiObj = builddeequ_rules
val mtd = hiObj.getClass.getMethod(name.toString,args.getClass)
mtd.invoke(hiObj,args)
}
import com.amazon.deequ.checks.{Check, CheckLevel}
object builddeequ_rules {
var checks = Check(CheckLevel.Warning, "Data unit test")
def isComplete(args: Any) {
val arg = args.asInstanceOf[Map[String,Any]]
val columnName = arg("column").toString
checks = checks.isComplete(columnName)
}
def isUnique(args: Any) {
val arg = args.asInstanceOf[Map[String,Any]]
val columnName = arg("column").toString
checks = checks.isUnique(columnName)
}
def isPositive(args: Any) {
val arg = args.asInstanceOf[Map[String,Any]]
val columnName = arg("column").toString
checks = checks.isPositive(columnName)
}
I am getting below error. Need help!
Error: type mismatch;
found : Any
required: Object
mtd.invoke(hiObj,args)
java.lang.Object is more or less scala.AnyRef. scala.Any is (simplyfying) a superset of objects and primitives. So the compiler is warning you, that you are trying to pass something that could potentially be primitive (Any) as java.lang.Object.
On bytecode level Any will quite often be just Object, sure, but Scala's type system make the distinction between things that are "natively" Objects and things that could involve autoboxing to make them Objects, and that's the error you see.
So the solution here would be to have this object annotated as AnyRef or even better, as java.lang.Object to clearly show that you want to use it for something Java/JVM-specific.
We are building some sync functionality using two-way json requests and this algorithm. All good and we have it running in prototype mode. Now I am trying to genericise the code, as we will be synching for several tables in the app. It would be cool to be able to define a class as "extends Synchable" and get the additional attributes and sync processing methods with a few specialisations/overrides. I have got this far:
abstract class Synchable [T<:Synchable[T]] (val ruid: String, val lastSyncTime: String, val isDeleted:Int) {
def contentEquals(Target: T): Boolean
def updateWith(target: T)
def insert
def selectSince(clientLastSyncTime: String): List[T]
def findByRuid(ruid: String): Option[T]
implicit val validator: Reads[T]
def process(clientLastSyncTime: String, updateRowList: List[JsObject]) = {
for (syncRow <- updateRowList) {
val validatedSyncRow = syncRow.validate[Synchable]
validatedSyncRow.fold(
valid = { result => // valid row
findByRuid(result.ruid) match { //- do we know about it?
case Some(knownRow) => knownRow.updateWith(result)
case None => result.insert
}
}... invalid, etc
I am new to Scala and know I am probably missing things - WIP!
Any pointers or suggestions on this approach would be much appreciated.
Some quick ones:
Those _ parameters you pass in and then immediately assign to vals: why not do it in one hit? e.g.
abstract class Synchable( val ruid: String = "", val lastSyncTime: String = "", val isDeleted: Int = 0) {
which saves you a line and is clearer in intent as well I think.
I'm not sure about your defaulting of Strings to "" - unless there's a good reason (and there often is), I think using something like ruid:Option[String] = None is more explicit and lets you do all sorts of nice monad-y things like fold, map, flatMap etc.
Looking pretty cool otherwise - the only other thing you might want to do is strengthen the typing with a bit of this.type magic so you'll prevent incorrect usage at compile-time. With your current abstract class, nothing prevents me from doing:
class SynchableCat extends Synchable { ... }
class SynchableDog extends Synchable { ... }
val cat = new SynchableCat
val dog = new SynchableDog
cat.updateWith(dog) // This won't end well
But if you just change your abstract method signatures to things like this:
def updateWith(target: this.type)
Then the change ripples down through the subclasses, narrowing down the types, and the compiler will omit a (relatively clear) error if I try the above update operation.
I'm trying to use Scala 2.10 reflection to find the most derived type of a method argument. For example, consider this program:
import reflect.runtime.universe._
object ReflectionTest {
def checkType[A : TypeTag](item: A) {
println("typeOf[A]: " + typeOf[A])
}
def main(args: Array[String]) {
val a = Array(1, "Hello")
for (item <- a) checkType(item)
}
}
Here a has type Array[Any] so each item being sent to checkType has type Any. As a result, checkType outputs
typeOf[A]: Any
typeOf[A]: Any
This makes sense to me since the TypeTag is generated by the compiler at the point of the call (where all it knows about the type is that it is Any). What I want, however, is to determine the actual type of each item. I'd like output something along the lines of
Int
String
I have looked over the documentation here
http://docs.scala-lang.org/overviews/reflection/overview.html
but the samples don't seem to cover this case and I find the discussion there of Environments, Universes, and Mirrors difficult to penetrate. It seems like what I'm trying to do should be fairly simple but perhaps I'm approaching it completely wrong.
Most obvious solution would be to use the class:
def checkType[A](item: A) {
println("typeOf[A]: " + item.getClass)
}
But if you want to work with Type, then some additional work is needed:
def checkType[A](item: A) {
val mirror = runtimeMirror(this.getClass.getClassLoader)
println("typeOf[A]: " + mirror.classSymbol(item.getClass).toType)
}
I've recently returned to scala after a long hiatus in python and trying to wrap my mind around the type system again. I'm trying to make a very simple web URL dispatcher to get familiar with the language again. So far I have:
trait Executable {
def execute(request: HttpRequest, response: HttpResponse): Future[HttpResponse]
}
class HelloWorldHandler extends Executable {
override def execute(request: HttpRequest, response: HttpResponse) = {
Future.value(response)
}
}
What I think I have here is the scala equivalent of an interface Executable and a class that implements that interface. Now I'd like to create a mapping of URLs to handlers like so:
val mapping: Map[String, _ <: Executable] = {
"/hello" -> new HelloWorldHandler()
}
When I compile this, I get the following error:
type mismatch;
found : (java.lang.String, pollcaster.handlers.HelloWorldHandler)
required: Map[String,pollcaster.Executable]
"/" -> new HelloWorldHandler()
^
I'm not sure where I went wrong in my understanding here but would appreciate any help in understanding how can I put a bunch of different classes that have the trait of Executable into a map object?
TIA
Scala doesn't have a map literal like that. The following should work, though:
val mapping: Map[String, _ <: Executable] = Map(
"/hello" -> new HelloWorldHandler(),
"/something" -> new SomeOtherHandler()
)
This is just using the Map object's apply method to create a new map.
You can create a Map from a generic Seq of Tuples2 but there is no automatic conversion from Tuple2 to Map. That makes perfectly sense: why would you create a map with a single key and a single value?