I have a method (scala 2.12) that does look like the following.
The goal is to pass to the method readValue from objectMapper (jackson) a string and a class that the string needs to be casted, which in this case is an Array[T].
T can be two different case classes and therefore that is the reason of why I try to parametrize it.
private def fromSeqToCastedSeq[T](files: Seq[File]): Seq[T] = {
files flatMap (file => {
val maps = objectMapper.readValue(file, classOf[Map[String, Any]])
val combinedString = objectMapper.writeValueAsString(maps.get("sqlDefinitions"))
val o = objectMapper.readValue(combinedString, classOf[Array[T]])
o})
Currently this does not compile with a scala.MatchError because it is not able to cast it at runtime.
Could someone help me understand if what I'm trying to achieve is possible?
Thanks.
As answered in Discord, you should be able to do this:
import scala.reflect.ClassTag
private def fromSeqToCastedSeq[T](files: Seq[File])(implicit ct: ClassTag[T]): Seq[T] = {
val arrayTClass = ct.wrap.runtimeClass.asInstanceOf[Class[Array[T]]]
files.flatMap { file =>
val maps = objectMapper.readValue(file, classOf[Map[String, Any]])
val combinedString = objectMapper.writeValueAsString(maps.get("sqlDefinitions"))
objectMapper.readValue(combinedString, arrayTClass)
}
}
Now, no idea if this will crash at runtime, is highly probably given this piece of code is extremely unsafe and unidiomatic.
Related
The following code snippet is a short scala macro bundle definition from a thoughtworks project:
private[SelfType] final class Macros(val c: whitebox.Context) {
import c.universe._
def apply[A: WeakTypeTag]: Tree = {
val a = weakTypeOf[A]
val selfTypes: List[Type] = {
val selfTypeBuilder = List.newBuilder[Type]
def buildSelfTypes(t: Type): Unit = {
val dealiased = t.dealias
dealiased match {
case RefinedType(superTypes, refinedScope) =>
superTypes.foreach(buildSelfTypes)
case typeRef: TypeRef =>
val symbol = dealiased.typeSymbol
if (symbol.isClass) {
selfTypeBuilder += symbol.asClass.selfType.asSeenFrom(dealiased, symbol)
}
case _ =>
}
}
buildSelfTypes(a)
selfTypeBuilder.result()
}
val out = selfTypes match {
case Nil =>
definitions.AnyTpe
case _ =>
internal.refinedType(selfTypes, c.internal.enclosingOwner)
}
q"_root_.com.thoughtworks.feature.SelfType.make[$a, $out]"
}
}
(courtesy of https://github.com/ThoughtWorksInc/feature.scala/blob/4d19cc19016d85f26925895f43f618e1b7552d09/SelfType/src/main/scala/com/thoughtworks/feature/SelfType.scala)
The last line as a quasiquote seems to contain a lot of boilerplate text:
q"_root_.com.thoughtworks.feature.SelfType.make[$a, $out]"
Assuming that this macro bundle is defined inside a trait as part of a family polymorphism design pattern: there is no deterministic q"_root_.com.thoughtworks.feature.SelfType.make[$a, $out]", it has to be derived from an object variable vvv of the macro bundle when it is being compiled. How do I use this variable to make the quasiquote shorter and more adaptive?
There may be multiple methods to achieve this (e.g. for each implementation, define a Liftable for SelfType object). But that's even more boilerplate. I'm looking for the shortest solution. Ideally, something like this:
val sym = Term(vvv)
q"$sym.make[$a, $out]"
If you have a static reference (e.g. the type/companion is imported), you can do:
q"${symbolOf[SelfType.type]}.make[$a, $out]"
You can also use symbolOf[A].companion if you have A: WeakTypeTag but no info about its companion. That might not work if the compiler doesn't consider the object A a companion to class A
Found my first solution:
val name = SelfTypes.getClass.getCanonicalName.stripSuffix("$")
val tree = c.parse(name)
q"$tree.make[$a, $out]"
Not sure if it is the most efficient or idiomatic solution, I'll let the question hanging there for a while
I have a situation where I'm trying to use implicit resolution on a singleton type. This works perfectly fine if I know that singleton type at compile time:
object Main {
type SS = String with Singleton
trait Entry[S <: SS] {
type out
val value: out
}
implicit val e1 = new Entry["S"] {
type out = Int
val value = 3
}
implicit val e2 = new Entry["T"] {
type out = String
val value = "ABC"
}
def resolve[X <: SS](s: X)(implicit x: Entry[X]): x.value.type = {
x.value
}
def main(args: Array[String]): Unit = {
resolve("S") //implicit found! No problem
}
}
However, if I don't know this type at compile time, then I run into issues.
def main(args: Array[String]): Unit = {
val string = StdIn.readLine()
resolve(string) //Can't find implicit because it doesn't know the singleton type at runtime.
}
Is there anyway I can get around this? Maybe some method that takes a String and returns the singleton type of that string?
def getSingletonType[T <: SS](string: String): T = ???
Then maybe I could do
def main(args: Array[String]): Unit = {
val string = StdIn.readLine()
resolve(getSingletonType(string))
}
Or is this just not possible? Maybe you can only do this sort of thing if you know all of the information at compile-time?
If you knew about all possible implementations of Entry in compile time - which would be possible only if it was sealed - then you could use a macro to create a map/partial function String -> Entry[_].
Since this is open to extending, I'm afraid at best some runtime reflection would have to scan the whole classpath to find all possible implementations.
But even then you would have to embed this String literal somehow into each implementations because JVM bytecode knows nothing about mappings between singleton types and implementations - only Scala compiler does. And then use that to find if among all implementations there is one (and exactly one) that matches your value - in case of implicits if there are two of them at once in the same scope compilation would fail, but you can have more than one implementation as long as the don't appear together in the same scope. Runtime reflection would be global so it wouldn't be able to avoid conflicts.
So no, no good solution for making this compile-time dispatch dynamic. You could create such dispatch yourself by e.g. writing a Map[String, Entry[_]] yourself and using get function to handle missing pices.
Normally implicits are resolved at compile time. But val string = StdIn.readLine() becomes known at runtime only. Principally, you can postpone implicit resolution till runtime but you'll be able to apply the results of such resolution at runtime only, not at compile time (static types etc.)
object Entry {
implicit val e1 = ...
implicit val e2 = ...
}
import scala.reflect.runtime.universe._
import scala.reflect.runtime
import scala.tools.reflect.ToolBox
val toolbox = ToolBox(runtime.currentMirror).mkToolBox()
def resolve(s: String): Any = {
val typ = appliedType(
typeOf[Entry[_]].typeConstructor,
internal.constantType(Constant(s))
)
val instanceTree = toolbox.inferImplicitValue(typ, silent = false)
val instance = toolbox.eval(toolbox.untypecheck(instanceTree)).asInstanceOf[Entry[_]]
instance.value
}
resolve("S") // 3
val string = StdIn.readLine()
resolve(string)
// 3 if you enter S
// ABC if you enter T
// "scala.tools.reflect.ToolBoxError: implicit search has failed" otherwise
Please notice that I put implicits into the companion object of type class in order to make them available in the implicit scope and therefore in the toolbox scope. Otherwise the code should be modified slightly:
object EntryImplicits {
implicit val e1 = ...
implicit val e2 = ...
}
// val instanceTree = toolbox.inferImplicitValue(typ, silent = false)
// should be replaced with
val instanceTree =
q"""
import path.to.EntryImplicits._
implicitly[$typ]
"""
In your code import path.to.EntryImplicits._ is import Main._.
Load Dataset from Dynamically generated Case Class
I would like to save a Type or TypeTag in a val for later use. At this time, I am having to specify a type in several locations in a block of code. I do not need to parameterize the code because only one type will be used. This is more of a curiosity than a necessity.
I tried using typeOf, classOf, getClass, and several other forms of accessing the class and type. The solution is likely simple but my knowledge of Scala typing or type references is missing this concept.
object Example extends App {
import scala.reflect.runtime.universe._
object TestClass { val str = "..." }
case class TestClass() { val word = ",,," }
def printType[A: TypeTag](): Unit = println(typeOf[A])
printType[List[Int]]() //prints 'List[Int]'
printType[TestClass]() //prints 'Example.TestClass'
val typeOfCompanion: ??? = ??? //TODO what goes here?
val typeOfCaseClass: ??? = ??? //TODO what goes here?
printType[typeOfCompanion]() //TODO should print something like 'Example.TestClass'
printType[typeOfCaseClass]() //TODO should print something like 'Example.TestClass'
}
The solution should be able to save a Type or TypeTag or what the solution is. Then, pass typeOfCompanion or typeOfCaseClass like printTypetypeOfCompanion for printing. Changing the printing portion of the code may be required; I am not certain.
You have to be more explicit here
import scala.reflect.runtime.universe._
def printType(a: TypeTag[_]): Unit = println(a)
val typeOfCompanion = typeTag[List[Int]]
printType(typeOfCompanion)
def printType[A: TypeTag](): Unit = println(typeOf[A])
is exactly the same as
def printType[A]()(implicit a: TypeTag[A]): Unit = println(typeOf[A])
(except for the parameter name). So it can be called as
val listTypeTag /* : TypeTag[List[Int]] */ = typeTag[List[Int]]
printType()(listTypeTag)
(you can remove the empty parameter list from printType if you want).
For the companion, you need to use a singleton type:
val companionTag = typeTag[TestClass.type]
val caseClassTag = typeTag[TestClass]
I want to call an arbitrary public method of an arbitrary stuff via reflection. I.e. let's say, I want to write method extractMethod to be used like:
class User { def setAvatar(avatar: Avatar): Unit = …; … }
val m = extractMethod(someUser, "setAvatar")
m(someAvatar)
From the Reflection. Overview document from Scala docs, I see the following direct way to do that:
import scala.reflect.ClassTag
import scala.reflect.runtime.universe._
def extractMethod[Stuff: ClassTag: TypeTag](
stuff: Stuff,
methodName: String): MethodMirror =
{
val stuffTypeTag = typeTag[Stuff]
val mirror = stuffTypeTag.mirror
val stuffType = stuffTypeTag.tpe
val methodSymbol = stuffType
.member(TermName(methodName)).asMethod
mirror.reflect(stuff)
.reflectMethod(methodSymbol)
}
However what I'm bothered with this solution is that I need to pass implicit ClassTag[Stuff] and TypeTag[Stuff] parameters (first one is needed for calling reflect, second one — for getting stuffType). Which may be quite cumbersome, especially if extractMethod is called from generics that are called from generics and so on. I'd accept this as necessity for some languages that strongly lack runtime type information, but Scala is based on JRE, which allows to do the following:
def extractMethod[Stuff](
stuff: Stuff,
methodName: String,
parameterTypes: Array[Class[_]]): (Object*) => Object =
{
val unboundMethod = stuff.getClass()
.getMethod(methodName, parameterTypes: _*)
arguments => unboundMethod(stuff, arguments: _*)
}
I understand that Scala reflection allows to get more information that basic Java reflection. Still, here I just need to call a method. Is there a way to somehow reduce requirements (e.g. these ClassTag, TypeTag) of the Scala-reflection-based extractMethod version (without falling back to pure-Java reflection), assuming that performance doesn't matter for me?
Yes, there is.
First, according to this answer, TypeTag[Stuff] is a strictly stronger requirement than ClassTag[Stuff]. Although we don't automatically get implicit ClassTag[Stuff] from implicit TypeTag[Stuff], we can evaluate it manually as ClassTag[Stuff](stuffTypeTag.mirror.runtimeClass(stuffTypeTag.tpe)) and then implicitly or explicitly pass it to reflect that needs it:
import scala.reflect.ClassTag
import scala.reflect.runtime.universe._
def extractMethod[Stuff: TypeTag](
stuff: Stuff,
methodName: String): MethodMirror =
{
val stuffTypeTag = typeTag[Stuff]
val mirror = stuffTypeTag.mirror
val stuffType = stuffTypeTag.tpe
val stuffClassTag = ClassTag[Stuff](mirror.runtimeClass(stuffType))
val methodSymbol = stuffType
.member(TermName(methodName)).asMethod
mirror.reflect(stuff)(stuffClassTag)
.reflectMethod(methodSymbol)
}
Second, mirror and stuffType can be obtained from stuff.getClass():
import scala.reflect.ClassTag
import scala.reflect.runtime.universe._
def extractMethod(stuff: Stuff, methodName: String): MethodMirror = {
val stuffClass = stuff.getClass()
val mirror = runtimeMirror(stuffClass.getClassLoader)
val stuffType = mirror.classSymbol(stuffClass).toType
val stuffClassTag = ClassTag[Stuff](mirror.runtimeClass(stuffType))
val methodSymbol = stuffType
.member(TermName(methodName)).asMethod
mirror.reflect(stuff)(stuffClassTag)
.reflectMethod(methodSymbol)
}
Therefore we obtained Scala-style reflection entities (i.e. finally MethodMirror) without requiring ClassTag and/or TypeTag to be passed explicitly or implicitly from the caller. Not sure, however, how it compares with the ways described in the question (i.e. passing tags from outside and pure Java) in the terms of performance.
This is more of a Scala concept doubt than Spark. I have this Spark initialization code :
object EntryPoint {
val spark = SparkFactory.createSparkSession(...
val funcsSingleton = ContextSingleton[CustomFunctions] { new CustomFunctions(Some(hashConf)) }
lazy val funcs = funcsSingleton.get
//this part I want moved to another place since there are many many UDFs
spark.udf.register("funcName", udf {funcName _ })
}
The other class, CustomFunctions looks like this
class CustomFunctions(val hashConfig: Option[HashConfig], sark: Option[SparkSession] = None) {
val funcUdf = udf { funcName _ }
def funcName(colValue: String) = withDefinedOpt(hashConfig) { c =>
...}
}
^ class is wrapped in Serializable interface using ContextSingleton which is defined like so
class ContextSingleton[T: ClassTag](constructor: => T) extends AnyRef with Serializable {
val uuid = UUID.randomUUID.toString
#transient private lazy val instance = ContextSingleton.pool.synchronized {
ContextSingleton.pool.getOrElseUpdate(uuid, constructor)
}
def get = instance.asInstanceOf[T]
}
object ContextSingleton {
private val pool = new TrieMap[String, Any]()
def apply[T: ClassTag](constructor: => T): ContextSingleton[T] = new ContextSingleton[T](constructor)
def poolSize: Int = pool.size
def poolClear(): Unit = pool.clear()
}
Now to my problem, I want to not have to explicitly register the udfs as done in the EntryPoint app. I create all udfs as needed in my CustomFunctions class and then register dynamically only the ones that I read from user provided config. What would be the best way to achieve it? Also, I want to register the required udfs outside the main app but that throws my the infamous TaskNotSerializable exception. Serializing the big CustomFunctions is not a good idea, hence wrapped it up in ContextSingleton but my problem of registering udfs outside cannot be solved that way. Please suggest the right approach.