Avoid capturing `this` when serializing functions in Scala - scala

When serializing functions in Scala, we should take care not to capture fields that are not serializable.
class Foo(s: Set[Int] = Set(1,2,3)) {
def replicate(): () => Foo = () => new Foo(s)
}
val foo = new Foo()
val fn = foo.replicate()
serialize(fn) // java.io.NotSerializableException: Foo
In the above code snippet serializing msg will fail because the function captures this and Foo is not serializable.
To avoid capturing this SIP-21 proposes spores for Scala.
Unfortunately, spores only exist for Scala 2.11.x (see this note).
Luckily, I found in this tread (this is also described in SIP-21) that one can avoid capturing the this reference, by manually creating local values as shown in the code snippet below.
class Foo(s: Set[Int] = Set(1,2,3)) {
def replicate(): () => Foo = {
() => {
{
val localS = this.s
new Foo(localS)
}
}
}
}
Unfortunately, in Scala 2.13.1 this does not work and still captures this (try out the complete code snippet below).
Hence, we cannot serialize the function returned by the replicate method.
Now, I found the following workaround, but it's a rather ugly one.
class Foo(s: Set[Int] = Set(1,2,3)) {
def indirection() = replicate(s)
def replicate(set: Set[Int]): () => Foo = () => new Foo(set)
}
val foo = new Foo()
val msg = foo.indirection()
toStr(msg) // Works :)
By adding an extra indirection, we avoid capturing this and we can serialize the function without requiring Foo to be serializable. However, I don't want to introduce these extra indirections in all the objects I want to serialize this way. How can I avoid this (in Scala 2.13.1) ?
The complete code snippet:
// Start writing your ScalaFiddle code here
import java.io._
import java.util.Base64
def serialize(o: Any) = {
val baos = new ByteArrayOutputStream();
val oos = new ObjectOutputStream( baos );
oos.writeObject( o );
oos.close();
Base64.getEncoder().encodeToString(baos.toByteArray())
}
def deserialize(s: String) = {
val data = Base64.getDecoder().decode( s );
val ois = new ObjectInputStream(new ByteArrayInputStream( data ) );
val o = ois.readObject();
ois.close();
o
}
//////////// TRY 1
try {
class Foo(val s: Set[Int] = Set(1,2,3)) {
def replicate(): () => Foo = () => new Foo(s)
}
val foo = new Foo()
val msg = foo.replicate()
val str = serialize(msg)
val reconstructedFoo = deserialize(str).asInstanceOf[() => Foo]()
println(s"Try 1: success, reconstructedFoo.s = ${reconstructedFoo.s}")
} catch {
case e: NotSerializableException => println("TRY 1: Not serializable.")
}
//////////// TRY 2
try {
class Foo(val s: Set[Int] = Set(1,2,3)) {
def replicate(): () => Foo = {
() => {
{
val localS = this.s
new Foo(localS)
}
}
}
}
val foo = new Foo()
val msg = foo.replicate()
val str = serialize(msg)
val reconstructedFoo = deserialize(str).asInstanceOf[() => Foo]()
println(s"Try 2: success, reconstructedFoo.s = ${reconstructedFoo.s}")
} catch {
case e: NotSerializableException => println("TRY 2: Not serializable.")
}
//////////// Solution (ugly)
try {
class Foo(val s: Set[Int] = Set(1,2,3)) {
def indirection() = replicate(s)
def replicate(set: Set[Int]): () => Foo = () => new Foo(set)
}
val foo = new Foo()
val msg = foo.indirection()
val str = serialize(msg)
val reconstructedFoo = deserialize(str).asInstanceOf[() => Foo]()
println(s"Try 3: success, reconstructedFoo.s = ${reconstructedFoo.s}")
} catch {
case e: NotSerializableException => println("TRY 3: Not serializable.")
}

Related

Scala 3 macros: create a new polynmorphic function using the reflect api

I intend to create the simple polymorphic function expression below using the Quotes.reflect API:
new PolyFunction {
def apply[X](a: X): X = a
}
What I have attempted is shown below with parts that I could not implement replaced by ???:
val name: String = "$anon"
val parents = List(TypeTree.of[Object], TypeTree.of[PolyFunction])
def decls(cls: Symbol): List[Symbol] =
List(
Symbol.newMethod(
cls,
"apply",
MethodType(List("a"))(
{ mt =>
???
},
mt => ???
)
)
)
val cls = Symbol.newClass(Symbol.spliceOwner, name, parents = parents.map(_.tpe), decls, selfType = None)
val applySym = cls.declaredMethod("apply").head
val applyDef = DefDef(applySym, argss => Some(???))
val clsDef = ClassDef(cls, parents, body = List(applyDef))
val closure = Block(List(clsDef),Apply(Select(New(TypeIdent(cls)), cls.primaryConstructor), Nil))
closure.asExpr
The problem, mainly, is I am unable to declare the type parameter X, and its reference in the first value parameter and the functions return type. I have noticed some reflection functions are annotated as experimental.
Try
import scala.annotation.experimental
import scala.quoted.*
inline def newPoly: PolyFunction = ${newPolyImpl}
#experimental
def newPolyImpl(using Quotes): Expr[PolyFunction] = {
import quotes.reflect.*
val name: String = "$anon"
val parents = List(TypeTree.of[Object], TypeTree.of[PolyFunction])
def decls(cls: Symbol): List[Symbol] =
List(
Symbol.newMethod(
cls,
"apply",
PolyType(List("X"))(_ => List(TypeBounds.empty), polyType => {
val typeParam = polyType.param(0)
MethodType(List("a"))(_ => List(typeParam), _ => typeParam)
})
)
)
val cls = Symbol.newClass(Symbol.spliceOwner, name, parents = parents.map(_.tpe), decls, selfType = None)
val applySym = cls.declaredMethod("apply").head
// argss=List(List(TypeTree[TypeRef(NoPrefix,type X)]), List(Ident(a)))
val applyDef = DefDef(applySym, argss => Some(argss(1)(0).asInstanceOf[Term]))
val clsDef = ClassDef(cls, parents, body = List(applyDef))
val closure = Block(List(clsDef), Apply(Select(New(TypeIdent(cls)), cls.primaryConstructor), Nil))
closure.asExprOf[PolyFunction]
}
Usage:
newPoly
//scalac: {
// class $anon extends java.lang.Object with scala.PolyFunction {
// def apply[X](a: X): X = a
// }
// new $anon()
//}
Scala 3.2.1
Method Override with Scala 3 Macros
Scala3: Crafting Types Through Metaprogramming?
`tq` equivalent in Scala 3 macros

Facing issue while using reflection class concept in scala

I have one main class like this:
class Test {
def exe(first:String, second:String, task:String):String = {
task match {
case "A" => {
val obj = new A(first)
obj.defineSecond(second)
}
case "B" => {
val obj = new B(first)
obj.defineSecond(second)
}
case "C" => {
val obj = new C(first)
obj.defineSecond(second)
}
....so many cases
}
}
}
Instead of writing case in my Test class everytime a new class is added, I tried using the concept of reflection in scala.
Below is what I trying:
val m = ru.runtimeMirror(getClass.getClassLoader)
val classTest = ru.typeOf[Test].typeSymbol.asClass
val cm = m.reflectClass(classTest)
But getting error as "class Test is inner class, use reflectClass on an InstaneMirror to obtain its classMirror".
Can anyone knows how can I can avoid adding cases to my main class everytime a new class is created, instead I can write my main class in a way it will work for every case.
I guess you haven't provided all necessary information in your question. It's written that "class Test is inner class" in your error message but Test is not inner in your code snippet. If you want your runtime-reflection code to be fixed please provide code snippet that reflects actual use case.
Meanwhile you can try a macro (working at compile time)
import scala.language.experimental.macros
import scala.reflect.macros.blackbox
class Test {
def exe(first: String, second: String, task: String): String = macro Test.exeImpl
}
object Test {
def exeImpl(c: blackbox.Context)(first: c.Tree, second: c.Tree, task: c.Tree): c.Tree = {
import c.universe._
val cases = Seq("A", "B", "C").map(name =>
cq"""${Literal(Constant(name))} => {
val obj = new ${TypeName(name)}($first)
obj.defineSecond($second)
}"""
)
q"$task match { case ..$cases }"
}
}
Usage:
class A(s: String) {
def defineSecond(s1: String): String = ""
}
class B(s: String) {
def defineSecond(s1: String): String = ""
}
class C(s: String) {
def defineSecond(s1: String): String = ""
}
new Test().exe("first", "second", "task")
//scalac: "task" match {
// case "A" => {
// val obj = new A("first");
// obj.defineSecond("second")
// }
// case "B" => {
// val obj = new B("first");
// obj.defineSecond("second")
// }
// case "C" => {
// val obj = new C("first");
// obj.defineSecond("second")
// }
//}

Serialization in Scala

Im trying to understand basics of serialization in Scala. When i run the first example below I get the following output on the last line: res1: A.Mao = A$$anonfun$main$1$Mao$1#78e67e0a
#SerialVersionUID(1L)
class Poo(val aa:Int) extends Serializable {
override def toString() = "Hola"
}
#SerialVersionUID(1L)
class Mao(val hi: Poo) extends Serializable
def serialize() = {
val test = new Mao(new Poo(1))
try{
val fout = new FileOutputStream("c:\\misc\\address.ser");
val oos = new ObjectOutputStream(fout);
oos.writeObject(test);
oos.close();
System.out.println("Done");
}catch {
case ex => ex.printStackTrace();
}
}
serialize()
def ReadObjectFromFile[A](filename: String)(implicit m:scala.reflect.Manifest[A]): A = {
val input = new ObjectInputStream(new FileInputStream(filename))
val obj = input.readObject()
obj match {
case x if m.erasure.isInstance(x) => x.asInstanceOf[A]
case _ => sys.error("Type not what was expected when reading from file")
}
}
ReadObjectFromFile[Mao]("c:\\misc\\address.ser")
If I change the example and use case classes instead things works as expected with the output
res1: A.Mao = Mao(Hola)
case class Poo(val aa:Int) {
override def toString() = "Hola"
}
case class Mao(val hi: Poo)
def serialize() = {
val test = new Mao(new Poo(1))
try{
val fout = new FileOutputStream("c:\\misc\\address.ser");
val oos = new ObjectOutputStream(fout);
oos.writeObject(test);
oos.close();
System.out.println("Done");
}catch {
case ex => ex.printStackTrace();
}
}
def ReadObjectFromFile[A](filename: String)(implicit m:scala.reflect.Manifest[A]): A = {
val input = new ObjectInputStream(new FileInputStream(filename))
val obj = input.readObject()
obj match {
case x if m.erasure.isInstance(x) => x.asInstanceOf[A]
case _ => sys.error("Type not what was expected when reading from file")
}
}
ReadObjectFromFile[Mao]("c:\\misc\\address.ser")
So my questions are:
What do I need to do to get class to give the same output as case class?
Why does case class work without adding any explicit information about serialization?
This has nothing to do with the de/serialization (which seems correct) - it's just the way the result is displayed:
Scala REPL (and Worksheets) use the value's toString method to display it. Case classes override the default toString() method, therefore the output is displayed nicely (as expected). For non-case classes, the defeault implementation of Object.toString() is called, and results in the class name and address that you see.
You can implement toString for the non-case class too, to get the same result:
class Mao(val hi: Poo) extends Serializable {
override def toString = s"Mao($hi)"
}
// program prints:
// Done
// Mao(Hola)

type parameter mismatch with WeakTypeTag reflection + quasiquoting (I think!)

Inspired by travisbrown, I'm trying to use a macro to create some "smart constructors".
Given
package mypkg
sealed trait Hello[A]
case class Ohayo[A,B](a: (A,B)) extends Hello[A]
and
val smartConstructors = FreeMacros.liftConstructors[Hello]
The macro should find all the subclasses of Hello, look at their constructors, and extract a few elements to populate this tree for the "smart constructor":
q"""
def $methodName[..$typeParams](...$paramLists): $baseType =
$companionSymbol[..$typeArgs](...$argLists)
"""
I hoped to get:
val smartConstructors = new {
def ohayo[A, B](a: (A, B)): Hello[A] = Ohayo[A, B](a)
}
but instead get:
error: type mismatch;
found : (A(in class Ohayo), B(in class Ohayo))
required: ((some other)A(in class Ohayo), (some other)B(in class Ohayo))
val liftedConstructors = FreeMacros.liftConstructors[Hello]
At a glance, the tree looks ok to me:
scala> q" new { ..$wellTyped }"
res1: u.Tree =
{
final class $anon extends scala.AnyRef {
def <init>() = {
super.<init>();
()
};
def ohayo[A, B](a: (A, B)): net.arya.constructors.Hello[A] = Ohayo[A, B](a)
};
new $anon()
}
but I guess it invisibly isn't. If I naively try to freshen up the typeParams with info.typeParams.map(p => TypeName(p.name.toString)), I get "can't splice A as type parameter" when I do the quasiquoting.
Where am I going wrong? Thanks for taking a look.
-Arya
import scala.language.experimental.macros
import scala.reflect.api.Universe
import scala.reflect.macros.whitebox
class FreeMacros(val c: whitebox.Context) {
import c.universe._
import FreeMacros._
def liftedImpl[F[_]](implicit t: c.WeakTypeTag[F[_]]): Tree = {
val atc = t.tpe
val childSymbols: Set[ClassSymbol] = subCaseClassSymbols(c.universe)(atc.typeSymbol.asClass)
val wellTyped = childSymbols.map(ctorsForSymbol(c.universe)(atc)).unzip
q"new { ..${wellTyped} }"
}
}
object FreeMacros {
def liftConstructors[F[_]]: Any = macro FreeMacros.liftedImpl[F]
def smartName(name: String): String = (
name.toList match {
case h :: t => h.toLower :: t
case Nil => Nil
}
).mkString
def subCaseClassSymbols(u: Universe)(root: u.ClassSymbol): Set[u.ClassSymbol] = {
val subclasses = root.knownDirectSubclasses
val cast = subclasses.map(_.asInstanceOf[u.ClassSymbol])
val partitioned = mapped.partition(_.isCaseClass)
partitioned match {
case (caseClasses, regularClasses) => caseClasses ++ regularClasses.flatMap(r => subCaseClassSymbols(u)(r))
}
}
def ctorsForSymbol(u: Universe)(atc: u.Type)(caseClass: u.ClassSymbol): (u.DefDef, u.DefDef) = {
import u._
import internal._
// these didn't help
// def clearTypeSymbol(s: Symbol): TypeSymbol = internal.newTypeSymbol(NoSymbol, s.name.toTypeName, s.pos, if(s.isImplicit)Flag.IMPLICIT else NoFlags)
// def clearTypeSymbol2(s: Symbol): TypeSymbol = internal.newTypeSymbol(NoSymbol, s.name.toTypeName, NoPosition, if(s.isImplicit)Flag.IMPLICIT else NoFlags)
// def clearTypeDef(d: TypeDef): TypeDef = internal.typeDef(clearTypeSymbol(d.symbol))
val companionSymbol: Symbol = caseClass.companion
val info: Type = caseClass.info
val primaryCtor: Symbol = caseClass.primaryConstructor
val method = primaryCtor.asMethod
val typeParams = info.typeParams.map(internal.typeDef(_))
// val typeParams = info.typeParams.map(s => typeDef(newTypeSymbol(NoSymbol, s.name.toTypeName, NoPosition, NoFlags)))
// val typeParams = info.typeParams.map(s => internal.typeDef(clearTypeSymbol2(s)))
val typeArgs = info.typeParams.map(_.name)
val paramLists = method.paramLists.map(_.map(internal.valDef(_)))
val argLists = method.paramLists.map(_.map(_.asTerm.name))
val baseType = info.baseType(atc.typeSymbol)
val List(returnType) = baseType.typeArgs
val methodName = TermName(smartName(caseClass.name.toString))
val wellTyped =
q"""
def $methodName[..$typeParams](...$paramLists): $baseType =
$companionSymbol[..$typeArgs](...$argLists)
"""
wellTyped
}
}
P.S. I have been experimenting with toolbox.untypecheck / typecheck per this article but haven't found a working combination.
you need using
clas.typeArgs.map(_.toString).map(name => {
TypeDef(Modifiers(Flag.PARAM),TypeName(name), List(),TypeBoundsTree(EmptyTree, EmptyTree))
}
replace
info.typeParams.map(p => TypeName(p.name.toString))
it si my code
object GetSealedSubClass {
def ol3[T]: Any = macro GetSealedSubClassImpl.ol3[T]
}
class GetSealedSubClassImpl(val c: Context) {
import c.universe._
def showInfo(s: String) =
c.info(c.enclosingPosition, s.split("\n").mkString("\n |---macro info---\n |", "\n |", ""), true)
def ol3[T: c.WeakTypeTag]: c.universe.Tree = {
//get all sub class
val subClass = c.weakTypeOf[T]
.typeSymbol.asClass.knownDirectSubclasses
.map(e => e.asClass.toType)
//check type params must ia s sealed class
if (subClass.size < 1)
c.abort(c.enclosingPosition, s"${c.weakTypeOf[T]} is not a sealed class")
// get sub class constructor params
val subConstructorParams = subClass.map { e =>
//get constructor
e.members.filter(_.isConstructor)
//if the class has many Constructor then you need filter the main Constructor
.head.map(s => s.asMethod)
//get function param list
}.map(_.asMethod.paramLists.head)
.map(_.map(e => q"""${e.name.toTermName}:${e.info} """))
val outfunc = subClass zip subConstructorParams map {
case (clas, parm) =>
q"def smartConstructors[..${
clas.typeArgs.map(_.toString).map(name => {
TypeDef(Modifiers(Flag.PARAM), TypeName(name), List(), TypeBoundsTree(EmptyTree, EmptyTree))
})
}](..${parm})=${clas.typeSymbol.name.toTermName} (..${parm})"
}
val outClass =
q"""
object Term{
..${outfunc}
}
"""
showInfo(show(outClass))
q"""{
$outClass
Term
}
"""
}
}
using like this
sealed trait Hello[A]
case class Ohayo[A, B](a: (A, B)) extends Hello[A]
object GetSealed extends App {
val a = GetSealedSubClass.ol3[Hello[_]]
val b=a.asInstanceOf[ {def smartConstructors[A, B](a: (A, B)): Ohayo[A, B]}].smartConstructors(1, 2).a
println(b)
}

How to get a name of a class member?

I want to be able to do something like this:
prepare form:
val formDescription = formBuilder(_.textField[User](_.firstName)
.textField[User](_.lastName)
).build
showForm(formDescription)
extract data from user filled form, using User:
//contains data of a form submitted by a user:
val formData: Map[String, String] = getFormData
val newUser = User(id = randomUuid, firstName = formData.extract[User](_.firstName))
One solution I see is to use a dynamic proxy that extends provided class and remembers what was invoked on him:
def getFieldName[T:Manifest](foo: T => Any) = {
val clazz = implicitly[Manifest[T]].erasure
val proxy = createDynamicProxy(clazz)
foo(proxy)
proxy.lastInvokedMethodName
}
Is there a better way to do it? Is there any lib that implements it already?
This reflective approach takes a case class and invokes its companion apply, calling getField and fetching default args if the field is not in the data.
import scala.reflect.runtime.{currentMirror => cm, universe => uni}
import uni._
def fromXML(xml: Node): Option[PluginDescription] = {
def extract[A]()(implicit tt: TypeTag[A]): Option[A] = {
// extract one field
def getField(field: String): Option[String] = {
val text = (xml \\ field).text.trim
if (text == "") None else Some(text)
}
val apply = uni.newTermName("apply")
val module = uni.typeOf[A].typeSymbol.companionSymbol.asModule
val ts = module.moduleClass.typeSignature
val m = (ts member apply).asMethod
val im = cm reflect (cm reflectModule module).instance
val mm = im reflectMethod m
def getDefault(i: Int): Option[Any] = {
val n = uni.newTermName("apply$default$" + (i+1))
val m = ts member n
if (m == NoSymbol) None
else Some((im reflectMethod m.asMethod)())
}
def extractArgs(pss: List[List[Symbol]]): List[Option[Any]] =
pss.flatten.zipWithIndex map (p => getField(p._1.name.encoded) orElse getDefault(p._2))
val args = extractArgs(m.paramss)
if (args exists (!_.isDefined)) None
else Some(mm(args.flatten: _*).asInstanceOf[A])
}
// check the top-level tag
xml match {
case <plugin>{_*}</plugin> => extract[PluginDescription]()
case _ => None
}
}
The idea was to do something like:
case class User(id: Int = randomUuid, firstName: String, lastName: String)
val user = extract[User]()
That's my own solution:
package utils
import javassist.util.proxy.{MethodHandler, MethodFilter, ProxyFactory}
import org.specs2.mutable._
import javassist.util.proxy.Proxy
import java.lang.reflect.{Constructor, Method}
class DynamicProxyTest extends Specification with MemberNameGetter {
"Dynamic proxy" should {
"extract field name" in {
memberName[TestClass](_.a) must ===("a")
memberName[TestClass](_.i) must ===("i")
memberName[TestClass](_.b) must ===("b")
memberName[TestClass](_.variable) must ===("variable")
memberName[TestClass](_.value) must ===("value")
memberName[TestClass](_.method) must ===("method")
}
}
}
trait MemberNameGetter {
def memberName[T: Manifest](foo: T => Any) = {
val mf = manifest[T]
val clazz = mf.erasure
val proxyFactory = new ProxyFactory
proxyFactory.setSuperclass(clazz)
proxyFactory.setFilter(new MethodFilter {
def isHandled(p1: Method) = true
})
val newClass = proxyFactory.createClass()
var lastInvokedMethod: String = null
val mh = new MethodHandler {
def invoke(p1: Any, p2: Method, p3: Method, p4: Array[AnyRef]) = {
lastInvokedMethod = p2.getName
p3.invoke(p1, p4: _*)
}
}
val constructor = defaultConstructor(newClass)
val parameters = defaultConstructorParameters(constructor)
// val proxy = constructor.newInstance("dsf", new Integer(0))
val proxy2 = constructor.newInstance(parameters: _*)
proxy2.asInstanceOf[Proxy].setHandler(mh)
foo(proxy2.asInstanceOf[T])
lastInvokedMethod
}
private def defaultConstructor(c: Class[_]) = c.getConstructors.head
private def defaultConstructorParameters(constructor: Constructor[_]) = {
val parameterTypes = constructor.getParameterTypes
parameterTypes.map{
case Integer.TYPE => Integer.valueOf(0)
case _ => null
}
}
}
case class TestClass(a: String, i: Int, b: Boolean) {
var variable = "asdf"
val value = "asdfasdfasd"
def method = "method"
}
val mh = new MethodHandler {
def invoke(p1: Any, p2: Method, p3: Method, p4: Array[AnyRef]) = {
lastInvokedMethod = p2.getName
p3.invoke(p1, p4: _*)
}
}
val constructor = defaultConstructor(newClass)
val parameters = defaultConstructorParameters(constructor)
// val proxy = constructor.newInstance("dsf", new Integer(0))
val proxy2 = constructor.newInstance(parameters: _*)
proxy2.asInstanceOf[Proxy].setHandler(mh)
foo(proxy2.asInstanceOf[T])
lastInvokedMethod
}
private def defaultConstructor(c: Class[_]) = c.getConstructors.head
private def defaultConstructorParameters(constructor: Constructor[_]) = {
val parameterTypes = constructor.getParameterTypes
parameterTypes.map{
case Integer.TYPE => Integer.valueOf(0)
case java.lang.Double.TYPE => java.lang.Double.valueOf(0)
case java.lang.Long.TYPE => java.lang.Long.valueOf(0)
case java.lang.Boolean.TYPE => java.lang.Boolean.FALSE
case _ => null
}
}
}
case class TestClass(a: String, i: Int, b: Boolean) {
var variable = "asdf"
val value = "asdfasdfasd"
def method = "method"
}