With reference to [Stackoverflow]
Scala: convert map to case class I tried to replicate one of the response and I see the following error:
Cannot construct instance of 'com.practice.scala.Test' (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator) at [Source: UNKNOWN; line: -1, column: -1]
Code:
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.scala.ScalaObjectMapper
case class Test(k1: Int, k2: String, k3: String)
object Workspace {
def fromMap[T](map: Map[String, Any])(implicit m: Manifest[T]): T = {
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.convertValue(map)
}
def main(args: Array[String]): Unit = {
val myMap = Map("k1" -> 1, "k2" -> "val2", "k3" -> "val3")
val result = fromMap[Test](myMap)
println(result)
}
}
What does this error says? Am I missing anything?
You need to register the DefaultScalaModule on your mapper:
// With 2.10 and later
val mapper = JsonMapper.builder()
.addModule(DefaultScalaModule)
.build()
// versions before 2.10 (also support for later 2.x but not 3.0)
val mapper = new ObjectMapper()
mapper.registerModule(DefaultScalaModule)
Source: https://github.com/FasterXML/jackson-module-scala#usage
Related
I want to make marshaller for case class which has fields referred to the same class.
case class TreeNode (name: String, parentNode: Option[TreeNode])
if i make serializer
implicit val nodeJson = jsonFormat2(TreeNode)
i get an error that no imlicits found for parameter Option[TreeNode]
is there are any way to solve such problem except writing serialization from scratch?
PS Some more attempts i made with encoder
private def toNode (node: TreeNode): Map[String, Any] = {
val parent = node.parentNode.map(toNode).orNull
Map[String, Any] ("name" -> node.name, "parentNode" -> parent)
}
implicit val treeNodeEncoder: Encoder[TreeNode] =
Encoder.forProduct2("name", "parentNode")(n =>
(n.name, n.parentNode.map(toNode).orNull)
)
it does not work as well, because Map[String, Any] has no implicit either
Assuming you're trying to make this work with circe, here is a working example:
import io.circe.{Encoder,Decoder}
import io.circe.generic.semiauto.{deriveEncoder,deriveDecoder}
import io.circe.syntax._
final case class TreeNode (name: String, parentNode: Option[TreeNode])
object TreeNode {
implicit val encoder: Encoder[TreeNode] = deriveEncoder
implicit val decoder: Decoder[TreeNode] = deriveDecoder
}
val a = TreeNode("a", None)
val b = TreeNode("b", Some(a))
val c = TreeNode("c", Some(b))
println(c.asJson)
Output
{
"name" : "c",
"parentNode" : {
"name" : "b",
"parentNode" : {
"name" : "a",
"parentNode" : null
}
}
}
Also note that representing trees starting from the leaves rather than the roots is unusual.
circe supports this out of the box, you just need to use the semiautomatic / automatic derivation mechanism provided by the library.
You can also modify the printer you want to use to control the identation and if you want to include or not nulls
Take a look to this code:
import io.circe.{Decoder, Encoder, Printer, parser}
import io.circe.syntax._
import io.circe.generic.semiauto._
final case class TreeNode(name: String, parentNode: Option[TreeNode] = None)
object TreeNode {
implicit final val decoder: Decoder[TreeNode] = deriveDecoder
implicit final val encoder: Encoder[TreeNode] = deriveEncoder
}
val data = TreeNode(name = "child", parentNode = Some(TreeNode(name = "parent")))
val printer = Printer.spaces2SortKeys.copy(dropNullValues = true)
val json = printer.print(data.asJson)
println(json)
println("------------------")
val result = parser.decode[TreeNode](json)
println(result)
You can see it running here
Thanks everyone for help, just to make it complete one more way i discovered, how to make it manually if necesserily
implicit val treeNodeEncoder: Encoder[TreeNode] = new Encoder[TreeNode] {
def apply(a: TreeNode): Json = {
val list = List(
("name" -> Json.fromString(a.name)),
("parentNode" -> a.parentNode.map(this.apply).orNull),
).filterNot(_._2 == null)
Json.obj(list: _*)
}
}
This way we can avoid to write parentNode as null and just skip it if it is None
I am using Avro4s. It's easy to serialise a
Map[String, T]
but I have a situation like
sealed trait Base
case object First extends Base
case object Second extends Base
and I need to serialise something like
Map[Base, T]
Any advice on the best way to achieve this? Thanks.
The thing is that according to the Avro spec
Map keys are assumed to be strings.
So the only type supported by Avro is Map[String,T]. It means that you need to write some custom code that will map your Map[Base, T] onto Map[String,T] and back. Something like this will probably work for you:
import scala.collection.breakOut
import scala.collection.immutable.Map
import scala.collection.JavaConverters._
import com.sksamuel.avro4s._
import org.apache.avro.Schema
import org.apache.avro.Schema.Field
object BaseMapAvroHelpers {
private val nameMap: Map[Base, String] = Map(First -> "first", Second -> "second")
private val revNameMap: Map[String, Base] = nameMap.toList.map(kv => (kv._2, kv._1)).toMap
implicit def toSchema[T: SchemaFor]: ToSchema[Map[Base, T]] = new ToSchema[Map[Base, T]] {
override val schema: Schema = Schema.createMap(implicitly[SchemaFor[T]].apply())
}
implicit def toValue[T: SchemaFor : ToValue]: ToValue[Map[Base, T]] = new ToValue[Map[Base, T]] {
override def apply(value: Map[Base, T]): java.util.Map[String, T] = value.map(kv => (nameMap(kv._1), kv._2)).asJava
}
implicit def fromValue[T: SchemaFor : FromValue]: FromValue[Map[Base, T]] = new FromValue[Map[Base, T]] {
override def apply(value: Any, field: Field): Map[Base, T] = {
val fromValueS = implicitly[FromValue[String]]
val fromValueT = implicitly[FromValue[T]]
value.asInstanceOf[java.util.Map[Any, Any]].asScala.map(kv => (revNameMap(fromValueS(kv._1)), fromValueT(kv._2)))(breakOut)
}
}
}
Usage example:
case class Wrapper[T](value: T)
def test(): Unit = {
import BaseMapAvroHelpers._
val map: Map[Base, String] = Map(First -> "abc", Second -> "xyz")
val wrapper = Wrapper(map)
val schema = AvroSchema[Wrapper[Map[Base, String]]]
println(s"Schema: $schema")
val bufOut = new ByteArrayOutputStream()
val out = AvroJsonOutputStream[Wrapper[Map[Base, String]]](bufOut)
out.write(wrapper)
out.flush()
println(s"Avro Out: ${bufOut.size}")
println(bufOut.toString("UTF-8"))
val in = AvroJsonInputStream[Wrapper[Map[Base, String]]](new ByteArrayInputStream(bufOut.toByteArray))
val read = in.singleEntity
println(s"read: $read")
}
and the output is something like:
Schema: {"type":"record","name":"Wrapper","namespace":"so","fields":[{"name":"value","type":{"type":"map","values":"string"}}]}
Avro Out: 40
{"value":{"first":"abc","second":"xyz"}}
read: Success(Wrapper(Map(First -> abc, Second -> xyz)))
I am not able to perform an implicit conversion from an RDD to a Dataframe in a Scala program although I am importing spark.implicits._.
Any help would be appreciated.
Main Program with the implicits:
object spark1 {
def main(args: Array[String]) {
val spark = SparkSession.builder().appName("e1").config("o1", "sv").getOrCreate()
import spark.implicits._
val conf = new SparkConf().setMaster("local").setAppName("My App")
val sc = spark.sparkContext
val data = sc.textFile("/TestDataB.txt")
val allSplit = data.map(line => line.split(","))
case class CC1(LAT: Double, LONG: Double)
val allData = allSplit.map( p => CC1( p(0).trim.toDouble, p(1).trim.toDouble))
val allDF = allData.toDF()
// ... other code
}
}
Error is as follows:
Error:(40, 25) value toDF is not a member of org.apache.spark.rdd.RDD[CC1]
val allDF = allData.toDF()
When you define the case class CC1 inside the main method, you hit https://issues.scala-lang.org/browse/SI-6649; toDF() then fails to locate the appropriate implicit TypeTag for that class at compile time.
You can see this in this simple example:
case class Out()
object TestImplicits {
def main(args: Array[String]) {
case class In()
val typeTagOut = implicitly[TypeTag[Out]] // compiles
val typeTagIn = implicitly[TypeTag[In]] // does not compile: Error:(23, 31) No TypeTag available for In
}
}
Spark's relevant implicit conversion has this type parameter: [T <: Product : TypeTag] (see newProductEncoder here), which means an implicit TypeTag[CC1] is required.
To fix this - simply move the definition of CC1 out of the method, or out of object entirely:
case class CC1(LAT: Double, LONG: Double)
object spark1 {
def main(args: Array[String]) {
val spark = SparkSession.builder().appName("e1").config("o1", "sv").getOrCreate()
import spark.implicits._
val data = spark.sparkContext.textFile("/TestDataB.txt")
val allSplit = data.map(line => line.split(","))
val allData = allSplit.map( p => CC1( p(0).trim.toDouble, p(1).trim.toDouble))
val allDF = allData.toDF()
// ... other code
}
}
I thought the toDF is in sqlContext.implicits._ so you need to import that not spark.implicits._. At least that is the case in spark 1.6
I want to convert any case class to a Map[String,Any] for example:
case class Person(name:String, address:Address)
case class Address(street:String, zip:Int)
val p = Person("Tom", Address("Jefferson st", 10000))
val mp = p.asMap
//Map("name" -> "Tom", "address" -> Map("street" -> "Jefferson st", "zip" -> 10000))
val p1 = mp.asCC[Person]
assert(p1 === p)
Possible duplications:
Here is a question that with reflection answer.
Here is a question for (converting from case class to map (without nesting)
I also found how to do it for a case claas without any nested case class inside it, here is the code from here:
package macros
import scala.language.experimental.macros
import scala.reflect.macros.blackbox.Context
trait Mappable[T] {
def toMap(t: T): Map[String, Any]
def fromMap(map: Map[String, Any]): T
}
object Mappable {
implicit def materializeMappable[T]: Mappable[T] = macro materializeMappableImpl[T]
def materializeMappableImpl[T: c.WeakTypeTag](c: Context): c.Expr[Mappable[T]] = {
import c.universe._
val tpe = weakTypeOf[T]
val companion = tpe.typeSymbol.companion
val fields = tpe.decls.collectFirst {
case m: MethodSymbol if m.isPrimaryConstructor => m
}.get.paramLists.head
val (toMapParams, fromMapParams) = fields.map { field =>
val name = field.asTerm.name
val key = name.decodedName.toString
val returnType = tpe.decl (name).typeSignature
(q"$key -> t.$name", q"map($key).asInstanceOf[$returnType]")
}.unzip
c.Expr[Mappable[T]] { q"""
new Mappable[$tpe] {
def toMap(t: $tpe): Map[String, Any] = Map(..$toMapParams)
def fromMap(map: Map[String, Any]): $tpe = $companion(..$fromMapParams)
}
""" }
}
}
Also it worth to mention Play Json library and ReactiveMongo Bson library do the same thing, but those project were really big to understand how to do this.
Let's say I have this example case class
case class Test(key1: Int, key2: String, key3: String)
And I have a map
myMap = Map("k1" -> 1, "k2" -> "val2", "k3" -> "val3")
I need to convert this map to my case class in several places of the code, something like this:
myMap.asInstanceOf[Test]
What would be the easiest way of doing that? Can I somehow use implicit for this?
Two ways of doing this elegantly. The first is to use an unapply, the second to use an implicit class (2.10+) with a type class to do the conversion for you.
1) The unapply is the simplest and most straight forward way to write such a conversion. It does not do any "magic" and can readily be found if using an IDE. Do note, doing this sort of thing can clutter your companion object and cause your code to sprout dependencies in places you might not want:
object MyClass{
def unapply(values: Map[String,String]) = try{
Some(MyClass(values("key").toInteger, values("next").toFloat))
} catch{
case NonFatal(ex) => None
}
}
Which could be used like this:
val MyClass(myInstance) = myMap
be careful, as it would throw an exception if not matched completely.
2) Doing an implicit class with a type class creates more boilerplate for you but also allows a lot of room to expand the same pattern to apply to other case classes:
implicit class Map2Class(values: Map[String,String]){
def convert[A](implicit mapper: MapConvert[A]) = mapper conv (values)
}
trait MapConvert[A]{
def conv(values: Map[String,String]): A
}
and as an example you'd do something like this:
object MyObject{
implicit val new MapConvert[MyObject]{
def conv(values: Map[String, String]) = MyObject(values("key").toInt, values("foo").toFloat)
}
}
which could then be used just as you had described above:
val myInstance = myMap.convert[MyObject]
throwing an exception if no conversion could be made. Using this pattern converting between a Map[String, String] to any object would require just another implicit (and that implicit to be in scope.)
Here is an alternative non-boilerplate method that uses Scala reflection (Scala 2.10 and above) and doesn't require a separately compiled module:
import org.specs2.mutable.Specification
import scala.reflect._
import scala.reflect.runtime.universe._
case class Test(t: String, ot: Option[String])
package object ccFromMap {
def fromMap[T: TypeTag: ClassTag](m: Map[String,_]) = {
val rm = runtimeMirror(classTag[T].runtimeClass.getClassLoader)
val classTest = typeOf[T].typeSymbol.asClass
val classMirror = rm.reflectClass(classTest)
val constructor = typeOf[T].decl(termNames.CONSTRUCTOR).asMethod
val constructorMirror = classMirror.reflectConstructor(constructor)
val constructorArgs = constructor.paramLists.flatten.map( (param: Symbol) => {
val paramName = param.name.toString
if(param.typeSignature <:< typeOf[Option[Any]])
m.get(paramName)
else
m.get(paramName).getOrElse(throw new IllegalArgumentException("Map is missing required parameter named " + paramName))
})
constructorMirror(constructorArgs:_*).asInstanceOf[T]
}
}
class CaseClassFromMapSpec extends Specification {
"case class" should {
"be constructable from a Map" in {
import ccFromMap._
fromMap[Test](Map("t" -> "test", "ot" -> "test2")) === Test("test", Some("test2"))
fromMap[Test](Map("t" -> "test")) === Test("test", None)
}
}
}
Jonathan Chow implements a Scala macro (designed for Scala 2.11) that generalizes this behavior and eliminates the boilerplate.
http://blog.echo.sh/post/65955606729/exploring-scala-macros-map-to-case-class-conversion
import scala.reflect.macros.Context
trait Mappable[T] {
def toMap(t: T): Map[String, Any]
def fromMap(map: Map[String, Any]): T
}
object Mappable {
implicit def materializeMappable[T]: Mappable[T] = macro materializeMappableImpl[T]
def materializeMappableImpl[T: c.WeakTypeTag](c: Context): c.Expr[Mappable[T]] = {
import c.universe._
val tpe = weakTypeOf[T]
val companion = tpe.typeSymbol.companionSymbol
val fields = tpe.declarations.collectFirst {
case m: MethodSymbol if m.isPrimaryConstructor ⇒ m
}.get.paramss.head
val (toMapParams, fromMapParams) = fields.map { field ⇒
val name = field.name
val decoded = name.decoded
val returnType = tpe.declaration(name).typeSignature
(q"$decoded → t.$name", q"map($decoded).asInstanceOf[$returnType]")
}.unzip
c.Expr[Mappable[T]] { q"""
new Mappable[$tpe] {
def toMap(t: $tpe): Map[String, Any] = Map(..$toMapParams)
def fromMap(map: Map[String, Any]): $tpe = $companion(..$fromMapParams)
}
""" }
}
}
This works well for me，if you use jackson for scala：
def from[T](map: Map[String, Any])(implicit m: Manifest[T]): T = {
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.convertValue(map)
}
Reference from：Convert a Map<String, String> to a POJO
I don't love this code, but I suppose this is possible if you can get the map values into a tuple and then use the tupled constructor for your case class. That would look something like this:
val myMap = Map("k1" -> 1, "k2" -> "val2", "k3" -> "val3")
val params = Some(myMap.map(_._2).toList).flatMap{
case List(a:Int,b:String,c:String) => Some((a,b,c))
case other => None
}
val myCaseClass = params.map(Test.tupled(_))
println(myCaseClass)
You have to be careful to make sure the list of values is exactly 3 elements and that they are the correct types. If not, you end up with a None instead. Like I said, not great, but it shows that it is possible.
commons.mapper.Mappers.mapToBean[CaseClassBean](map)
Details: https://github.com/hank-whu/common4s
Here's an update to Jonathon's answer for Scala 3 (which no longer has TypeTag). Be aware that this won't work for case classes nested inside of other classes. But for top-level case classes it seems to work fine.
import scala.reflect.ClassTag
object Reflect:
def fromMap[T <: Product : ClassTag](m: Map[String, ?]): T =
val classTag = implicitly[ClassTag[T]]
val constructor = classTag.runtimeClass.getDeclaredConstructors.head
val constructorArgs = constructor.getParameters()
.map { param =>
val paramName = param.getName
if (param.getType == classOf[Option[_]])
m.get(paramName)
else
m.get(paramName)
.getOrElse(throw new IllegalArgumentException(s"Missing required parameter: $paramName"))
}
constructor.newInstance(constructorArgs: _*).asInstanceOf[T]
And a test for the above:
case class Foo(a: String, b: Int, c: Option[String] = None)
case class Bar(a: String, b: Int, c: Option[Foo])
class ReflectSuite extends munit.FunSuite:
test("fromMap") {
val m = Map("a" -> "hello", "b" -> 42, "c" -> "world")
val foo = Reflect.fromMap[Foo](m)
assertEquals(foo, Foo("hello", 42, Some("world")))
val n = Map("a" -> "hello", "b" -> 43)
val foo2 = Reflect.fromMap[Foo](n)
assertEquals(foo2, Foo("hello", 43))
val o = Map("a" -> "yo", "b" -> 44, "c" -> foo)
val bar = Reflect.fromMap[Bar](o)
assertEquals(bar, Bar("yo", 44, Some(foo)))
}
test("fromMap should fail when required parameter is missing") {
val m = Map("a" -> "hello", "c" -> "world")
intercept[java.lang.IllegalArgumentException] {
Reflect.fromMap[Foo](m)
}
}