When I execute in the REPL, it works (due to implicit mapWrites in the scope):
scala> Map("a"->1l, "b"->2l)
res0: scala.collection.immutable.Map[String,Long] = Map(a -> 1, b -> 2)
scala> Map("c" -> res0, "d" -> res0)
res1: scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Long]] = Map(c -> Map(a -> 1, b -> 2), d -> Map(a -> 1, b -> 2))
scala> import play.api.libs.json._
import play.api.libs.json._
scala> Json.toJson(res1)
res2: play.api.libs.json.JsValue = {"c":{"a":1,"b":2},"d":{"a":1,"b":2}}
Why my code still doesn't compile (it's the same type as in the REPL) ?
No Json deserializer found for type Map[String,Map[String,Long]]. Try to implement an implicit Writes or Format for this type.
[edit] I've found a workaround but i don't understand why i need it :
implicit def mapWrites = Writes[Map[String,Map[String,Long]]] ( m => Writes.mapWrites(Writes.mapWrites[Long]).writes(m))
Play 2.1 JSON API does not provide a serializer for the Type Map[String, Object].
Define case class and Format for the specific type instead of Map[String, Object]:
// { "val1" : "xxx", "val2" : ["a", "b", "c"] }
case class Hoge(val1: String, val2: List[String])
implicit val hogeFormat = Json.format[Hoge]
If you don't want to create case class.
The following code provides JSON serializer/deserializer for Map[String, Object]:
implicit val objectMapFormat = new Format[Map[String, Object]] {
def writes(map: Map[String, Object]): JsValue =
Json.obj(
"val1" -> map("val1").asInstanceOf[String],
"val2" -> map("val2").asInstanceOf[List[String]]
)
def reads(jv: JsValue): JsResult[Map[String, Object]] =
JsSuccess(Map("val1" -> (jv \ "val1").as[String], "val2" -> (jv \ "val2").as[List[String]]))
}
Jackson doesn't know how to deserialize the Scala collection class scala.collection.immutable.Map, because it doesn't implement any of the Java collection interfaces.
You can either deserialize to a Java collection:
val mapData = mapper.readValue(jsonContent, classOf[java.util.Map[String,String]])
or add the Scala module to the mapper:
mapper.registerModule(DefaultScalaModule)
val mapData = mapper.readValue(jsonContent, classOf[Map[String,String]])
or you can also try like this
import play.api.libs.json._
val a1=Map("val1"->"a", "val2"->"b")
Json.toJSon(a1)
Because a1 is just Map[String,String] that works OK.
But if I have something more complex like where I have Map[String,Object], that doesn't work:
val a = Map("val1" -> "xxx", "val2"-> List("a", "b", "c"))
Json.toJSon(a1)
>>> error: No Json deserializer found for type scala.collection.immutable.Map[String,Object]
I found that I can do something like the following:
val a2 = Map("val1" -> Json.toJson("a"), "val2" -> Json.toJson(List("a", "b", "c")))
Json.toJson(a2)
And that works.`
Related
How to Convert Seq[Option[Map[String, Any]]] to Option[Map[String, Any]]
So skip any option that is None, and keep the valid Maps and merge them. If every option is None, then the final option should be None as well.
A flatMap along with groupMapReduce, followed by an Option filter should do the job:
val listOfMaps: List[Option[Map[String, Any]]] =
List(Some(Map("a"->"p", "b"->2)), None, Some(Map("a"->"q", "c"->"r")))
val mergedMap = listOfMaps.
flatMap(_.getOrElse(Map.empty[String, Any])).
groupMapReduce(_._1)(t => List[Any](t._2))(_ ::: _)
// mergedMap: Map[String, List[Any]] =
// Map("a" -> List("p", "q"), "b" -> List(2), "c" -> List("r"))
Option(mergedMap).filter(_.nonEmpty)
// res1: Option[Map[String, List[Any]]] =
// Some(Map("a" -> List("p", "q"), "b" -> List(2), "c" -> List("r")))
A few notes:
groupMapReduce is available only on Scala 2.13+.
If you must stick to Seq instead of List, simply replace method ::: with ++ in groupMapReduce.
It is assumed that merging of the Maps means aggregating the Map values of a common key into a List. Replace groupMapReduce with toMap if keeping only one of the Map values of a common key is wanted instead.
This solution treats Some(Map(.empty[String, Any])) the same as None.
This is a one-line solution
val in: Seq[Option[Map[String, Any]]] = ???
val out: Option[Map[String, Any]] =
in.flatten.headOption.map(_ => in.flatten.reduce(_ ++ _))
The in.flatten.headOption is a simple way of getting Some if at least one of the elements is Some or None if they are all None.
The reduce just combines all the Maps into one.
This can clearly be optimised by avoiding the duplicate in.flatten call.
Here is an example with flatten,fold, ++ and a match at the end to provide Some or None.
baz.scala
package baz
object baz {
// fixtures
val x0 = Seq[Option[Map[String, Any]]]()
val x1 = Seq[Option[Map[String, Any]]](None)
val x2 = Seq[Option[Map[String, Any]]](Some(Map("a" -> 1, "b" -> "two")))
val x3 = Seq[Option[Map[String, Any]]](Some(Map("a" -> 1, "b" -> "two")), Some(Map("c" -> 3.0)), None)
def f(x: Seq[Option[Map[String, Any]]]) =
x.flatten.fold(Map[String, Any]())((a,b) => a ++ b) match { case m if m.isEmpty => None case m => Some(m) }
}
Sample run
bash-3.2$ scalac baz.scala && scala -classpath .
Welcome to Scala 2.13.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_66).
Type in expressions for evaluation. Or try :help.
scala> import baz.baz._
import baz.baz._
scala> x0 -> f(x0)
res0: (Seq[Option[Map[String,Any]]], Option[Map[String,Any]]) = (List(),None)
scala> x1 -> f(x1)
res1: (Seq[Option[Map[String,Any]]], Option[Map[String,Any]]) = (List(None),None)
scala> x2 -> f(x2)
res2: (Seq[Option[Map[String,Any]]], Option[Map[String,Any]]) = (List(Some(Map(a -> 1, b -> two))),Some(Map(a -> 1, b -> two)))
scala> x3 -> f(x3)
res3: (Seq[Option[Map[String,Any]]], Option[Map[String,Any]]) = (List(Some(Map(a -> 1, b -> two)), Some(Map(c -> 3.0)), None),Some(Map(a -> 1, b -> two, c -> 3.0)))
scala> :quit
The following code got the two errors on the last map(...). What parameter is missing in the map() function? How to resolve the error of "encoder"?
Error:
Error:(60, 11) Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.
.map(r => Cols(r.getInt(0), r.getString(1), r.getString(2), r.getString(3), r.getDouble(4), r.getDate(5), r.getString(6), r.getString(7), r.getDouble(8), r.getString(9)))
Error:(60, 11) not enough arguments for method map: (implicit evidence$6: org.apache.spark.sql.Encoder[Cols])org.apache.spark.sql.Dataset[Cols].
Unspecified value parameter evidence$6.
.map(r => Cols(r.getInt(0), r.getString(1), r.getString(2), r.getString(3), r.getDouble(4), r.getDate(5), r.getString(6), r.getString(7), r.getDouble(8), r.getString(9)))
Code:
case class Cols (A: Int,
B: String,
C: String,
D: String,
E: Double,
F: Date,
G: String,
H: String,
I: Double,
J: String
)
class SqlData(sqlContext: org.apache.spark.sql.SQLContext, jdbcSqlConn: String) {
def getAll(source: String) = {
sqlContext.read.format("jdbc").options(Map(
"driver" -> "com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url" -> jdbcSqlConn,
"dbtable" -> s"MyFunction('$source')"
)).load()
.select("A", "B", "C", "D", "E", "F", "G", "H", "I", "J")
// The following line(60) got the errors.
.map((r) => Cols(r.getInt(0), r.getString(1), r.getString(2), r.getString(3), r.getDouble(4), r.getDate(5), r.getString(6), r.getString(7), r.getDouble(8), r.getString(9)))
}
}
Update:
I have the following function
def compare(sqlContext: org.apache.spark.sql.SQLContext, dbo: Dataset[Cols], ods: Dataset[Cols]) = {
import sqlContext.implicits._
dbo.map((r) => ods.map((s) => { // Errors occur here
0
}))
and it got the same error.
Why it still has the error after I imported sqlContext.implicits._?
I create a new parameter sqlContext simply for importing. Is there a better way do it?
Combining all the comments into an answer:
def getAll(source: String): Dataset[Cols] = {
import sqlContext.implicits._ // this imports the necessary implicit Encoders
sqlContext.read.format("jdbc").options(Map(
"driver" -> "com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url" -> jdbcSqlConn,
"dbtable" -> s"MyFunction('$source')"
)).load().as[Cols] // shorter way to convert into Cols, thanks #T.Gaweda
}
I have a JSON string, say:
val json = JSONObject(Map("a" -> 1)).toString()
I want to convert this json to map again. I tried:
val map = json.toMap[String, Int]
This gives me the following error:
Error:(46, 25) Cannot prove that Char <:< (String, Int).
val map = json.toMap[String, Int]
^
Error:(46, 25) not enough arguments for method toMap: (implicit ev: <:<[Char,(String, Int)])scala.collection.immutable.Map[String,Int].
Unspecified value parameter ev.
val map = json.toMap[String, Int]
^
What is the correct way of doing this?
val map = json.toMap.asInstanceOf[Map[String, Int]]
Using play-json you can convert Json to Map using validate. Validate returns JsResult which can be JsSuccess(data,path) or JsError(errors). Pattern match to get the map out of it.
If you are using sbt project then add play-json dependency to your project.
build.sbt
libraryDependencies ++= Seq("com.typesafe.play" %% "play-json" % "2.54.")
Scala REPL
scala> import play.api.libs.json._
import play.api.libs.json._
scala> val map = Map("java" -> 1, "scala" -> 2)
map: scala.collection.immutable.Map[String,Int] = Map(java -> 1, scala -> 2)
scala> Json.toJson(map).validate[Map[String, Int]]
res3: play.api.libs.json.JsResult[Map[String,Int]] = JsSuccess(Map(java -> 1, scala -> 2),)
scala> val result = Json.toJson(map).validate[Map[String, Int]]
result: play.api.libs.json.JsResult[Map[String,Int]] = JsSuccess(Map(java -> 1, scala -> 2),)
scala> result match { case JsSuccess(data, _) => data case JsError(errors) => Unit}
res4: Object = Map(java -> 1, scala -> 2)
If this is a one-off then you could do something like this:
val json = JSONObject(Map("a" -> 1)).toString() //> json : String = {"a" : 1}
val map = json.substring(1, json.length - 1).split(":").map(_.trim) match {
case Array(s: String, i: String) => Map(s -> i.toInt)
} //> map : scala.collection.immutable.Map[String,Int] = Map("a" -> 1)
However, if you are doing more with Json then you probably want to explore the use of a library since I think the support for Json in the standard library is limited. See this post for a good overview of available libraries.
Is it possible to create an object from a Map[String, Any] where each pair is a (StringField -> Value) ?
For example,
case class Example(a: String, b: Int)
val obj = new Example( Map('a' -> 'blah', 'b' -> 1) ) //?
val orMaybe = Example( Map('a' -> 'blah', 'b' -> 1 ) //?
You can pass a tuple as the parameter list to a function using .tupled:
(Example.apply _).tupled("a" -> 1)
With that, and knowing that myMap.map passes each element of the Map as a Tuple, you could do the following:
Map("a" -> 1).map((Example.apply _).tupled).head
Yes, it's possible in several ways. One of them would be to use reflection, another (a simpler one) would be to use json as an intermediate.
E.g. with json4s:
import org.json4s._
import org.json4s.native.Serialization
import org.json4s.native.Serialization.{read, write}
scala> implicit val formats = DefaultFormats
formats: org.json4s.DefaultFormats.type = org.json4s.DefaultFormats$#1f2bf363
case class Example(a: String, b: Int)
scala> val obj = read[Example](write( Map("a" -> "blah", "b" -> 1) ))
obj: Example = Example(blah,1)
scala: val orMaybe = read[Example](write( Map("a" -> "blah", "b" -> 1 )))
orMaybe: Example = Example(blah,1)
Let's say I have this example case class
case class Test(key1: Int, key2: String, key3: String)
And I have a map
myMap = Map("k1" -> 1, "k2" -> "val2", "k3" -> "val3")
I need to convert this map to my case class in several places of the code, something like this:
myMap.asInstanceOf[Test]
What would be the easiest way of doing that? Can I somehow use implicit for this?
Two ways of doing this elegantly. The first is to use an unapply, the second to use an implicit class (2.10+) with a type class to do the conversion for you.
1) The unapply is the simplest and most straight forward way to write such a conversion. It does not do any "magic" and can readily be found if using an IDE. Do note, doing this sort of thing can clutter your companion object and cause your code to sprout dependencies in places you might not want:
object MyClass{
def unapply(values: Map[String,String]) = try{
Some(MyClass(values("key").toInteger, values("next").toFloat))
} catch{
case NonFatal(ex) => None
}
}
Which could be used like this:
val MyClass(myInstance) = myMap
be careful, as it would throw an exception if not matched completely.
2) Doing an implicit class with a type class creates more boilerplate for you but also allows a lot of room to expand the same pattern to apply to other case classes:
implicit class Map2Class(values: Map[String,String]){
def convert[A](implicit mapper: MapConvert[A]) = mapper conv (values)
}
trait MapConvert[A]{
def conv(values: Map[String,String]): A
}
and as an example you'd do something like this:
object MyObject{
implicit val new MapConvert[MyObject]{
def conv(values: Map[String, String]) = MyObject(values("key").toInt, values("foo").toFloat)
}
}
which could then be used just as you had described above:
val myInstance = myMap.convert[MyObject]
throwing an exception if no conversion could be made. Using this pattern converting between a Map[String, String] to any object would require just another implicit (and that implicit to be in scope.)
Here is an alternative non-boilerplate method that uses Scala reflection (Scala 2.10 and above) and doesn't require a separately compiled module:
import org.specs2.mutable.Specification
import scala.reflect._
import scala.reflect.runtime.universe._
case class Test(t: String, ot: Option[String])
package object ccFromMap {
def fromMap[T: TypeTag: ClassTag](m: Map[String,_]) = {
val rm = runtimeMirror(classTag[T].runtimeClass.getClassLoader)
val classTest = typeOf[T].typeSymbol.asClass
val classMirror = rm.reflectClass(classTest)
val constructor = typeOf[T].decl(termNames.CONSTRUCTOR).asMethod
val constructorMirror = classMirror.reflectConstructor(constructor)
val constructorArgs = constructor.paramLists.flatten.map( (param: Symbol) => {
val paramName = param.name.toString
if(param.typeSignature <:< typeOf[Option[Any]])
m.get(paramName)
else
m.get(paramName).getOrElse(throw new IllegalArgumentException("Map is missing required parameter named " + paramName))
})
constructorMirror(constructorArgs:_*).asInstanceOf[T]
}
}
class CaseClassFromMapSpec extends Specification {
"case class" should {
"be constructable from a Map" in {
import ccFromMap._
fromMap[Test](Map("t" -> "test", "ot" -> "test2")) === Test("test", Some("test2"))
fromMap[Test](Map("t" -> "test")) === Test("test", None)
}
}
}
Jonathan Chow implements a Scala macro (designed for Scala 2.11) that generalizes this behavior and eliminates the boilerplate.
http://blog.echo.sh/post/65955606729/exploring-scala-macros-map-to-case-class-conversion
import scala.reflect.macros.Context
trait Mappable[T] {
def toMap(t: T): Map[String, Any]
def fromMap(map: Map[String, Any]): T
}
object Mappable {
implicit def materializeMappable[T]: Mappable[T] = macro materializeMappableImpl[T]
def materializeMappableImpl[T: c.WeakTypeTag](c: Context): c.Expr[Mappable[T]] = {
import c.universe._
val tpe = weakTypeOf[T]
val companion = tpe.typeSymbol.companionSymbol
val fields = tpe.declarations.collectFirst {
case m: MethodSymbol if m.isPrimaryConstructor ⇒ m
}.get.paramss.head
val (toMapParams, fromMapParams) = fields.map { field ⇒
val name = field.name
val decoded = name.decoded
val returnType = tpe.declaration(name).typeSignature
(q"$decoded → t.$name", q"map($decoded).asInstanceOf[$returnType]")
}.unzip
c.Expr[Mappable[T]] { q"""
new Mappable[$tpe] {
def toMap(t: $tpe): Map[String, Any] = Map(..$toMapParams)
def fromMap(map: Map[String, Any]): $tpe = $companion(..$fromMapParams)
}
""" }
}
}
This works well for me,if you use jackson for scala:
def from[T](map: Map[String, Any])(implicit m: Manifest[T]): T = {
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.convertValue(map)
}
Reference from:Convert a Map<String, String> to a POJO
I don't love this code, but I suppose this is possible if you can get the map values into a tuple and then use the tupled constructor for your case class. That would look something like this:
val myMap = Map("k1" -> 1, "k2" -> "val2", "k3" -> "val3")
val params = Some(myMap.map(_._2).toList).flatMap{
case List(a:Int,b:String,c:String) => Some((a,b,c))
case other => None
}
val myCaseClass = params.map(Test.tupled(_))
println(myCaseClass)
You have to be careful to make sure the list of values is exactly 3 elements and that they are the correct types. If not, you end up with a None instead. Like I said, not great, but it shows that it is possible.
commons.mapper.Mappers.mapToBean[CaseClassBean](map)
Details: https://github.com/hank-whu/common4s
Here's an update to Jonathon's answer for Scala 3 (which no longer has TypeTag). Be aware that this won't work for case classes nested inside of other classes. But for top-level case classes it seems to work fine.
import scala.reflect.ClassTag
object Reflect:
def fromMap[T <: Product : ClassTag](m: Map[String, ?]): T =
val classTag = implicitly[ClassTag[T]]
val constructor = classTag.runtimeClass.getDeclaredConstructors.head
val constructorArgs = constructor.getParameters()
.map { param =>
val paramName = param.getName
if (param.getType == classOf[Option[_]])
m.get(paramName)
else
m.get(paramName)
.getOrElse(throw new IllegalArgumentException(s"Missing required parameter: $paramName"))
}
constructor.newInstance(constructorArgs: _*).asInstanceOf[T]
And a test for the above:
case class Foo(a: String, b: Int, c: Option[String] = None)
case class Bar(a: String, b: Int, c: Option[Foo])
class ReflectSuite extends munit.FunSuite:
test("fromMap") {
val m = Map("a" -> "hello", "b" -> 42, "c" -> "world")
val foo = Reflect.fromMap[Foo](m)
assertEquals(foo, Foo("hello", 42, Some("world")))
val n = Map("a" -> "hello", "b" -> 43)
val foo2 = Reflect.fromMap[Foo](n)
assertEquals(foo2, Foo("hello", 43))
val o = Map("a" -> "yo", "b" -> 44, "c" -> foo)
val bar = Reflect.fromMap[Bar](o)
assertEquals(bar, Bar("yo", 44, Some(foo)))
}
test("fromMap should fail when required parameter is missing") {
val m = Map("a" -> "hello", "c" -> "world")
intercept[java.lang.IllegalArgumentException] {
Reflect.fromMap[Foo](m)
}
}