I'm trying to figure out how to get Play's toJSon method to serialize java.sql.Timestamp/java.sql.Date objects as date/time strings rather than seconds since epoch, which seems to be the default. I've tried two methods to accomplish this:
1) I changed the Jackson JSON configuration default as seen here in the Global onStart handler:
import play.api._
import play.libs.Json
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.SerializationFeature
object Global extends GlobalSettings {
override def onStart(app: play.api.Application){
println("really started")
var om = new ObjectMapper()
om.configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false)
Json.setObjectMapper(om)
}
}
But this doesn't seem to have any effect. I can tell the code is executing based on the println statement but the serialization is unaffected.
2) Write a custom Writer for the java.sql.Date object:
implicit val sqlDateWrites: Writes[java.sql.Date] = new Writes[java.sql.Date] {
def writes(d: java.sql.Date): JsValue = JsString("WTF")
}
However this doesn't work either. I'm not sure if it's an error in how I'm writing it, or if I am just including it in the wrong place (I'm declaring it in the same file that I'm calling "toJson" in.
Any help would be appreciated.
https://gist.github.com/fancellu/f4b72e853766acf26bf16a7fb37cb8ac
Give this code a go, stores as ISO-8601 format
You're mixing up Play's Java JSON library and its Scala library.
If you're using Scala, only use play.api.libs.json. If you're in Java, play.libs.Json.
To create a Writes[java.sql.Date], call Writes.sqlDateWrites(pattern) with whatever pattern you're using.
val sqlDateWrite = Writes.sqlDateWrites(myPattern)
Then, when you create your Writes for whatever object you're converting:
case class Foo(id: Long, createdAt: java.sql.Date)
implicit val fooWrites: Writes[Foo] = (
(__ \ "id").write[Long] and
(__ \ "createdAt").write[java.sql.Date](sqlDateWrite)
)(unlift(Foo.unapply))
Related
No instance of play.api.libs.json.Format is available for akka.actor.typed.ActorRef[org.knoldus.eventSourcing.UserState.Confirmation] in the implicit scope (Hint: if declared in the same file, make sure it's declared before)
[error] implicit val userCommand: Format[AddUserCommand] = Json.format
I am getting this error even though I have made Implicit instance of Json Format for AddUserCommand.
Here is my code:
trait UserCommand extends CommandSerializable
object AddUserCommand{
implicit val format: Format[AddUserCommand] = Json.format[AddUserCommand]
}
final case class AddUserCommand(user:User, reply: ActorRef[Confirmation]) extends UserCommand
Can anyone please help me with this error and how to solve it?
As Gael noted, you need to provide a Format for ActorRef[Confirmation]. The complication around this is that the natural serialization, using the ActorRefResolver requires that an ExtendedActorSystem be present, which means that the usual approaches to defining a Format in a companion object won't quite work.
Note that because of the way Lagom does dependency injection, this approach doesn't really work in Lagom: commands in Lagom basically can't use Play JSON.
import akka.actor.typed.scaladsl.adapter.ClassicActorSystemOps
import play.api.libs.json._
class PlayJsonActorRefFormat(system: ExtendedActorSystem) {
def reads[A] = new Reads[ActorRef[A]] {
def reads(jsv: JsValue): JsResult[ActorRef[A]] =
jsv match {
case JsString(s) => JsSuccess(ActorRefResolver(system.toTyped).resolveActorRef(s))
case _ => JsError(Seq(JsPath() -> Seq(JsonValidationError(Seq("ActorRefs are strings"))))) // hopefully parenthesized that right...
}
}
def writes[A] = new Writes[ActorRef[A]] {
def writes(a: ActorRef[A]): JsValue = JsString(ActorRefResolver(system.toTyped).toSerializationFormat(a))
}
def format[A] = Format[ActorRef[A]](reads, writes)
}
You can then define a format for AddUserCommand as
object AddUserCommand {
def format(arf: PlayJsonActorRefFormat): Format[AddUserCommand] = {
implicit def arfmt[A]: Format[ActorRef[A]] = arf.format
Json.format[AddUserCommand]
}
}
Since you're presumably using JSON to serialize the messages sent around a cluster (otherwise, the ActorRef shouldn't be leaking out like this), you would then construct an instance of the format in your Akka Serializer implementation.
(NB: I've only done this with Circe, not Play JSON, but the basic approach is common)
The error says that it cannot construct a Format for AddUserCommand because there's no Format for ActorRef[Confirmation].
When using Json.format[X], all the members of the case class X must have a Format defined.
In your case, you probably don't want to define a formatter for this case class (serializing an ActorRef doesn't make much sense) but rather build another case class with data only.
Edit: See Levi's answer on how to provide a formatter for ActorRef if you really want to send out there the actor reference.
Spark 2.3.0 with Scala 2.11. I'm implementing a custom Aggregator according to the docs here. The aggregator requires 3 types for input, buffer, and output.
My aggregator has to act upon all previous rows in the window so I declared it like this:
case class Foo(...)
object MyAggregator extends Aggregator[Foo, ListBuffer[Foo], Boolean] {
// other override methods
override def bufferEncoder: Encoder[ListBuffer[Mod]] = ???
}
One of the override methods is supposed to return the encoder for the buffer type, which in this case is a ListBuffer. I can't find any suitable encoder for org.apache.spark.sql.Encoders nor any other way to encode this so I don't know what to return here.
I thought of creating a new case class which has a single property of type ListBuffer[Foo] and using that as my buffer class, and then using Encoders.product on that, but I am not sure if that is necessary or if there is something else I am missing. Thanks for any tips.
You should just let Spark SQL do its work and find the proper encoder using ExpressionEncoder as follows:
scala> spark.version
res0: String = 2.3.0
case class Mod(id: Long)
import org.apache.spark.sql.Encoder
import scala.collection.mutable.ListBuffer
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
scala> val enc: Encoder[ListBuffer[Mod]] = ExpressionEncoder()
enc: org.apache.spark.sql.Encoder[scala.collection.mutable.ListBuffer[Mod]] = class[value[0]: array<struct<id:bigint>>]
I cannot see anything in org.apache.spark.sql.Encoders that could be used to directly encode a ListBuffer, or for that matter even a List
One option seems to be going with putting it in a case class, as you suggested:
import org.apache.spark.sql.Encoders
case class Foo(field: String)
case class Wrapper(lb: scala.collection.mutable.ListBuffer[Foo])
Encoders.product[Wrapper]
Another option could be to use kryo:
Encoders.kryo[scala.collection.mutable.ListBuffer[Foo]]
Or finally you could look at ExpressionEncoders, which extend Encoder:
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
ExpressionEncoder[scala.collection.mutable.ListBuffer[Foo]]
This is the best solution, as it keeps everything transparent to catalyst and therefore allows it to do all of its wonderful optimisations.
One thing I noticed whilst having a play:
ExpressionEncoder[scala.collection.mutable.ListBuffer[Foo]].schema == ExpressionEncoder[List[Foo]].schema
I haven't tested any of the above whilst executing aggregations, so there may be runtime issues. Hope this is helpful.
I am working a on Flink project and would like to parse the source JSON string data to Json Object. I am using jackson-module-scala for the JSON parsing. However, I encountered some issues with using the JSON parser within Flink APIs (map for example).
Here are some examples of the code, and I cannot understand the reason under the hood why it is behaving like this.
Situation 1:
In this case, I am doing what the jackson-module-scala's official exmaple code told me to do:
create a new ObjectMapper
register the DefaultScalaModule
DefaultScalaModule is a Scala object that includes support for all currently supported Scala data types.
call the readValue in order to parse the JSON to Map
The error I got is: org.apache.flink.api.common.InvalidProgramException:Task not serializable.
object JsonProcessing {
def main(args: Array[String]) {
// set up the execution environment
val env = StreamExecutionEnvironment.getExecutionEnvironment
// get input data
val text = env.readTextFile("xxx")
val mapper = new ObjectMapper
mapper.registerModule(DefaultScalaModule)
val counts = text.map(mapper.readValue(_, classOf[Map[String, String]]))
// execute and print result
counts.print()
env.execute("JsonProcessing")
}
}
Situation 2:
Then I did some Google, and came up with the following solution, where registerModule is moved into the map function.
val mapper = new ObjectMapper
val counts = text.map(l => {
mapper.registerModule(DefaultScalaModule)
mapper.readValue(l, classOf[Map[String, String]])
})
However, what I am not able to understand is: why this is going to work, with calling method of outside-defined object mapper? Is it because the ObjectMapper itself is Serializable as stated here ObjectMapper.java#L114?
Now, the JSON parsing is working fine, but every time, I have to call mapper.registerModule(DefaultScalaModule) which I think could cause some performance issue (Does it?). I also tried another solution as follows.
Situation 3:
I created a new case class Jsen, and use it as the corresponding parsing class, registering the Scala modules. And it is also working fine.
However, this is not so flexible if your input JSON is varying very often. It is not maintainable to manage the class Jsen.
case class Jsen(
#JsonProperty("a") a: String,
#JsonProperty("c") c: String,
#JsonProperty("e") e: String
)
object JsonProcessing {
def main(args: Array[String]) {
...
val mapper = new ObjectMapper
val counts = text.map(mapper.readValue(_, classOf[Jsen]))
...
}
Additionally, I also tried using JsonNode without calling registerModule as follows:
...
val mapper = new ObjectMapper
val counts = text.map(mapper.readValue(_, classOf[JsonNode]))
...
It is working fine as well.
My main question is: what is actually causing the problem of Task not serializable under the hood of registerModule(DefaultScalaModule)?
How to identify whether your code could potentially cause this unserializable problem during coding?
The thing is that Apache Flink is designed to be distributed. It means that it needs to be able to run your code remotely. So it means that all your processing functions should be serializable. In the current implementation this is ensure early on when you build your streaming process even if you will not run this in any distributed mode. This is a trade-off with an obvious benefit of providing you feedback down to the very line that breaks this contract (via exception stack trace).
So when you write
val counts = text.map(mapper.readValue(_, classOf[Map[String, String]]))
what you actually write is something like
val counts = text.map(new Function1[String, Map[String, String]] {
val capturedMapper = mapper
override def apply(param: String) = capturedMapper.readValue(param, classOf[Map[String, String]])
})
The important thing here is that you capture the mapper from the outside context and store it as a part of your Function1 object that has to be serializble. And this means that the mapper has to be serializable. The designers of Jackson library recognized that kind of a need and since there is nothing fundamentally non-serizliable in a mapper they made their ObjectMapper and the default Modules serializable. Unfortunately for you the designers of Scala Jackson Module missed that and made their DefaultScalaModule deeply non-serialiazable by making ScalaTypeModifier and all sub-classes non-serializable. This is why your second code works while the first one doesn't: "raw" ObjectMapper is serializable while ObjectMapper with pre-registered DefaultScalaModule is not.
There are a few possible workarounds. Probably the easiest one is to wrap ObjectMapper
object MapperWrapper extends java.io.Serializable {
// this lazy is the important trick here
// #transient adds some safety in current Scala (see also Update section)
#transient lazy val mapper = {
val mapper = new ObjectMapper
mapper.registerModule(DefaultScalaModule)
mapper
}
def readValue[T](content: String, valueType: Class[T]): T = mapper.readValue(content, valueType)
}
and then use it as
val counts = text.map(MapperWrapper.readValue(_, classOf[Map[String, String]]))
This lazy trick works because although an instance of DefaultScalaModule is not serializable, the function to create an instance of DefaultScalaModule is.
Update: what about #transient?
what are the differences here, if I add lazy val vs. #transient lazy val?
This is actually a tricky question. What the lazy val is compiled to is actually something like this:
object MapperWrapper extends java.io.Serializable {
// #transient is set or not set for both fields depending on its presence at "lazy val"
[#transient] private var mapperValue: ObjectMapper = null
[#transient] #volatile private var mapperInitialized = false
def mapper: ObjectMapper = {
if (!mapperInitialized) {
this.synchronized {
val mapper = new ObjectMapper
mapper.registerModule(DefaultScalaModule)
mapperValue = mapper
mapperInitialized = true
}
}
mapperValue
}
def readValue[T](content: String, valueType: Class[T]): T = mapper.readValue(content, valueType)
}
where #transient on the lazy val affects both backing fields. So now you can see why lazy val trick works:
locally it works because it delays initialization of the mapperValue field until first access to the mapper method so the field is safe null when the serialization check is performed
remotely it works because MapperWrapper is fully serializable and the logic of how lazy val should be initialized is put into a method of the same class (see def mapper).
Note however that AFAIK this behavior of how lazy val is compiled is an implementation detail of the current Scala compiler rather than a part of the Scala specification. If at some later point a class similar to .Net Lazy will be added to the Java standard library, Scala compiler potentially might start generating different code. This is important because it provides a kind of trade-off for #transient. The benefit of adding #transient now is that it ensures that code like this works as well:
val someJson:String = "..."
val something:Something = MapperWrapper.readValue(someJson:String, ...)
val counts = text.map(MapperWrapper.readValue(_, classOf[Map[String, String]]))
Without #transient the code above will fail because we forced initialization of the lazy backing field and now it contains a non-serializable value. With #transient this is not an issue as that field will not be serialized at all.
A potential drawback of #transient is that if Scala changes the way code for lazy val is generated and the field is marked as #transient, it might actually be not de-serialized in the remote-work scenario.
Also there is a trick with object because for objects Scala compiler generates custom de-serialization logic (overrides readResolve) to return the same singleton object. It means that the object including the lazy val is not really de-serialized and the value from the object itself is used. It means that #transient lazy val inside object is much more future-proof than inside class in remote scenario.
I have two case classes. The main one, Request, contains two maps.
The first map has a string for both key and value.
The second map has a string key, and value which is an instance of the second case class, KVMapList.
case class Request (var parameters:MutableMap[String, String] = MutableMap[String, String](), var deps:MutableMap[String, KVMapList] = MutableMap[String, KVMapList]())
case class KVMapList(kvMap:MutableMap[String, String], list:ListBuffer[MutableMap[String, String]])
The requirement is to transform Request into a JSON representation.
The following code is trying to do this:
import com.fasterxml.jackson.annotation.PropertyAccessor
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
import com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility
import com.fasterxml.jackson.databind.ObjectMapper
def test(req:Request):String {
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.setVisibility(PropertyAccessor.ALL, Visibility.ANY)
var jsonInString: String = null
try {
jsonInString = mapper.writeValueAsString(request)
}
catch {
=case e: IOException => {
e.printStackTrace
}
jsonString
}
This however is not working. Even when the Request class is populated, the output is :
{"parameters":{"underlying":{"some-value":""},"empty":false,"traversableAgain":true},"deps":{"sizeMapDefined":false,"empty":false,"traversableAgain":true}}
Using the JSON object mapper with corresponding Java classes is straightforward, but have not yet got it working in Scala. Any assistance is very much appreciated.
Jackson is more of a bad old memory in Scala to some degree. You should use a native Scala library for JSON processing, particularly one really good at compile time derivation of JSON serializers, such as circe.
I'm aware this doesn't directly answer your question, but after using circe I would never go back to anything else.
import io.circe.generic.auto._
import io.circe.parser._
import io.circe.syntax._
val req = new Request(...)
val json = req.asJson.noSpaces
val reparsed = decode[Request](json)
On a different note, using mutable maps inside case classes is as non-idiomatic as it gets, and it should be quite trivial to implement immutable ops for your maps using the auto-generated copy method.
case class Request(parameters: Map[String, String] {
def +(key: String, value: String): Request = {
this.copy(parameters = parameters + (key -> value))
}
}
You should really avoid mutability wherever possible, and it looks like avoiding it here wouldn't be much work at all.
I am not sure what this ScalaObjectMapper does, doesn't look like it is useful.
If you add mapper.registerModule(DefaultScalaModule) in the beginning, it should work ... assuming that by MutableMap you mean mutable.Map, and not some sort of home-made class (because, if you do, you'd have to provide a serializer for it yourself).
(DefaultScalaModule is in jackson-module-scala library. Just add it to your build if you don't already have it).
I'm following along the Play! 2.1 coast-to-coast tutorial at http://mandubian.com/2013/01/13/JSON-Coast-to-Coast/ but cannot get even the most trivial example working.
When I compile my project I get an error:
could not find implicit value for parameter reducer: play.api.libs.functional.Reducer[play.api.libs.json.JsString,B]
My controller code is as follows:
package controllers
import play.api._
import play.api.mvc._
import play.api.libs.json._
import play.api.libs.json.Reads._
import play.api.libs.functional.syntax._
object MyController extends Controller{
val validate = (
(__ \ 'title).json.pick[JsString] and
(__ \ 'desc).json.pick[JsString]
).reduce
def test() = Action { implicit request =>
Ok("test")
}
}
What am I missing to get this working?
The syntax here is not quite right. 'pick' returns a JsValue (the Play! equivalent of valid Json types including String, Array, etc).
To validate multiple json fields you need to use 'pickBranch' which returns a JsObject (which is basically the equivalent of a Map[String, JsValue]). I'm guessing that reduce is a merge operation for several JsObjects.
I actually still haven't found a good use case for 'pick'. The '\' syntax seems to do the equivalent job with less code and clutter.