I'm receiving an JSON graph (payload) from the client, so at the web api I get a JsValue. I simply want to take that, decorate it with a couple of fields and store it in Mongo. Something like this:
case class Plan(_id: ObjectId, name: String, payload: JsValue)
{
"_id" = 12345,
"name" : "test model",
"payload" : {a JSON graph}
}
From JsValue to database...
builder += "payload" -> JSON.parse(Json.stringify(model.payload))
From database back to a JsValue...
payload = Json.parse(dbo.as[MongoDBList]("payload").toString))
While it works to go from JsValue -> String -> MongoDBObject, I have two valid typed objects and I have to use an untyped intermediate format to go from one to another.
If you just want to store the graph as a string "payload", you can of course do that.
You may want to consider using ReactiveMongo instead of Casbah, along with Play-ReactiveMongo, which provides direct-to-JSON capability. I have not used Play-ReactiveMongo.
Related
I have a JSON object that I've transformed that I need to filter down to only a subset of its original keys. I've looked through the docs for the Json object in circe but it doesn't appear to expose any API around filtering the object. Do I have to use a cursor for this? I considered creating a decoder from a case class however my keys have a special character . in them. Here is some more code/data for context.
{
"field.nested.this": "value",
"field.nested.that": "value",
"field.nested.where": "value"
}
What's the best approach to create a new JSON instance that doesn't contain the field.nested.that field?
I'm not sure if this is what you need:
object Circe extends App {
import io.circe._
import io.circe.literal._
import io.circe.syntax._
//I'm using a json literal here.
//If you have a runtime string from an external source
// you would need to parse it with `io.circe.parser.parse` first
val json: Json = json"""
{
"field.nested.this": "value",
"field.nested.that": "value",
"field.nested.where": "value"
}
"""
val maybeJsonFiltered =
json.asObject.map(_.filterKeys(_ != "field.nested.that").asJson)
println(maybeJsonFiltered)
// Some({
// "field.nested.this" : "value",
// "field.nested.where" : "value"
// })
}
Alternatively you could also parse it as a map (json.as[Map[String, String]]) or a custom case class with only the fields you need, and encode them back to json. You will probably need a #JsonKey annotation for all your fields with ..
I am trying to prevent empty values being inserted into my mongoDB collection. The field in question looks like this:
MongoDB Field
"stadiumArr" : [
"Old Trafford",
"El Calderon",
...
]
Sample of (mapped) case class
case class FormData(_id: Option[BSONObjectID], stadiumArr: Option[List[String]], ..)
Sample of Scala form
object MyForm {
val form = Form(
mapping(
"_id" -> ignored(Option.empty[BSONObjectID]),
"stadiumArr" -> optional(list(text)),
...
)(FormData.apply)(FormData.unapply)
)
}
I am also using the Repeated Values functionality in Play Framework like so:
Play Template
#import helper._
#(myForm: Form[models.db.FormData])(implicit request: RequestHeader, messagesProvider: MessagesProvider)
#repeatWithIndex(myForm("stadiumArr"), min = 5) { (stadium, idx) =>
#inputText(stadium, '_label -> ("stadium #" + (idx + 1)))
}
This ensures that whether there are at least 5 values or not in the array; there will still be (at least) 5 input boxes created. However if one (or more) of the input boxes are empty when the form is submitted an empty string is still being added as value in the array, e.g.
"stadiumArr" : [
"Old Trafford",
"El Calderon",
"",
"",
""
]
Based on some other ways of converting types from/to the database; I've tried playing around with a few solutions; such as:
implicit val arrayWrite: Writes[List[String]] = new Writes[List[String]] {
def writes(list: List[String]): JsValue = Json.arr(list.filterNot(_.isEmpty))
}
.. but this isn't working. Any ideas on how to prevent empty values being inserted into the database collection?
Without knowing specific versions or libraries you're using it's hard to give you an answer, but since you linked to play 2.6 documentation I'll assume that's what you're using there. The other assumption I'm going to make is that you're using reactive-mongo library. Whether or not you're using the play plugin for that library or not is the reason why I'm giving you two different answers here:
In that library, with no plugin, you'll have defined a BSONDocumentReader and a BSONDocumentWriter for your case class. This might be auto-generated for you with macros or not, but regardless how you get it, these two classes have useful methods you can use to transform the reads/writes you have to another one. So, let's say I defined a reader and writer for you like this:
import reactivemongo.bson._
case class FormData(_id: Option[BSONObjectID], stadiumArr: Option[List[String]])
implicit val formDataReaderWriter = new BSONDocumentReader[FormData] with BSONDocumentWriter[FormData] {
def read(bson: BSONDocument): FormData = {
FormData(
_id = bson.getAs[BSONObjectID]("_id"),
stadiumArr = bson.getAs[List[String]]("stadiumArr").map(_.filterNot(_.isEmpty))
)
}
def write(formData: FormData) = {
BSONDocument(
"_id" -> formData._id,
"stadiumArr" -> formData.stadiumArr
)
}
}
Great you say, that works! You can see in the reads I went ahead and filtered out any empty strings. So even if it's in the data, it can be cleaned up. That's nice and all, but let's notice I didn't do the same for the writes. I did that so I can show you how to use a useful method called afterWrite. So pretend the reader/writer weren't the same class and were separate, then I can do this:
val initialWriter = new BSONDocumentWriter[FormData] {
def write(formData: FormData) = {
BSONDocument(
"_id" -> formData._id,
"stadiumArr" -> formData.stadiumArr
)
}
}
implicit val cleanWriter = initialWriter.afterWrite { bsonDocument =>
val fixedField = bsonDocument.getAs[List[String]]("stadiumArr").map(_.filterNot(_.isEmpty))
bsonDocument.remove("stadiumArr") ++ BSONDocument("stadiumArr" -> fixedField)
}
Note that cleanWriter is the implicit one, that means when the insert call on the collection happens, it will be the one chosen to be used.
Now, that's all a bunch of work, if you're using the plugin/module for play that lets you use JSONCollections then you can get by with just defining play json Reads and Writes. If you look at the documentation you'll see that the reads trait has a useful map function you can use to transform one Reads into another.
So, you'd have:
val jsonReads = Json.reads[FormData]
implicit val cleanReads = jsonReads.map(formData => formData.copy(stadiumArr = formData.stadiumArr.map(_.filterNot(_.isEmpty))))
And again, because only the clean Reads is implicit, the collection methods for mongo will use that.
NOW, all of that said, doing this at the database level is one thing, but really, I personally think you should be dealing with this at your Form level.
val form = Form(
mapping(
"_id" -> ignored(Option.empty[BSONObjectID]),
"stadiumArr" -> optional(list(text)),
...
)(FormData.apply)(FormData.unapply)
)
Mainly because, surprise surprise, form has a way to deal with this. Specifically, the mapping class itself. If you look there you'll find a transform method you can use to filter out empty values easily. Just call it on the mapping you need to modify, for example:
"stadiumArr" -> optional(
list(text).transform(l => l.filter(_.nonEmpty), l => l.filter(_.nonEmpty))
)
To explain a little more about this method, in case you're not used to reading the signatures in the scaladoc.
def
transform[B](f1: (T) ⇒ B, f2: (B) ⇒ T): Mapping[B]
says that by calling transform on some mapping of type Mapping[T] you can create a new mapping of type Mapping[B]. In order to do this you must provide functions that convert from one to the other. So the code above causes the list mapping (Mapping[List[String]]) to become a Mapping[List[String]] (the type did not change here), but when it does so it removes any empty elements. If I break this code down a little it might be more clear:
def convertFromTtoB(list: List[String]): List[String] = list.filter(_.nonEmpty)
def convertFromBtoT(list: List[String]): List[String] = list.filter(_.nonEmpty)
...
list(text).transform(convertFromTtoB, convertFromBtoT)
You might wondering why you need to provide both, the reason is because when you call Form.fill and the form is populated with values, the second method will be called so that the data goes into the format the play form is expecting. This is more obvious if the type actually changes. For example, if you had a text area where people could enter CSV but you wanted to map it to a form model that had a proper List[String] you might do something like:
def convertFromTtoB(raw: String): List[String] = raw.split(",").filter(_.nonEmpty)
def convertFromBtoT(list: List[String]): String = list.mkString(",")
...
text.transform(convertFromTtoB, convertFromBtoT)
Note that when I've done this in the past sometimes I've had to write a separate method and just pass it in if I didn't want to fully specify all the types, but you should be able to work from here given the documentation and type signature for the transform method on mapping.
The reason I suggest doing this in the form binding is because the form/controller should be the one with the concern of dealing with your user data and cleaning things up I think. But you can always have multiple layers of cleaning and whatnot, it's not bad to be safe!
I've gone for this (which always seems obvious when it's written and tested):
implicit val arrayWrite: Writes[List[String]] = new Writes[List[String]] {
def writes(list: List[String]): JsValue = Json.toJson(list.filterNot(_.isEmpty).toIndexedSeq)
}
But I would be interested to know how to
.map the existing Reads rather than redefining from scratch
as #cchantep suggests
I am trying to parse MongoDB Extended JSON using Circe JSON Parser, works fine in most of the cases except for special datatypes, for eg. in below case class i have priorityOrder which is of long datatype.
case class relinfo(id:String,assetId:String,insureeId:String,queue:String,priorityOrder:Long) extends baseDomain
But when it is converted to MongoDB JSON format, it is converted to special mongo format described below (check priorityOrder field)
{
"_id" : "4abf009d-64b1-496c-b0e8-9061f5e183a0",
"id" : "4abf009d-64b1-496c-b0e8-9061f5e183a0",
"assetId" : "e26d5310-ab0c-4672-9971-4babd3420302",
"insureeId" : "cdee05a1-a09c-4e10-81df-c3f112298cc3",
"queue" : "Low",
"priorityOrder" : {
"$numberLong" : "1930926795621"
}
}
The challenge is during de-serialization process, if i try to take this JSON and convert back to concrete object type using circe parser then it fails to map priorityOrder attribute, is there any way i can write custom encoder/decoder that will treat long data type in a special manner. The custom encoder/decoder will read value from "$numberLong" nested type and convert that value to Long datatype.
I get this exception from circe parser
Left(DecodingFailure(Long, List(El(DownField(priorityOrder),true,false))))
I was able to figure out solution to this problem by creating custom decoder for long datatype. Here is the code for individuals in similar boat
implicit val decodeLong: Decoder[Long] = new Decoder[Long] {
final def apply(c: HCursor): Decoder.Result[Long] =
{
val longval = c.downField("$numberLong").as[String] match
{
case Right(x) => x.toLong
case _ => throw new Exception("Unable to find $numberLong")
}
Right(longval)
}
}
I'm new to play, scala, and reactivemongo and was wondering if someone could explain to me the following code in easy terms to understand.
def createFromJson = Action.async(parse.json) { request =>
import play.api.libs.json.Reads._
val transformer: Reads[JsObject] =
Reads.jsPickBranch[JsString](__ \ "name") and
Reads.jsPickBranch[JsNumber](__ \ "age") and
Reads.jsPut(__ \ "created", JsNumber(new java.util.Date().getTime())) reduce
request.body.transform(transformer).map { result =>
collection.insert(result).map { lastError =>
Logger.debug(s"Successfully inserted with LastError: $lastError")
Created
}
}.getOrElse(Future.successful(BadRequest("invalid json")))}
I know that it creates a user from a JSON user with name and age attributes. What I don't understand is the way that input JSON is read in this method. ALSO the concept of Action.async(par.json), request => getorElse, Future, etc.
ALSO any easier/simpler ways of writing this method would be greatly appreciated.
Thanks in advance!
I believe you found this code in a template I have made following excellent reactive mongo documentation.
http://reactivemongo.org/releases/0.11/documentation/index.html
http://reactivemongo.org/releases/0.11/documentation/tutorial/play2.html
I feel a bit obliged to explain it. Let's run through the code.
def createFromJson = Action.async(parse.json) { request =>
The function createFromJson will return an Action (play stuff) that is asynchronous (returns a future of a result) that handles a body in json format. To do that it will use the request.
Documentation: https://www.playframework.com/documentation/2.5.x/ScalaAsync
A json can be anything that follows the json formats, for example an array a String, an object, ...
Our transformer is going to take only the data that we are interested in from the json and will return a clean json object
val transformer: Reads[JsObject] =
Reads.jsPickBranch[JsString](__ \ "name") and
Reads.jsPickBranch[JsNumber](__ \ "age") and
Reads.jsPut(__ \ "created", JsNumber(new java.util.Date().getTime())) reduce
As you see, it will pick the branch name as a string and the branch age as a number. It will also add to the final json object a field created with the time of creation.
As you see we are not transforming it to a Person instance, it is just a JsObject instance as it is defined in
val transformer: Reads[JsObject] ....
Play offers you a few ways to handle json in a simpler way. This examples tries to show the power of manipulating directly the json values without converting to a model.
For example if you have a case class
case class Person(name: String, age: Int)
You could create automatically a reads from it,
val personReads: Person[Person] = Json.reads[Person]
But to just store it in Mongo
DB there is no reason to build this instance and then transform it to json again.
Of course if you need to do some logic with the models before inserting them, you might need to create the model.
Documentation:
https://www.playframework.com/documentation/2.5.x/ScalaJson
https://www.playframework.com/documentation/2.5.x/ScalaJsonCombinators
https://www.playframework.com/documentation/2.5.x/ScalaJsonAutomated
https://www.playframework.com/documentation/2.5.x/ScalaJsonTransformers
With this in mind the rest of the code should be clear
request.body.transform(transformer).map { result =>
collection.insert(result).map { lastError =>
Logger.debug(s"Successfully inserted with LastError: $lastError")
Created
}
}
From the request, we take the body (a JsValue) we transform it into a JsObject (result) and we insert it in the collection.
Insert returns a Future with the last error, when the Person is stored last error will be logged and a Created (201 code) will be returned to the client of the API.
The last bit should be also clear by now
}.getOrElse(Future.successful(BadRequest("invalid json")))
If there is any problem parsing and transforming the json body of the request into our JsObject an already completed future with the result BadRequest (400 code) will be returned to the client.
It is a future because Action.Async needs future of result as the return type.
Enjoy scala.
I have a class which I want to begin indexing into ElasticSearch using the Scala client elastic4s. I have extended DocumentMap to allow me to insert the documents. The simple values like String, Int etc are working but I cannot seem to get a List of another class to map correctly.
The documents look similar to this:
case class AThing(UserName: String, Comment: String, Time: String)
extends DocumentMap {
override def map: Map[String, Any] = Map(
"UserName" -> UserName,
"Comment" -> Comment,
"Time" -> Time
)
}
case class ThingsThatHappened(Id: String, Things: Seq[AThing] = Nil)
extends DocumentMap {
override def map: Map[String, Any] = Map(
"Id" -> Id,
"Things" -> Things
)
}
It will map the Id field fine within elasticsearch but then I get a an incorrect value which looks similar to this, when the document is inserted into elasticsearch:
List(AThing(id_for_the_thing,user_name_a,typed_in_comment,2015-03-12))
Obviously this is wrong and I am expecting something a kin to this JSON structure once it has been inserted into elasticsearch, such as:
"events" : [
{
"UserName" :"user_name_a",
"Comment": "typed_in_comment",
"Time": "2015-03-12"
}
]
Does anyone know a way to map an array of complex types when indexing data using elastic4s?
Elastic4s or the java client (currently) isn't smart enough to figure out that you have a nested sequence or array, but it would work if it was a nested java map (still a bit rubbish from the Scala point of view).
I think the best thing to do is use the new Indexable typeclass that was added in 1.4.13
So, given
case class AThing(UserName: String, Comment: String, Time: String)
Then create a type class and bring it into scope
implicit object AThingIndexable extends Indexable[AThing] {
def json = ... create json here using Jackson or similar which will handle nested sequences properly
}
Then you should be able to do:
client.execute { index into "myIndex/AThings" source aThing }
It's not quite as automatic as using the DocumentMap but gives you more control.
See a unit test here with it in action
First of all you need to create index in elastic4s. I assume you did this.
client.execute {
create index "myIndex" mappings (
"AThings" as(
"UserName" typed StringType,
"Comemnt" typed StringType,
"Time" typed StringType,
)
)
}
if you create this index, then you can put case class into this directly.
val aThings = AThings("username","comment","time")
client.execute {index into "myIndex/AThings" doc aThings}