Modify Future[JsArray] in Scala - mongodb

I use the Play 2.0 framework for getting data from my MongoDB.
This is done by the following code:
def getTopicsInAppFormat = Action.async {
// let's do our query
val cursor: Cursor[TopicModel] = topicCollection.find(Json.obj()).cursor[TopicModel]
// gather all the JsObjects in a list
val futureTopicsList: Future[List[TopicModel]] = cursor.collect[List]()
// transform the list into a JsArray
val futurePersonsJsonArray: Future[JsArray] = futureTopicsList.map { topics =>
Json.arr(topics)
}
// everything's ok! Let's reply with the array
futurePersonsJsonArray.map {
topics =>
Ok(topics(0))
}
}
But the problem is that I want the function to return an alternative representation of the data. So, I want for example to change the amount of attributes, etc. What is a good way to achieve that? I tried to modify the data (respectively create a new array with the new format) in the last step, right before the Ok() function. However, I didn't have any progress with that :/
EDIT:
At the moment I'm trying to create new JSON objects but I'm stuck while getting the data from the original one...
My current code looks like this:
futurePersonsJsonArray.map {
topics =>
/* Access a attribute */
println(topics(0).\("content"))
/* However: error: JsUndefined('content' is undefined on object */
/* Could be used to set the new attributes */
val myNewJson = Json.obj(
"name" -> "JohnDoe",
"age" -> "123",
"created" -> new java.util.Date().getTime())
Ok(/*topics(0)*/myNewJson)
}

You're probably misinterpreting the format of topics. The message you're getting is telling you that there is no content property on the first element of the topics array. This is a simplified example:
val myObject = Json.obj("a" -> 1, "b" -> 2)
myObject \ "a" //JsNumber(1)
myObject \ "c" //JsUndefined('c' is undefined on object...)
This makes sense, since we get undefined in Javascript when trying to read a property that doesn't exist. In the Play Json library, \ always returns a JsValue, and one of its subtypes is JsUndefined.
You should reexamine the format of the objects in the topics array and reassess how you can access its values.

Now, my final solution for the problem "modify the data just before returning" looks like this:
futurePersonsJsonArray.map {
topics =>
def createNewFormat(uID: JsValue, name: JsValue, content: JsValue, lat: JsValue, lng: JsValue): JsObject = {
return Json.obj(
"id" -> uID,
[...]
}
/* Access the needed attributes */
val uIDs = topics(0).\\("uID")
val names = topics(0).\\("name")
val content = topics(0).\\("content")
val gps = topics(0).\\("gps")
var dataArray = new JsArray()
/* create new Array */
for( i <- 0 to uIDs.length-1){
dataArray = dataArray.append(createNewFormat(uIDs(i),names(i),content(i),gps(i)(0),gps(i)(1)))
}
/* put this into a new JSObject */
var returnThis = Json.obj("data" -> dataArray)
Ok(returnThis)
}
Maybe this does help someone with similar problems :)

Related

Iterate through complex Scala collections loaded from config file

I've read other threads on SO about iterating through collections from config files in Scala but they always assume the type in question is either a ConfigList or an ObjectList. In my case it is a more complex structure and I could not figure out how to access its elements.
In my config file I need to have a group of tuples without being too finicky on the collection type. For instance, I'm open to use a List of tuples or a Map[String, List], etc. Like the following (from application.conf):
myprogr {
groupsOfSomeStuff
{
group_1 -> {
name = "name1",
url = "url1",
something = "whatever"
},
...,
group_n -> {
name = "namen",
url = "urln",
something = "whatever"
}
}
}
At the moment with the conf file above, I can only print the whole groupsOfSomeStuff but I can not access any of its individual elements:
var conf = ConfigFactory.load()
println(conf.getObject("myprogr.groupsOfSomeStuff"))
which returns:
SimpleConfigObject({"group_1 ->":{"something":"whatever","name":"name1","url":"url1"}, ..., "group_n ->":{"something":"whatever","name":"namen","url":"urln"})
If I try to print conf.getObjectList or conf.getConfList I get an error at run time cause what gets extracted from the conf file is not a list but an object. The same happens if I substitute the "->" in the conf file with ":" or with "=" (since as I wrote, I'm open to different types of collections).
If I try to assign conf.getObject("myprogr.groupsOfSomeStuff") to a var of type SimpleConfigObject (with the intention of iterate through the elements of its "value" Map attribute), I get a compile-time error "SimpleConfigObject is not accessible from this position".
How can I iterate through the group_1, ..., group_n elements and individually access the name, url and something parts of each entry's value?
Thanks a million in advance! ;)
object TestConfig extends App {
import scala.collection.JavaConverters._
case class Foo(name: String, url: String, something: String)
val config = ConfigFactory.parseResources("test.conf")
val path = "myprogr.groupsOfSomeStuff"
val fooList: List[Foo] = config.getObject(path).keySet().asScala.map { key =>
val member = config.getObject(s"$path.$key").toConfig
Foo(member.getString("name"), member.getString("url"), member.getString("something"))
}.toList
println(fooList)
}
It should print List(Foo(name1,url1,whatever), Foo(namen,urln,whatever))
I hope this is what you are trying to do.

Scala/Play/Squeryl Retrieve multiple params

I have the following url : http://localhost/api/books/?bookId=21&bookId=62?authorId=2
I want to retrieve all the bookId values with Scala and then use Squeryl to do a fetch in a the database.
I'm using the PlayFrameWork as the WebServer, so here's my code :
val params = request.queryString.map { case (k, v) => k -> v(0) } // Retrieve only one the first occurence of a param
So params.get("bookId") will only get the last value in the bookId params. e-g : 62.
To retrieve all my bookId params i tried this :
val params = request.queryString.map { case (k, v) => k -> v } so i can get a Seq[String], but what about the authorId which is not a Seq[String]? .
At the end i want to fetch the bookIds and authorId in my DB using Squeryl :
(a.author_id === params.get("authorId").?) and
(params.get("bookId").map(bookIds: Seq[String] => b.bookId in bookIds))
In my controller i get the params and open the DB connection :
val params = request.queryString.map { case (k, v) => k -> v(0) }
DB.withTransaction() { where(Library.whereHelper(params)}
In my model i use the queries :
def whereHelper(params : Map[String,String]) = {
(a.author_id === params.get("authorId").?) and
(params.get("bookId").map{bookIds: Seq[String] => b.bookId in bookIds})
}
Since bookIds is a list, i need to use the Seq[String]. There's a way to use request.queryString.map { case (k, v) => k -> v } for both a string (authorId) and a list of strings (bookIds) ?
Thanks,
If I really understand what you are trying to do, you want to know how to get the parameters from queryString. This is pretty simple and you can do the following at your controller:
def myAction = Action { request =>
// get all the values from parameter named bookId and
// transforming it to Long. Maybe you don't want the map
// and then you can just remove it.
val bookIds: Seq[Long] = request.queryString("bookId").map(_.toLong)
// Notice that now I'm using getQueryString which is a helper
// method to access a queryString parameter. It returns an
// Option[String] which we are mapping to a Option[Long].
// Again, if you don't need the mapping, just remove it.
val authorId: Option[Long] = request.getQueryString("authorId").map(_.toLong)
DB.withTransaction() { where(Library.whereHelper(authorId, bookIds) }
// Do something with the result
}
At your model you will have:
def whereHelper(authorId: Option[Long], booksId: List[Long]) = authorId match {
case Some(author_id) =>
(a.author_id === author_id) and
(b.bookId in bookIds)
case None =>
(b.bookId in bookIds)
}
I've left explicit types to help you understand what is happen. Now, since you have both values, you can just use the values at your query.
Edit after chat:
But, since you want to receive a params: Map[String, Seq[String]] at your models and is just having problems about how to get the authorId, here is what you can do:
def whereHelper(params: Map[String, Seq[String]]) = {
// Here I'm being defensive to the fact that maybe there is no
// "booksIds" key at the map. So, if there is not, an Seq.empty
// will be returned. map method will run only if there is something
// at the Seq.
val booksIds = params.getOrElse("booksIds", Seq.empty).map(_.toLong)
// The same defensive approach is being used here, and also getting
// the head as an Option, so if the Seq is empty, a None will be
// returned. Again, the map will be executed only if the Option
// is a Some, returning another Some with the value as a Long.
val authorId = params.getOrElse("authorId", Seq.empty).headOption
authorId.map(_.toLong) match {
case Some(author_id) =>
(a.author_id === author_id) and
(b.bookId in booksIds)
case None =>
(b.bookId in booksIds)
}
}
Of course, more parameters you have, more complicated this method will be.

Specs2 JSONMatchers: mapping over Array elements?

I'm using the Specs2 JSONMatcher to validate that a GET request is being correctly converted from its internal representation (there are some manipulations we do before generating the JSON). What I need to do is, make sure that an element in the JSON array matches the corresponding object from our repository.
What I've tried:
val response = response.entity.asString // Spray's way of getting the JSON response
repository.all.map { obj =>
resp must */ ("id" -> obj.id)
resp must */ ("state" -> generateState(obj)
}
The problem is that the */ matcher just finds that "state": "whatever" (assuming generateState returns "whatever") exists somewhere in the JSON document, not necessarily in the same one matched by the ID
I tried using the indices but the repository.all method doesn't always return the elements in the same order, so there's no way of matching by index.
What I'd like to do is, iterate over the elements of the JSON array and match each one separately. Say an /## operator (or something) that takes matchers for each element:
resp /## { elem =>
val id = elem("id")
val obj = repository.lookup(id)
elem /("state" -> generateState(obj))
}
Does anyone have a way to do this or something similar?
Probably the easiest thing to do for now (until a serious refactoring of JsonMatchers) is to do some parsing and recursively use a JsonMatchers in a Matcher[String]:
"""{'db' : { 'id' : '1', 'state' : 'ok_1'} }""" must /("db" -> stateIsOk)
// a string matcher for the json string in 'db'
def stateIsOk: Matcher[String] = { json: String =>
// you need to provide a way to access the 'id' field
// then you can go on using a json matcher for the state
findId(json) must beSome { id: String =>
val obj = repository.lookup(id)
json must /("state" -> generate(obj))
}
}
// I am using my own parse function here
def findId(json: String): Option[String] =
parse(json).flatMap { a =>
findDeep("id", a).collect { case JSONArray(List(v)) => v.toString }
}
// dummy system
def generate(id: String) = "ok_"+id
case object repository {
def lookup(id: String) = id
}
What I did in the end is use responseAs[JArray], JArray#arr and JObject#values to convert the JSON structures into Lists and Maps, and then used the standard List and Map matchers. Much more flexible.

Play reactivemongo doesnt stop

In my play/swagger/reactivemongo application i use the following function in the controller to get a list of results with "EntityID" 8.
#ApiOperation(value = "Gets the item of a specific ID", notes = "Returns an Item", responseClass = "Item", httpMethod = "GET")
#ApiErrors(Array(
new ApiError(code = 400, reason = "Invalid ID supplied"),
new ApiError(code = 404, reason = "Item not found")))
def index = Action { implicit request =>
Async {
// test id
var myVar: Int = 8
val cursor: Cursor[JsObject] = collection.find(Json.obj("EntityID" -> myVar))
.sort(Json.obj("_id" -> -1))
.cursor[JsObject]
// gather all the JsObjects in a list
val futureUsersList: Future[List[JsObject]] = cursor.toList
// transform the list into a JsArray
val futureUsersListJs: Future[JsArray] = futureUsersList.map { Measurements_live =>
Json.arr(Measurements_live)
}
futureUsersListJs.map { Measurements_live =>
Ok(Measurements_live)
}
}
}
The measurements model:
case class Measurements_live(
EntityID: Int,
SensorID: Int,
Datetime: Date,
Value: String)
object JsonFormats {
import play.api.libs.json.Json
import play.api.data._
import play.api.data.Forms._
implicit val measureFormat = Json.format[Measurements_live]
val measureForm = Form(
mapping(
"EntityID" -> number,
"SensorID" -> number,
"Datetime" -> date,
"Value" -> text)(Measurements_live.apply _)(Measurements_live.unapply _))
}
The problems is that it wont stop loading. There are a total of 35000 objects in the database.
I've played with cursor.close() that stops the cursor and that will return some results. What i want is that the cursor automatically closes when all results are returned.
My first instinct is "that's a lot of objects to be fetching in one HTTP request!".
I can't say exactly why this doesn't work but I suspect it's the collating into a Future[List[JsObject]] that is killing the thing.
I would use the version of Cursor.toList() that takes an upTo parameter, set that to a reasonable number (say 100) and then allow the client to specify which page of results they want to see as a parameter. You could even let the client set the page size if you trust them not to get too greedy.
Obviously you'll need to add another clause to your find to get the correct offset for the pagination; if your client just tells you the highest _id value that they've already got, your find() query would still be very simple.

How to query with '$in' over '_id' in reactive mongo and play

I have a project set up with playframework 2.2.0 and play2-reactivemongo 0.10.0-SNAPSHOT. I'd like to query for few documents by their ids, in a fashion similar to this:
def usersCollection = db.collection[JSONCollection]("users")
val ids: List[String] = /* fetched from somewhere else */
val query = ??
val users = usersCollection.find(query).cursor[User].collect[List]()
As a query I tried:
Json.obj("_id" -> Json.obj("$in" -> ids)) // 1
Json.obj("_id.$oid" -> Json.obj("$in" -> ids)) // 2
Json.obj("_id" -> Json.obj("$oid" -> Json.obj("$in" -> ids))) // 3
for which first and second return empty lists and the third fails with error assertion 10068 invalid operator: $oid.
NOTE: copy of my response on the ReactiveMongo mailing list.
First, sorry for the delay of my answer, I may have missed your question.
Play-ReactiveMongo cannot guess on its own that the values of a Json array are ObjectIds. That's why you have to make a Json object for each id that looks like this: {"$oid": "526fda0f9205b10c00c82e34"}. When the ReactiveMongo Play plugin sees an object which first field is $oid, it treats it as an ObjectId so that the driver can send the right type for this value (BSONObjectID in this case.)
This is a more general problem actually: the JSON format does not match exactly the BSON one. That's the case for numeric types (BSONInteger, BSONLong, BSONDouble), BSONRegex, BSONDateTime, and BSONObjectID. You may find more detailed information in the MongoDB documentation: http://docs.mongodb.org/manual/reference/mongodb-extended-json/ .
I managed to solve it with:
val objectIds = ids.map(id => Json.obj("$oid" -> id))
val query = Json.obj("_id" -> Json.obj("$in" -> objectIds))
usersCollection.find(query).cursor[User].collect[List]()
since play-reactivemongo format considers BSONObjectID only when "$oid" is followed by string
implicit object BSONObjectIDFormat extends PartialFormat[BSONObjectID] {
def partialReads: PartialFunction[JsValue, JsResult[BSONObjectID]] = {
case JsObject(("$oid", JsString(v)) +: Nil) => JsSuccess(BSONObjectID(v))
}
val partialWrites: PartialFunction[BSONValue, JsValue] = {
case oid: BSONObjectID => Json.obj("$oid" -> oid.stringify)
}
}
Still, I hope there is a cleaner solution. If not, I guess it makes it a nice pull request.
I'm wondering if transforming id to BSONObjectID isn't more secure this way :
val ids: List[String] = ???
val bsonObjectIds = ids.map(BSONObjectID.parse(_)).collect{case Success(t) => t}
this will only generate valid BSONObjectIDs (and discard invalid ones)
If you do it this way :
val objectIds = ids.map(id => Json.obj("$oid" -> id))
your objectIds may not be valid ones depending on string id really being the stringify version of a BSONObjectID or not
If you import play.modules.reactivemongo.json._ it work without any $oid formatters.
import play.modules.reactivemongo.json._
...
val ids: Seq[BSONObjectID] = ???
val selector = Json.obj("_id" -> Json.obj("$in" -> ids))
usersCollection.find(selector).cursor[User].collect[Seq]()
I tried with the following and it worked for me:
val listOfItems = BSONArray(51, 61)
val query = BSONDocument("_id" -> BSONDocument("$in" -> listOfItems))
val ruleListFuture = bsonFutureColl.flatMap(_.find(query, Option.empty[BSONDocument]).cursor[ResponseAccDataBean]().
collect[List](-1, Cursor.FailOnError[List[ResponseAccDataBean]]()))