Using ReactiveMongo 0.11 for Scala 2.11. I have an issue where my queries are failing to descend. The following is my Index and ReactiveMongo query:
collection.indexesManager.ensure(Index(Seq("userId" -> IndexType.Ascending, "lastActivity" -> IndexType.Descending), background = true))
def listEfforts(userId: String, page: Int, pageSize: Int): Future[\/[ErrMsg, List[EffortDesc]]] = {
val query = BSONDocument("userId" -> userId)
val sort = BSONDocument("lastActivity" -> -1)
val skipN = (page - 1) * pageSize
val queryOptions = new QueryOpts(skipN = skipN, batchSizeN = pageSize, flagsN = 0)
collection.find(query).options(queryOptions).sort(sort).cursor[EffortDesc](ReadPreference.primaryPreferred).collect[List](pageSize).flatMap {
case list => Future(\/.right(list))
}
}
What's happening is my results are all ascending, even though my sort variable has been set to -1. lastActivity is a Unix timestamp field in milliseconds. I've tried other debugging issues (like recompiling, etc.)
Any idea what could be causing this? Thanks for your help!
Found the issue. If I put a IndexType.Descending on lastActivity field and then additionally sort as "descending" (via "lastActivity" -> -1) MongoDB will first return a descended sort according to its index and then sort it again.
I'm not sure if this is normal/expected behavior in Mongo but changing -1 to 1 fixed the issue.
Using either
("fieldName", BSONInteger.apply(1))
or
("fieldName", BSONInteger.apply(-1))
works for me.
Related
I am trying to implement Pagination with Reactive Mongo.
def get(page: Int, pageSize: Int, filter: JsObject = Json.obj()): Future[Seq[Thing]] = {
val actualPage = if(page > 1) page else 1
collection
.find(filter)
.options(QueryOpts(skipN = (actualPage - 1) * pageSize, batchSizeN = pageSize))
.cursor[Thing]()
.collect[Seq](pageSize, Cursor.FailOnError[Seq[Thing]]())
}
This works and returns me a sequence of documents filtering by the page and pageSize. The problem is I have no idea how many results match the specified query, so I cant tell how many page's exist.
Is there a way I can also get a $count as part of the same query? Or do I need to resort to querying the database a second time with the below?
collection.count(filter)
Ideally I would change my return type to Future[Page[Seq[Thing]]] where Page would contain the total result count. e.g.
case class Page[T](results: Seq[T], count: Int)
I am trying to get an aggregate in ReactiveMongo 0.12 and Play Framework 2.6 (using JSON collections - not BSON) by filtering dates from a collection called "visitors". A typical document may look like this:
{ "_id": ObjectID("59c33152ca2abb344c575152"), "placeId": ObjectID("59c33152ca2abb344c575152"), "date": ISODate("2017-03-26T00:00:00Z"), "visitors": 1200 }
So from here I want to aggregate this data to get various visitor totals, averages, etc, grouping by placeId (which identifies the place in another collection) and filtering by dates after 15-05-2016.
I've based this on this similar question - without the match it works but with it - it does not. There isn't an error but it just doesn't work:
def getVisitorAggregate(col: JSONCollection) = {
import col.BatchCommands.AggregationFramework.{Group, Match, SumField, AvgField, MinField, MaxField}
val format = new java.text.SimpleDateFormat("dd-MM-YYYY")
val myDate = "15-05-2016"
val parseDate: Date = format.parse(myDate)
val longDate: Long = parseDate.getTime
col.aggregate(
Group(JsString("$placeId"))(
"totalVisitors" -> SumField("visitors"),
"avgVisitors" -> AvgField("visitors"),
"minVisitors" -> MinField("visitors"),
"maxVisitors" -> MaxField("visitors")
),
List(Match(Json.obj("date" -> Json.obj("$gte" -> JsNumber(longDate)))))
)
.map(_.head[VisitorAggregate])
}
I have looked and tested for many hours online and I cannot find the correct syntax but this will be simple for someone who knows I'm sure. Thanks
ISODate is a mongodb type, and Model.aggregate does not cast the arguments, so "date" -> Json.obj("$gte" -> JsNumber(longDate)) is wrong.
You need to use a type that will be converted to the ISODate, I am pretty sure it is not JsNumber.
It is a BSONDateTime type would you use BSON, but you do not.
According to documentation it must be a
JsObject with a $date JsNumber field with the timestamp (milliseconds)
as value
So solution can be (I did not verify):
Match(Json.obj("date" -> Json.obj("$gte" -> Json.obj("$date" -> JsNumber(longDate)))))
I hate to answer my own question here but now that I have figured this out I really want to clarify to others how this is done using Aggregate. Ultimately there were two parts to this question.
1) what is the syntax of querying dates?
As #AndriyKuba mentioned and I had seen in the documentation yet not fully understood; the query is formulated like this:
Json.obj("date" -> Json.obj("$gte" -> Json.obj("$date" -> JsNumber(longDate))))
2) how do I match a query within an Aggregate?
This is more of a question of the order of the query. I was originally trying to use match after grouping and aggregating the data - which is (obviously) only going to filter the data after. As I wanted to first get a date range and then aggregate that data I had to match first - this also meant that some of the syntax had to change accordingly:
def getVisitorAggregate(col: JSONCollection) = {
import col.BatchCommands.AggregationFramework.{Group, Match, SumField, AvgField, MinField, MaxField}
val format = new java.text.SimpleDateFormat("dd-MM-YYYY")
val myDate = "15-05-2016"
val parseDate: Date = format.parse(myDate)
val longDate: Long = parseDate.getTime
col.aggregate(
Match(Json.obj("date" -> Json.obj("$gte" -> Json.obj("$date" -> JsNumber(longDate))))),
List(Group(JsString("$rstId"))(
"totalVisitors" -> SumField("visitors"),
"avgVisitors" -> AvgField("visitors"),
"minVisitors" -> MinField("visitors"),
"maxVisitors" -> MaxField("visitors")
))
)
.map(_.head[VisitorAggregate])
}
Really frustrating that there isn't more documentation out there on using the Play Framework with ReactiveMongo as there are a lot of instances of trying to fathom syntax and logic.
Having some issues getting ReactiveMongo 0.11 to paginate my query. The behavior is that it is returning all results, instead of the page-by-page results.
Here's my query:
def listConvos(userId: String, page: Int, pageSize: Int) = {
val query = BSONDocument("userId" -> userId)
val sort = BSONDocument("lastActivity" -> -1)
val skipN = (page-1) * pageSize
val queryOptions = new QueryOpts(skipN = skipN, batchSizeN = pageSize, flagsN = 0)
collection.find(query).options(queryOptions).sort(sort).cursor[ConvoDesc](ReadPreference.primaryPreferred).collect[List]()
Note that pagination starts at 1. Using the IntelliJ debugger, the values for the variables above are:
userId = "29aosidfj43903p"
query = BSONDocument(<non-empty>)
sort = BSONDocument(<non-empty>)
queryOptions = QueryOpts(10,10,0)
page = 2
pageSize = 10
I can also confirm I've set a compound index on the userId and lastActivity fields. Thanks for your help!
The solution was to also include the page size in the collect method like so:
collection.find(query).options(queryOptions).sort(sort).cursor[ConvoDesc](ReadPreference.primaryPreferred).collect[List](pageSize)
I've probably missed something obvious, but within the ReactiveMongo API (v0.8) how do you set a limit on the number of documents returned by a query?
I'd like to return the single most recent document added to a collection. This is my code so far:
def getLatest()(implicit reader: reactivemongo.bson.handlers.RawBSONReader[T]): Future[Option[T]] = {
collection.find (QueryBuilder(
queryDoc = Some(BSONDocument()),
sortDoc = Some(BSONDocument("_id" -> BSONInteger(-1)))
)).headOption().mapTo[Option[T]]
}
headOption() works to retrieve a single result, but I'm not explicitly using any kind of Mongo limit clause so I'm worried about this query's impact on the DB. Please help me improve this code. Thanks in advance.
In 0.8 you have to set the batchSize option to 1 in order to tell MongoDB to close the database cursor automatically:
val maybedoc = collection.find(BSONDocument(), QueryOpts().batchSize(1)).headOption
// or using QueryBuilder like you do
val maybedoc2 = collection.find (QueryBuilder(
queryDoc = Some(BSONDocument()),
sortDoc = Some(BSONDocument("_id" -> BSONInteger(-1)))
), QueryOpts().batchSize(1)).headOption()
In 0.9 collections have been refactored and greatly simplified. Now you can do this:
val maybedoc = collection.
find(BSONDocument()).
sort(BSONDocument("_id" -> -1)).
one[BSONDocument]
The one[T] method in 0.9 sets the batchSize flag for you and returns an Option[T].
Yes, the headOption() function limits the query to just one result:
def headOption()(implicit ec: ExecutionContext) :Future[Option[T]] = {
collect[Iterable](1).map(_.headOption)
}
https://github.com/zenexity/ReactiveMongo/blob/0.8/src/main/scala/api/cursor.scala#L180
I'm trying to build a list page like the one in the "Computers" sample. My environment is Play 2.0 and PostrgreSQL 9.0
I have the following method in my User object:
def list(page: Int = 0, pageSize: Int = 10, orderBy: Int = 1, filter: String = "%"): Page[User] = {
val offset = pageSize * page
val mode = if (orderBy > 0) "ASC NULLS FIRST" else "DESC NULLS LAST"
Logger.debug("Users.list with params: page[%d] pageSize[%d] orderBy[%d] filter[%s] order[%s]".format(page, pageSize, orderBy, filter, mode))
DB.withConnection {
implicit connection =>
val users = SQL(
"""
select * from publisher
where name ilike {filter}
order by {orderBy} %s
limit {pageSize} offset {offset}
""".format(mode)
).on(
'pageSize -> pageSize,
'offset -> offset,
'filter -> filter,
'orderBy -> scala.math.abs(orderBy)
).as(User.simple *)
val totalRows = SQL(
"""
select count(*) from publisher
where name like {filter}
"""
).on(
'filter -> filter
).as(scalar[Long].single)
Page(users, page, offset, totalRows)
}
}
Doesn't matter which value of 'orderBy' I provide, the order is always based on id of the entities.
The query generated by Anorm is valid PostgreSQL and it works fine when running it against the database directly. But it seems like if Anorm parser was ignoring the order in which the results are returned, and instead returns a list ordered by 'id'.
I've even tried to simplify the query to a "select * from publisher order by 2 ASC/DESC", but nothing is fixed, the ordering is ignored by Anorm on return.
Any suggestion on how to solve this issue?
Thanks to Guillaume on the mailing list of Play I found a workaround.
All placeholders work except the one in order by. The worse part is that when you follow the logs, the driver generates the correct query and PostgreSQL is receiving it. I'm not sure what's the deal, very confusing, but if I remove that placeholder, it just works.
Depressing :(
I solved it like this:
val users = SQL(
"""
select * from publisher
where name ilike {filter}
order by %d %s
limit {pageSize} offset {offset}
""".format(scala.math.abs(orderBy), mode)
).on(
'pageSize -> pageSize,
'offset -> offset,
'filter -> filter
).as(User.simple *)
Now you'll be screaming "SQL INJECTION". Relax. Although it may be possible somehow, orderBy is an integer (which we turn into abs value for more safety). If you try to call the controller that provides orderBy with a string, Play returns a 404 error. So only integers are allowed. And if there is no column corresponding to the given integer, the order by is ignored. So, not ideal, but not so bad.