I am trying to get an aggregate in ReactiveMongo 0.12 and Play Framework 2.6 (using JSON collections - not BSON) by filtering dates from a collection called "visitors". A typical document may look like this:
{ "_id": ObjectID("59c33152ca2abb344c575152"), "placeId": ObjectID("59c33152ca2abb344c575152"), "date": ISODate("2017-03-26T00:00:00Z"), "visitors": 1200 }
So from here I want to aggregate this data to get various visitor totals, averages, etc, grouping by placeId (which identifies the place in another collection) and filtering by dates after 15-05-2016.
I've based this on this similar question - without the match it works but with it - it does not. There isn't an error but it just doesn't work:
def getVisitorAggregate(col: JSONCollection) = {
import col.BatchCommands.AggregationFramework.{Group, Match, SumField, AvgField, MinField, MaxField}
val format = new java.text.SimpleDateFormat("dd-MM-YYYY")
val myDate = "15-05-2016"
val parseDate: Date = format.parse(myDate)
val longDate: Long = parseDate.getTime
col.aggregate(
Group(JsString("$placeId"))(
"totalVisitors" -> SumField("visitors"),
"avgVisitors" -> AvgField("visitors"),
"minVisitors" -> MinField("visitors"),
"maxVisitors" -> MaxField("visitors")
),
List(Match(Json.obj("date" -> Json.obj("$gte" -> JsNumber(longDate)))))
)
.map(_.head[VisitorAggregate])
}
I have looked and tested for many hours online and I cannot find the correct syntax but this will be simple for someone who knows I'm sure. Thanks
ISODate is a mongodb type, and Model.aggregate does not cast the arguments, so "date" -> Json.obj("$gte" -> JsNumber(longDate)) is wrong.
You need to use a type that will be converted to the ISODate, I am pretty sure it is not JsNumber.
It is a BSONDateTime type would you use BSON, but you do not.
According to documentation it must be a
JsObject with a $date JsNumber field with the timestamp (milliseconds)
as value
So solution can be (I did not verify):
Match(Json.obj("date" -> Json.obj("$gte" -> Json.obj("$date" -> JsNumber(longDate)))))
I hate to answer my own question here but now that I have figured this out I really want to clarify to others how this is done using Aggregate. Ultimately there were two parts to this question.
1) what is the syntax of querying dates?
As #AndriyKuba mentioned and I had seen in the documentation yet not fully understood; the query is formulated like this:
Json.obj("date" -> Json.obj("$gte" -> Json.obj("$date" -> JsNumber(longDate))))
2) how do I match a query within an Aggregate?
This is more of a question of the order of the query. I was originally trying to use match after grouping and aggregating the data - which is (obviously) only going to filter the data after. As I wanted to first get a date range and then aggregate that data I had to match first - this also meant that some of the syntax had to change accordingly:
def getVisitorAggregate(col: JSONCollection) = {
import col.BatchCommands.AggregationFramework.{Group, Match, SumField, AvgField, MinField, MaxField}
val format = new java.text.SimpleDateFormat("dd-MM-YYYY")
val myDate = "15-05-2016"
val parseDate: Date = format.parse(myDate)
val longDate: Long = parseDate.getTime
col.aggregate(
Match(Json.obj("date" -> Json.obj("$gte" -> Json.obj("$date" -> JsNumber(longDate))))),
List(Group(JsString("$rstId"))(
"totalVisitors" -> SumField("visitors"),
"avgVisitors" -> AvgField("visitors"),
"minVisitors" -> MinField("visitors"),
"maxVisitors" -> MaxField("visitors")
))
)
.map(_.head[VisitorAggregate])
}
Really frustrating that there isn't more documentation out there on using the Play Framework with ReactiveMongo as there are a lot of instances of trying to fathom syntax and logic.
Related
I am using the AWS Dynamo DB library for Scala - com.amazonaws.services.dynamodbv2.
Earlier I had a table with a primary key and I was using GetItem to get specific item from it like so :
val item = Try(Option(ddb.getItem(new GetItemRequest().withTableName(table).withKey(Collections.singletonMap(keyField, new AttributeValue(key)))).getItem).map(_.get(valField).getS))
But now I need to start using a new sort key (timestamp of the created date) on top.
This way I can have multiple identical primary keys with different timestamp sort keys.
This is meant so I will be able to get the closest item to my current sort timestamp property.
I think I need KeyConditionExpression where my input timestamp is bigger or equal to the new sort key,
and I saw a property ScanIndexForward which can be set to true in combination with
limit = 1
so I will get only one item and it will be the closest(?)
And this should get me the desired item I hope, but I am not so sure how to approach this in Scala and with the AWS library.
In case anyone interested,
this is what worked out for me -
val queryRequest = new QueryRequest().withTableName(table)
.withKeyConditions(createKeyConditionsMap(keyField, keyValue, sortField, sortValue))
.withScanIndexForward(false)
.withLimit(1)
val x = ddb.query(queryRequest).getItems()
def createKeyConditionsMap(keyField: String, keyValue: String, sortField: String, sortValue: String) = {
Map(keyField -> new Condition().withComparisonOperator("EQ").withAttributeValueList(new AttributeValue(keyValue)),
sortField -> new Condition().withComparisonOperator("LE").withAttributeValueList(new AttributeValue().withN(sortValue))).asJava
}
Using ReactiveMongo 0.11 for Scala 2.11. I have an issue where my queries are failing to descend. The following is my Index and ReactiveMongo query:
collection.indexesManager.ensure(Index(Seq("userId" -> IndexType.Ascending, "lastActivity" -> IndexType.Descending), background = true))
def listEfforts(userId: String, page: Int, pageSize: Int): Future[\/[ErrMsg, List[EffortDesc]]] = {
val query = BSONDocument("userId" -> userId)
val sort = BSONDocument("lastActivity" -> -1)
val skipN = (page - 1) * pageSize
val queryOptions = new QueryOpts(skipN = skipN, batchSizeN = pageSize, flagsN = 0)
collection.find(query).options(queryOptions).sort(sort).cursor[EffortDesc](ReadPreference.primaryPreferred).collect[List](pageSize).flatMap {
case list => Future(\/.right(list))
}
}
What's happening is my results are all ascending, even though my sort variable has been set to -1. lastActivity is a Unix timestamp field in milliseconds. I've tried other debugging issues (like recompiling, etc.)
Any idea what could be causing this? Thanks for your help!
Found the issue. If I put a IndexType.Descending on lastActivity field and then additionally sort as "descending" (via "lastActivity" -> -1) MongoDB will first return a descended sort according to its index and then sort it again.
I'm not sure if this is normal/expected behavior in Mongo but changing -1 to 1 fixed the issue.
Using either
("fieldName", BSONInteger.apply(1))
or
("fieldName", BSONInteger.apply(-1))
works for me.
I'm currently trying to create a percolator query with Elastic4s. I've got about this far but I can't seem to find any examples so I'm not sure how this quite works. So I've got:
val percQuery = percolate in esIndex / esType query myQuery
esClient.execute(percQuery)
Every time it runs it doesn't match anything. I figured out I need to be able to percolate on an Id but I can't seem to find any examples on how to do it, not even in the docs. I know with Elastic4s creating queries other than a percolator query lets you specify an id field like:
val query = index into esIndex / esType source myDoc id 12345
I've tried this way for percolate but it doesn't like the id field, does anyone know how this can be done?
I was using Dispatch Http to do this previously but I'm trying to move away from it. Before, I was doing this to submit the percolator query:
url(s"$esUrl/.percolator/$queryId)
.setContentType("application/json", "utf-8")
.setBody(someJson)
.POST
notice the queryId just need something similar to that but in elastic4s.
So you want to add a document and return the queries that are waiting for that id to be added? That seems an odd use for percolate as it will be a one time use only, as only one document can be added per id. You can't do a percolate currently on id in elastic4s, and I'm not sure if you can even do it in elasticsearch itself.
This is the best attempt I can come up with, where you have your own "id" field, which could mirror the 'proper' _id field.
object Test extends App {
import ElasticDsl._
val client = ElasticClient.local
client.execute {
create index "perc" mappings {
"idtest" as(
"id" typed StringType
)
}
}.await
client.execute {
register id "a" into "perc" query {
termQuery("id", "a")
}
}.await
client.execute {
register id "b" into "perc" query {
termQuery("id", "b")
}
}.await
val resp1 = client.execute {
percolate in "perc/idtest" doc("id" -> "a")
}.await
// prints a
println(resp1.getMatches.head.getId)
val resp2 = client.execute {
percolate in "perc/idtest" doc("id" -> "b")
}.await
// prints b
println(resp2.getMatches.head.getId)
}
Written using elastic4s 1.7.4
So after much more researching I figured out how this works with elastic4s. To do this in Elastic4s you actually have to use register instead of percolate like so:
val percQuery = register id queryId into esIndex query myQuery
This will register a percolator query at the id.
The following code snippet retrieves all the users that have been modified before a given date:
val date = DateTime(...).getMillis
users.find(
Json.obj("lastUpdate" -> Json.obj("$lt" -> Json.obj("$date" -> date))),
None, None, page, perPage
)
How do I retrieve all the users that have been modified within a period starting from lastUpdate? In other words, I need to add some days to lastUpdate and compare it with a given date:
users.find(
Json.obj("lastUpdate" -> /* how do I add N days to lastUpdate before comparing it with date? */
Json.obj("$lt" -> Json.obj("$date" -> date))
),
None, None, page, perPage
)
You can't. A simple query in MongoDB can't use values from the document it's querying. You can only use values that are constant throughout the query.
Valid Query:
{
FirstName : "bar"
}
Invalid Query:
{
FirstName : LastName
}
You can however acheive that by other means, for example MongoDB's aggregation framework
I am both new to scala and cashbah. I am trying to
update a document if exists (by _id) and create if doesn't exist.
while updating, update some key values
while updating, update some keys which values are sets, include some data to those sets.
To achive this, I've written this:
DBObject = MongoDBObject("_id" -> uri.toString) ++
$addToSet("appearsOn" -> sourceToAppend) ++
$addToSet("hasElements" -> elementsToAppend) ++
$addToSet("hasTriples" -> triplesToAppend) ++
MongoDBObject("uDate" -> new DateTime)
/* Find and replace here! */
OntologyDocument.dao.collection.findAndModify(
query = MongoDBObject({"_id" -> uri.toString}),
update = update,
upsert = true,
fields = null,
sort = null,
remove = false,
returnNew = true
)
Documents looked by _id, some new items added to appearsOn hasElements hasTriples and uDate is updated.
sourceToAppend elementsToAppend and triplesToAppend are List[String]
When I run this, I got this error:
java.lang.IllegalArgumentException: fields stored in the db can't start with '$' (Bad Key: '$addToSet')
at com.mongodb.DBCollection.validateKey(DBCollection.java:1444) ~[mongo-java-driver-2.11.1.jar:na]
I didn't get it. What is wrong with this query? $addToSet isn't a field, why casbah thinks it is a field? What am I doing wrong here?
The reason its failing is because the update query is invalid (it wont work in the js shell).
$set is implicit for values in the update document, but you can't mix it with other update operators eg $addToSet. If you want to mix $set with other set operators then you can if you are explicit:
val update = $set("uDate" -> new DateTime) ++
$addToSet("appearsOn" -> sourceToAppend,
"hasElements" -> elementsToAppend,
"hasTriples" -> triplesToAppend)
You can't $set "_id" but as thats in the query and its an upsert - it will merge so don't include it in the update statement - otherwise it will error.
Finally, #AsyaKamsky is right if you dont need the returned document - use an update its also atomic.