I am trying to recreate an aggregation pipeline I built in mongo compass in Java. It consists of an $addFields which converts string to date using $toDate. I am trying to do the same in Java with Spring Data Mongo Aggregation Pipeline but have failed thus far and need help
I tried with some other example with date from string and projection but I don't know how to work it with $addFields.
This is the mongo pipeline that I need to be recreated in Java with Spring Data.
{
"event.currentTimeStamp": {
"$toDate": "$event.timestamp"
}
}
This works fine from shell and Compass.
This is my sample Document .
{
"_id":"5c0f0f97ece8cc0009c0a8f8",
"event":{
"jobid":"e4955ab0-003a-40fd-ac5a-4363c3e0f604",
"username":"sn",
"timestamp":"Mon Dec 10 20:15:02 EST 2018",
"system":"Mercury",
"eventStatus":"START",
"eventType":"UPLOAD"
}
I need the timestamp field to be converted to date and added as a field so that I can do other aggregation operations on them.
NOTE: Hoping to get an answer as a form of aggregation as that's what I am doing. Thanks.
Please find the below answer. Hope this will help you.
db.getCollection('test1').aggregate([
{$addFields : {"date" : {"$toDate" : "$event.timestamp"}}}
])
Related
I'm trying to perform a date range query on the _id field using compass.
I've tried what I found here with the following filter:
{_id: { $gte: ObjectId.fromDate(new Date('2019-01-01')) } }
What am I missing? I'd like to get a list of all documents from some date forward (in this example from 1 Jan 2019 to present). Unfortunately there isn't a timestamp in the document fields so I need to extract it from the object id.
You need to pass a date object to the ObjectId.fromDate, not a string. Try this:
ObjectId.fromDate(new Date('2019-01-01'))
This function works only in the shell and doesn't exist in the drivers.
EDIT after comments:
Here is a solution that works in Compass as well:
{
$expr: {
$gte: [
{"$toDate":"$_id"},
ISODate("2021-01-01T00:00:00.000Z")
]
}
}
Keep in mind, however, that it requires a version of mongo of 4.0+. You can checkout the docs here.
Also, checkout this related topic: Can I query MongoDB ObjectId by date?
It is not about Compass, but it provides solutions for generating the ObjectIds from a date without being dependent on ObjectId.fromDate().
I found a solution, maybe not the best but it does the job. Hopefully this can help someone with the same problem if there isn't a cleaner solution.
I converted the date I needed to an ObjectId outside of compass online here.
Then I wrote the query with that ObjectId:
{_id: { $gte: ObjectId(' object id here ') } }
As suggested in the comments, see this related topic. It's not specific to Compass but it provides a solution for generating ObjectIds from a date without being dependent on ObjectId.fromDate().
(* MongoDB shell version: 2.6.12 *)
I have a collection logs and a collection users. logs has a field userId which represents the user who sent the log. users has a field _id.
I use Robo 3T to undertake queries to data in remote server.
Now, I'm interested in the logs whose url is /subscribe, and I want to see those user information behind. So I write the following query:
db.getCollection('logs').find({ "url" : "/subscribe" }).aggregate({
$lookup:{
from:"users",
localField:"userId",
foreignField:"_id",
as:"logs_users"
}
})
But I get an error:
Does anyone know how to solve this?
Edit 1: I got a new error:
You should not but use aggregate after the find, instead, use aggregate directly and use $match operator to find the documents and then $lookup
db.getCollection('logs').aggregate([
{
$match:{"url" : "/subscribe"}
},
{
$lookup:{
from:"users",
localField:"userId",
foreignField:"_id",
as:"logs_users"
}
}
])
Update: To use $lookup your MongoDB version should be equal or greater than 3.2, as this operator(joins) don't work in older versions of MongoDB
I'm trying to write a Mongo aggregation using the out operator as described in the docs. This is the aggregation I'm writing:
db.mycollectionname.aggregate([
{ $match: {} },
{ $project: {}},
{ $out: {to: "projets", mode: "insertDocuments"}}
])
When I execute this I get the following error: $out stage requires a string argument, but found object - clear in and of itself but it goes against what the docs say. When I provide a string to the $out stage, I don't get the error but that's not what I want.
Mongo version: 3.6.9
(I have more logic under the $project pipeline stage which I removed for brevity, it doesn't have any impact).
Can someone help me understand why this differs from what the docs say? And how I can provide the arguments I want to pass to the out stage (an object containing "to" and "mode") as a string?
Many thanks,
Chris.
You should look at the version specific documentation:
https://docs.mongodb.com/v3.6/reference/operator/aggregation/out/
$out in MongoDB 3.6 and MongoDB 4.0 only require a single string. In MongoDB 4.2, $out can use a dictionary to set the mode.
I think the problem is with your MongoDb version, you are using 3.6.9 but the document says:
MongoDB 4.2 adds a new syntax structure that implements expanded functionality and flexibility around merging aggregation pipeline results into a target collection, including support for sharded collections and output modes that preserve the existing collection data.
Just update your version and it will work. :)
I am trying a fairly complex aggregate command on two collections involving $lookup pipeline. This normally works just fine on simple aggregation as long as index is set on foreignField.
But my $lookup is more complex as the indexed field is not just a normal Int64 field but actually an array of Int64. When doing a simple find(), it is easy to verify using explain() that the index is being used. But explaining the aggregate pipeline does not explain whether index is being used in the $lookup pipeline. All my timing tests seem to indicate that the index is not being used. MongoDB version is 3.6.2. Db compatibility is set to 3.6.
As I said earlier, I am not using simple foreignField lookup but the 3.6-specific pipeline + $match + $expr...
Could using pipeline be showstopper for the index? Does anyone have any deep experience with the new $lookup pipeline syntax and/or the index on an array field?
Examples
Either of the following works fine and if explained, shows that index on followers is being used.
db.col1.find({followers: {$eq : 823778}})
db.col1.find({followers: {$in : [823778]}})
But the following one does not seem to make use of the index on followers [there are more steps in the pipeline, stripped for readability].
db.col2.aggregate([
{$match:{field: "123"}},
{$lookup:{
from: "col1",
let : {follower : "$follower"},
pipeline: [{
$match: {
$expr: {
$or: [
{ $eq : ["$follower", "$$follower"] },
{ $in : ["$$follower", "$followers"]}
]
}
}
}],
as: "followers_all"
}
}])
This is a missing feature which is going to part of 3.8 version.
Currently eq matches in lookup sub pipeline are optimised to use indexes.
Refer jira fixed in 3.7.1 ( dev version).
Also, this may be relevant as well for non-multi key indexes.
I have a database containing this document:
{"_id":{"$id":"xxx"},"duration":{"sec":137,"usec":0},"name":"test"}
If I call db.collection.aggregate with this pipeline:
{$project:{_id: 0, name: 1, duration: 1, seconds: "$duration.sec"}}
I get this result:
{"result":[{"duration":{"sec":137,"usec":0},"name":"test"}],"ok":1}
Why does the result not have a 'seconds' field? Have I used the wrong projection syntax?
I'm not entirely sure of the version of mongodb the server is running. I'm using the 1.3.1 php driver with php 5.4.3, but the server may be older than that - perhaps by about half a year?
According to the MongoDB documentation on $project:
You may also use $project to rename fields. Consider the following
example:
db.article.aggregate(
{ $project : {
title : 1 ,
page_views : "$pageViews" ,
bar : "$other.foo"
}} );
This operation renames the pageViews field to page_views, and renames the foo field in the other sub-document as the top-level
field bar.
That example seems to match-up pretty good with what you are trying to do.
I know 10gen officially released the aggregation framework with MongoDB v2.2. Check out the current production release, which I believe is 2.2.3. If you are running on a prior development version, there could be something odd going on with aggregation.
As Bryce said, I'm currently using MongoDB 2.6 through the shell and the $project pipeline is working for renaming nested fields as you do.
db.article.aggregate({$project:{'_id': 0, 'name': 1, 'duration': 1, 'seconds': '$duration.sec'}}
I've not tried yet trough the python or php drivers but my former pipelines with the last pymongo worked very well.