Note
I am completely new to MongoDB so my terminology might not be fully correct.
Context
I have some data with various dates in MongoDB and I am trying to use a $switch function to provide column aliasing to form some new calculated columns however I am running into some issues (perhaps syntax or wrong implementation) and want to know why and how to resolve.
Question
Basically I want to create a new calculated date column which takes the value of the a date field if its not null, else it takes the value of another date field if its not null and so on, however it defaults to some message "blank" if there is no values in any of those fields.
Schema
A single document with the collection follows this rough structure
{
_id:ObjectId("619756f12c115f24df503c26")
uniqueid:"12345678"
date1:"2021-11-02 20:04:50.253"
date2:"2021-11-03 18:10:57.520"
date3:"2021-11-08 07:08:00.000"
date4:"2021-11-08 14:40:00.000"
date5:"2021-11-08 08:34:00.000"
}
Tried
As part of an aggregation pipeline I tried using the $project and then returning all the columns along with the new calculated columns however it comes up with an error at this stage saying stage must be a properly formatted document leading me to believe I either my syntax is wrong or perhaps using the wrong operators to do this.
If I remove the calculated column code then the preview seems to work (further convincing me its something to do with my calculated column implementation).
db.collection.aggregate([
{
$project:
{
"uniqueid": 1,
"date1": 1,
"date2": 1,
"date3": 1,
"date4":1,
"date5": 1,
"cal_date1": {
$switch: {
branches: [
{ case: {"$date2": {$ne: null}}, then: "$date2"},
{ case: {"$date3": {$ne: null}}, then: "$date3"},
{ case: {"$date4": {$ne: null}}, then: "$date4"},
{ case: {"$date5": {$ne: null}}, then: "$date5"}
],
default: "blank"
}
},
"cal_date2": {
$switch: {
branches: [
{ case: {"$date4": {$ne: null}}, then: "$date4"},
{ case: {"$date5": {$ne: null}}, then: "$date5"}
],
default: "blank"
}
},
"cal_date3": {
$switch: {
branches: [
{ case: {"$date5": {$ne: null}}, then: "$date5"}
],
default: "blank"
}
}
}
}
])
Update1: 2021-11-24 T12:15pm UTC
I have added in a missing curly bracket at the end of each of the case expressions. The error now is unknown operator: $date2
change this case: {"$date4": {$ne: null}}, to case: {$ne:["$date4",null]}, in all of aggregate
Description
It seems that the syntax of my statement is what was causing the issue (not sure why). If someone could elaborate that would be good.
Answer
Here is the code:
db.collection.aggregate([
{
$project:
{
"uniqueid": 1,
"date1": 1,
"date2": 1,
"date3": 1,
"date4":1,
"date5": 1,
"cal_date1": {
$switch: {
branches: [
{ case: {$ne:["$date2",null]}, then: "$date2"},
{ case: {$ne:["$date3",null]}, then: "$date3"},
{ case: {$ne:["$date4",null]}, then: "$date4"},
{ case: {$ne:["$date5",null]}, then: "$date5"}
],
default: "blank"
}
},
"cal_date2": {
$switch: {
branches: [
{ case: {$ne:["$date4",null]}, then: "$date4"},
{ case: {$ne:["$date4",null]}, then: "$date5"}
],
default: "blank"
}
},
"cal_date3": {
$switch: {
branches: [
{ case: {$ne:["$date5",null]}, then: "$date5"}
],
default: "blank"
}
}
}
}
])
Notice that the {"$date_value": {$ne: null}} within each statement has been changed to {$ne:["$date_value",null]} possibly due to the $ne operator requiring 2 arguments?
Related
Context
This question uses the same collection/ document schema construct from my other question: MongoDB: Creating calculated fields using the switch function (column aliasing)
Data Schema
Note: cal_date1 is the calculated value shown in my other post linked above. When performing these steps they are done in a pipeline so that (I hope) the calculated columns can be thought of as part of the general document schema as shown below.
{
_id:ObjectId("619756f12c115f24df503c26"),
uniqueid:"12345678",
date1:"2021-11-02 20:04:50.253",
date2:"2021-11-03 18:10:57.520",
date3:"2021-11-08 07:08:00.000",
date4:"2021-11-08 14:40:00.000",
date5:"2021-11-08 08:34:00.000",
cal_date1: "2021-11-03 18:10:57.520"
}
Questions
How can I calculate the difference in date values between two particular dates (with some conditional logic behind it).
Extension
Is there a way of calculate the working days between two dates where "working" is defined as dates that are inclusive of Mon-Fri (no Sat, Sun)?
Tried
I have been playing with the $dateDiff operator inside a switch function as shown below however come across the error unknown operator: $cal_date1
db.collection.aggregate([
{
$project:
{
"uniqueid": 1,
"date1": 1,
"date2": 1,
"date3": 1,
"date4":1,
"date5": 1,
"cal_date1": {
$switch: {
branches: [
{ case: {$ne:["$date2",null]}, then: "$date2"},
{ case: {$ne:["$date3",null]}, then: "$date3"},
{ case: {$ne:["$date4",null]}, then: "$date4"},
{ case: {$ne:["$date5",null]}, then: "$date5"}
],
default: "blank"
}
},
"cal_date2": {
$switch: {
branches: [
{ case: {$ne:["$date4",null]}, then: "$date4"},
{ case: {$ne:["$date4",null]}, then: "$date5"}
],
default: "blank"
}
},
"cal_date3": {
$switch: {
branches: [
{ case: {$ne:["$date5",null]}, then: "$date5"}
],
default: "blank"
}
}
}
}
])
--updated code
"cal_days_between_date1_caldate1": {
$switch:
{
branches: [
{case: { $eq: ["$date1", null]}, then: "blank"},
{case: { $eq: ["$cal_date1", "blank"]}, then: "blank"}
],
default: {
$dateDiff: {
startDate: {
$dateFromString: {
dateString: "$date1"
}
},
endDate: {
$dateFromString: {
dateString: "$cal_date1"
}
},
unit: "day"
}
}
}
}
Update: 2021-11-24: T3:08pm UTC
Have altered the case expression slightly to try and get a boolean output and now have a new error of unknown operator: $dateDiff
Update: 2021-11-30: T8:36pm UTC
Have added in the code which contains the referenced fields (top of code chunk) along with the additions of the $dateFromString operator. The calculated date code is meant to act as the next step in the pipeline which I output along with the other fields.
The problem is that you are trying to use Date operators on fields that are strings.
You should transforms your strings to dates before with $dateFromString. Check this playground and docs.
Also there was some mistakes in the $eq arrays
db.collection.aggregate({
"$project": {
"cal_days_between_date1_caldate1": {
$switch: {
branches: [
{
case: {
$eq: [
"$date1",
null
]
},
then: "blank"
},
{
case: {
$eq: [
"$cal_date1",
"blank"
]
},
then: "blank"
}
],
default: {
$dateDiff: {
startDate: {
$dateFromString: {
dateString: "$date1"
}
},
endDate: {
$dateFromString: {
dateString: "$cal_date1"
}
},
unit: "day"
}
},
}
}
}
})
I have a scenario where I want to pull documents that have a lastAlertSentDate field that's over 30 days old. This will run in a daily cron job. Upon querying, this field will then be reset to NOW. So it's meant to act as a "rotating 30 day window" if you will.
The complication here is that the field won't exist if it hasn't been set yet. In this edge case, we'll then have to use a createdDate field of the document to do the 30-day comparison against.
So effectively, I want something like, "If lastAlertSentDate exists, then get all docs where it's older than 30days from now. ---Otherwise, get all docs where createdDate is older than 30days from now"
So the logic between both fields are the same, it's just the field itself that can be different. Because of this, I was thinking to first USE addFields a dateToUseField and then do a match on the second stage based on this.
[
{
'$addFields': {
'dateToUse': {
'$cond': {
'if': {
'$ne': [
'$lastAlertSentDate', undefined
]
},
'then': '$lastAlertSentDate',
'else': '$createdDate'
}
}
}
}, {
'$match': {
'dateToUse': {
'$lte': '30_DAYS_PRIOR'
}
}
}
]
So the else part doesn't seem to work. It doesn't assign $createdDate to dateToUse.
What am I missing? Also, how can I condense this? I'm sure I don't need the addFields first and I can do everything within the $match
You have two options here:
Use a $or query with two predicates, where each of them is a $and predicate:
Either lastAlertSentDate does not exists and createdDate > n
Or lastAlertSentDate exists and it is > n
Playground Link
db.collection.find({
$or: [
{
$and: [
{
"lastAlertSentDate": {
"$exists": false
}
},
{
"createdDate": {
$gt: 5
}
}
]
},
{
$and: [
{
"lastAlertSentDate": {
"$exists": true
}
},
{
"lastAlertSentDate": {
$gt: 5
}
}
]
}
]
})
Use an aggregation using the $ifNull
Playground Link
db.collection.aggregate([
{
$match: {
$expr: {
$gt: [
{
"$ifNull": [
"$lastAlertSentDate",
"$createdDate"
]
},
5
]
}
}
}
])
I have written a find query, which works, the find query returns records where name and level exist
db.docs.find( { $and: [{name:{$exists:true}},{level:{ $exists:true}} ] },{_id:0, name:1}).sort({"name":1})
and now want to combine it with something like the code below which also works, but needs to be merged with the above to pull the correct data
db.docs.aggregate(
[
{
$project:
{
_id:0,
name: 1,
Honours:
{
$cond: { if: { $gte: [ "$level", 8 ] }, then: "True", else: "False" }
}
}
}
]
)
The find query returns records where name and level exist, but I need to enhance the result with new column called Honours, showing True of False depending on whether the level is gte (greater than or equal to 8)
So I am basically trying to combine the above find filter with the $cond function (which I found and modified example here : $cond)
I tried the below and a few other permutations to try and make find and sort with the $project and$cond aggregate, but it returned errors. I am just very new to how to construct mongodb syntax to make it all fit together. Can anyone please help?
db.docs.aggregate(
[{{ $and: [{name:{$exists:true}},{level:{ $exists:true}} ] },{_id:0, name:1}).sort({"name":1}
{
$project:
{
_id:0,
name: 1,
Honours:
{
$cond: { if: { $gte: [ "$level", 8 ] }, then: "True", else: "False" }
}
}
}
}
]
)
Try below aggregation pipeline :
db.docs.aggregate([
/** $match is used to filter docs kind of .find(), lessen the dataset size for further stages */
{
$match: {
$and: [{ name: { $exists: true } }, { level: { $exists: true } }]
}
},
/** $project works as projection - w.r.t. this projection it will lessen the each document size for further stages */
{
$project: {
_id: 0,
name: 1,
Honours: {
$cond: { if: { $gte: ["$level", 8] }, then: "True", else: "False" }
}
}
},
/** $sort should work as .sort() */
{ $sort: { name: 1 } }
]);
I can't get my head around a mongodb aggregation framework construction that adds up some values for each "_id" field documents... IF those values exist for the field "Wert".
E.g I have a document with _id field and a conditional ProduktTeilsummeDemonstrator":[] or "ProduktTeilsummeDemonstrator":[{Wert:342},{Wert:142}] that array can be empty or not, if it is empty, I want to add a new field "ProduktTeilsumme":0, else, I want to add up all values in that array to the new field...
The data that I have looks like this:
[{"_id":230,"ProduktSummeDemonstrator":713,"ProduktTeilsummeDemonstrator":[],"ProduktTeilsumme":null},{"_id":855,"ProduktSummeDemonstrator":1744,"ProduktTeilsummeDemonstrator":[],"ProduktTeilsumme":null},{"_id":767,"ProduktSummeDemonstrator":1010,"ProduktTeilsummeDemonstrator":[{"Zeitstempel":"2018-07-09T15:07:32.472Z","Wert":24},{"Zeitstempel":"2018-07-09T15:07:32.472Z","Wert":102},{"Zeitstempel":"2018-07-09T14:52:32.473Z","Wert":15},{"Zeitstempel":"2018-07-09T14:52:32.472Z","Wert":20},{"Zeitstempel":"2018-07-09T15:07:32.472Z","Wert":90},{"Zeitstempel":"2018-07-09T14:52:32.472Z","Wert":104},{"Zeitstempel":"2018-07-09T15:07:32.473Z","Wert":29},{"Zeitstempel":"2018-07-09T14:52:32.472Z","Wert":94},{"Zeitstempel":"2018-07-09T14:52:32.473Z","Wert":33},{"Zeitstempel":"2018-07-09T15:07:32.473Z","Wert":245},{"Zeitstempel":"2018-07-09T14:52:32.473Z","Wert":243},{"Zeitstempel":"2018-07-09T15:07:32.473Z","Wert":11}],"ProduktTeilsumme":null},{"_id":9,"ProduktSummeDemonstrator":94,"ProduktTeilsummeDemonstrator":[],"ProduktTeilsumme":null}]
I tried out different things with $reduce or $cond expressions, but somehow it won't add up: (Previously before that calculation stage I am grouping by ID and also filtering based on some time field condition..)
{
$project: {
ProduktSummeDemonstrator: "$ProduktSummeDemonstrator",
ProduktTeilsummeDemonstrator: {
$filter: {
input: "$res",
as: "res",
cond: { $and: [
{ $gte: ["$$res.Zeitstempel", new Date(req.params.start) ] },
{ $lte: ["$$res.Zeitstempel", new Date(req.params.end) ] }
] }
}
},
ProduktTeilsumme: {/*
$reduce: {
input: "$ProduktTeilsummeDemonstrator",
initialValue:0,
in: {
$add: ["$$value","$$this.Wert"]
}
} */
$cond: {
if: { $eq: [ "", "$ProduktTeilsummeDemonstrator" ] },
then: 0,
else: {
$reduce: {
input: "$ProduktTeilsummeDemonstrator",
initialValue: 0,
in: {
$add: ["$$value","$$this.Wert"]
}
}
}
}
}
}
}
at least for "_id":767 I should get some values back, but I am getting "null" always.
You have to use multiple project stages if you want to keep both the array and added value. One for $filtering ProduktTeilsummeDemonstrator followed by adding up array values.
Something like
[
{"$project":{
"ProduktSummeDemonstrator":1,
"ProduktTeilsummeDemonstrator":{
"$filter":{
"input":"$ProduktTeilsummeDemonstrator",
"as":"res",
"cond":{
"$and":[
{"$gte":["$$res.Zeitstempel", new Date(req.params.start)]},
{"$lte":["$$res.Zeitstempel", new Date(req.params.end)]}
]
}
}
}
}},
{"$project":{
"ProduktSummeDemonstrator":1,
"ProduktTeilsummeDemonstrator":1,
"ProduktTeilsumme":{"$sum":"$ProduktTeilsummeDemonstrator.Wert"}
}}
]
I am writing a query in MongoDB using aggregation such that if condition 1 matches then I want to project some fields and when condition 2 matches I want to project some different fields and when the third condition 3 reaches I want to project some other different fields.
My Query is like below
{
$match: {
$and: [
{
{field_a: "henesa"}
},
{
$expr: {
$or: [
{ Condition 1}, {Condition 2}, {condition 3}
]
}
}
]
}
},
{$project: { /* Here How To Decide which params to send */}}
Can anyone please tell me how can I do that.
You can use <field>: <expression> syntax of $projection at the projection stage.
In <expression> part you can use conditional operators to project values based on your criteria. E.g.
{ $project: {
field: { $switch: {
branches: [
{ case: Condition 1, then: "$field1" },
{ case: Condition 2, then: "$field2" },
...
]
} }
} }
Or more complex combination of $cond if you need to handle cases when more than one $or conditions met.