Aggregate and $ne Not Working as Expected - mongodb

Question

I'm trying the aggregation below, but do not appear to be getting the expected result using $ne and null.
I have tried other solutions like using a combination of $cond, $not, and $eq to no avail. Using $gt:[ "$unloadeddate", null] seems to give some results, but that does not seem to be proper syntax and I am concerned it's not trustworthy to run properly over entire dataset.
Also, querying as follows:
db.getCollection('esInvoices').find({"esBlendTickets.loadeddate":{$ne:null}})
... returns results so not sure why the same query in the aggregate function not working.
Any help is appreciated!!
The first part works...
"unloadeddate": { "$switch": {
branches:[ {
case: {
"$ne":[ "$esBlendTickets.ticketdate", null]
},
then: "$esBlendTickets.ticketdate"
}, {
case: {
"$ne":[ "$esDeliveryTickets.ticketdate", null]
},
then: "$esDeliveryTickets.ticketdate"
}],
default: null
}
},
"loadeddate": { "$switch": {
branches:[ {
case: {
"$ne":[ "$esBlendTickets.loadeddate", null]
},
then: "$esBlendTickets.loadeddate"
}, {
case: {
"$ne":[ "$esDeliveryTickets.loadeddate", null]
},
then: "$esDeliveryTickets.loadeddate"
}],
default: null
}
},
... but this second part (essentially the same logic except for the resulting value) does not work as expected...
"stagename": { "$switch": {
branches:[ {
case: {
"$ne":[ "$esDeliveryTickets.ticketdate", null]
},
then: "Invoiced"
}, {
case: {
"$ne":[ "$esBlendTickets.ticketdate", null]
},
then: "Invoiced"
}],
default: "Invoiced-Only"
}
}
Complete Aggregation:
db.esInvoices.aggregate([ {
$addFields: {
// A single invoice will not have both a blend ticket and delivery ticket associated so looping tough each case should work.
"unloadeddate": { "$switch": {
branches:[ {
case: {
"$ne":[ "$esBlendTickets.ticketdate", null]
},
then: "$esBlendTickets.ticketdate"
}, {
case: {
"$ne":[ "$esDeliveryTickets.ticketdate", null]
},
then: "$esDeliveryTickets.ticketdate"
}],
default: null
}
},
"loadeddate": { "$switch": {
branches:[ {
case: {
"$ne":[ "$esBlendTickets.loadeddate", null]
},
then: "$esBlendTickets.loadeddate"
}, {
case: {
"$ne":[ "$esDeliveryTickets.loadeddate", null]
},
then: "$esDeliveryTickets.loadeddate"
}],
default: null
}
},
"stagedate": "$InvoiceHeader.InvDate",
"stagename": { "$switch": {
branches:[ {
case: {
"$ne":[ "$esDeliveryTickets.ticketdate", null]
},
then: "Invoiced"
}, {
case: {
"$ne":[ "$esBlendTickets.ticketdate", null]
},
then: "Invoiced"
}],
default: "Invoiced-Only"
}
}
}}])

Answer 1

Think I just ran into the same problem you were having. Using Mongo 3.4 at the moment. From what I can tell the query $ne behaves differently from the aggregate $ne when you are comparing it to null. Threw me off for a little bit.
Specifically, a { $ne: [ '$field', null ] } predicate in the aggregate pipeline will return true when $field is undefined.
However, when using queries (not $aggregate), the { field: { $ne: null } predicate will return false for those same documents when $field is undefined.
My work-around was to use a $project with { field: { $ifNull: [ '$field': null ] } } in a prior step to turn undefined instances of that field into explicit nulls, which will then make the aggregate $ne work as I needed. For whatever reason, $ifNull works with null, undefined, and missing fields. Not sure why $ne is different.
Here's an example to reproduce.
db.test.insertMany([
{ a: 1, b: 'string' },
{ a: 2, b: null },
{ a: 3 },
])
db.test.find({ b: { $ne: null }}, { _id: 0 })
/*
returns:
{
"a" : 1.0,
"b" : "string"
}
*/
db.test.aggregate([
{ $project: {
_id: 0,
a: 1,
b: 1,
switched: { $switch: {
branches: [
{ case: { $ne: [ '$b', null ] }, then: 'cased' },
],
default: 'default',
}}
}}
])
/*
returns:
{
"a" : 1.0,
"b" : "string",
"switched" : "cased"
},
{
"a" : 2.0,
"b" : null,
"switched" : "default"
},
{
"a" : 3.0,
"switched" : "cased" <--
}
*/

Answer 2

The problem is missing fields are undefined while null fields are, well, nulls. When you write "$ne":[ "$esDeliveryTickets.ticketdate", null] you don't filter the former but only the latter.
But undefined is "smaller" than null. In order to filter them both you just need to make lte/gt instead of eq/ne. So this query returns all existing values:
See example:
db.test.insertOne(
{
"a" : 10,
"b" : null
})
...
db.getCollection('mytest').find( {},
{
"a" : { $lte : ["$a", null] },
"b" : { $lte : ["$b", null] },
"c" : { $lte : ["$c", null] },
})
// {
// "_id" : ObjectId("606724f38ec4d26b981b5a1c"),
// "a" : false,
// "b" : true,
// "c" : true
// }
In your case:
"stagename": { "$switch": {
branches:[ {
case: {
"$gt":[ "$esDeliveryTickets.ticketdate", null]
},
then: "Invoiced"
}, {
case: {
"$gt":[ "$esBlendTickets.ticketdate", null]
},
then: "Invoiced"
}],
default: "Invoiced-Only"
}
}
and this one returns only missing (or null) values:
"stagename": { "$switch": {
branches:[ {
case: {
"$lte":[ "$esDeliveryTickets.ticketdate", null]
},
then: "Invoiced"
}, {
case: {
"$lte":[ "$esBlendTickets.ticketdate", null]
},
then: "Invoiced"
}],
default: "Invoiced-Only"
}
}