mongodb lookup match-expr as second pipeline step - mongodb

I'm relatively new to mongodb aggregations and I have a tiny problem:
I want to make a join between two collections. The problem is, that the foreign field is placed in an inner array. That means that I have to unwind the array in order to make a proper $match. In my $match, I use an $epxr and an $eq in order to make the join on the unwinded documents (which is intended and therefor not a problem). The $expr is needed to access variables from the orginal collection:
[
...
{
$lookup: {
from: 'foreignCollection',
as: 'field',
let: {
localField: '$someComparisonField'
},
pipeline: [
{
$unwind: '$arr'
},
{
$match: {
$expr: {
$eq: [ '$arr.foreignField', '$$localField' ]
}
}
}
]
}
}
]
However field is always an empty array in my result set. I seriously don't know what I'm doing wrong :D
Can someone help me?
Edit:
As requested, some sample data for the two involved collections:
orginalCollection:
{
...
someComparisonField: 1
},
{
...
someComparisonField: 2
}
foreignCollection:
{
...
arr: [
{
...
foreignField: 1
},
{
...
foreignField: 1
},
{
...
foreignField: 2
},
]
},
{
...
arr: [
{
...
foreignField: 1
},
{
...
foreignField: 2
},
{
...
foreignField: 2
},
]
},
{
...
arr: [
{
...
foreignField: 2
},
{
...
foreignField: 1
},
{
...
foreignField: 2
},
]
},
Edit 2:
I forgot to add a small detail: with $eq in the inner pipeline, I am accessing a fixed index which means that the foreignField actually looks like this: foreignField: [ <value> ].

With some investigation, I came to the following conclusion:
Before reading further, I recommend reading both of my edits to the questions. Especially my 2nd edit.
It seems like $eq within an $expr simply can't compare a fixed index.
This expression simply doesn't work somehow:
{
$expr: {
$eq: [ '$arr.foreignField.0': '$$localField' ]
}
}
To work arround this, I created an projection before this stage which extracts the value from the desired fixed index and stores it in a new field. So now the particular stages from the inner pipeline look like this:
...
{
$project: {
value: { $arrayElemAt: [ '$arr.foreignField', 0 ] }
}
},
{
$match: {
$expr: {
$eq: [ '$value', '$$localField' ]
}
}
}
...

Related

$lookup with pipeline match and projection does not work for guid

I have two collections that I want to join with $lookup based on two id fields. Both fields are from type guid and looke like this in mongodb compass: 'Binary('cavTZa/U2kqfHtf08sI+Fg==', 3)'
This syntax in the compass aggregation pipeline builder gives the expected result:
{
from: 'clients',
localField: 'ClientId',
foreignField: '_id',
as: 'ClientData'
}
But i want to add some projection and tried to change it like this:
{
from: 'clients',
'let': {
id: '$_id.clients'
},
pipeline: [
{
$match: {
$expr: {
$eq: [
'$ClientId',
'$$id'
]
}
}
},
{
$project: {
Name: 1,
_id: 0
}
}
],
as: 'ClientData'
}
But the result here is that every client from collection 'clients' is added to every document in the starting table. I have to use MongoDB 3.6 so the new lookup syntax from >=5.0 is not available.
Any ideas for me? Does $eq work for binary stored guid data?
In the first example, you say that the local field is ClientId and the foreign field is _id. But that's not what you used in your second example.
This should work better:
{
from: 'clients',
'let': {
ClientId: '$ClientId'
},
pipeline: [
{
$match: {
$expr: {
$eq: [
'$$ClientId',
'$_id'
]
}
}
},
{
$project: {
Name: 1,
_id: 0
}
}
],
as: 'ClientData'
}

How to use lookup when both the tables have the same key names?

I have two collections and both of them have the same key names platform, channel and month. When I am using lookup, the let step is taking the variables from the first collection and not the collection b. Using the dot operator also does not work.
Another problem is that month and year operators are not working in let.
How do I fix these?
{
$lookup: {
from: "b",
let: {
rev_platform: "$b.platform",
rev_channel: "$b.channel",
rev_month: {$month:'$b.month'},
rev_year: {$year:'$b.month'}
},
pipeline: [
{ $match:
{ $expr:
{ $and:
[
{ $eq: [ "$platform", "$$rev_platform" ] },
{ $eq: [ "$channel", "$$rev_channel" ] },
{ $eq: [ "$month", "$$rev_month" ] },
{ $eq: [ "$year", "$$rev_year" ] },
]
}
}
},
],
as: "fromB"
}
},
https://docs.mongodb.com/manual/reference/operator/aggregation/replaceRoot/#mongodb-pipeline-pipe.-replaceRoot
By default, ROOT = top level document.
"References the start of the field path being processed in the aggregation pipeline stage. Unless documented otherwise, all stages start with CURRENT the same as ROOT.
CURRENT is modifiable. However, since $ is equivalent to $$CURRENT., rebinding CURRENT changes the meaning of $ accesses."
To change the root that is used for referencing values, you can "replaceRoot".
https://docs.mongodb.com/manual/reference/operator/aggregation/replaceRoot/
Perhaps something along these lines can 'namespace' your variables....
db.people.aggregate( [
{ $replaceRoot: { newRoot: {document_a:$$ROOT}} }
] )
It's also helpful to remember the syntax/limitations of $join:
{
$lookup:
{
from: <collection to join>,
let: { <var_1>: <expression>, …, <var_n>: <expression> },
pipeline: [ <pipeline to execute on the joined collection> ], // Cannot include $out or $merge
as: <output array field>
}
}

MongoDB $lookup creates array

So I am trying to join two collections together:
Collections are:
shows
episodes
I am using the $lookup value inside the shows collection.
[{$lookup: {
from: 'episode',
localField: 'url',
foreignField: 'show_b',
as: 'match_docs'
}}]
However I am getting all of the episodes from each show inside the match_docs in theory that is fine, however I need to be able to limit it to the latest episode limit:1 for each show ordered by pubDate
If anyone knows how I could limit the match_docs to only lookup once that would be great
I have also tried
{
from: 'episode',
localField: 'url',
foreignField: 'show_b',
pipeline: [
{$sort:{id:1}},{$limit:1},
],
as: 'match_docs'
}
With no success.
That would be easy with the second syntax of $lookup:
[
{
$lookup: {
from: 'episodes', # make sure your collection name is correct
let: {url: '$url'},
pipeline: [
{
$match: {
$expr: {
$eq: ['$show_b', '$$url']
}
}
},
{
$sort: {
pubDate: -1
}
},
{
$limit: 1
}
],
as: 'match_docs'
}
}
]

MongoDB lookup with object relation instead of array

I have a collection matches like this. I'm using players object {key: ObjectId, key: ObjectID} instead of classic array [ObjectId, ObjectID] for reference players collection
{
"_id": ObjectId("5eb93f8efd259cd7fbf49d55"),
"date": "01/01/2020",
"players": {
"home": ObjectId("5eb93f8efd259cd7fbf49d59"),
"away": ObjectId("5eb93f8efd259cd7fbf49d60")
}
},
{...}
And players collection:
{
"_id": ObjectId("5eb93f8efd259cd7fbf49d59"),
"name": "Roger Federer"
"country": "Suiza"
},
{
"_id": ObjectId("5eb93f8efd259cd7fbf49d60"),
"name": "Rafa Nadal"
"country": "España"
},
{...}
What's the better way to do mongoDB lookup? something like this is correct?
const rows = await db.collection('matches').aggregate([
{
$lookup: {
from: "players",
localField: "players.home",
foreignField: "_id",
as: "players.home"
}
},
{
$lookup: {
from: "players",
localField: "players.away",
foreignField: "_id",
as: "players.away"
},
{ $unwind: "$players.home" },
{ $unwind: "$players.away" },
}]).toArray()
I want output like this:
{
_id: 5eb93f8efd259cd7fbf49d55,
date: "12/05/20",
players: {
home: {
_id: 5eb93f8efd259cd7fbf49d59,
name: "Roger Federer",
country: "Suiza"
},
away: {
_id: 5eb93f8efd259cd7fbf49d60,
name: "Rafa Nadal",
country: "España"
}
}
}
{...}
You can try below aggregation query :
db.matches.aggregate([
{
$lookup: {
from: "players",
localField: "players.home",
foreignField: "_id",
as: "home"
}
},
{
$lookup: {
from: "players",
localField: "players.away",
foreignField: "_id",
as: "away"
}
},
/** Check output of lookup is not empty array `[]` & get first doc & write it to respective field, else write the same value as original */
{
$project: {
date: 1,
"players.home": { $cond: [ { $eq: [ "$home", [] ] }, "$players.home", { $arrayElemAt: [ "$home", 0 ] } ] },
"players.away": { $cond: [ { $eq: [ "$away", [] ] }, "$players.away", { $arrayElemAt: [ "$away", 0 ] } ] }
}
}
])
Test : mongoplayground
Changes or Issues with current Query :
1) As you're using two $unwind stages one after the other, If anyone of the field either home or away doesn't have a matching document in players collection then in the result you don't even get actual match document also, But why ? It's because if you do $unwind on [] (which is returned by lookup stage) then unwind will remove that parent document from result, To overcome this you need to use preservenullandemptyarrays option in unwind stage.
2) Ok, there is another way to do this without actually using $unwind. So do not use as: "players.home" or as: "players.away" cause you're actually writing back to original field, Just in case if you don't find a matching document an empty array [] will be written to actual fields either to "home" or "away" wherever there is not match (In this case you would loose actual ObjectId() value existing in that particular field in matches doc). So write output of lookup to a new field.
Or even more efficient way, instead of two $lookup stages (Cause each lookup has to go through docs of players collection again & again), you can try one lookup with multiple-join-conditions-with-lookup :
db.matches.aggregate([
{
$lookup: {
from: "players",
let: { home: "$players.home", away: "$players.away" },
pipeline: [
{
$match: { $expr: { $or: [ { $eq: [ "$_id", "$$home" ] }, { $eq: [ "$_id", "$$away" ] } ] } }
}
],
as: "data"
}
}
])
Test : mongoplayground
Note : Here all the matching docs from players which match with irrespective of away or home field will be pushed to data array. So to keep DB operation simple you can get that array from DB along with actual matches document & Offload some work to code which is to map respective objects from data array to players.home & players.away fields.

Mongo Aggregate Objects with $lookup using non matching values

I've got an Object Mission referring to another object Position with a key _p_position.
Mission objects look like:
{
_id: "ijjn97678",
_p_position: "Position$qwerty123",
...
}
Position objects look like:
{
_id: "qwerty123",
...
}
I don't know if it is Mongo or Parse convention but as one can see a Position$ is added on relational position attribute in missions.
I'd like to aggregate both into a single Object to get a results similar to the following:
{
_id: "ijjn97678",
_p_position: "Position$qwerty123",
positions: [
{
_id: "qwerty123"
}
]
}
using:
missions.aggregate([
{
$lookup: {
as: "position",
from: "Position",
foreignField: "_id",
localField: "_p_position",
},
},
])
But I need to remove Position$ from _p_position. Is there a way I can compute "_p_position" before it is used to find a matching Position's id ?
PS: I only have reading rights on DB
You can use $addFields to add another field which will be then passed to $lookup stage. To get the part that's following the dollar sign you need: $indexOfBytes and $substr operators. Additionally dollar sign itself is a special character in Aggregation Framework (represents a field reference) so you need $literal to force it to be considered as regular field
db.missions.aggregate([
{
$addFields: {
value: {
$let: {
vars: { index: { $indexOfBytes: [ "$_p_position", { $literal: "$" } ] } },
in: { $substr: [ "$_p_position", { $add: [ "$$index", 1 ] } , { $strLenBytes: "$_p_position" } ] }
}
}
}
},
{
$lookup: {
from: "Position",
localField: "value",
foreignField: "_id",
as: "position"
}
}
])