I'm trying to lookup in multiple collections, based on a specific field.
for example if field type equals to 1, lookup the collection from Admin and if type equals to 2, lookup from Client.I know the following query is incorrect, but i just want to show what i mean.
db.User.aggregate([
{
"$lookup":{
"localField":"ID",
"from":{"$cond": { if: { "type":1 } ,then: "Admin", else: "Client"} },
"foreignField":"ID",
"as":"newUser"
},
{
"$unwind":"$newUser"
}
}])
Any help will be appreciated.
Bad news, you cant, the only solution is to use $facet and have 2 separated pipelines.
As you probably imagine this is not a great solution as it wastes resources on the redundant pipeline.
I'm not sure if you can involve some code but if you can it is your best option.
$facet pipeline draft:
db.User.aggregate([
{
$facet: {
user: [
{
"$lookup":{
"localField":"ID",
"from":Client,
"foreignField":"ID",
"as":"newUser"
},
},
{
"$unwind":"$newUser"
}],
admin: [
{
"$lookup":{
"localField":"ID",
"from":Admin,
"foreignField":"ID",
"as":"newUser"
},
},
{
"$unwind":"$newUser"
}],
}
},
{
$match: {
use "correct" user here..
}
}
])
Related
Let's say I have a Mongo database that contains objects such as :
[
{
"test": {
"123123": {
"someField": null
}
}
},
{
"test": {
"323143": {
"someField": "lalala"
},
"121434": {
"someField": null
}
}
},
{
"test": {
"4238023": {
"someField": "afafa"
}
}
},
]
As you can see, the keys right under "test" can vary.
I want to find all documents that have at least one someField that is not null.
Something like find : "test.*.someField": { $ne: null } ( * represents any value here)
How can i do this in mongoose ? I'm thinking an aggregation pipeline will be needed here but not exactly sure how.
Constraints :
I don't have much control over the db schema in this scenario.
Ideally i don't want to have to do this logic in nodeJS, I would like to query directly via the db.
The trickiest part here is that you cannot search keys that match a pattern. Luckily there is a workaround. Yes, you do need an aggregation pipeline.
Let's look at an individual document:
{
"test": {
"4238023": {
"someField": "afafa"
}
}
}
We need to query someField, but to get to it, we need to somehow circumvent 4238023 because it varies with each document. What if we could break that test object down and look at it presented like so:
{
"k": "4238023",
"v": {
"someField": "afafa"
}
}
Suddenly, it get a heck of a lot easier to query it. Well, mongodb aggreation offers a function called $objectToArray which does exactly that.
So what we are going to do is:
Convert the test object into an array for each document.
Match only documents where AT LEAST ONE v.someField is not null.
Put it back together to look as your original documents, minus the ones that do not match the null criterion.
So, here is the pipeline you need:
db.collection.aggregate([
{
"$project": {
"arr": {
"$objectToArray": "$$ROOT.test"
}
}
},
{
"$match": {
arr: {
$elemMatch: {
"v.someField": {
$ne: null
}
}
}
}
},
{
"$project": {
"_id": 1,
"test": {
$arrayToObject: "$arr"
}
}
}
])
Playground: https://mongoplayground.net/p/b_VNuOLgUb2
Note that in mongoose you will run this aggregation the same way you would do it in a terminal... well plus the .then.
YourCollection.aggregate([
...
...
])
.then(result => console.log(result))
I have a little problem with my aggregations. I have a large collection of flat documents with the following schema:
{
_id:ObjectId("5dc027d38da295b969eca568"),
emp_no:10001,
salary:60117,
from_date:"1986-06-26",
to_date:"1987-06-26"
}
It's all about annual employee salaries. The data is exported from relational database so there are multiple documents with the same value of "emp_no" but the rest of their attributes vary. I need to aggregate them by values of attribute "emp_no" so as a result I will have something like this:
//one document
{
_id:ObjectId("5dc027d38da295b969eca568"),
emp_no:10001,
salaries: [
{
salary:60117,
from_date:"1986-06-26",
to_date:"1987-06-26"
},
{
salary:62102,
from_date:"1987-06-26",
to_date:"1988-06-25"
},
...
]
}
//another document
{
_id:ObjectId("5dc027d38da295b969eca579"),
emp_no:10002,
salaries: [
{
salary:65828,
from_date:"1996-08-03",
to_date:"1997-08-03"
},
...
]
}
//and so on
Last but not least there are almost 2.9m of documents so aggregating by "emp_no" manually would be a bit of a problem. Is there a way I can aggregate them using just mongo queries? How do I do this kind of thing? Thank you in advance for any help
The group stage of aggregation pipeline can be used to get this type of aggregates. Specify the attribute you want to group by as the value of _id field in the group stage.
How does the below query work for you?
db.collection.aggregate([
{
"$group": {
"_id": "$emp_no",
"salaries": {
"$push": {
"salary": "$salary",
"from_data": "$from_date",
"to_data": "$to_date"
}
},
"emp_no": {
"$first": "$emp_no"
},
"first_document_id": {
"$first": "$_id"
}
}
},
{
"$project": {
"_id": "$first_document_id",
"salaries": 1,
"emp_no": 1
}
}
])
So I have one collection that I'd like to query/aggegate. The query is made up of several parts that are OR'ed together. For every part of the query, I have a specific set of fields that need to be shown.
So my hope was to do this with an aggregate, that will $match the queries OR'ed together all at once, and then use $project with $cond to see what fields are needed. The problem here is that $cond uses expressions, while the $match uses queries. Which is a problem since some query features are not available as an expression. So a simple conversion is not an option.
So I need another solution..
- I could just make an aggregate per separate query, because there I know what fields to match, and them merger the results together. But this will not work if I use pagination in the queries (limit/skip etc).
- find some other way to tag every document so I can (afterwards) remove any fields not needed. It might not be super efficient, but would work. No clue yet how to do that
- figure out a way to make queries that are only made of expressions. For my purpose that might be good enough, and it would mean a rewrite of the query parser. It could work, but is not ideal.
So This is the next incarnation right here. It will deduplicate and merge records and finally transform it back again to something resembling a normal query result:
db.getCollection('somecollection').aggregate(
[
{
"$facet": {
"f1": [
{
"$match": {
<some query 1>
},
{
"$project: {<some fixed field projection>}
}
],
"f2": [
{
"$match": {
<some query 1>
}
},
{
"$project: {<some fixed field projection>}
}
]
}
},
{
$project: {
"rt": { $concatArrays: [ "$f1", "$f2"] }
}
},
{ $unwind: { path: "$rt"} },
{ $replaceRoot: {newRoot:"$rt"}},
{ $group: {_id: "$_id", items: {$push: {item:"$$ROOT"} } }},
{
$project: {
"rt": { $mergeObjects: "$items" }
}
},
{ $replaceRoot: {newRoot:"$rt.item"}},
]
);
There might still be some optimisation to be, so any comments are welcome
I found an extra option using $facet. This way, I can make a facet for every group opf fields/subqueries. This seems to work fine, except that the result is a single document with a bunch of arrays. not yet sure how to convert that back to multiple documents.
okay, so now I have it figured out. I'm not sure yet about all of the intricacies of this solution, but it seems to work in general. Here an example:
db.getCollection('somecollection').aggregate(
[
{
"$facet": {
"f1": [
{
"$match": {
<some query 1>
},
{
"$project: {<some fixed field projection>
}
],
"f2": [
{
"$match": {
<some query 1>
}
},
{
"$project: {<some fixed field projection>
}
]
}
},
{
$project: {
"rt": { $concatArrays: [ "$f1", "$f2"] }
}
},
{ $unwind: { path: "$rt"} },
{ $replaceRoot: {newRoot:"$rt"}}
]
);
I am attempting to do a mongodb regex query on a field. I'd like the query to prioritize a full match if it finds one and then partials afterwards.
For instance if I have a database full of the following entries.
{
"username": "patrick"
},
{
"username": "robert"
},
{
"username": "patrice"
},
{
"username": "pat"
},
{
"username": "patter"
},
{
"username": "john_patrick"
}
And I query for the username 'pat' I'd like to get back the results with the direct match first, followed by the partials. So the results would be ordered ['pat', 'patrick', 'patrice', 'patter', 'john_patrick'].
Is it possible to do this with a mongo query alone? If so could someone point me towards a resource detailing how to accomplish it?
Here is the query that I am attempting to use to perform this.
db.accounts.aggregate({ $match :
{
$or : [
{ "usernameLowercase" : "pat" },
{ "usernameLowercase" : { $regex : "pat" } }
]
} })
Given your precise example, this could be accomplished in the following way - if your real world scenario is a little bit more complex you may hit problems, though:
db.accounts.aggregate([{
$match: {
"username": /pat/i // find all documents that somehow match "pat" in a case-insensitive fashion
}
}, {
$addFields: {
"exact": {
$eq: [ "$username", "pat" ] // add a field that indicates if a document matches exactly
},
"startswith": {
$eq: [ { $substr: [ "$username", 0, 3 ] }, "pat" ] // add a field that indicates if a document matches at the start
}
}
}, {
$sort: {
"exact": -1, // sort by our primary temporary field
"startswith": -1 // sort by our seconday temporary
}
}, {
$project: {
"exact": 0, // get rid of the "exact" field,
"startswith": 0 // same for "startswith"
}
}])
Another way would be using $facet which may prove a bit more powerful by enabling more complex scenarios but slower (several people here will hate me, though, for this proposal):
db.accounts.aggregate([{
$facet: { // run two pipelines against all documents
"exact": [{ // this one will capture all exact matches
$match: {
"username": "pat"
}
}],
"others": [{ // this one will capture all others
$match: {
"username": { $ne: "pat", $regex: /pat/i }
}
}]
}
}, {
$project: {
"result": { // merge the two arrays
$concatArrays: [ "$exact", "$others" ]
}
}
}, {
$unwind: "$result" // flatten the resulting array into separate documents
}, {
$replaceRoot: { // restore the original document structure
"newRoot": "$result"
}
}])
Part of my collection is as follows:
# Order1
{
"countries": [
{
"code":"us"
}
],
"cities": {
"includes":["a", "b"],
"excludes":[]
}
}
# Order2
{
"countries": [
{
"code":"all"
}
],
"cities": {
"includes":["all"],
"excludes":["x"]
}
}
# Order3
{
"countries": [
{
"code":"all"
}
],
"cities": {
"includes":["all"],
"excludes":[]
}
}
How do I query to get both including and excluding orders correctly?
Example 1:
URL parameters: country=us&city=a
Expected result: Order1, Order2, Order3
Example 2:
URL parameters: country=us&city=x
Expected result: Order3
Example 3:
URL parameters: country=de&city=z
Expected result: Order2, Order3
One of things I've tried is similar to this:
$and: [{
$or: [{
'countries.code': 'all'
}, {
'countries.code': URL_PARAMETER_FOR_COUNTRY
}]
}, {
$and: [{
$or: [{
'cities.includes': 'all'
}, {
'cities.includes': URL_PARAMETER_FOR_CITY
}]
},
{
'cities.excludes': {
$ne: URL_PARAMETER_FOR_CITY
}
}]
}]
I think you were basically on the right track but you're getting confused with the "nesting" of logic and the usage of operators like $and. All MongoDB query arguments are already an "AND* condition, so there is generally no need to specify this.
Also when you have more than one condition for an $or on exactly the same field, it's a lot cleaner to write this with $in instead:
db.collection.find({
"countries.code": { "$in": ["all",COUNTRY_PARAM] },
"cities.includes": { "$in": ["all",CITY_PARAM] },
"cities.excludes": { "$ne": CITY_PARAM }
})
Even if you wanted to roll in "mutliple" tests for the "excludes" such as an "all" value, then just apply the "list non-equivalent" statement which is $nin:
db.collection.find({
"countries.code": { "$in": ["all",COUNTRY_PARAM] },
"cities.includes": { "$in": ["all",CITY_PARAM] },
"cities.excludes": { "$nin": ["all",CITY_PARAM }
})
That meets all described combinations of returning the matching documents.
The reason query arguments are an "AND" condition without specifying it is because generally that is what people mean, hence it makes the expression much more simplified than the terse representation, and far less confusing.
AND with AND means "AND" on all conditions, so no need to partition.