Mongodb aggregate match query with priority on full match - mongodb

I am attempting to do a mongodb regex query on a field. I'd like the query to prioritize a full match if it finds one and then partials afterwards.
For instance if I have a database full of the following entries.
{
"username": "patrick"
},
{
"username": "robert"
},
{
"username": "patrice"
},
{
"username": "pat"
},
{
"username": "patter"
},
{
"username": "john_patrick"
}
And I query for the username 'pat' I'd like to get back the results with the direct match first, followed by the partials. So the results would be ordered ['pat', 'patrick', 'patrice', 'patter', 'john_patrick'].
Is it possible to do this with a mongo query alone? If so could someone point me towards a resource detailing how to accomplish it?
Here is the query that I am attempting to use to perform this.
db.accounts.aggregate({ $match :
{
$or : [
{ "usernameLowercase" : "pat" },
{ "usernameLowercase" : { $regex : "pat" } }
]
} })

Given your precise example, this could be accomplished in the following way - if your real world scenario is a little bit more complex you may hit problems, though:
db.accounts.aggregate([{
$match: {
"username": /pat/i // find all documents that somehow match "pat" in a case-insensitive fashion
}
}, {
$addFields: {
"exact": {
$eq: [ "$username", "pat" ] // add a field that indicates if a document matches exactly
},
"startswith": {
$eq: [ { $substr: [ "$username", 0, 3 ] }, "pat" ] // add a field that indicates if a document matches at the start
}
}
}, {
$sort: {
"exact": -1, // sort by our primary temporary field
"startswith": -1 // sort by our seconday temporary
}
}, {
$project: {
"exact": 0, // get rid of the "exact" field,
"startswith": 0 // same for "startswith"
}
}])
Another way would be using $facet which may prove a bit more powerful by enabling more complex scenarios but slower (several people here will hate me, though, for this proposal):
db.accounts.aggregate([{
$facet: { // run two pipelines against all documents
"exact": [{ // this one will capture all exact matches
$match: {
"username": "pat"
}
}],
"others": [{ // this one will capture all others
$match: {
"username": { $ne: "pat", $regex: /pat/i }
}
}]
}
}, {
$project: {
"result": { // merge the two arrays
$concatArrays: [ "$exact", "$others" ]
}
}
}, {
$unwind: "$result" // flatten the resulting array into separate documents
}, {
$replaceRoot: { // restore the original document structure
"newRoot": "$result"
}
}])

Related

get document with same 3 fields in a collection

i have a collection with more then 1000 documents and there are some documents with same value in some fields, i need to get those
the collection is:
[{_id,fields1,fields2,fields3,etc...}]
what query can i use to get all the elements that have the same 3 fields for example:
[
{_id:1,fields1:'a',fields2:1,fields3:'z'},
{_id:2,fields1:'a',fields2:1,fields3:'z'},
{_id:3,fields1:'f',fields2:2,fields3:'g'},
{_id:4,fields1:'f',fields2:2,fields3:'g'},
{_id:5,fields1:'j',fields2:3,fields3:'g'},
]
i need to get
[
{_id:2,fields1:'a',fields2:1,fields3:'z'},
{_id:4,fields1:'f',fields2:2,fields3:'g'},
]
in this way i can easly get a list of "duplicate" that i can delete if needed, it's not really important get id 2 and 4 or 1 and 3
but 5 would never be included as it's not 'duplicated'
EDIT:
sorry but i forgot to mention that there are some document with null value i need to exclude those
This is the perfect use case of window field. You can use $setWindowFields to compute $rank in the grouping/partition you want. Then, get those rank not equal to 1 to get the duplicates.
db.collection.aggregate([
{
$match: {
fields1: {
$ne: null
},
fields2: {
$ne: null
},
fields3: {
$ne: null
}
}
},
{
"$setWindowFields": {
"partitionBy": {
fields1: "$fields1",
fields2: "$fields2",
fields3: "$fields3"
},
"sortBy": {
"_id": 1
},
"output": {
"duplicateRank": {
"$rank": {}
}
}
}
},
{
$match: {
duplicateRank: {
$ne: 1
}
}
},
{
$unset: "duplicateRank"
}
])
Mongo Playground
I think you can try this aggregation query:
First group by the feilds you want to know if there are multiple values.
It creates an array with the _ids that are repeated.
Then get only where there is more than one ($match).
And last project to get the desired output. I've used the first _id found.
db.collection.aggregate([
{
"$group": {
"_id": {
"fields1": "$fields1",
"fields2": "$fields2",
"fields3": "$fields3"
},
"duplicatesIds": {
"$push": "$_id"
}
}
},
{
"$match": {
"$expr": {
"$gt": [
{
"$size": "$duplicatesIds"
},
1
]
}
}
},
{
"$project": {
"_id": {
"$arrayElemAt": [
"$duplicatesIds",
0
]
},
"fields1": "$_id.fields1",
"fields2": "$_id.fields3",
"fields3": "$_id.fields2"
}
}
])
Example here

How to remove same property name from sub object and push the sub objects as element of an array in mongo db

The coursesMarks property is present in every object. So I want to push the value inside coursesMarks property, to an array and return the array to the user.
[
{
"coursesMarks": {
"_id": "634a9be567a1f07be02f71d8",
"courseCode": "cse1201",
"courseTitle": "SP"
}
},
{
"coursesMarks": {
"_id": "634a9be567a1f07be02f71db",
"courseCode": "cse1203",
"courseTitle": "DS"
}
}
]
Then expected output is:
[
{
"courses":
{
"_id": "634a9be567a1f07be02f71d8",
"courseCode": "cse1201",
"courseTitle": "SP"
},
{
"_id": "634a9be567a1f07be02f71db",
"courseCode": "cse1203",
"courseTitle": "DS"
}
}
]
I've asked a clarifying question in the comments. But if we assume that the sample data provided is a single document where the array is stored in a field named arr, then a pipeline similar to the following may be what you are looking for:
[
{
$addFields: {
courses: {
$map: {
input: "$arr",
in: "$$this.coursesMarks"
}
}
}
},
{
$unset: "arr"
}
]
Playground example here
Edit
Based on the additional information about the structure of the data, you are looking to $group things in this particular case. Therefore the relevant addition to your pipeline should look something like this:
[
...
{
$group: {
_id: null,
courses: {
$push: "$coursesMarks"
}
}
},
{
$unset: "_id"
}
]
Playground demonstration here. It includes an empty $match stage at the beginning to represent whatever additional matching logic you currently have.

Mongodb lookup like search: local field as array of objects

I have two collections userProfile and skills,
Eg:userProfile
{
"_id": "5f72c6d4e23732390c96b031",
"name":"name"
"other_skills": [
"1","2"
],
"primary_skills": [
{
"_id": "607ffd1549e13876fef7f2c5",
"years": 4.5,
"skill_id": "1"
},
{
"_id": "607ffd1549e13876fef7f2c6",
"years": 2,
"skill_id": "2"
},
{
"_id": "607ffd1549e13876fef7f2c7",
"years": 1,
"skill_id": "3"
}
]
}
Eg:Skills
{
"_id":1,
"name": "Ruby on Rails",
}
{
"_id":2,
"name": "PHP",
}
{
"_id":3,
"name": "php",
}
I want to retrieve the userprofile based on the skills
eg: input of skill php i want to retrieve the userprofiles that matches either in primary_skills or other_skills
But I got confused about the implementation, I think it can do with pipeline in lookup and the elemMatch. This is the query I tried so far
const skills = ['php','PHP']
userProfile.aggrigate([{
$lookup:{
from:'skills',
let:{'primary_skills':'$primary_skills'},
pipeline:[
{
$match:{
primary_skills:{
$elemMatch:{
name:'' //not sure how to write match
}
}
}
}
]
}
}])
Can somebody help me with this, Thanks in advance
I'll first show you how to correct your pipeline to work, however this approach is very inefficient as you will have to $lookup on every single user in your db which is obviously a lot of overhead.
Here is how to properly match your condition:
const skills = ['php','PHP']
db.userProfile.aggregate([
{
$lookup: {
from: "skills",
let: {
"primary_skills": {
$map: {
input: "$primary_skills",
as: "skill",
in: "$$skill.skill_id"
}
},
"other_skills": "$other_skills"
},
pipeline: [
{
$match: {
$expr: {
"$in": [
"$_id",
{
"$concatArrays": [
"$$other_skills",
"$$primary_skills"
]
}
]
}
}
}
],
as: "skills"
}
},
{
$match: {
'skills.name': {$in: skills}
}
}
])
Mongo Playground
As I've said I recommend you do not do this. what I suggest you do is split it into 2 calls, first fetch the relevant skill ids. and then query on users.
By doing this you can also utilize indexes for much faster queries, like so:
const skills = ['php', 'PHP'];
const matchedSkillIds = await skills.distinct('_id', {name: {$in: skills}});
const users = await userProfile.find({
$or: [
{
'primary_skills.skill_id': {$in: matchedSkillIds}
},
{
'other_skills': {$in: matchedSkillIds}
}
]
})
Finally if you do insist on doing it in one query at the very least start the pipeline from the skill collection.

Mongo DB aggregate match not returning value

I have the following mongo db schema and I am trying to build an aggregate query that searches under github_open_issues under the repo key and can return me a match for all the values with repoA as the value. I have tried the following as my query however its not returning any result. Im a bit confused why this is not working as I have another db with a schema similar to this and this type of query works there but here something seems to be different and is not working. I have also put together this interactive example mongoplayground
query
db.collection.aggregate([
{
"$unwind": "$github_open_issues"
},
{
"$match": {
"github_open_issues.repo": {
"$in": [
"repoA"
]
}
}
},
])
schema
[
{
"github_open_issues": {
"0": {
"git_url": "https://github.com/",
"git_assignees": "None",
"git_open_date": "2019-09-26",
"git_id": 253113,
"repo": "repoA",
"git_user": "userA",
"state": "open"
},
"1": {
"git_url": "https://github.com/",
"git_assignees": "None",
"git_open_date": "2019-11-15",
"git_id": 294398,
"repo": "repoB",
"git_user": "userB",
"state": "open"
},
"2": {
"git_url": "https://github.com/",
"git_assignees": "None",
"git_open_date": "2021-04-12",
"git_id": 661208,
"repo": "repoA",
"state": "open"
}
},
"unique_label_seen": {
"568": {
"label_name": "some label",
"times_seen": 12,
"535": {
"label_name": "another label",
"times_seen": 1
}
}
}
}
]
$objectToArray convert github_open_issues object to array in key-value format
$filter to iterate loop of above converted array and filter your search condition
$match to filter github_open_issues not empty
$arrayToObject convert github_open_issues array to object
db.collection.aggregate([
{
$addFields: {
github_open_issues: {
$filter: {
input: { $objectToArray: "$github_open_issues" },
cond: { $in: ["$$this.v.repo", ["repoA"]] }
}
}
}
},
{ $match: { github_open_issues: { $ne: [] } } },
{ $addFields: { github_open_issues: { $arrayToObject: "$github_open_issues" } } }
])
Playground
You query is correct but you data in schema placed wrong inside github_open_issues.repo your objects are place by numbers like {"0": {values... }, "1":{values... }} which cannot get your desired value. You can check the playground now playground

how to lower case the value of unique:true keys in mongodb?

I have created a mongodb and by mistake have entered duplicate values in the form of capital and small case letters.
I have made the index unique. MongoDB is case sensitive and hence, considered the capital letter and small letter as different values.
Now my problem is the database have got around 32 GB. and I came across this issue. Kindly help me.
Here is the sample:
db.tt.createIndex({'email':1},{unique:true})
> db.tt.find().pretty()
{
"_id" : ObjectId("591d706c0ef9acde11d7af66"),
"email" : "g#gmail.com",
"src" : [
{
"acc" : "ln"
},
{
"acc" : "drb"
}
]
}
{
"_id" : ObjectId("591d70740ef9acde11d7af68"),
"email" : "G#gmail.com",
"src" : [
{
"acc" : "ln"
},
{
"acc" : "drb"
},
{
"acc" : "dd"
}
]
}
How I can make the email as lowercase and assign the src values to the original one. Kindly help me.
you can achive this using $toLower aggregation operator like this :
db.tt.aggregate([
{
$project:{
email:{
$toLower:"$email"
},
src:1
}
},
{
$unwind:"$src"
},
{
$group:{
_id:"$email",
src:{
$addToSet:"$src"
}
}
},
{
$project:{
_id:0,
email:"$_id",
src:1
}
},
{
$out:"anotherCollection"
}
])
$addToSet allow to keep oly one distinct occurence of src items
this will write this document to a new collection named anotherCollection:
{ "email" : "g#gmail.com", "src" : [ { "acc" : "dd" }, { "acc" : "drb" }, { "acc" : "ln" } ] }
Note that with $out, you can averwrite directly your tt collection, however before doing this make sure to understand what your doing because all previous data will be lost
The most efficient way I can think of to merge the data is run an aggregation and loop the result to write back to the collection in bulk operations:
var ops = [];
db.tt.aggregate([
{ "$unwind": "$src" },
{ "$group": {
"_id": { "$toLower": "$email" },
"src": { "$addToSet": "$src" },
"ids": { "$addToSet": "$_id" }
}}
]).forEach(doc => {
var id = doc.ids.shift();
ops = [
...ops,
{
"deleteMany": {
"filter": { "_id": { "$in": doc.ids } }
}
},
{
"updateOne": {
"filter": { "_id": id },
"update": {
"$set": { "email": doc._id },
"$addToSet": { "src": { "$each": doc.src } }
}
}
},
];
if ( ops.length >= 500 ) {
db.tt.bulkWrite(ops);
ops = [];
}
});
if ( ops.length > 0 )
db.tt.bulkWrite(ops);
In steps, that's $unwind the array items so they can be merged via $addToSet, under a $group on using $toLower on the email value. You also want to keep the set of unique source document ids.
In the loop you shift the first _id value off of doc.ids and update that document with the lowercase email and the revised "src" set. Using $addToSet here makes the operation write safe with any other updates that might occur to the document.
Then the other operation in the loop deletes the other documents that shared the same converted case email, so there are no duplicates. Actually do that one first. The default "ordered" operations make sure this is fine.
And do it in the shell, since it's a one-off operation and is really just as simple as listing as shown.