Query Mongoose with priority query conditions - mongodb

I need to find all documents where first query argument matches then if it can't find more documents that match that query it should apply another query argument.
so for example:
db.users.find({
$or: [
{ type: typeToSearch }, // First find all users that has type=typeToSearch
{ active: true } // then if it can't find more users with type=typeToSearch then look for active one
]})
.limit(20)
What actually this query does it will find active users first (depending on the order in a collection).
What I need is - if I have 18 users that have given type then they should be returned first and then 2 any active.

That's a cool feature you are looking for! Nothing in Mongoose will help you with this out of the box, and poking around in npm I don't see anything that will help you there either.
For your two queries you have to do something like this:
const fancyQuery = async limit => {
const first = await db.users.find({ type: typeToSearch }).limit(20)
let second = [];
if (first.length < limit)
second = await db.users.find({ active: true,
type:{$ne:typeToSearch}} })
.limit(20-first.length)
return [...first, ...second]
}
The only other path I can think of using the query api, is to fetch 40 items and then to filter the results with javascript. I think you'd need to change your query a little to prevent the active = true part of the query from also refetching all the same documents as the type query:
db.users.find({
$or: [
{ type: typeToSearch },
{ active: true,
type: {$ne: typeToSearch}
}
]})
.limit(40)
You'd filter the results first by type and then by not type and active up to 20 items.
You might also be able to use an aggregation pipeline to accomplish this, but I don't have an answer like that at my finger-tips.

Related

MongoDB Aggregate conditional match criteria

I am looking to set up a series of filter options on the ui and then have the backend perform the search due to the amount of data that could come back with no filters. Not all filter criteria may be present in each search case. With $and, if one of the conditions is not present, nothing returns, with $or, if only 1 resolves to true, it can end up returning all documents.
Here's along the lines of what I've tried:
await Visit.aggregate()
.match(
{
$and: [
{client: clientId},
{$or: [
{user: userId},
{visitStart: {$gte: new Date(isStart)}},
{visitEnd: {$lte: new Date(isEnd)}}
]}
]
}
)
.sort({ date: 'asc'})
.exec((err, result) => {
if (err) {
res.send(err);
} else {
res.json(result);
}
});
This at least only returns documents relevant to the main criteria, the client. But if any of the 3 conditional criteria resolves to true then it will return more than the requested search. For example, if I specify a userId, a start date, and an end date, as long as any document matches before the end date (even if it does not meet the other 2 criteria) it gets returned in the results. I'm sure there has to be a clean way to make such a conditional search, any thoughts would be appreciated.

MongoDB query is performing slow when using $in operator

With some complex queries and aggregation, I got the list of object ids as below:
var objectIdsCollection = Array of ObjectIds
I have another collection, it's say Collection1 and It has OtherCollectionObjectId. Now, I need to filter this Collection1 based on object ids I got in an array.
Collection1 {
_id: ObjectId('xyz'),
name: ..
someOtherAttributes: ...
OtherCollectionObjectId: ObjectId('abc')
}
Below is the query which I am trying. I need to use await because, the result of this query is dependent on another query.
let queryData = await Document1.aggregate([
{
$match: {
OtherCollectionObjectId: {
$in: objectIdsCollection,
},
Deleted: false,
},
},
]);
But, this query is performing way slow. Sometimes it takes around 1 minute to fetch the results.
I tried couple of other suggestions from internet but nothing seems to work for this kind of scenario.
Please let me know anything which can improve the performance of this query.
Thanks

MongoDB big collection aggregation is slow

I'm having a problem with the time of my mongoDB query, from a node backend using mongoose. i have a collection called people that has 10M records, and every record is queried from the backend and inserted from another part of the system that's written in c++ and needs to be very fast.
this is my mongoose schema:
{
_id: {type: String, index: {unique: true}}, // We generate our own _id! Might it be related to the slowness?
age: { type: Number },
id_num: { type: String },
friends: { type: Object }
}
schema.index({'id_num': 1}, { unique: true, collation: { locale: 'en_US', strength: 2 } })
schema.index({'age': 1})
schema.index({'id_num': 'text'});
Friends is an object looking like that: {"Adam": true, "Eve": true... etc.}.
there's no meaning to the value, and we use dictionaries to avoid duplicates fast on C++.
also, we didn't encounter a set/unique-list type of field in mongoDB.
The Problem:
We display people in a table with pagination. the table has abilities of sort, search, and select number of results.
At first, I queried all people and searched, sorted and paged it on the js. but when there are a lot of documents, It's turning problematic (memory problems).
The next thing i did was to try to fit those manipulations (searching, sorting & paging) on my query.
I used mongo's text search- but it not matches a partial word. is there any way to search a partial insensitive string? (I prefer not to use regex, to avoid unexpected problems)
I have to sort before paging, so I tried to use mongo sort. the problem is, that when the user wants to sort by "Friends", we want to return the people sorted by their number of friends (number of entries in the object).
The only way i succeeded pulling it off was using $addFields in aggregation:
{$addFields: {$size: {$ifNull: [{$objectToArray: '$friends'}, [] ]}}}
this addition is taking forever! when sorting by friends, the query takes about 40s for 8M people, and without this part it takes less than a second.
I used limit and skip for pagination. it works ok, but we have to wait until the user requests the second page and make another very long query.
In the end, this is the the interesting code part:
const { sortBy, sortDesc, search, page, itemsPerPage } = req.query
// Search never matches partial string
const match = search ? {$text: {$search: search}} : {}
const sortByInDB = ['age', 'id_num']
let sort = {$sort : {}}
const aggregate = [{$match: match}]
// if sortBy is on a simple type, we just use mongos sort
// else, we sortBy friends, and add a friends_count field.
if(sortByInDB.includes(sortBy)){
sort.$sort[sortBy] = sortDesc === 'true' ? -1 : 1
} else {
sort.$sort[sortBy+'_count'] = sortDesc === 'true' ? -1 : 1
// The problematic part of the query:
aggregate.push({$addFields: {friends_count: {$size: {
$ifNull: [{$objectToArray: '$friends'},[]]
}}}})
}
const numItems = parseInt(itemsPerPage)
const numPage = parseInt(page)
aggregate.push(sort, {$skip: (numPage - 1)*numItems}, {$limit: numItems})
// Takes a long time (when sorting by "friends")
let users = await User.aggregate(aggregate)
I tried indexing all simple fields, but the time is still too much.
The only other solution i could think of, is making mongo calculate a field "friends_count" every time a document is created or updated- but i have no idea how to do it, without slowing our c++ that writes to the DB.
Do you have any creative idea to help me? I'm lost, and I have to shorten the time drastically.
Thank you!
P.S: some useful information- the C++ area is writing the people to the DB in a bulk once in a while. we can sync once in a while and mostly rely on the data to be true. So, if that gives any of you any idea for a performance boost, i'd love to hear it.
Thanks!

How to filter minimongo collection with more parameters in meteor

I need help with filtering reactively minimongo collection with 3+ parameters. Firs, I have loaded minimongo from server's mongo collection with publish/subscribe. Now, I want to filter that collection and let user see only objects from collection that are same like filters. First of all I have search query filter that checks if input string is same as some filed in collection:
Job.find({ $or: [ { Name: regex }, { Description: regex }, ]});
This part is doing his job well. And now second filter: I have field in object that is true if that particular job is remote friendly and false if it is not, I wanted to if user enables that filter he only see just remote friendly job positions, and if he disables that, he can see all jobs available ( + search query of course):
if (remoteQuery === true ) {
return Job.find({ $or: [ { Name: regex }, { Description: regex }, ] , Remote: true});
}
This is working, but it is bad way for doing this. And now biggest problem comes with last filter: I have one more field that is storing "job" (collection object) type. Type can be 1, 2 , 3 or 4. So how could I say to minimongo e.g. "Show only jobs that have "Front-end" inside (search query), that are remote friendly, and have third filter 2 and 3 inside"
Job.find({ $or: [ { Name: "Front-end"}, { Description: "Front-end"}, ] , Remote: true, positionType: [2,3],});
Something like this? Thanks!
Sounds like you are looking for the MongoDB query $in operator:
The $in operator selects the documents where the value of a field equals any value in the specified array.
Therefore your 3rd query could look like:
Job.find({
positionType: {
$in: [2, 3] // Matches documents where `positionType` is either 2 or 3
},
// Other conditions…
});

Update with expression instead of value

I am totally new to MongoDB... I am missing a "newbie" tag, so the experts would not have to see this question.
I am trying to update all documents in a collection using an expression. The query I was expecting to solve this was:
db.QUESTIONS.update({}, { $set: { i_pp : i_up * 100 - i_down * 20 } }, false, true);
That, however, results in the following error message:
ReferenceError: i_up is not defined (shell):1
At the same time, the database did not have any problem with eating this one:
db.QUESTIONS.update({}, { $set: { i_pp : 0 } }, false, true);
Do I have to do this one document at a time or something? That just seems excessively complicated.
Update
Thank you Sergio Tulentsev for telling me that it does not work. Now, I am really struggling with how to do this. I offer 500 Profit Points to the helpful soul, who can write this in a way that MongoDB understands. If you register on our forum I can add the Profit Points to your account there.
I just came across this while searching for the MongoDB equivalent of SQL like this:
update t
set c1 = c2
where ...
Sergio is correct that you can't reference another property as a value in a straight update. However, db.c.find(...) returns a cursor and that cursor has a forEach method:
Queries to MongoDB return a cursor, which can be iterated to retrieve
results. The exact way to query will vary with language driver.
Details below focus on queries from the MongoDB shell (i.e. the
mongo process).
The shell find() method returns a cursor object which we can then iterate to retrieve specific documents from the result. We use
hasNext() and next() methods for this purpose.
for( var c = db.parts.find(); c.hasNext(); ) {
print( c.next());
}
Additionally in the shell, forEach() may be used with a cursor:
db.users.find().forEach( function(u) { print("user: " + u.name); } );
So you can say things like this:
db.QUESTIONS.find({}, {_id: true, i_up: true, i_down: true}).forEach(function(q) {
db.QUESTIONS.update(
{ _id: q._id },
{ $set: { i_pp: q.i_up * 100 - q.i_down * 20 } }
);
});
to update them one at a time without leaving MongoDB.
If you're using a driver to connect to MongoDB then there should be some way to send a string of JavaScript into MongoDB; for example, with the Ruby driver you'd use eval:
connection.eval(%q{
db.QUESTIONS.find({}, {_id: true, i_up: true, i_down: true}).forEach(function(q) {
db.QUESTIONS.update(
{ _id: q._id },
{ $set: { i_pp: q.i_up * 100 - q.i_down * 20 } }
);
});
})
Other languages should be similar.
//the only differnce is to make it look like and aggregation pipeline
db.table.updateMany({}, [{
$set: {
col3:{"$sum":["$col1","$col2"]}
},
}]
)
You can't use expressions in updates. Or, rather, you can't use expressions that depend on fields of the document. Simple self-containing math expressions are fine (e.g. 2 * 2).
If you want to set a new field for all documents that is a function of other fields, you have to loop over them and update manually. Multi-update won't help here.
Rha7 gave a good idea, but the code above is not work without defining a temporary variable.
This sample code produces an approximate calculation of the age (leap years behinds the scene) based on 'birthday' field and inserts the value into suitable field for all documents not containing such:
db.employers.find({age: {$exists: false}}).forEach(function(doc){
var new_age = parseInt((ISODate() - doc.birthday)/(3600*1000*24*365));
db.employers.update({_id: doc._id}, {$set: {age: new_age}});
});
Example to remove "00" from the beginning of a caller id:
db.call_detail_records_201312.find(
{ destination: /^001/ },
{ "destination": true }
).forEach(function(row){
db.call_detail_records_201312.update(
{ _id: row["_id"] },
{ $set: {
destination: row["destination"].replace(/^001/, '1')
}
}
)
});