How to put First Document of a collection inside $cond for custom pagination in nestJs and mongodb - mongodb

I'm Implementing Keyset pagination in a nestjs and mongodb project. And I'm trying to customize the original pagination solution. I want to make a condition if the startId document is the first document present in the collection.
This is the Code I'm trying. AND THE ISSUE WITH THIS CODE IS THAT IT RETURNS ALL THE DOCUMENTS WHATEVER THE ID YOU GIVE IN THE QUERY. I KNOW MY LOGIC COULD BE WRONG BUT ONE THING ABOUT I'M SURE AND THAT IS I'M WRITING THE SYNTAX, IS WRONG, AS THIS IS MY FIRST TIME EXPERIECE WITH MONGO QUERIES
async findAll( page?: number, documentsToSkip = 0, limitOfDocuments?: number,
startId?: string,
): Promise<userDocument[]> {
return await this.userModel
.find({
$cond: {
if: { $first: { _id: startId } },
then: { $gte: startId },
else: { $gt: startId },
},
})
.skip(documentsToSkip)
.limit(limitOfDocuments);
}
Explanation of the above code. For example I'm dealing with Users
1- If the startId(passed in query) is equal to the _id of the first document present in the user collection then the below then should be executed
then: { $gte: startId },
2- If the startId(passed in query) is not equal to the _id of the first document present in the user collection then the below else should be executed. Lets's say the pagination limit is set to 10 documents per page. And If I'm providing the id of the 11th document then else should be executed
else: { $gt: startId },
REASON FOR All THIS
The keyset solution present on the internet uses this
_id: { $gt: startId } and with this the document of the startedId is automatically skipped. SO I'M TRYING TO DO IF THE PAGINATION STARTS FROM THE FIRST DOCUMENT THEN THE FIRST DOCUMENT ITSELF SHOULD BE PRESENT AND VISIBLE. BUT THEN IF USER MOVES TO THE SECOND PAGE OF THE PAGINATION THE LAST DOCUMENT OF THE FIRST PAGE SHOULD NOT BE VISIBLE AT THE SECOND PAGE, AS THE LAST DOCUMENT ID BECOMES THE STARTING ID FOR THE SECOND PAGE DOCUMENTS

I've made the solution for my particular scenario. The issue was I hade to customize original pagination according to the first document present in the collection. But $first wasn't working for me. AS $first returns the first element of an Array and not the first document itself.
SOLUTION
1- To find the first document use findOne() method without any parameter. And it will simply return the first document of the collection.
let firstDocument: userDocument = await this.userModel.findOne();
2-For simple customization of the Pagination
if (startId == firstDocument._id) {
return await this.userModel
.find({
_id: { $gte: startId },
})
.skip(documentsToSkip)
.limit(limitOfDocuments);
} else {
return await this.userModel
.find({
_id: { $gt: startId },
})
.skip(documentsToSkip)
.limit(limitOfDocuments);
}
3- I had to make other changes as well so my original function now look different In case If someone want to know
async findAll(
page?: number,
limitOfDocuments?: number,
startId?: string,
): Promise<PaginatedUsersDto> {
let firstDocument: userDocument = await this.userModel.findOne();
let count = await this.userModel.count();
let totalPages = (count / limitOfDocuments).toFixed();
console.log('First Doucment in User Colelction', firstDocument);
if (startId == firstDocument._id) {
this.paginatedUsers = await this.userModel
.find({
_id: { $gte: startId },
})
.sort({ _id: 1 })
.skip((page - 1) * limitOfDocuments)
.limit(limitOfDocuments);
} else {
this.paginatedUsers = await this.userModel
.find({
_id: { $gt: startId },
})
.skip((page - 1) * limitOfDocuments)
.limit(limitOfDocuments);
}
let paginatedObj = {
paginatedUsers: this.paginatedUsers,
totalPages: totalPages,
count: count,
};
return paginatedObj;
}

Related

Can't remove object in array using Mongoose

This has been extensively covered here, but none of the solutions seems to be working for me. I'm attempting to remove an object from an array using that object's id. Currently, my Schema is:
const scheduleSchema = new Schema({
//unrelated
_id: ObjectId
shifts: [
{
_id: Types.ObjectId,
name: String,
shift_start: Date,
shift_end: Date,
},
],
});
I've tried almost every variation of something like this:
.findOneAndUpdate(
{ _id: req.params.id },
{
$pull: {
shifts: { _id: new Types.ObjectId(req.params.id) },
},
}
);
Database:
Database Format
Within these variations, the usual response I've gotten has been either an empty array or null.
I was able slightly find a way around this and accomplish the deletion by utilizing the main _id of the Schema (instead of the nested one:
.findOneAndUpdate(
{ _id: <main _id> },
{ $pull: { shifts: { _id: new Types.ObjectId(<nested _id>) } } },
{ new: true }
);
But I was hoping to figure out a way to do this by just using the nested _id. Any suggestions?
The problem you are having currently is you are using the same _id.
Using mongo, update method allows three objects: query, update and options.
query object is the object into collection which will be updated.
update is the action to do into the object (add, change value...).
options different options to add.
Then, assuming you have this collection:
[
{
"_id": 1,
"shifts": [
{
"_id": 2
},
{
"_id": 3
}
]
}
]
If you try to look for a document which _id is 2, obviously response will be empty (example).
Then, if none document has been found, none document will be updated.
What happens if we look for a document using shifts._id:2?
This tells mongo "search a document where shifts field has an object with _id equals to 2". This query works ok (example) but be careful, this returns the WHOLE document, not only the array which match the _id.
This not return:
[
{
"_id": 1,
"shifts": [
{
"_id": 2
}
]
}
]
Using this query mongo returns the ENTIRE document where exists a field called shifts that contains an object with an _id with value 2. This also include the whole array.
So, with tat, you know why find object works. Now adding this to an update query you can create the query:
This one to remove all shifts._id which are equal to 2.
db.collection.update({
"shifts._id": 2
},
{
$pull: {
shifts: {
_id: 2
}
}
})
Example
Or this one to remove shifts._id if parent _id is equal to 1
db.collection.update({
"_id": 1
},
{
$pull: {
shifts: {
_id: 2
}
}
})
Example

How to return an aggregate query in an Apollo resolver (Meteor/Apollo/Graphql)?

I'm trying to run a query with an aggregate in Meteor and I've not been able to figure out how to return an aggregation query.
In my GraphQL schema:
type UserRelation {
date: String!
userID: String!
}
type UserData {
userID: String!
description: String!
friends_list: [UserRelation]!
}
extend type Query {
getFriends(userID: String!, friendID: String): [UserRelation]
}
In my resolvers.js:
import UserData from "./userData";
export default {
Query: {
getFriends(obj, args, context) {
if (args.friendID) return UserData.rawCollection().aggregate([
{
$match: {
_id: 'cuS7KebEQDRv2iFpJ'
}
},
{
$unwind: '$friends_list'
},
{
$match: {
'friends_list._id': '8GW4gjwWhkEexfndd'
}
}
]);
return UserData.findOne({ userID: args.userID }, {fields: {_id: 0, friends_list: 1 }}).friends_list;
}
}
}
In this hardcoded example, I have the following document in my database:
And I want to return the entire document with ONLY the user with a matched _id and whose friends_list has another user with the second matched _id. So this way I can return the entire document with only that 1 element in the friends_list array instead of all other elements as well.
Now this query works fine in Robo3T (a MongoDB GUI), but there's an issue with the return statement when I run it in Graphiql as an error is thrown which states:
"Expected Iterable, but did not find one for field \"Query.getFriends\"." When I log that statement, I see a Mongo AggregationCursor. But I'm not sure how I should "convert" this to a type that the resolver can return properly.
Any help is appreciated, thank you!
I don't know if I actually figured it out because I need to test it more, but this is working so far, in my resolvers.js I have:
import UserData from "./userData";
async function getFriend(userID, friendID) {
return UserData.rawCollection().aggregate([
{
$match: {
_id: userID
}
},
{
$unwind: '$friends_list'
},
{
$match: {
'friends_list._id': friendID
}
}
]).toArray();
}
export default {
Query: {
async getFriends(obj, args, context) {
if (args.friendID){
const result = await getFriend(args.userID, args.friendID);
return result[0].friends_list;
}
return UserData.findOne({ userID: args.userID }, {fields: {_id: 0, friends_list: 1 }}).friends_list;
}
}
}
By making it async I was able to get this done. Not sure what implications this has, but so far it works. If anyone is able to review this and give some critique to improve this it would be sincerely appreciated, as any information about this has been excruciatingly difficult to find, as I have not seen any use-cases of this so far.

Most efficient way to put fields of an embedded document in its parent for an entire MongoDB collection?

I am looking for the most efficient way to modify all the documents of a collection from this structure:
{
[...]
myValues:
{
a: "any",
b: "content",
c: "can be found here"
}
[...]
}
so it becomes this:
{
[...]
a: "any",
b: "content",
c: "can be found here"
[...]
}
Basically, I want everything under the field myValues to be put in its parent document for all the documents of a collection.
I have been looking for a way to do this in a single query using dbCollection.updateMany(), put it does not seem possible to do such thing, unless the content of myValues is the same for all documents. But in my case the content of myValues changes from one document to the other. For example, I tried:
db.getCollection('myCollection').updateMany({ myValues: { $exists: true } }, { $set: '$myValues' });
thinking it would perhaps resolve the myValues object and use that object to set it in the document. But it returns an error saying it's illegal to assign a string to the $set field.
So what would be the most efficient approach for what I am trying to do? Is there a way to update all the documents of the collection as I need in a single command?
Or do I need to iterate on each document of the collection, and update them one by one?
For now, I iterate on all documents with the following code:
var documents = await myCollection.find({ myValues: { $exists: true } });
for (var document = await documents.next(); document != null; document = await documents.next())
{
await myCollection.updateOne({ _id: document._id }, { $set: document.myValues, $unset: { myValues: 1} });
}
Since my collection is very large, it takes really long to execute.
You can consider using $out as an alternative, single-command solution. It can be used to replace existing collection with the output of an aggregation. Knowing that you can write following aggregation pipeline:
db.myCollection.aggregate([
{
$replaceRoot: {
newRoot: {
$mergeObjects: [ "$$ROOT", "$myValues" ]
}
}
},
{
$project: {
myValues: 0
}
},
{
$out: "myCollection"
}
])
$replaceRoot allows you to promote an object which merges the old $$ROOT and myValues into root level.

Limiting results in MongoDB but still getting the full count?

For speed, I'd like to limit a query to 10 results
db.collection.find( ... ).limit(10)
However, I'd also like to know the total count, so to say "there were 124 but I only have 10". Is there a good efficient way to do this?
By default, count() ignores limit() and counts the results in the entire query.
So when you for example do this, var a = db.collection.find(...).limit(10);
running a.count() will give you the total count of your query.
Doing count(1) includes limit and skip.
The accepted answer by #johnnycrab is for the mongo CLI.
If you have to write the same code in Node.js and Express.js, you will have to use it like this to be able to use the "count" function along with the toArray's "result".
var curFind = db.collection('tasks').find({query});
Then you can run two functions after it like this (one nested in the other)
curFind.count(function (e, count) {
// Use count here
curFind.skip(0).limit(10).toArray(function(err, result) {
// Use result here and count here
});
});
cursor.count() should ignore cursor.skip() and cursor.limit() by default.
Source: http://docs.mongodb.org/manual/reference/method/cursor.count/#cursor.count
You can use a $facet stage which processes multiple aggregation pipelines within a single stage on the same set of input documents:
// { item: "a" }
// { item: "b" }
// { item: "c" }
db.collection.aggregate([
{ $facet: {
limit: [{ $limit: 2 }],
total: [{ $count: "count" }]
}},
{ $set: { total: { $first: "$total.count" } } }
])
// { limit: [{ item: "a" }, { item: "b" }], total: 3 }
This way, within the same query, you can get both some documents (limit: [{ $limit: 2 }]) and the total count of documents ({ $count: "count" }).
The final $set stage is an optional clean-up step, just there to project the result of the $count stage, such that "total" : [ { "count" : 3 } ] becomes total: 3.
There is a solution using push and slice: https://stackoverflow.com/a/39784851/4752635
I prefe
First for filtering and then grouping by ID to get number of filtered elements. Do not filter here, it is unnecessary.
Second query which filters, sorts and paginates.
Solution with pushing $$ROOT and using $slice runs into document memory limitation of 16MB for large collections. Also, for large collections two queries together seem to run faster than the one with $$ROOT pushing. You can run them in parallel as well, so you are limited only by the slower of the two queries (probably the one which sorts).
I have settled with this solution using 2 queries and aggregation framework (note - I use node.js in this example, but idea is the same):
var aggregation = [
{
// If you can match fields at the begining, match as many as early as possible.
$match: {...}
},
{
// Projection.
$project: {...}
},
{
// Some things you can match only after projection or grouping, so do it now.
$match: {...}
}
];
// Copy filtering elements from the pipeline - this is the same for both counting number of fileter elements and for pagination queries.
var aggregationPaginated = aggregation.slice(0);
// Count filtered elements.
aggregation.push(
{
$group: {
_id: null,
count: { $sum: 1 }
}
}
);
// Sort in pagination query.
aggregationPaginated.push(
{
$sort: sorting
}
);
// Paginate.
aggregationPaginated.push(
{
$limit: skip + length
},
{
$skip: skip
}
);
// I use mongoose.
// Get total count.
model.count(function(errCount, totalCount) {
// Count filtered.
model.aggregate(aggregation)
.allowDiskUse(true)
.exec(
function(errFind, documents) {
if (errFind) {
// Errors.
res.status(503);
return res.json({
'success': false,
'response': 'err_counting'
});
}
else {
// Number of filtered elements.
var numFiltered = documents[0].count;
// Filter, sort and pagiante.
model.request.aggregate(aggregationPaginated)
.allowDiskUse(true)
.exec(
function(errFindP, documentsP) {
if (errFindP) {
// Errors.
res.status(503);
return res.json({
'success': false,
'response': 'err_pagination'
});
}
else {
return res.json({
'success': true,
'recordsTotal': totalCount,
'recordsFiltered': numFiltered,
'response': documentsP
});
}
});
}
});
});

How to limit number of updating documents in mongodb

How to implement somethings similar to db.collection.find().limit(10) but while updating documents?
Now I'm using something really crappy like getting documents with db.collection.find().limit() and then updating them.
In general I wanna to return given number of records and change one field in each of them.
Thanks.
You can use:
db.collection.find().limit(NUMBER_OF_ITEMS_YOU_WANT_TO_UPDATE).forEach(
function (e) {
e.fieldToChange = "blah";
....
db.collection.save(e);
}
);
(Credits for forEach code: MongoDB: Updating documents using data from the same document)
What this will do is only change the number of entries you specify. So if you want to add a field called "newField" with value 1 to only half of your entries inside "collection", for example, you can put in
db.collection.find().limit(db.collection.count() / 2).forEach(
function (e) {
e.newField = 1;
db.collection.save(e);
}
);
If you then want to make the other half also have "newField" but with value 2, you can do an update with the condition that newField doesn't exist:
db.collection.update( { newField : { $exists : false } }, { $set : { newField : 2 } }, {multi : true} );
Using forEach to individually update each document is slow. You can update the documents in bulk using
ids = db.collection.find(<condition>).limit(<limit>).map(
function(doc) {
return doc._id;
}
);
db.collection.updateMany({_id: {$in: ids}}, <update>})
The solutions that iterate over all objects then update them individually are very slow.
Retrieving them all then updating simultaneously using $in is more efficient.
ids = People.where(firstname: 'Pablo').limit(10000).only(:_id).to_a.map(&:id)
People.in(_id: ids).update_all(lastname: 'Cantero')
The query is written using Mongoid, but can be easily rewritten in Mongo Shell as well.
Unfortunately the workaround you have is the only way to do it AFAIK. There is a boolean flag multi which will either update all the matches (when true) or update the 1st match (when false).
As the answer states there is still no way to limit the number of documents to update (or delete) to a value > 1. A workaround to use something like:
db.collection.find(<condition>).limit(<limit>).forEach(function(doc){db.collection.update({_id:doc._id},{<your update>})})
If your id is a sequence number and not an ObjectId you can do this in a for loop:
let batchSize= 10;
for (let i = 0; i <= 1000000; i += batchSize) {
db.collection.update({$and :[{"_id": {$lte: i+batchSize}}, {"_id": {$gt: i}}]}),{<your update>})
}
let fetchStandby = await db.model.distinct("key",{});
fetchStandby = fetchStandby.slice(0, no_of_docs_to_be_updated)
let fetch = await db.model.updateMany({
key: { $in: fetchStandby }
}, {
$set:{"qc.status": "pending"}
})
I also recently wanted something like this. I think querying for a long list of _id just to update in an $in is perhaps slow too, so I tried to use an aggregation+merge
while (true) {
const record = db.records.findOne({ isArchived: false }, {_id: 1})
if (!record) {
print("No more records")
break
}
db.records.aggregate([
{ $match: { isArchived: false } },
{ $limit: 100 },
{
$project: {
_id: 1,
isArchived: {
$literal: true
},
updatedAt: {
$literal: new Date()
}
}
},
{
$merge: {
into: "records",
on: "_id",
whenMatched: "merge"
}
}
])
print("Done update")
}
But feel free to comment if this is better or worse that a bulk update with $in.