Best way to doing this in mongodb? - mongodb

Question

db.post
{user:'50001', content:...,},
{user:'50002', content:...,},
{user:'50003', content:...,},
{user:'50001', content:...,},
{user:'50002', content:...,},
db.catagory
{user:'50001', type:'public',...}
{user:'50002', type:'vip',...}
{user:'50003', type:'public',...}
What I want, is pickup the post of user what type:public.
I can do it like:
publicUsers = db.catagory.distinct(user, {type:'public'})
db.post.find({user: {$in: publicUsers }}).sort({_id:-1})
or use lookup in aggregate.
and output
{user:'50001', content:...,},
{user:'50003', content:...,},
{user:'50001', content:...,},
Is that some other better way to do this faster?
Consider of the large request situation, should I create a new db like post_public for finding?

Answer 1

The aggregation pipeline is your best bet. It's quite performant, as it's intended for just these types of situations.
Another Edit
Ok, this is the best solution I have, but it has some limitations set by Mongo.
Firstly, a sort operation on non-indexed fields cannot use more than 100MB of RAM. That's quite a bit of space for text based posts, but something to be aware of (there's a workaround that uses disk space if needed, but it will slow down the operation).
Second, and more relevant in this case, Mongo sets a hard cap on the size of a single document to 16MB. Given that you wanted the posts in a single array, a large dataset can potentially pass that if no limits on the output are imposed. If you decide you don't want all posts in a single array, by removing the last three stages of the pipeline ($group, $addFields, and $project), the result will be the sorted posts each within their own document. You will need to limit the number of documents manually to get the n number of posts you want if you go that route. IMO, you should do this. Those last three stages add additional processing burden but offer very little practical benefit on the output. The only difference is iterating over a nested array, vs iterating over the root array.
db.categories.aggregate([
{$project: {
user: 1,
type: 1
}},
{$match: {
type: "public"
}},
{$lookup: {
from: 'posts',
localField: 'user',
foreignField: 'user',
as: 'userPosts'
}},
{$unwind: {
path: "$userPosts",
preserveNullAndEmptyArrays: false
}},
{$sort: {
"userPosts.posted": -1,
}},
{$group: {
_id: "public",
posts: {
$push: "$userPosts"
}
}},
{$addFields: {
"latestPosts": {
$slice: ["$posts", 5]
}
}},
{$project: {
posts: 0
}}
]);
You'll access the resulting posts like so:
.then(result => {
let latestPosts = result[0].latestPosts;
});
Edit
I built a mock database and came up with this solution:
db.categories.aggregate([{
$project: {
_id: 0,
user: 1,
type: 1,
}
}, {
$match: {
type: "public"
}
}, {
$group: {
_id: {
$concat: ["$user", "-", "$type"]
},
user: {
$first: "$user"
},
type: {
$first: "$type"
}
}
}, {
$sort: {
user: -1
}
}, {
$limit: 10
}, {
$lookup: {
from: 'posts',
localField: 'user',
foreignField: 'user',
as: 'userPosts'
}
}]);
The result will be an array of objects representing each unique user-type combo with the type specified in the $match filter. The object will contain a child array with all the posts made by that user. All the users will be sorted ind descending order and limited to ten.
If all you want is a single operator to get distinct values from categories:
db.categories.aggregate([{$project: {
_id: 0,
user: 1,
type: 1
}}, {$group: {
_id: {$concat: ["$user","-","$type"]},
user: {$first: "$user"},
type: {$first: "$type"}
}}]);
The goal is to first filter out the users with non-public types so they don't appear in the result, join the posts belonging to that user to a new field called "user_posts", and then sort all the users in ascending order. I don't know the entire structure of your database so my specific example may not be the best fit, but it would be a good starting point.
Edit
Some other ways to increase performance for large amounts of data:
By querying in pieces. Say for example you're displaying paginated user posts. Then you would only need to query the posts for the page currently being viewed. You can reduce any noticeable loading times by pre-fetching the next few pages as well.
Using a "virtual" list. This will display every post in one long list, but only fetch a certain size chunk that is currently being viewed, or will soon be viewed. Similar to the first pagination example. If you're familiar with the RecyclerView in Android development, that's the idea.
Restructuring the way posts are stored in your database. It would be a much faster operation if Mongo didn't need to iterate through so much unrelated data when looking for posts of a specific user or type. The solution is to store posts nested below a common ancestor, such as their poster, or their view type. This does mean that you may have duplicate entries spread between documents. This is one of the inherent downsides of a no-SQL database. You will need to find a solution for handling synchronizing the documents spread throughout the database.
Creating a new index on document fields that you use to sort your array. Indices help Mongo organize the documents internally for more efficient searches.
Running a heavy query once, then $out-ing the result. In this way you can store the result of expensive operations in a more permanent fashion than a simple cache.
A variant of number 5 is to $out the result of a large sorting operation, and then continuing to insert new entries at the end of the now sorted collection.

Answer 2

Finally, I do it like this.
db.post.aggregate([
{$match:{
uid:{$in: db.catagory.distinct('id',{type:'public'})} # directly write here
_id:{$lt: ObjectId("5c6d5494b9f43f020d4122a0")} # for turn the page
}},
{$sort:{_id:-1}}, # sort by post _id
{$limit:20} # page size
])
It's faster than lookup or others.