Let's say I have a list of items. I need to find (return a cursor) exactly 8 items. First I need to see how many featured items are there. If I can get 8 featured items, then no issue. But if the count is less than 8, I need to randomly items until I get 8.
Is it possible to do this in mongodb?
If you sort the cursor by your featured field you can pick up the featured ones first and then fill in with others:
const noMoreThan8Docs = MyCollection.find({},{ sort: { featured: -1 }, limit: 8 });
This assumes that featured is a boolean key. Booleans sort false-then-true so you need to reverse the sort.
I'm not sure how random the documents that are selected after the featured ones will be. However, since you're using Meteor and Meteor uses random _ids (unlike MongoDB native) you can sort on that key as well.
const noMoreThan8Docs = MyCollection.find({},{ sort: { featured: -1, _id: 1 }, limit: 8 });
This is also not truly random since the same non-featured documents will tend to sort first. If you want to really randomize the non-featured items you'll want to do a random find of those and append them if you have less than 8 featured documents.
I think what you want to do is pad out the list of items to make sure you always return 8. You can do this in the helper method,
var rows = MyTable.find({search: "Something"}).limit(8).fetch();
for (var i=rows.length;i<8;i++) {
rows.push({name: "Empty data row "+i}):
}
return rows;
Related
I'm using MongoDB 4.0 on mongoDB Atlas cluster (3 replicas - 1 shard).
Assuming i have a collection that contains multiple documents.
Each of this documents holding an array out of subdocuments that represent cities in a certain year with additional information. An example document would look like that (i removed unessesary information to simplify example):
{_id:123,
cities:[
{name:"vienna",
year:1985
},
{name:"berlin",
year:2001
}
{name:"vienna",
year:1985
}
]}
I have a compound index on and year. What is the fastest way to count the occurrences of name and year combinations?
I already tried the following aggregation:
[{$unwind: {
path: '$cities'
}}, {$group: {
_id: {
name: 'cities.name',
year: '$cities.year'
},
count: {
$sum: 1
}
}}, {$project: {
count: 1,
name: '$_id.name',
year: '$_id.year',
_id: 0
}}]
Another approach i tried was a map-reduce in the following form - the map reduce performed a bit better ~30% less time needed.
map function:
function m() {
for (var i in this.cities) {
emit({
name: this.cities[i].name,
year: this.cities[i].year
},
1);
}
}
reduce function (also tried to replace sum with length, but surprisingly sum is faster):
function r(id, counts) {
return Array.sum(counts);
}
function call in mongoshell:
db.test.mapReduce(m,r,{out:"mr_test"})
Now i was asking myself - Is it possible to access the index? As far as i know it is a B+ tree that holds the pointers to the relevant documents on disk, therefore from a technical point of view I think is would be possible to iterate through all leaves of the index tree and just counting the pointers? Does anybody if this is possible?
Does anybody knows another way to solve this approach in a high performant way? (It is not possible to change the design, because of other dependencies of the software, we are running this on a very big dataset). Has anybody maybe experience in solve such task via shards?
The index will not be very helpful in this situation.
MongoDB indexes were designed for identifying documents that match a given critera.
If you create an index on {cities.name:1, cities.year:1}
This document:
{_id:123,
cities:[
{name:"vienna",
year:1985
},
{name:"berlin",
year:2001
}
{name:"vienna",
year:1985
}
]}
Will have 2 entries in the b-tree that refer to this document:
vienna|1985
berlin|2001
Even if it were possible to count the incidence of a specific key in the index, this does not necessarily correspond.
MongoDB does not provide a method to examine the raw entries in an index, and it explicitly refuses to use an index on a field containing an array for counting.
The MongoDB count command and helper functions all count documents, not elements inside of them. As you noticed, you can unwind the array and count the items in an aggregation pipeline, but at that point you've already loaded all of the documents into memory, so it's too late to make use of an index.
Im trying to return the values from a 'repeating item' input. But 'inputs.value' doesn't work. I think I need to creat a loop and index for every item on the list but not sure.
I think I need to create a loop and index for every item on the list..
Yes. You are right on that. The 'Repeating Item' is internally stored as an object array. So you need to iterate that array to process it. The individual items are objects and hence you will not be available on the inputs object directly, but via inputs.lineitems, where lineitems is the property name of the repeating item prototype.
For example:
You are creating a repeating items list of items which you want to order. So, you have two inputs inside the repeating items prototype, say itemName and itemQuantity. You name the repeating items property name as LineItems. You want to display it as an output table and also display total quantity ordered. The output table is named Orders and the total label is named Total.
You could then iterate this to process it further, like this:
var result = [], totalItems = 0;
Where result is an array that you would want to map to your output table, and totalItems is where you would cache the total quantity.
inputs.LineItems.forEach(function(item, idx) {
totalItems += item.itemQuantity;
result.push({
'ItemNumber': idx + 1,
'Item': item.itemName,
'Quantity': item.itemQuantity
});
});
Where, you are iterating the repeating items via inputs.LineItems and increment total accordingly . You also prepare the result array to map to the Orders table later on.
This is what you return:
return {
Total: totalItems,
Orders: result
};
Where, Total is the output label you defined earlier, and Orders is the out put table name you defined earlier.
Here is a demo for you to understand it better:
https://jscalc.io/calc/YicDJYCSlYTGYFMS
To see the source, just click on the ellipsis (three dots shown after the 'Powered by JSCalc.io' text) and click "make a copy".
Hope this helps.
I'm trying to create an android app which pulls first 1-10 documents in the mongodb collection and show those item in a list, then later when the list reaches the end i want to pull 11-20 documents in the mongodb collection and it goes on.
def get_all_tips(from_item, max_items):
db = client.MongoTip
tips_list = db.tips.find().sort([['_id', -1]]).limit(max_items).skip(from_item)
if tips_list.count() > 0:
from bson import json_util
return json_util.dumps(tips_list, default=json_util.default), response_ok
else:
return "Please move along nothing to see here", response_invalid
pass
But the above code does not work the way i intended but rather it returns from from_item to max_items
Example: calling get_all_tips(3,4)
It returns:
Document 3, Document 4, Document 5, Document 6
I'm expecting:
Document 3, Document 4
In your code you are specifying two parameters.
from_item: which is the starting index of the documents to return
max_items: number of items to return
Therefore calling get_all_tips(3,4) will return 4 documents starting from document 3 which is exactly what's happening.
Proposed fixes:
If you want it to return documents 3 and 4 call get_all_tips(3,2) instead, which means return a maximum of two documents starting from 3.
If you'd rather like to specify the start and end indexes in your function, I recommend the following changes:
def get_all_tips(from_item, to_item):
if to_item < from_item:
return "to_item must be greater than from item", bad_request
db = client.MongoTip
tips_list = db.tips.find().sort([['_id', -1]]).limit(to_item - from_item).skip(from_item)
That being said, I'd like to point out that MongoDb documentation does not recommend use of skip for pagination for large collections.
MongoDb 3.2 cursor.skip
I want to search in the first 1000 records of my document whose name is CityDB. I used the following code:
db.CityDB.find({'index.2':"London"}).limit(1000)
but it does not work, it return the first 1000 of finding, but I want to search just in the first 1000 records not all records. Could you please help me.
Thanks,
Amir
Note that there is no guarantee that your documents are returned in any particular order by a query as long as you don't sort explicitely. Documents in a new collection are usually returned in insertion order, but various things can cause that order to change unexpectedly, so don't rely on it. By the way: Auto-generated _id's start with a timestamp, so when you sort by _id, the objects are returned by creation-date.
Now about your actual question. When you first want to limit the documents and then perform a filter-operation on this limited set, you can use the aggregation pipeline. It allows you to use $limit-operator first and then use the $match-operator on the remaining documents.
db.CityDB.aggregate(
// { $sort: { _id: 1 } }, // <- uncomment when you want the first 1000 by creation-time
{ $limit: 1000 },
{ $match: { 'index.2':"London" } }
)
I can think of two ways to achieve this:
1) You have a global counter and every time you input data into your collection you add a field count = currentCounter and increase currentCounter by 1. When you need to select your first k elements, you find it this way
db.CityDB.find({
'index.2':"London",
count : {
'$gte' : currentCounter - k
}
})
This is not atomic and might give you sometimes more then k elements on a heavy loaded system (but it can support indexes).
Here is another approach which works nice in the shell:
2) Create your dummy data:
var k = 100;
for(var i = 1; i<k; i++){
db.a.insert({
_id : i,
z: Math.floor(1 + Math.random() * 10)
})
}
output = [];
And now find in the first k records where z == 3
k = 10;
db.a.find().sort({$natural : -1}).limit(k).forEach(function(el){
if (el.z == 3){
output.push(el)
}
})
as you see your output has correct elements:
output
I think it is pretty straight forward to modify my example for your needs.
P.S. also take a look in aggregation framework, there might be a way to achieve what you need with it.
Given the Schema Displayed Below & MongoDB shell version: 2.0.4:
How would I go about counting the number of items in the impressions array ?
I am thinking I have to do a Map Reduce, which seems to be a little complex, I thought that the count Function would do it. Can someone Demonstrate this?
A simple example of counting is:
var count = 0;
db.engagement.find({company_name: "me"},{impressions:1}).forEach(
function (doc) {
count += doc.impressions.length;
}
)
print("Impressions: " + count);
If you have a large number of documents to process, you would be better maintaining the count as an explicit field. You could either update the count when pushing to the impressions array, or use an incremental MapReduce to re-count for updated documents as needed.