Is there any possibility of knowing the total number of items for the method Artist/Top without fetching all metadata?
Currently if you call: http://api.deezer.com/artist/27/top will return "total": 5
However if you set the limit parameter the total will be the number of returned items
http://api.deezer.com/artist/27/top?limit=22 returns "total": 22
For other methods the total field will return the total number of items that can be fetched.
Related
var product = db.GetCollection<Product>("Product");
var lookup1 = new BsonDocument(
"$lookup",
new BsonDocument {
{ "from", "Variant" },
{ "localField", "Maincode" },
{ "foreignField", "Maincode" },
{ "as", "variants" }
}
);
var pipeline = new[] { lookup1};
var result = product.Aggregate<Product>(pipeline).ToList();
The data of collection a is very large so it takes me 30 seconds to put the data in the list.
What should I do to make a faster lookup?
What that query is doing is retrieving every document from the Product collection, and then for each document found, perform a find query in the Variant collection. If there is no index on the Maincode field in the Variant collection, it will be reading the entire collection for each document.
This means that if there are, say, 1000 total products, with 3000 total variants (3 per product, on average), this query will be reading all 1000 documents from Product, and if that index isn't there, it would read all 3000 documents from Variant 1000 times, i.e. it will be examining 3 million documents.
Some ways to possibly speed this up:
create an index on {Maincode:1} in the Variant collection
This will reduce the number of documents that must be read in order to complete the lookup
change the schema
If the variants are stored in the same document with the product, there is no need for a lookup
filter the products prior to lookup
Again, reducing the documents read during the lookup
use a cursor to retrieve the documents in batches
If you perform any necessary sorting first, and the lookup last, you can return the documents to the application in batches, which would allow the application to display or begin processing the first batch before the second batch is available. This doen't make the query itself faster, but it can reduced the perceived wait in the application.
New to Mongodb I am trying understand the 100mb limit MongoDb for aggregate pipelines. Trying to find out what this actually means? Does it apply to the size of the database collection we are performing the aggregate on?
Bit of background we have the following query on an inventory ledger where we are taking a data set, running a group sum to find out which products are still in-stock (ie amount sum is greater than 0). Based on the result where the product is in stock we return those records by running a lookup in the original collection. The query is provided below.
Assume the inventory objects contains about 10 sub fields/record pair. And assume for 1000records/1mb.
QUESTION
My question is if the inventory collection size reaches 100mb as a JSON object array does this mean the call with fail? ie the max we can run the aggregate on is 100mb x 1000 records = 100,000 records?
BTW we are on a server that does not support writing to disk hence the question.
db.inventory.aggregate([
{
$group: {
_id: {
"group_id": "$product"
},
"quantity": {
$sum: "$quantity"
}
}
},
{
"$match": {
"quantity": {
$gt: 0
}
}
},
{
$lookup: {
from: "inventory",
localField: "_id.group_id",
foreignField: "$product",
as: "records"
}
}
])
The 100MB limit is a restriction on the amount of memory used by an aggregation stage.
The pipeline in your question first needs to read every document from the collection. It does this by requesting the documents from the storage engine, which will read each document from the disk and store it in the in-memory cache. The cache does not count against the 100MB limit.
The aggregation process will receive documents individually from the storage engine, and pass it through the pipeline to the first blocking stage (group is a blocking stage).
The group stage will examine the input document, update the fields in matching group, and then discard the input document.
This means the memory required by the group stage will be the sum of:
the size of 1-2 documents
total storage size for each result group
any scratch space needed for the operations to build each result
The specific group stage in the question is return a product identifier and an integer.
Using the Object.bsonsize funtion in the mongo shell, we can see that a null product ID produces a 43-byte object:
> Object.bsonsize({_id:{group_id:null},quantity:0})
43
So the total memory required will be
<number of distinct `product` values> x (<size of a product value> + 43)
Note that the values will be stored in BSON, so a string will be length+5, a UUID would be 21 bytes, etc.
i have a query and when i validate it i see that the count command returns a different results from the aggregate result.
i have an array of sub-documents like so:
{
...
wished: [{'game':'dayz','appid':'1234'}, {'game':'half-life','appid':'1234'}]
...
}
i am trying to query a count of all games in the collection and return the name along with the count of how many times i found that game name.
if i go
db.user_info.count({'wished.game':'dayz'})
it returns 106 as the value and
db.user_info.aggregate([{'$unwind':'$wished'},{'$group':{'_id':'$wished.game','total':{'$sum':1}}},{'$sort':{'total':-1}}])
returns 110
i don't understand why my counts are different. the only thing i can think of is that it has to do with the data being in an array of sub-documents as opposed to being in an array or just in a document.
The $unwind statement will cause one user with multiple wished games to appear as several users. Imagine this data:
{
_id: 1,
wished: [{game:'a'}, {game:'b'}]
}
{
_id: 2,
wished: [{game:'a'}, {game:'c'}, {game:'a'}]
}
The count can NEVER be more than 2.
But with this same data, an $unwind will give you 5 different documents. Summing them up will then give you a:3, b:1, c:1.
I want to get the records which has minimum value of 1
I write like this in server side, It is showing
the [object][object} has no such method min
return Videos.find({}).min({reportlen: 1});
Any alternatives???
Use query selector $gte
// Matches documents where reportlen is greater or equal to 1
Videos.find({reportlen:{$gte:1}})
I'm using mongoDB. I have a collection with:
String user_name,
Integer score
I would like to make a query that gets a user_name. The query should be sorted by score which returns the range of the 50 documents which the requested user_name is one of them.
For example, if I have 110 documents with the user_name X1-X110 with the scores 1-110 respectively and the input user_name was X72 I would like to get the range: X51-X100
EDIT:
An example of 3 documents:
{ "user_name": "X1", "score": 1}
{ "user_name": "X2", "score": 2}
{ "user_name": "X3", "score": 3}
Now if I have 110 documents as described above, and I want to find X72 I want to get the following documents:
{ "user_name": "X50", "score": 50}
{ "user_name": "X51", "score": 51}
...
{ "user_name": "X100", "score": 100}
How can I do it?
Clarification: I don't have each document rank stored. What I do have is document scores, which aren't necessarily consecutive (the example is a little bit misleading). Here's a less misleading example:
{ "user_name": "X1", "score": 17}
{ "user_name": "X2", "score": 24}
{ "user_name": "X3", "score": 38}
When searching for "X72" I would like to get a slice of size 50 in which "X72" resides according to its rank. Again, the rank is not the element score, but the element index in a hypothetical array sorted by scores.
Check out the MongoDB cursor operations sort, limit and skip. When used in conjunction, they can be used to get elements n to m which match your query:
cursor = db.collcetion.find({...}).sort({score:1}).limit(100).skip(50);
This should return documents 51 to 100 in order of score.
When I understood you correctly, you want to query the users which are scorewise in the neighbourhood of another player.
With three queries you can select the user, the 25 users above it and the 25 users below.
First, you need to get the user itself and its score.
user = db.collection.findOne({user_name: "X72"});
Then you select the next 25 players with scores above them:
cursor db.collection.find(score: { $gt:user.score}).sort(score: -1 ).limit(25);
//... iterate cursor
Then you select the next 25 players with scores below them:
cursor db.collection.find(score: { $lt:user.score}).sort(score: 1 ).limit(25);
//... iterate cursor
Unfortunately, there is no direct way to achieve what you want. You will need some processing at your client end to figure out the range.
First fetch the score by doing simple findOne / find
db.sample.findOne({"user_name": "X72"})
Next, using the score value (72 in this case), calculate the range in your client
lower = 72/50 => lower = 1.44
extract the number before decimal and set it to lower
lower = 1
upper = lower+1 => upper = 2
Now multiply the lower and upper values by 50 in your client, which would give you below values.
lower = 50
upper = 100
pass the lower and upper values to find and get the desired list.
db.sample.find({score:{$gt:50,$lte:100}}).sort({score:1})
Partial solution with one query:
I tried to do this with one query, but unfortunately I could not complete it. I am providing details below in hope that someone may be able to expand on this and complete what I started. Following are the steps that I planned:
project the documents to divide all scores by 50 and store in a new field _score. (This is as far as I got)
extract the value before decimal from _score [Stuck here] (Currently, I did not find any way to do this)
group values based on _score. (each group will give you one slot)
find and return the group where your score belongs (by using $match in aggregation pipeline)
db.sample.aggregate([{$project:{_id:1, user_name:1,score:1,_score:{$divide:["$score",50]}}}])
I would be really interested to see how this is done!!!