Using https://docs.aws.amazon.com/redshift/latest/dg/r_LISTAGG.html
I noticed in case the aggregation is > than 65K you get an error
Result size exceeds LISTAGG limit code: 8001
is there any setting to return NULL rather than crashing the query?
No, that is the max size of a varchar data type. As for work arounds it will depend on what you are looking to achieve and what other approaches will meet the need.
Related
My sample use case is to Query data about people who have not blocked the user
and there is no limit to the number of people that can block the user
So my query looks something like
db.collection().find( { followedPersonId: { $nin: [ blockerId1, blockerId2, blockerId3.....] } } )
So the number of Array items in the $nin operator can grow to a potentially large number, So is there a limit to the size of this array in MongoDB?
The command size limit is currently set at 48 MB. If your query is bigger than that the driver should fail when trying to serialize it, and the server should fail if it was asked to parse it.
Since your query is technically a query document, I imagine the lower 16 MB BSON document size limit would also apply to it. The 48 MB limit applies to find-and-modify queries that specify a query document and an update document.
Short answer is NO!
When you try to do this the query will become huge. And performance will be less.
Will the ids be in crores?
If yes then I will suggest not to go with this method.
I'm reaching some sort of RAM limit when doing this query, here's the error:
The operation: #<Moped::Protocol::Query
#length=100
#request_id=962
#response_to=0
#op_code=2004
#flags=[]
#full_collection_name="test_db.cases"
#skip=1650
#limit=150
#selector={"$query"=>{}, "$orderby"=>{"created_at"=>1}}
#fields=nil>
failed with error 17144: "Runner error: Overflow sort stage buffered data usage of 33555783 bytes exceeds internal limit of 33554432 bytes"
See https://github.com/mongodb/mongo/blob/master/docs/errors.md
for details about this error.
There are two solutions I can think of:
1) up the buffer limit. this requires mongo 2.8 which is some unstable release that i'd have to install manually.
2) break apart the query? chunk it? this is what the query looks like:
upload_set = Case.all.order_by(:created_at.asc).skip(#set_skipper).limit(150).each_slice(5).to_a
#set_skipper grows by 150 every time the method is called.
Any help?
From http://docs.mongodb.org/manual/reference/limits/
Sorted Documents
MongoDB will only return sorted results on fields without an index if
the combined size of all documents in the sort operation, plus a small
overhead, is less than 32 megabytes.
Did you try using an index on created_at ? That should remove that limitation.
I will execute a big query so i want to know what is the maximum length of a mongodb query ?
The maximum size of a document, which is what you are constructing when you create a query is 16MB. You can see that, and other limits here:
http://docs.mongodb.org/manual/reference/limits/
It appears there is an issue with the count operation on a geospatial query that contains more than 100 results. If I run the following query I still get a count of 100 no matter what.
db.locations.find({"loc":{$nearSphere:[50, 50]}}).limit(1000).count()
I understand that the default size limit on a query that uses the "near" syntax is 100 but it appears you cannot return more than that. Am I doing something wrong or is there a workaround for this?
Try "within" instead of "near".
This works for me,
center = [50, 50];
radius = 1/111.12; //convert it to KM.
db.places.count({"loc" : {"$within" : {"$center" : [center, radius]}}})
I found this at https://jira.mongodb.org/browse/SERVER-856
For what it's worth, even though the count reported for the cursor is 100 (regardless of what limit you give the query, even "10") you actually get the "limit" number of results back when you run through the query.
Here's a link to an issue where they assert it isn't broken:
https://jira.mongodb.org/browse/SERVER-739
Hope that helps!
It is not an issue with the mongo query. Try using cursor.size() rather than cursor.count(). Count does not take into account the limit where size does. So if you use .size() instead of count() you should get a print out of the correct number of returned items.
Check out this stack overflow Q and A for a clear solution to your issue.
Difference between cursor.count() and cursor.size() in MongoDB
Let's say i put a limit and skip on the MongoDB query...I want to know the total results if there was not a limit on there.
Of course, I could do this the shitty way...which is to query twice.
In MongoDB the default behavior of count() is to ignore skip and limit and count the number of results in the entire original query. So running count will give you exactly what you want.
Passing a Boolean true to count or calling size instead would give you a count WITH skip or limit.
There is no way to get the count without executing the query twice.