I'm using Houston for an Admin section to my Meteor app, but there are a lot of fields I don't need to see. I'm thinking I want to avoid publishing those fields from mongo in the first place, rather than eliminate them in a convoluted way in the template.
Here's the coffeescript in Houston in server/publications.coffee:
Houston._publish name, (sort, filter, limit, unknown_arg) ->
check sort, Match.Optional(Object)
check filter, Match.Optional(Object)
check limit, Match.Optional(Number)
check unknown_arg, Match.Any
return unless Houston._user_is_admin #userId
try
collection.find(filter, sort: sort, limit: limit)
catch e
console.log e
Here's what I've tried to replace it with. This doesn't cause any errors, but it doesn't stop the fields from showing.
Houston._publish name, (sort, filter, limit, unknown_arg) ->
check sort, Match.Optional(Object)
check filter, Match.Optional(Object)
check limit, Match.Optional(Number)
check unknown_arg, Match.Any
return unless Houston._user_is_admin(#userId)
try
return collection.find(filter,
sort: sort
limit: limit
,
fields:
category: 0
userId: 0
)
catch e
console.log e
I'm not sure if my error is in my coffeescript conversion (I've double-checked indentation), or in my general approach.
Thanks for any help,
Charlie Magee
Related
When looking for pagination techniques on the internet, one usually finds two ways :
Offset-based pagination :
Collection.find(
{ where_this: "equals that" },
{ skip: 15, limit: 5 }
)
Cursor-based pagination :
Collection.find(
{ where_this: "equals that", _id: { $gt: cursor }},
{ sort: { _id: 1 }}
)
But is there a way to have cursor-based pagination without sorting the collection according to that cursor ? Like, telling Mongo "Alright, I want the 5 next items after that _id, no matter in which order the _ids are, just give me 5 items after you see that _id". Something along those lines :
Collection.find(
{ where_this: "equals that", _id: { $must_come_after: cursor }},
{ sort: { other_field: 1 }}
)
It is not always possible to use the field you're sorting with as the cursor. First of all, because these fields can be of different types and you might allow your app users to sort, for example, tables as they please. With a strongly-typed API framework like GraphQL, this would be a mess to handle. Secondly, you could have two or more equal values for that field following each other in the sorted collection. If your pages split in the middle, asking for the next page will either give you duplicates or ignore items.
Is there a way to do that ? How is it usually done to allow custom sorting fields without offset-based pagination ? Thanks.
When we talk about "paginating a result set", the implicit assumption is that the result set stays the same throughout the process. In order for the result set to stay the same, it must normally be sorted. Without specifying an order, the database is free to return the documents in any order, and this order may change from one retrieval to the next. Reordering of documents that a user is paginating through creates a poor user experience.
you could have two or more equal values for that field following each other in the sorted collection. If your pages split in the middle, asking for the next page will either give you duplicates or ignore items.
The database can return the documents which compare equal in any order, and this order can change between adjacent queries. This is why when sorting on a field that has a low cardinality, it is a good idea to add another field with a high cardinality to the sort expression to ensure the documents are returned in a stable order.
But is there a way to have cursor-based pagination without sorting the collection according to that cursor ?
You can encode offset in a cursor identifier and use skip/limit.
So right now I want to use cursor to find all restaurants which are good for groups, have a price range less than or equal to three and have more than 50 reviews that have reviews with at least 5 "useful" votes;
db.business.find({$and:[{"attributes": /.*GoodForGroups: True.*/i},{review_count:{$gte:50}},{"attributes.RestaurantsPriceRange2":{$lte:3}}]})
But so far this query turn up nothing, so is there anything wrong with it?
Also this is what I can think of for my cursor so far, feel free to change it if you see anything wrong with it.
var myCursor=db.business.find({$and:[{"attributes": /.GoodForGroups: True./i},{review_count:{$gte:50}},{"attributes.RestaurantsPriceRange2":{$lte:3}}]},{business_id:true, _id: false})
while(myCursor.hasNext()){
db.reviews.find({$and:[{business_id:myCursor.next().business_id},{"useful":{$gte: 5}}]});
}
Firstly, you don't need the $and in your query, mongodb by default perform a $and.
Secondly, you are trying to query keys of your document, it is not possible.
Thirdly, your GoodForGroups should have a boolean value. You don't need a regex here.
Please run:
db.business.find({"attributes.GoodForGroups": true},{review_count:{$gte:50}},{"attributes.RestaurantsPriceRange2":{$lte:3}})
I want to know what is the need of first curly brace {} after find here in this query.
db.mycol.find({},{"title":1,_id:0})
It is an empty query, in the sense of a limiting boundary. So
db.mycol.find({},{"title":1,_id:0})
would basically translate to
Show me the title, but not the _id (as you would do by default) for all documents of the mycol collection in the current database.
Let's say you want all the titles written by Alan Turing. So you could modify the query part like this:
db.mycol.find({"author":"Alan Turing"},{"title":1,_id:0})
In general, MongoDB's find operation can be described like this
db.collection.find(query,projection)
For more detailed information, you might want to read find's documentation.
The first Curly braces is used as a where condition in MySql
Check out this Link - SQL to MongoDB Mapping Chart
MySQL
SELECT user_id, status FROM users WHERE status = "A"
MongoDB
db.users.find(
{ status: "A" },
{ user_id: 1, status: 1, _id: 0 }
)
This is called projection and tells which fields to show. For example here you show only title. It is something like Select title from mycol; If you will not specify projection it will return everything, which is close to select * from mycol;
_id is always shown, unless you hide it explicitly with _id: 0. Another thing is that you can not mix 0 and 1 (except for _id). But one more time - read FAQ.
You should refer this link for better explanation
http://docs.mongodb.org/manual/reference/operator/projection/positional/
first of all, projection does not return the first result it finds it tells mongo what to return.
.findOne(query) will LIMIT result to one or find(query).limit(1) will also do the same.
You say you are trying to "get all" your data.
a standard find type query will get your started...
find({mongo:query},{mongo:projection})
but limits results to a cursor from the mongo shell*
Let's say I have two collections, A and B, and a single document in A is related to N documents in B. For example, the schemas could look like this:
Collection A:
{id: (int),
propA1: (int),
propA2: (boolean)
}
Collection B:
{idA: (int), # id for document in Collection A
propB1: (int),
propB2: (...),
...
propBN: (...)
}
I want to return properties propB2-BN and propA2 from my API, and only return information where (for example) propA2 = true, propB6 = 42, and propB1 = propA1.
This is normally fairly simple - I query Collection B to find documents where propB6 = 42, collect the idA values from the result, query Collection A with those values, and filter the results with the Collection A documents from the query.
However, adding skip and limit parameters to this seems impossible to do while keeping the behavior users would expect. Naively applying skip and limit to the first query means that, since filtering occurs after the query, less than limit documents could be returned. Worse, in some cases no documents could be returned when there are actually still documents in the collection to be read. For example, if the limit was 10 and the first 10 Collection B documents returned pointed to a document in Collection A where propA2 = false, the function would return nothing. Then the user would assume there's nothing left to read, which may not be the case.
A slightly less naive solution is to simply check if the return count is < limit, and if so, repeat the queries until the return count = limit. The problem here is that skip/limit queries where the user would expect exclusive sets of documents returned could actually return the same documents.
I want to apply skip and limit at the mongo query level, not at the API level, because the results of querying collection B could be very large.
MapReduce and the aggregation framework appear to only work on a single collection, so they don't appear to be alternatives.
This seems like something that'd come up a lot in Mongo use - any ideas/hints would be appreciated.
Note that these posts ask similar sounding questions but don't actually address the issues raised here.
Sounds like you already have a solution (2).
You cannot optimize/skip/limit on first query, depending on search you can perhaps do it on second query.
You will need a loop around it either way, like you write.
I suppose, the .skip will always be costly for you, since you will need to get all the results and then throw them away, to simulate the skip, to give the user consistent behavior.
All the logic would have to go to your loop - unless you can match in a clever way to second query (depending on requirements).
Out of curiosity: Given the time passed, you should have a solution by now?!
I have browsed through various examples but have failed to find what I am looking for.. What I want is to search for a specific document by _id and skip multiple times between a collection by using one query? Or some alternative which is fast enough to my case.
Following query would skip first one and return second in advance:
db.posts.find( { "_id" : 1 }, { comments: { $slice: [ 1, 1 ] } } )
That would be skip 0, return 1 and leaves the rest out from result..
But what If there would be like 10000 comments and I would want to use same pattern, but return that array values like this:
skip 0, return 1, skip 2, return 3, skip 4, return 5
So that would return collection which comments would be size of 5000, because half of them is skipped away. Is this possible? I applied large number like 10000 because I fear that using multiple queries to apply this would not be performance wise.. (example shown in here: multiple queries to accomplish something similar). Thnx!
I went through several resources and concluded that currently this is impossible to make with one query.. Instead, I agreed on that there are only two options to overcome this problem:
1.) Make a loop of some sort and run several slice queries while increasing the position of a slice. Similar to resource I linked:
var skip = NUMBER_OF_ITEMS * (PAGE_NUMBER - 1)
db.companies.find({}, {$slice:[skip, NUMBER_OF_ITEMS]})
However, depending on the type of a data, I would not want to run 5000 individual queries to get only half of the array contents, so I decided to use option 2.) Which seems for me relatively fast and performance wise.
2.) Make single query by _id to row you want and before returning results to client or some other part of your code, skip your unwanted array items away by using for loop and then return the results. I made this at java side since I talked to mongo via morphia. I also used query explain() to mongo and understood that returning single line with array which has 10000 items while specifying _id criteria is so fast, that speed wasn't really an issue, I bet that slice skip would only be slower.