I'm using lacinia library for my GraphQL Clojure server.
For this simple schema:
input QueryConfig {
startDate: String!
endDate: String!
}
type MyData{
x: Float!
y: Float!
z: Float!
}
schema {
query: Query
}
type Query {
myQuery(config:QueryConfig) : [MyData]
}
I'm posting the following query:
{
myQuery(config:{startDate:"2020-01-01",endDate:"2020-01-01"}){
x
}
}
In my resolver code, I would like to know what fields were asked (x in the example above), So I won't over fetch results from my DB and retrieve only the x values.
In the resolver code the context contains :graphql-query key which contains the entire query. However, parsing the query text seems awkward.
(defn my-resolver
[context args value]
;TODO find out what fields were asked and fetch from DB
)
What would be the right way of getting the query fields?
Lacinia does allow you to preview nested selections:
A field resolver can “preview” what fields will be selected below it
in the selections tree. This is a tool frequently used to optimize
data retrieval operations.
Example code from docs:
(require
'[com.walmartlabs.lacinia.executor :as executor])
(defn resolve-hero
[context args _]
(if (executor/selects-field? context :character/friends)
(fetch-hero-with-friends args)
(fetch-hero args)))
From the docs:
It is also possible to get all the fields that will be selected, using
selections-seq. This a lazy, breadth-first navigation of all fields in
the selection tree.
So you can use: (executor/selections-seq context) to get all fields that were sent by the query.
Related
In MongoDB (using mongosh or command-line mongo cli), you can query documents, for example using db.mycollection.find({"something":true}) and get the following result:
{
"someDate": ISODate("2022-10-24T17:21:44.980Z"),
"something": true,
"hello": "world"
}
This result, however, is not valid JSON (Due to ISODate). How can I change the query above to make MongoDB return canonical (valid) JSON?
I'm looking for a recursive and generalized way to do this, even for deeply nested documents.
There are a number of existing answers, I'll clarify a few:
Use aggregate to produce the output in JSON format: Playground
db.collection.aggregate([
{
$match: {
something: true
}
},
{
$project: {
_id: 1,
someDate: {
$dateToString: {
format: "%Y-%m-%dT%H:%M:%S:%LZ",
date: "$someDate"
}
},
something: 1,
hello: 1
}
}
])
Loop through your query in your application: (example, node.js)
db.mycollection.find({"something":true}).forEach(function(doc) {
doc.someDate = doc.someDate.toISOString() // or even .toJSON()
})
// Or with await
const records = await db.mycollection.find({"something":true}).map(doc => {
doc.someDate = doc.someDate.toISOString()
return doc
}).toArray()
The details of where you are running this command are very important, can you please share those?
I am guessing that you are probably running this via the (older) mongo utility (as opposed to the newer mongosh). But confirmation of that, and the database version that you are using, would both be helpful. I will retain this assumption for the purposes of this answer.
This result, however, is not valid JSON (Due to ISODate).
The database itself doesn't return some text that has ISODate. In fact, it doesn't return or "speak" JSON at all. Rather, the database communicates via "Binary JSON" or BSON for short. In fact, the "Does MongoDB use BSON or JSON?" section of this page specifically mentions the following:
Firstly, BSON documents may contain Date or Binary objects that are not natively representable in pure JSON.
So when you see things like ISODate() that is the client application wrapping and representing the rich BSON document in a more limited (and text-based) JSON-like form. Importantly, this is often for readability purposes. You should be able to natively pass around and operate on information (documents) returned by the database directly in the application without doing any sort of transformation and without losing rich type information. Additional reading material about BSON is here.
Getting back to the original question, if you want to have the shell print out a valid JSON document than you can do that via additional helpers. In the older mongo utility, a reproduction of the situation described in the question is:
> db.mycollection.findOne({"something": true})
{
"_id" : 1,
"someDate" : ISODate("2022-11-30T14:38:37.711Z"),
"something" : true,
"hello" : "world"
}
The shell itself can understand and operate on ISODate() (and other functions of that nature). If you did want to remove things like ISODate() for some reason, then you can leverage the JSON.stringify() functionality (reformatted with line indents for readability):
> JSON.stringify( db.mycollection.findOne({"something": true}) )
{
"_id":1,
"someDate":"2022-11-30T14:38:37.711Z",
"something":true,
"hello":"world"
}
The newer mongosh shell offers even more utility here:
> EJSON.serialize( db.mycollection.findOne() )
{
_id: 1,
someDate: { '$date': '2022-11-30T14:38:37.711Z' },
something: true,
hello: 'world'
}
Via these EJSON functions, mongosh is attempting to preserve type information when it prints the data in this format. Notice that in the earlier example the date was just represented as a string, but here the shell is using Extended JSON to capture the fact that the type of someDate is a Date.
I'm running a basic query to a small DB that is structured as follow:
carMake:
carModel:
carColor:
Let's say first 2 fields have specific information:
carMake: "Ford"
carModel: "Mustang"
But for the third field, I can query all values with the specific color: red, blue etc. The question I have is how can I query the third field using:
carColor: "All Colors"
If "All Colors" is not a specific value? I've used: .whereField(CAR_COLOR, isGreaterThanOrEqualTo or arrayContains or arrayContainsAny, etc) and it doesn't work. I get no results. Any ideas? Thank you!
Based on the comments discussion, it sounds like you want to be able to include whereField in your query chain, but not actually execute it if certain conditions are true/false.
I suggest adding an extension to Query:
extension Query {
func whereField(useCondition: Bool, _ field: String, arrayContains: String) -> Query {
if useCondition {
return self.whereField(field, arrayContains: arrayContains)
} else {
return self
}
}
}
Then, you could do something like this:
db.collection("car").whereField(useCondition: !allColors, "carColor", arrayContains: searchedColor)
Note that you may have to adjust this to fit the variety of whereField that you need.
I've searched and found no Go solution to the problem, not with or without using mgo.v2, not on StackOverflow and not on any other site. This Q&A is in the spirit of knowledge sharing / documenting.
Let's say we have a users collection in MongoDB modeled with this Go struct:
type User struct {
ID bson.ObjectId `bson:"_id"`
Name string `bson:"name"`
Country string `bson:"country"`
}
We want to sort and list users based on some criteria, but have paging implemented due to the expected long result list.
To achieve paging of the results of some query, MongoDB and the mgo.v2 driver package has built-in support in the form of Query.Skip() and Query.Limit(), e.g.:
session, err := mgo.Dial(url) // Acquire Mongo session, handle error!
c := session.DB("").C("users")
q := c.Find(bson.M{"country" : "USA"}).Sort("name", "_id").Limit(10)
// To get the nth page:
q = q.Skip((n-1)*10)
var users []*User
err = q.All(&users)
This however becomes slow if the page number increases, as MongoDB can't just "magically" jump to the xth document in the result, it has to iterate over all the result documents and omit (not return) the first x that need to be skipped.
MongoDB provides the right solution: If the query operates on an index (it has to work on an index), cursor.min() can be used to specify the first index entry to start listing results from.
This Stack Overflow answer shows how it can be done using a mongo client: How to do pagination using range queries in MongoDB?
Note: the required index for the above query would be:
db.users.createIndex(
{
country: 1,
name: 1,
_id: 1
}
)
There is one problem though: the mgo.v2 package has no support specifying this min().
How can we achieve efficient paging that uses MongoDB's cursor.min() feature using the mgo.v2 driver?
Unfortunately the mgo.v2 driver does not provide API calls to specify cursor.min().
But there is a solution. The mgo.Database type provides a Database.Run() method to run any MongoDB commands. The available commands and their documentation can be found here: Database commands
Starting with MongoDB 3.2, a new find command is available which can be used to execute queries, and it supports specifying the min argument that denotes the first index entry to start listing results from.
Good. What we need to do is after each batch (documents of a page) generate the min document from the last document of the query result, which must contain the values of the index entry that was used to execute the query, and then the next batch (the documents of the next page) can be acquired by setting this min index entry prior to executing the query.
This index entry –let's call it cursor from now on– may be encoded to a string and sent to the client along with the results, and when the client wants the next page, he sends back the cursor saying he wants results starting after this cursor.
Doing it manually (the "hard" way)
The command to be executed can be in different forms, but the command name (find) must be first in the marshaled result, so we'll use bson.D (which preserves order in contrast to bson.M):
limit := 10
cmd := bson.D{
{Name: "find", Value: "users"},
{Name: "filter", Value: bson.M{"country": "USA"}},
{Name: "sort", Value: []bson.D{
{Name: "name", Value: 1},
{Name: "_id", Value: 1},
},
{Name: "limit", Value: limit},
{Name: "batchSize", Value: limit},
{Name: "singleBatch", Value: true},
}
if min != nil {
// min is inclusive, must skip first (which is the previous last)
cmd = append(cmd,
bson.DocElem{Name: "skip", Value: 1},
bson.DocElem{Name: "min", Value: min},
)
}
The result of executing a MongoDB find command with Database.Run() can be captured with the following type:
var res struct {
OK int `bson:"ok"`
WaitedMS int `bson:"waitedMS"`
Cursor struct {
ID interface{} `bson:"id"`
NS string `bson:"ns"`
FirstBatch []bson.Raw `bson:"firstBatch"`
} `bson:"cursor"`
}
db := session.DB("")
if err := db.Run(cmd, &res); err != nil {
// Handle error (abort)
}
We now have the results, but in a slice of type []bson.Raw. But we want it in a slice of type []*User. This is where Collection.NewIter() comes handy. It can transform (unmarshal) a value of type []bson.Raw into any type we usually pass to Query.All() or Iter.All(). Good. Let's see it:
firstBatch := res.Cursor.FirstBatch
var users []*User
err = db.C("users").NewIter(nil, firstBatch, 0, nil).All(&users)
We now have the users of the next page. Only one thing left: generating the cursor to be used to get the subsequent page should we ever need it:
if len(users) > 0 {
lastUser := users[len(users)-1]
cursorData := []bson.D{
{Name: "country", Value: lastUser.Country},
{Name: "name", Value: lastUser.Name},
{Name: "_id", Value: lastUser.ID},
}
} else {
// No more users found, use the last cursor
}
This is all good, but how do we convert a cursorData to string and vice versa? We may use bson.Marshal() and bson.Unmarshal() combined with base64 encoding; the use of base64.RawURLEncoding will give us a web-safe cursor string, one that can be added to URL queries without escaping.
Here's an example implementation:
// CreateCursor returns a web-safe cursor string from the specified fields.
// The returned cursor string is safe to include in URL queries without escaping.
func CreateCursor(cursorData bson.D) (string, error) {
// bson.Marshal() never returns error, so I skip a check and early return
// (but I do return the error if it would ever happen)
data, err := bson.Marshal(cursorData)
return base64.RawURLEncoding.EncodeToString(data), err
}
// ParseCursor parses the cursor string and returns the cursor data.
func ParseCursor(c string) (cursorData bson.D, err error) {
var data []byte
if data, err = base64.RawURLEncoding.DecodeString(c); err != nil {
return
}
err = bson.Unmarshal(data, &cursorData)
return
}
And we finally have our efficient, but not so short MongoDB mgo paging functionality. Read on...
Using github.com/icza/minquery (the "easy" way)
The manual way is quite lengthy; it can be made general and automated. This is where github.com/icza/minquery comes into the picture (disclosure: I'm the author). It provides a wrapper to configure and execute a MongoDB find command, allowing you to specify a cursor, and after executing the query, it gives you back the new cursor to be used to query the next batch of results. The wrapper is the MinQuery type which is very similar to mgo.Query but it supports specifying MongoDB's min via the MinQuery.Cursor() method.
The above solution using minquery looks like this:
q := minquery.New(session.DB(""), "users", bson.M{"country" : "USA"}).
Sort("name", "_id").Limit(10)
// If this is not the first page, set cursor:
// getLastCursor() represents your logic how you acquire the last cursor.
if cursor := getLastCursor(); cursor != "" {
q = q.Cursor(cursor)
}
var users []*User
newCursor, err := q.All(&users, "country", "name", "_id")
And that's all. newCursor is the cursor to be used to fetch the next batch.
Note #1: When calling MinQuery.All(), you have to provide the names of the cursor fields, this will be used to build the cursor data (and ultimately the cursor string) from.
Note #2: If you're retrieving partial results (by using MinQuery.Select()), you have to include all the fields that are part of the cursor (the index entry) even if you don't intend to use them directly, else MinQuery.All() will not have all the values of the cursor fields, and so it will not be able to create the proper cursor value.
Check out the package doc of minquery here: https://godoc.org/github.com/icza/minquery, it is rather short and hopefully clean.
In order to iterate over all the documents in a MongoDB (2.6.9) collection using Grails (2.5.0) and the MongoDB Plugin (3.0.2) I created a forEach like this:
class MyObjectService {
def forEach(Closure func) {
def criteria = MyObject.createCriteria()
def ids = criteria.list { projections { id() } }
ids.each { func(MyObject.get(it)) }
}
}
Then I do this:
class AnalysisService{
def myObjectService
#Async
def analyze(){
MyObject.withStatelessSession {
myObjectService.forEach { myObject ->
doSomethingAwesome(myObject)
}
}
}
}
This works great...until I hit a collection that is large (>500K documents) at which point a CommandFailureException is thrown because the size of the aggregation result is greater than 16MB.
Caused by CommandFailureException: { "serverUsed" : "foo.bar.com:27017" , "errmsg" : "exception: aggregation result exceeds maximum document size (16MB)" , "code" : 16389 , "ok" : 0.0}
In reading about this, I think that one way to handle this situation is to use the option allowDiskUse in the aggregation function that runs on the MongoDB side so that the 16MB memory limit won't apply and I can get a larger aggregation result.
How can I pass this option to my criteria query? I've been reading the docs and the Javadoc for the Grails MongoDB plugin, but I can't seem to find it. Is there is another way to approach the generic problem (iterate over all members of a large collection of domain objects)?
This is not possible with the current implementation of MongoDB Grails plugin. https://github.com/grails/grails-data-mapping/blob/master/grails-datastore-gorm-mongodb/src/main/groovy/org/grails/datastore/mapping/mongo/query/MongoQuery.java#L957
If you look at the above line, then you will see that the default options are being used for building AggregationOptions instance so there is no method to provide an option.
But there is another hackish way to do it using the Groovy's metaclass. Let's do it..:-)
Store the original method reference of builder() method before writing criteria in your service:
MetaMethod originalMethod = AggregationOptions.metaClass.static.getMetaMethod("builder", [] as Class[])
Then, replace the builder method to provide your implementation.
AggregationOptions.metaClass.static.builder = { ->
def builderInstance = new AggregationOptions.Builder()
builderInstance.allowDiskUse(true) // solution to your problem
return builderInstance
}
Now, your service method will be called with criteria query and should not results in the aggregation error you are getting since we have not set the allowDiskUse property to true.
Now, reset the original method back so that it should not affect any other call (optional).
AggregationOptions.metaClass.static.addMetaMethod(originalMethod)
Hope this helps!
Apart from this, why do you pulling all IDs in forEach method and then re getting the instance using get() method? You are wasting the database queries which will impact the performance. Also, if you follow this, you don't have to do the above changes.
An example with the same: (UPDATED)
class MyObjectService {
void forEach(Closure func) {
List<MyObject> instanceList = MyObject.createCriteria().list {
// Your criteria code
eq("status", "ACTIVE") // an example
}
// Don't do any of this
// println(instanceList)
// println(instanceList.size())
// *** explained below
instanceList.each { myObjectInstance ->
func(myObjectInstance)
}
}
}
(I'm not adding the code of AnalysisService since there is no change)
*** The main point is here at this point. So whenever you write any criteria in domain class (without projection and in mongo), after executing the criteria code, Grails/gmongo will not immediately fetch the records from the database unless you call some methods like toString(), 'size()ordump()` on them.
Now when you apply each on that instance list, you will not actually loading all instances into memory but instead you are iterating over Mongo Cursor behind the scene and in MongoDB, cursor uses batches to pull record from database which is extremely memory safe. So you are safe to directly call each on your criteria result which will not blow up the JVM unless you called any of the methods which triggers loading all records from the database.
You can confirm this behaviour even in the code: https://github.com/grails/grails-data-mapping/blob/master/grails-datastore-gorm-mongodb/src/main/groovy/org/grails/datastore/mapping/mongo/query/MongoQuery.java#L1775
Whenever you write any criteria without projection, you will get an instance of MongoResultList and there is a method named initializeFully() which is being called on toString() and other methods. But, you can see the MongoResultList is implementing iterator which is in turn calling MongoDB cursor method for iterating over the large collection which is again, memory safe.
Hope this helps!
For this app, I'm using Node.js, MongoDB, Mongoose & Express
So I have a Param Object that contains an array of Pivots, and I want to read certain data from the pivots as outlined below
---in models.js-------------------------
var Pivot = new Schema({
value : String
, destination : String
, counter : Number
});
var Param = new Schema({
title : String
, desc : String
, pivots : [Pivot]
});
------------- in main.js --------------
var Param = db.model('Param');
app.get('/:title/:value', function(req, res){
Param.findOne({"title":req.param('title')}, function(err, record){
console.log(record.pivots);
record.pivots.find({"value":req.param('value')}, function(err, m_pivot){
pivot.counter++;
res.redirect(m_pivot.destination);
});
record.save();
});
});
I know that the code works until console.log(record.pivots), since i got a doc collection with the right pivot documents inside.
However, there does not seem to be a find method to let me match an embedded document by the 'value' property defined in the schema. Is it possible to search through this array of embedded documents using .find() or .findOne() , and if not, is there some easy way to access it through mongoose?
varunsrin,
This should do it
app.get('/:title/:value', function(req, res) {
Param.findOne({'pivots.value': req.param('value'), "title":req.param('title')}},
function(err, record) {
record.pivot.counter++;
res.redirect(m_pivot.destination);
record.save();
});
});
Note the pluralization of the query to match the field name in your schema
You can querying using embedded document properties like this:
{'pivot.value': req.param('value')}}
Update in response to comment:
app.get('/:title/:value', function(req, res) {
Param.findOne({'pivot.value': req.param('value'), "title":req.param('title')}},
function(err, record) {
record.pivot.counter++;
res.redirect(m_pivot.destination);
record.save();
});
});
I solved it temporarily using a simple for loop to parse the object array as follows:
for (var i=0; i <record.pivots.length; i++){
if (record.pivots[i].value == req.param('value')){
res.redirect(record.pivots.destination);
}
}
However, I still think that Mongoose must have a simpler way of interacting with embedded documents - and this loop is somewhat slow, especially when the number of embedded documents grows large.
If anyone has any suggestions for a faster way to search this object array either in js or with a mongoose function, please post below.
the biggest problem with this is that if your req has some fields empty (that should act as wildcard), you will not find anything since mongo tries to match empty params as well, so searching for {"user":"bob", "color":""} is not the same as {"user":"bob", "color":"red"} or {"user":"bob"}. this means that you have to first create a query object and filter out any unused parameters before you pass it in, and if you create a query object, you can no longer do something like "user.name=..." because mongo interperets this as an error since it does not first resolve the object literal into a string.
Any ideas on this problem?
ps. You'd think it would be easy enough to make an object like:
user.name="bob"; user.color:"green"; user.signup.time="12342561"
and then just use user as a query object :/
I think you are looking for the "$in" keyword?
As in:
{a: {$in: [10, "hello"]}}
source: MongoDB Queries CheatSheet