I've got a schema similar to this:
{
'policy': [
{ 'date' : { type: Date } }
]
}
I'm attempting to find these based on the first policy's date, based on a range. At first when my queries like this were silently failing (i.e. returning no results):
{
'$and' : [
{ 'policy.0.date' : { '$gt' : <lower-bound Date> } },
{ 'policy.0.date' : { '$lt' : <higher-bound Date> } }
]
}
I wasn't getting any errors, and I knew there were records there, so I pared it down to looking for all who were above a certain date:
{ 'policy.0.date' : { $gt : startDate } }
After attempting that, I got this error:
[Error: Can't use $gt with Array.]
I've even tried to querying without being based on the first element { 'policy.date' : { $gt : startDate } }, but that returns no records either.
Any ideas on how to query on the date field without resorting to $where?
It would seem actual DB queries return correctly, the issues lies in the version of mongoose used.
Related
I have a Mongo Document in below format:
{
"id":"eafa3720-28e2-11ed-bf07"
"type":"test"
"serviceType_details": [
{
"is_custom_service_type": false,
"bill_amount": 100
}
]
}
"serviceType_details" Key doesn't have any definite schema.
Now I want to export it using MongoDB aggregate to Parquet so that I could use Presto to query it.
My Pipeline Code:
db.test_collection.aggregate([
{
$match: {
"id": "something"
}
},
{
$addFields: {
...
},
}
{
"$out" : {
"format" : {
"name" : "parquet",
"maxFileSize" : "10GB",
"maxRowGroupSize" : "100MB"
}
}
}
])
Now I want to export the value of "serviceType_details" in json string not as array ( when using current code parquet recognises it as an array)
I have tried $convert,$project and it's not working.
Currently the generated Parquet schema looks something like this:
I want the generated Parquet schema for "serviceType_details" to have as string and value should be stringify version of array which is present in mongo document.
Reason for me to have need it as string is because in each document "serviceType_details" details have completely different schema, its very difficult to maintain Athena table on top of it.
You can use the $function operator to define custom functions to implement behaviour not supported by the MongoDB Query Language
It could be done using "$function" like this:
db.test_collection.aggregate([
{
$match: {
"id": "something"
}
},
{
$addFields: {
newFieldName: {
$function: {
body: function(field) {
return (field != undefined && field != null) ? JSON.stringify(field) : "[]"
},
args: ["$field"],
lang: "js"
}
},
},
}
{
"$out" : {
"format" : {
"name" : "parquet",
"maxFileSize" : "10GB",
"maxRowGroupSize" : "100MB"
}
}
}
])
Executing JavaScript inside an aggregation expression may decrease performance. Only use the $function operator if the provided pipeline operators cannot fulfill your application's needs.
I am passing in two dates formatted as MM-DD-YYYY which is a date range. I need to query all records within that range and include specified fields. I've had no luck.
Part of a record in Mongo:
{
"_id": "some ID",
"date": {
"$date": "2015-06-26T13:02:12.121Z"
},
Query:
var Start = '09-07-2015'
var End = '09-14-2015'
If I do:
var query = Order.find({
date : {
$lt : End,
$gt : Start
}
});
I get the full document within the week ranges as expected. However, I want to specify the fields to return rather than full document. So I've tried using grouping and project to specify those fields:
var query = Order.aggregate(
{
$match :
{
date: {
$gte: start,
$lt: end
}
},
$group:
{
cust_ID: '$request.headers.customer_id',
wholesaler_ID: '$request.headers.wholesalerID'
}
}
);
Likewise: I've also tried it using project to get the results I want. I thought maybe it won't match on a date string like 09-07-2015, so I included the ISO date directly. Still no luck... the query comes back undefined or empty:
var query = Order.aggregate(
{
$project:
{
date: 'date',
cust_ID: '$request.headers.custID',
wholesaler_ID: '$request.headers.wholesalerID'
}
},
{
$match :
{
date: {
$gte: "2014-12-09T21:02:56.872Z",
$lt: "2015-12-09T21:02:56.872Z"
}
}
}
);
var query = Order.find({
date : {
$lt : End,
$gt : Start
}}, {cust_ID:1, wholeseller_ID:1}
);
This will work.
I just tested this using Robomongo
db.getCollection('offerdb').find({time_posted:{$gt: '2015-10-21T21:40:04+05:30', $lte:'2015-12-14T05:53:14+05:30'}},{_id:1, merchant_id:1})
Works like a charm for me.
Try this command in mongo shell
use dbname
db.collection_name.find({date: {$gte: ISODate('2015-09-07 00:00:00'), $lte: ISODate('2015-09-14 23:59:59.999999')}},{'cust_ID':1,'_id':0,'wholeseller_ID':1})
In my DynamoDB every document has several fields, one of the fields is a document called "engines" that holds several documents (all the engines) that hold several fields, as the picture shows below:
I would like to get all the couples of (engine,definitions) that their definition date is greater than a specific date.
I tried:
cursor=collection.find(
{'engines': { "$elemMatch" :
{ "definitions" :
{'$gt': startdate} } } }
,{'engines':{'$elemMatch':1}},{'engines':{'$elemMatch':{'definitions':1}}} )
but I get:
TypeError: skip must be an instance of int
Can someone help with the query?
You've mixed up the closing } and ended up passing {'engines':{'$elemMatch':{'definitions':1}}} as a skip argument value.
I think you meant:
cursor = collection.find(
{
'engines': {
"$elemMatch": {
"definitions": {
'$gt': startdate
}
}
}
},
{
'engines': {
'$elemMatch': {
'definitions': 1
}
}
}
)
I try to aggregate and group objects in mongodb by month. I basically copy query from mongo docs.
db.users.aggregate(
{
$group : {
_id: {
month : { $month : "$registrationDate" }
},
count: { $sum: 1 }
}
}
);
Type of registrationDate is date.
Short version of object in users collection.
{
"_id" : ObjectId("50ab08399b57f2be03000000"),
...
"registrationDate" : ISODate("2012-11-20T05:34:01.000Z"),
...
}
Then I get an exception
exception: can't convert from BSON type NumberDouble to Date
The problem is that you have some documents in your collection where the type of registrationDate is not a date but a double-precision floating point number. You can find these documents with db.users.find( { registrationDate: { $type:1 } } ). Fix these documents and it should work. Alternatively you can add the following step to the front of your aggregation to exclude those documents where the registrationDate is not a Date: {$match: { registrationDate: { $type:9 } } }
In MongoDB, using $type, it is possible to filter a search based on if the field matches a BSON data type (see DOCS).
For eg.
db.posts.find({date2: {$type: 9}}, {date2: 1})
which returns:
{
"_id" : ObjectId("4c0ec11e8fd2e65c0b010000"),
"date2" : "Fri Jul 09 2010 08:25:26 GMT"
}
I need a query that will tell me what the actual type of the field is, for every field in a collection. Is this possible with MongoDB?
Starting from MongoDB 3.4, you can use the $type aggregation operator to return a field's type.
db.posts.aggregate(
[
{ "$project": { "fieldType": { "$type": "$date2" } } }
]
)
which yields:
{
"_id" : ObjectId("4c0ec11e8fd2e65c0b010000"),
"fieldType" : "string"
}
type the below query in mongo shell
typeof db.employee.findOne().first_name
Syntax
typeof db.collection_name.findOne().field_name
OK, here are some related questions that may help:
Get all field names in a collection using map-reduce.
Here's a recursive version that lists all possible fields.
Hopefully that can get you started. However, I suspect that you're going to run into some issues with this request. There are two problems here:
I can't find a "gettype" function for JSON. You can query by $type, but it doesn't look like you can actually run a gettype function on a field and have that maps back to the BSON type.
A field can contain data of multiple types, so you'll need a plan to handle this. Even if it's not apparent Mongo could store some numbers as ints and others floats without you really knowing. In fact, with the PHP driver, this is quite possible.
So if you assume that you can solve problem #1, then you should be able to solve problem #2 using a slight variation on "Get all field Names".
It would probably look something like this:
"map" : function() { for (var key in this) { emit(key, [ typeof value[key] ]); } }
"reduce" : function(key, stuff) { return (key, add_to_set(stuff) ); }
So basically you would emit the key and the type of key value (as an array) in the map function. Then from the reduce function you would add unique entries for each type.
At the end of the run you would have data like this
{"_id":[255], "name" : [1,5,8], ... }
Of course, this is all a lot of work, depending on your actual problem, you may just want to ensure (from your code) that you're always putting in the right type of data. Finding the type of data after the data is in the DB is definitely a pain.
Taking advantage of the styvane query, I added a $group listing to make it easier to read when we have different data types.
db.posts.aggregate(
[
{ "$project": { _id:0, "fieldType": { "$type": "$date2" } } },
{"$group": { _id: {"fieldType": "$fieldType"},count: {$sum: 1}}}
])
And have this result:
{ "_id" : { "fieldType" : "missing" }, "count" : 50 }
{ "_id" : { "fieldType" : "date" }, "count" : 70 }
{ "_id" : { "fieldType" : "string" }, "count" : 10 }
Noting that a=5;a.constructor.toString() prints function Number() { [native code] }, one can do something similar to:
db.collection.mapReduce(
function() {
emit(this._id.constructor.toString()
.replace(/^function (\S+).+$/, "$1"), 1);
},
function(k, v) {
return Array.sum(v);
},
{
out: { inline: 1 }
});