How to sort results by a custom expression which I use in find?
The collection contains documents with the following attributes for example:
{
"_id" : ObjectId("5ef1cd704b35c6d6698f2050"),
"Name" : "TD",
"Date" : ISODate("2021-06-23T09:37:51.976Z"),
"A" : "19.36",
"B" : 2.04,
}
I'm using the following find query to get the records with Date since "2022-01-01" and the ratio between A and B is lower than 0.1:
db.getCollection('my_collection').find(
{
"experationDate" :
{
$gte: new ISODate("2022-01-01 00:00:00.000Z")
},
"$expr":
{
"$lte": [
{ "$divide": ["$A", "$B"] },
0.1
]
}
})
Now, I can't find the right way to sort the results by this ratio.
You can use aggregate in this way:
Search the documents you want using $match, add a field named ratio and use it to sort. And finally, not shown the field using $project:
db.collection.aggregate([
{ "$match": {
"Date": { "$gte": ISODate("2020-01-01") },
"$expr": { "$lte": [ { "$divide": [ "$B", { "$toDouble": "$A" } ] }, 0.1 ] } }
},
{
"$set": {
"ratio": { "$divide": [ "$B", { "$toDouble": "$A" } ] }
}
},
{
"$sort": { "ratio": 1 }
},
{
"$project": { "ratio": 0 }
}
])
Example here
By te way, I've used other values to get results. Ratio between 2.04 and 19.36 is greater than 0.1. You have dividied A/B but I think you mean B/A.
By the way, this is not important, you can change values but the query will still works ok.
Also, maybe this could work better. Is the same query, but could be more efficient (maybe, I don't know) because prevent to divide each value into collection twice:
First filter by date, then add the field ratio to each document found (and in this way is not necessary divide every document). Another filter using the ratio, the sort, and not output the field.
db.collection.aggregate([
{
"$match": { "Date": { "$gte": ISODate("2020-01-01") } }
},
{
"$set": { "ratio": { "$divide": [ "$B", { "$toDouble": "$A" } ] } }
},
{
"$match": { "ratio": { "$lte": 0.1 } }
},
{
"$sort": { "ratio": 1 }
},
{
"$project": { "ratio": 0 }
}
])
Example
I have a MongoDB collection containing elements like this:
{
"name": "test",
"instances": [
{
"year": 2015
},
{
"year": 2016
},
]
}
How can I get the minimum and maximum value for year within the document named test? E.g. I want to aggregate all documents inside that array, but I can't find the syntax for that. Thanks in advance!
Both $min and $max takes an array as a parameter and in your case path instances.year returns an array so your query can look like below:
db.col.aggregate([
{
$match: { name: "test" }
},
{
$project: {
minYear: { $min: "$instances.year" },
maxYear: { $max: "$instances.year" }
}
}
])
You can use below aggregation
db.collection.aggregate([
{ "$project": {
"maxYear": {
"$arrayElemAt": [
"$instances",
{
"$indexOfArray": [
"$instances.year",
{ "$max": "$instances.year" }
]
}
]
},
"minYear": {
"$arrayElemAt": [
"$instances",
{
"$indexOfArray": [
"$instances.year",
{ "$min": "$instances.year" }
]
}
]
}
}}
])
I am trying to split a document which has the following fields of string type:
{
"_id" : "17121",
"firstName": "Jello",
"lastName" : "New",
"bio" :"He is a nice person."
}
I want to split the above document into three new documents For Example:
{
"_id": "17121-1",
"firstName": "Jello"
}
{
"_id": "17121-2",
"firstName": "New"
}
{
"_id": "17121-3",
"bio": "He is a nice person."
}
Can anyone suggest how to proceed?
db.coll1.find().forEach(function(obj){
// I want to extract every single field. How to iterate on the field within this Bson object(obj) to collect every field.?
});
or any suggestion to do with aggregation pipeline in MongoDB.
You can use the below aggregation query.
The below query will convert each document fields into key value document array followed by $unwind while keeping the index and $replaceRoot with merge to produce the desired output.
$objectToArray to produce array (keyvalarr) with key (name of the array field)-value (array field) pair.
$match to remove the _id key value document.
$arrayToObject to produce the named key value while adding new _id key value pair and flatten array key values.
db.coll.aggregate([
{
"$project": {
"keyvalarr": {
"$objectToArray": "$$ROOT"
}
}
},
{
"$unwind": {
"path": "$keyvalarr",
"includeArrayIndex": "index"
}
},
{
"$match": {
"keyvalarr.k": {
"$ne": "_id"
}
}
},
{
"$replaceRoot": {
"newRoot": {
"$arrayToObject": [
{
"k": "_id",
"v": {
"$concat": [
{
"$substr": [
"$_id",
0,
-1
]
},
"-",
{
"$substr": [
"$index",
0,
-1
]
}
]
}
},
"$keyvalarr"
]
}
}
}
])
Anu. Here are two options you can use.
The first option is pretty straightforward, but it requires you to hardcode _id' indexes yourself.
db.users.aggregate([
{
$project: {
pairs : [
{ firstName: '$firstName', _id : { $concat : [ { $substr : [ '$_id', 0, 50 ] }, '-1' ] } },
{ lastName: '$lastName', _id : { $concat : [ '$_id', '-2' ] } },
{ bio: '$bio', _id : { $concat : [ { $substr : [ '$_id', 0, 50 ] }, '-3' ] } }
]
}
},
{
$unwind : '$pairs'
},
{
$replaceRoot: { newRoot: '$pairs' }
}
])
The second option does a little bit more job and is somewhat more tricky. But it is probably easier to extend if you ever need to add another field.
db.users.aggregate([
{
$project: {
pairs : [
{ firstName: '$firstName' },
{ lastName: '$lastName' },
{ bio: '$bio' }
]
}
},
{
$addFields: {
pairsReference : '$pairs'
}
},
{
$unwind: '$pairs'
},
{
$addFields: {
'pairs._id' : { $concat: [ { $substr : [ '$_id', 0, 50 ] }, '-', { $substr: [ { $indexOfArray : [ '$pairsReference', '$pairs' ] }, 0, 2 ] } ] }
}
},
{
$replaceRoot: { newRoot: '$pairs' }
}
])
You can redirect results of both queries into another collection by using $out stage.
UPD:
The only reason you get the error is that one of the _ids is not a string.
Replace the first parameter of $concat ($_id) with the following expression:
{ $substr : [ '$_id', 0, 50 ] }
In aggregation project array element from index x to y where x is defined inside collection and y is an array element corresponding to the index of the array element which is matched. Please see below example then it will become quite easier to understand what i am trying to say.
coupons collection
{"coupon_id": "coupon01", "codes": ["FLAT30", "FLAT50", "FLAT70", "FLAT90"], "curr_index": 0}
For example see below example code here i am trying to get coupon codes starting from curr_index to (curr_index + n) where n is the number in coupon_ctrs corresponding to the index of coupons_ids example- for id "584559bd1f65d363bd5d25fd" n is 1, for id "58455a5c1f65d363bd5d2600" n is 2 and for the id "584878eb0005202c64b0cc5d" n is 3.
coupons_ctrs = [1, 2, 3];
coupons_ids = ["584559bd1f65d363bd5d25fd", "58455a5c1f65d363bd5d2600", "584878eb0005202c64b0cc5d"];
int n = 2;
couponmodel.aggregate(
{ $match : { '_id': { $in : coupons_ids }} },
{ $project: {_id:0, codes : /* How to use slice here so that codes array will be returned between cur_index and (curr_index + coupons_ctr corresponding to the index coupon id is found, example- for _id "584559bd1f65d363bd5d25fd" it should be 1 and so on) */} },
function(err, docs) {
if (err) {
} else {
}
});
UPDATE
As suggested by Styvane i could use $zip and it would work perfect but as i am using mongoDB 3.2.11 so i can't use it, so what can be the solution for using functionality of $zip in mongodb 3.2.11 ??
Can anyone please tell me how can i include this coupon_ctrs array and use it inside aggregation pipeline.
You need to apply a $slice expression to each element in the "coupons_ctrs" array using the $map operator which means that we use the literal "coupons_ctrs" array as "input" to $map.
let coupons_ids = [
"584559bd1f65d363bd5d25fd",
"58455a5c1f65d363bd5d2600",
"584878eb0005202c64b0cc5d"
];
let coupons_ctrs = [1, 2, 3];
db.couponmodel.aggregate(
[
{ "$match" : { "_id": { "$in" : coupons_ids } } },
{ "$project": {
"codes": {
"$map": {
"input": coupons_ctrs,
"as": "n",
"in": {
"$slice": [
"$codes",
"$curr_index",
{ "$add": [ "$curr_index", "$$n" ] }
]
}
}
}
}}
]
)
Which yields:
{
"codes" : [
[ "FLAT30" ],
[ "FLAT30", "FLAT50" ],
[ "FLAT30", "FLAT50", "FLAT70" ]
]
}
In MongoDB 3.4 we can use the $zip operator to do this:
db.couponmodel.aggregate(
[
{ "$project": {
"codes": {
"$map": {
"input": {
"$zip": {
"inputs": [ coupons_ids, coupons_ctrs ]
}
},
"as": "item",
"in": {
"coupon_id": { "$arrayElemAt": [ "$$item", 0 ] },
"value": {
"$slice": [
"$codes",
"$curr_index",
{ "$add": [
"$curr_index",
{ "$arrayElemAt": [ "$$item", 1 ] }
] }
]
}
}
}
}
}}
]
)
which return something like this:
{
"codes" : [
{
"coupon_id" : "584559bd1f65d363bd5d25fd",
"value" : [ "FLAT30" ]
},
{
"coupon_id" : "58455a5c1f65d363bd5d2600",
"value" : [ "FLAT30", "FLAT50" ],
},
{
"coupon_id" : "584878eb0005202c64b0cc5d",
"value" : [ "FLAT30", "FLAT50", "FLAT70" ]
}
]
}
I am trying to convert a string that contains a numerical value to its value in an aggregate query in MongoDB.
Example of document
{
"_id": ObjectId("5522XXXXXXXXXXXX"),
"Date": "2015-04-05",
"PartnerID": "123456",
"moop": "1234"
}
Example of the aggregate query I use
{
aggregate: 'my_collection',
pipeline: [
{$match: {
Date :
{$gt:'2015-04-01',
$lt: '2015-04-05'
}}
},
{$group:
{_id: "$PartnerID",
total:{$sum:'$moop'}
}}]}
where the results are
{
"result": [
{
"_id": "123456",
"total": NumberInt(0)
}
}
How can you convert the string to its numerical value?
MongoDB aggregation not allowed to change existing data type of given fields. In this case you should create some programming code to convert string to int. Check below code
db.collectionName.find().forEach(function(data) {
db.collectionName.update({
"_id": data._id,
"moop": data.moop
}, {
"$set": {
"PartnerID": parseInt(data.PartnerID)
}
});
})
If your collections size more then above script will slow down the performance, for perfomace mongo provide mongo bulk operations, using mongo bulk operations also updated data type
var bulk = db.collectionName.initializeOrderedBulkOp();
var counter = 0;
db.collectionName.find().forEach(function(data) {
var updoc = {
"$set": {}
};
var myKey = "PartnerID";
updoc["$set"][myKey] = parseInt(data.PartnerID);
// queue the update
bulk.find({
"_id": data._id
}).update(updoc);
counter++;
// Drain and re-initialize every 1000 update statements
if (counter % 1000 == 0) {
bulk.execute();
bulk = db.collectionName.initializeOrderedBulkOp();
}
})
// Add the rest in the queue
if (counter % 1000 != 0) bulk.execute();
This basically reduces the amount of operations statements sent to the sever to only sending once every 1000 queued operations.
Using MongoDB 4.0 and newer
You have two options i.e. $toInt or $convert. Using $toInt, follow the example below:
filterDateStage = {
'$match': {
'Date': {
'$gt': '2015-04-01',
'$lt': '2015-04-05'
}
}
};
groupStage = {
'$group': {
'_id': '$PartnerID',
'total': { '$sum': { '$toInt': '$moop' } }
}
};
db.getCollection('my_collection').aggregate([
filterDateStage,
groupStage
])
If the conversion operation encounters an error, the aggregation operation stops and throws an error. To override this behavior, use $convert instead.
Using $convert
groupStage = {
'$group': {
'_id': '$PartnerID',
'total': {
'$sum': {
'$convert': { 'input': '$moop', 'to': 'int' }
}
}
}
};
Using Map/Reduce
With map/reduce you can use javascript functions like parseInt() to do the conversion. As an example, you could define the map function to process each input document:
In the function, this refers to the document that the map-reduce operation is processing. The function maps the converted moop string value to the PartnerID for each document and emits the PartnerID and converted moop pair. This is where the javascript native function parseInt() can be applied:
var mapper = function () {
var x = parseInt(this.moop);
emit(this.PartnerID, x);
};
Next, define the corresponding reduce function with two arguments keyCustId and valuesMoop. valuesMoop is an array whose elements are the integer moop values emitted by the map function and grouped by keyPartnerID.
The function reduces the valuesMoop array to the sum of its elements.
var reducer = function(keyPartnerID, valuesMoop) {
return Array.sum(valuesMoop);
};
db.collection.mapReduce(
mapper,
reducer,
{
out : "example_results",
query: {
Date: {
$gt: "2015-04-01",
$lt: "2015-04-05"
}
}
}
);
db.example_results.find(function (err, docs) {
if(err) console.log(err);
console.log(JSON.stringify(docs));
});
For example, with the following sample collection of documents:
/* 0 */
{
"_id" : ObjectId("550c00f81bcc15211016699b"),
"Date" : "2015-04-04",
"PartnerID" : "123456",
"moop" : "1234"
}
/* 1 */
{
"_id" : ObjectId("550c00f81bcc15211016699c"),
"Date" : "2015-04-03",
"PartnerID" : "123456",
"moop" : "24"
}
/* 2 */
{
"_id" : ObjectId("550c00f81bcc15211016699d"),
"Date" : "2015-04-02",
"PartnerID" : "123457",
"moop" : "21"
}
/* 3 */
{
"_id" : ObjectId("550c00f81bcc15211016699e"),
"Date" : "2015-04-02",
"PartnerID" : "123457",
"moop" : "8"
}
The above Map/Reduce operation will save the results to the example_results collection and the shell command db.example_results.find() will give:
/* 0 */
{
"_id" : "123456",
"value" : 1258
}
/* 1 */
{
"_id" : "123457",
"value" : 29
}
You can easily convert the string data type to numerical data type.
Don't forget to change collectionName & FieldName.
for ex : CollectionNmae : Users & FieldName : Contactno.
Try this query..
db.collectionName.find().forEach( function (x) {
x.FieldName = parseInt(x.FieldName);
db.collectionName.save(x);
});
Eventually I used
db.my_collection.find({moop: {$exists: true}}).forEach(function(obj) {
obj.moop = new NumberInt(obj.moop);
db.my_collection.save(obj);
});
to turn moop from string to integer in my_collection following the example in Simone's answer MongoDB: How to change the type of a field?.
String can be converted to numbers in MongoDB v4.0 using $toInt operator. In this case
db.col.aggregate([
{
$project: {
_id: 0,
moopNumber: { $toInt: "$moop" }
}
}
])
outputs:
{ "moopNumber" : 1234 }
Here is a pure MongoDB based solution for this problem which I just wrote for fun. It's effectively a server-side string-to-number parser which supports positive and negative numbers as well as decimals:
db.collection.aggregate({
$addFields: {
"moop": {
$reduce: {
"input": {
$map: { // split string into char array so we can loop over individual characters
"input": {
$range: [ 0, { $strLenCP: "$moop" } ] // using an array of all numbers from 0 to the length of the string
},
"in":{
$substrCP: [ "$moop", "$$this", 1 ] // return the nth character as the mapped value for the current index
}
}
},
"initialValue": { // initialize the parser with a 0 value
"n": 0, // the current number
"sign": 1, // used for positive/negative numbers
"div": null, // used for shifting on the right side of the decimal separator "."
"mult": 10 // used for shifting on the left side of the decimal separator "."
}, // start with a zero
"in": {
$let: {
"vars": {
"n": {
$switch: { // char-to-number mapping
branches: [
{ "case": { $eq: [ "$$this", "1" ] }, "then": 1 },
{ "case": { $eq: [ "$$this", "2" ] }, "then": 2 },
{ "case": { $eq: [ "$$this", "3" ] }, "then": 3 },
{ "case": { $eq: [ "$$this", "4" ] }, "then": 4 },
{ "case": { $eq: [ "$$this", "5" ] }, "then": 5 },
{ "case": { $eq: [ "$$this", "6" ] }, "then": 6 },
{ "case": { $eq: [ "$$this", "7" ] }, "then": 7 },
{ "case": { $eq: [ "$$this", "8" ] }, "then": 8 },
{ "case": { $eq: [ "$$this", "9" ] }, "then": 9 },
{ "case": { $eq: [ "$$this", "0" ] }, "then": 0 },
{ "case": { $and: [ { $eq: [ "$$this", "-" ] }, { $eq: [ "$$value.n", 0 ] } ] }, "then": "-" }, // we allow a minus sign at the start
{ "case": { $eq: [ "$$this", "." ] }, "then": "." }
],
default: null // marker to skip the current character
}
}
},
"in": {
$switch: {
"branches": [
{
"case": { $eq: [ "$$n", "-" ] },
"then": { // handle negative numbers
"sign": -1, // set sign to -1, the rest stays untouched
"n": "$$value.n",
"div": "$$value.div",
"mult": "$$value.mult",
},
},
{
"case": { $eq: [ "$$n", null ] }, // null is the "ignore this character" marker
"then": "$$value" // no change to current value
},
{
"case": { $eq: [ "$$n", "." ] },
"then": { // handle decimals
"n": "$$value.n",
"sign": "$$value.sign",
"div": 10, // from the decimal separator "." onwards, we start dividing new numbers by some divisor which starts at 10 initially
"mult": 1, // and we stop multiplying the current value by ten
},
},
],
"default": {
"n": {
$add: [
{ $multiply: [ "$$value.n", "$$value.mult" ] }, // multiply the already parsed number by 10 because we're moving one step to the right or by one once we're hitting the decimals section
{ $divide: [ "$$n", { $ifNull: [ "$$value.div", 1 ] } ] } // add the respective numerical value of what we look at currently, potentially divided by a divisor
]
},
"sign": "$$value.sign",
"div": { $multiply: [ "$$value.div" , 10 ] },
"mult": "$$value.mult"
}
}
}
}
}
}
}
}
}, {
$addFields: { // fix sign
"moop": { $multiply: [ "$moop.n", "$moop.sign" ] }
}
})
I am certainly not advertising this as the bee's knees or anything and it might have severe performance implications for larger datasets over a client based solutions but there might be cases where it comes in handy...
The above pipeline will transform the following documents:
{ "moop": "12345" } --> { "moop": 12345 }
and
{ "moop": "123.45" } --> { "moop": 123.45 }
and
{ "moop": "-123.45" } --> { "moop": -123.45 }
and
{ "moop": "2018-01-03" } --> { "moop": 20180103.0 }
Three things need to care for:
parseInt() will store double data type in mongodb. Please use new NumberInt(string).
in Mongo shell command for bulk usage, yield won't work. Please DO NOT add 'yield'.
If you already change string to double by parseInt(). It looks like you have no way to change the type to int directly. The solution is a little bit wired: change double to string first and then change back to int by new NumberInt().
If you can edit all documents in aggregate :
"TimeStamp": {$toDecimal: {$toDate: "$Your Date"}}
And for the client, you set the query :
Date.parse("Your date".toISOString())
That's what makes you whole work with ISODate.
Try:
"TimeStamp":{$toDecimal: { $toDate:"$Datum"}}
Though $toInt is really useful, it was added on mongoDB 4.0, I've run into this same situation in a database running 3.2 which upgrading to use $toInt was not an option due to some other application incompatibilities, so i had to come up with something else, and actually was surprisingly simple.
If you $project and $add zero to your string, it will turn into a number
{
$project : {
'convertedField' : { $add : ["$stringField",0] },
//more fields here...
}
}
It should be saved. It should be like this :
db. my_collection.find({}).forEach(function(theCollection) {
theCollection.moop = parseInt(theCollection.moop);
db.my_collection.save(theCollection);
});
Collation is what you need:
db.collectionName.find().sort({PartnerID: 1}).collation({locale: "en_US", numericOrdering: true})
db.user.find().toArray().filter(a=>a.age>40)