I have an aggregation pipeline in which i want to add new field based on certain condition. My pipeline is like this
[
{ // match stage
$or:[
{
$and: [
{placement: {'$nin': [-1,-2]}},
{contract_proposal_metadata : {$exists: true}}
]
},
{
risk_info_request_metadata: {$exists: true}
}
]
}
]
Now i want to add a new field record_type based on the condition that if contract_proposal_metadata exists so record type will be 'renewal' and if risk_info_request_metadata is exists then record_type will be request.
How can i achieve this?
You are not adding new field conditionally. You are always adding the field, just with different values.
There is $cond operator which returns 1 of 2 values depending on condition in the first argument.
You already know $exist for the $match stage, and the equivalent operator to use in aggregation expression is $type
[
{ // match stage
.....
},
{ // adding the field
$addFields: {
record_type: { $cond: {
if: { $eq: [ { $type: "$contract_proposal_metadata" }, "missing" ] },
then: "request",
else: "renewal"
} }
}
}
]
You need to use aggregate update
db.collection.update({
placement: { //Your match goes here
"$nin": [
-1,
-2
]
},
},
[
{
$set: {
status: {
$switch: {
branches: [
{ //update condition goes here
case: {
$ifNull: [
"$contract_proposal_metadata",
false
]
},
then: "renewal"
},
{
case: {
$ifNull: [
"$risk_info_request_metadata",
false
]
},
then: "request"
},
],
default: ""
}
}
}
}
],
{
multi: true
})
It supported from mongo 4.2+
$exists cannot be used, hence $ifnull used
playground
How do I return records from my 'greetsthings' collection where the 'things' doesn't have a specific substring?
{
"_id" : ObjectId("5d24e6e5e8b6b11536a8519b"),
"message" : "hello",
"meta" : {
"info" : "friendly score 923",
"things" : "cat bat dragon",
}
},
{
"_id" : ObjectId("5d24e6e5e8b6b11536a8519c"),
"message" : "hello",
"meta" : {
"info" : "confused score 622",
"things" : "cat monkey dragon",
}
}
I'm trying to query/filter for all 'hello message' records of {$match:{ message: { $eq: 'hello' }}} by a substring in meta.things
And by comparing the substring of monkey to meta.things to filter the results
When I try this $filter it errors with "errmsg" : "input to $filter must be an array not object"
db.getCollection('greetsthings').aggregate(
[{$match:{ message: { $eq: 'hello' }}},
{$project: {
message: 1,
"meta": {
$filter: {$gte : [{$indexOfCP: ["$meta.things", "monkey"]},0]}
}
}
}
])
How can I return only records where the message is hello and monkey is in the meta.things string?
You have "syntax" error with "$meta.things". $expr allows compute operation with document's fields:
db.greetsthings.aggregate([
{
$match: {
$expr: {
$and: [
{
$eq: [
"$message",
"hello"
]
},
{
$gt: [
{
$indexOfCP: [
"$meta.things",
"monkey"
]
},
-1
]
}
]
}
}
}
])
MongoPlayground
And also negation, how can I return records where the message is hello but dragon isn't in the meta.things string?
Searches a string for an occurence of a substring and returns the UTF-8 code point index (zero-based) of the first occurence. If the substring is not found, returns -1.
https://docs.mongodb.com/manual/reference/operator/aggregation/indexOfCP/
$eq: [
{
$indexOfCP: [
"$meta.things",
"dragon"
]
},
-1
]
Please try this :
db.getCollection('greetsthings').aggregate(
[{ $match: { message: 'hello' } },
{
$addFields: {
shouldExists: { $gte: [{ $indexOfCP: ['$meta.things', "monkey"] }, 0] }, // Adds try if word monkey exists in string
shouldNotExists: { $gte: [{ $indexOfCP: ['$meta.things', "dragon"] }, 0] } // Adds try if word dragon exists in string
}
}, { $match: { shouldExists: true, shouldNotExists: false } },
{ $project: { shouldExists: 0, shouldNotExists: 0 } }
])
Test : MongoDB-Playground
I have mongodb document with the following data:
amenities[
{
amenity_id:52,
Amenity:"AC"
},
{
amenity_id:23,
Amenity:"Free Parking"
}
]
I want to match each amenity_id element of the array with particular value and return true or false using condition $cond. I used
"$project":{'Free Parking':{'$cond':{'if':{'$in':['amenities.amenityId',[23]]},'then':'True','else':'False'}}
If a document contains amenity_id= 52 then a query has to return False.
It is returning false irrespective of the menityId. The amenity Id could be list hence using $in. How can i test each element ?
Considering your input collection is
[
{
amenities: [
{
_id: 52,
Amenity: "AC"
},
{
_id: 23,
Amenity: "Free Parking"
}
]
}
]
using aggregate pipeline $map
db.collection.aggregate([
{
$project: {
amenites: {
$map: {
input: "$amenities",
as: "item",
in: {
Amenity: "$$item.Amenity",
_id: "$$item._id",
isValid: {
$cond: {
if: {
$eq: [
"$$item._id",
23
]
},
then: true,
else: false
},
}
}
}
}
}
}
])
You'll get result as:
[
{
"amenites": [
{
"Amenity": "AC",
"_id": 52,
"isValid": false
},
{
"Amenity": "Free Parking",
"_id": 23,
"isValid": true
}
]
}
]
use $unwind operation in mongodb it will unwind the array field in to multiple documents
Please refer this url - https://docs.mongodb.com/manual/reference/operator/aggregation/unwind/
ex:
{
_id :1,
field: [1,2,3]
}
this will be converted to the following using $unwind,
[
{_id, field:1},
{_id, field:2},
{_id, field:3}
]
after $unwind you can run a match query, with $in query,
Ex:
{
$match: {
field: {$in: [1,2]}
}
}
Assume that there is 4 users in collections.
> db.users.find().pretty()
{
"_id" : ObjectId("5d369b451b48d91cba76c618"),
"user_id" : 1,
"final_score" : 65,
"max_score" : 15,
"min_score" : 15,
}
{
"_id" : ObjectId("5d369b451b48d91cba76c619"),
"user_id" : 2,
"final_score" : 70,
"max_score" : 15,
"min_score" : 15,
}
{
"_id" : ObjectId("5d369b451b48d91cba76c61a"),
"user_id" : 3,
"final_score" : 60,
"max_score" : 15,
"min_score" : 15,
}
{
"_id" : ObjectId("5d369b451b48d91cba76c61b"),
"user_id" : 4,
"final_score" : 83,
"max_score" : 15,
"min_score" : 15,
}
I want to extract users that meet below conditions.
final_score >= user_id=3's final_score + each document's max_score
final_score <= user_id=3's final_score - each document's min_score
To represent with MySQL, it is very simple.
SELECT * FROM users
WHERE final_score <= 60 + users.max_score AND final_score >= 60 - users.min_score
But I wonder that how can I querying with mongodb?
Thanks.
EDIT
I think it can be execute with this.
So I made query like this.
db.users.find({
'final_score': {
'$lte': '60 + this.max_score',
'$gte': '60 - this.min_score'
}
})
But it return nothing
The difficulty here comes from the fact that you need to run two separate pipelines (one to get the value for user 3 and second one to filter all documents). In Aggregation Framework you can do that using $facet operator which allows you to run multiple pipelines and then keep processing data in subsequent steps. To compare the data you can use $filter and to get original shape as a result you need to transform nested array into separate documents using $unwind and $replaceRoot
db.users.aggregate([
{
$facet: {
user3: [
{ $match: { user_id: 3 } }
],
docs: [
{ $match: {} }
]
}
},
{
$addFields: {
user3: { $arrayElemAt: [ "$user3", 0 ] }
}
},
{
$project: {
docs: {
$filter: {
input: "$docs",
cond: {
$and: [
{ $lte: [ "$$this.final_score", { $add: [ "$user3.final_score", "$$this.max_score" ] } ] },
{ $gte: [ "$$this.final_score", { $subtract: [ "$user3.final_score", "$$this.max_score" ] } ] },
]
}
}
}
}
},
{
$unwind: "$docs"
},
{
$replaceRoot: {
newRoot: "$docs"
}
}
])
Mongo Playground
From your description, I guess you already know the score of user3 is 60.
In this case:
db.collection.aggregate([
{
$addFields: {
match: {
$and: [
{
$gte: [
"$final_score",
{
$subtract: [
60,
"$min_score"
]
}
]
},
{
$lte: [
"$final_score",
{
$add: [
60,
"$max_score"
]
}
]
}
]
}
}
},
{
$match: {
match: true
}
},
{
$project: {
match: 0
}
}
])
mongoplayground
I am trying to convert a string that contains a numerical value to its value in an aggregate query in MongoDB.
Example of document
{
"_id": ObjectId("5522XXXXXXXXXXXX"),
"Date": "2015-04-05",
"PartnerID": "123456",
"moop": "1234"
}
Example of the aggregate query I use
{
aggregate: 'my_collection',
pipeline: [
{$match: {
Date :
{$gt:'2015-04-01',
$lt: '2015-04-05'
}}
},
{$group:
{_id: "$PartnerID",
total:{$sum:'$moop'}
}}]}
where the results are
{
"result": [
{
"_id": "123456",
"total": NumberInt(0)
}
}
How can you convert the string to its numerical value?
MongoDB aggregation not allowed to change existing data type of given fields. In this case you should create some programming code to convert string to int. Check below code
db.collectionName.find().forEach(function(data) {
db.collectionName.update({
"_id": data._id,
"moop": data.moop
}, {
"$set": {
"PartnerID": parseInt(data.PartnerID)
}
});
})
If your collections size more then above script will slow down the performance, for perfomace mongo provide mongo bulk operations, using mongo bulk operations also updated data type
var bulk = db.collectionName.initializeOrderedBulkOp();
var counter = 0;
db.collectionName.find().forEach(function(data) {
var updoc = {
"$set": {}
};
var myKey = "PartnerID";
updoc["$set"][myKey] = parseInt(data.PartnerID);
// queue the update
bulk.find({
"_id": data._id
}).update(updoc);
counter++;
// Drain and re-initialize every 1000 update statements
if (counter % 1000 == 0) {
bulk.execute();
bulk = db.collectionName.initializeOrderedBulkOp();
}
})
// Add the rest in the queue
if (counter % 1000 != 0) bulk.execute();
This basically reduces the amount of operations statements sent to the sever to only sending once every 1000 queued operations.
Using MongoDB 4.0 and newer
You have two options i.e. $toInt or $convert. Using $toInt, follow the example below:
filterDateStage = {
'$match': {
'Date': {
'$gt': '2015-04-01',
'$lt': '2015-04-05'
}
}
};
groupStage = {
'$group': {
'_id': '$PartnerID',
'total': { '$sum': { '$toInt': '$moop' } }
}
};
db.getCollection('my_collection').aggregate([
filterDateStage,
groupStage
])
If the conversion operation encounters an error, the aggregation operation stops and throws an error. To override this behavior, use $convert instead.
Using $convert
groupStage = {
'$group': {
'_id': '$PartnerID',
'total': {
'$sum': {
'$convert': { 'input': '$moop', 'to': 'int' }
}
}
}
};
Using Map/Reduce
With map/reduce you can use javascript functions like parseInt() to do the conversion. As an example, you could define the map function to process each input document:
In the function, this refers to the document that the map-reduce operation is processing. The function maps the converted moop string value to the PartnerID for each document and emits the PartnerID and converted moop pair. This is where the javascript native function parseInt() can be applied:
var mapper = function () {
var x = parseInt(this.moop);
emit(this.PartnerID, x);
};
Next, define the corresponding reduce function with two arguments keyCustId and valuesMoop. valuesMoop is an array whose elements are the integer moop values emitted by the map function and grouped by keyPartnerID.
The function reduces the valuesMoop array to the sum of its elements.
var reducer = function(keyPartnerID, valuesMoop) {
return Array.sum(valuesMoop);
};
db.collection.mapReduce(
mapper,
reducer,
{
out : "example_results",
query: {
Date: {
$gt: "2015-04-01",
$lt: "2015-04-05"
}
}
}
);
db.example_results.find(function (err, docs) {
if(err) console.log(err);
console.log(JSON.stringify(docs));
});
For example, with the following sample collection of documents:
/* 0 */
{
"_id" : ObjectId("550c00f81bcc15211016699b"),
"Date" : "2015-04-04",
"PartnerID" : "123456",
"moop" : "1234"
}
/* 1 */
{
"_id" : ObjectId("550c00f81bcc15211016699c"),
"Date" : "2015-04-03",
"PartnerID" : "123456",
"moop" : "24"
}
/* 2 */
{
"_id" : ObjectId("550c00f81bcc15211016699d"),
"Date" : "2015-04-02",
"PartnerID" : "123457",
"moop" : "21"
}
/* 3 */
{
"_id" : ObjectId("550c00f81bcc15211016699e"),
"Date" : "2015-04-02",
"PartnerID" : "123457",
"moop" : "8"
}
The above Map/Reduce operation will save the results to the example_results collection and the shell command db.example_results.find() will give:
/* 0 */
{
"_id" : "123456",
"value" : 1258
}
/* 1 */
{
"_id" : "123457",
"value" : 29
}
You can easily convert the string data type to numerical data type.
Don't forget to change collectionName & FieldName.
for ex : CollectionNmae : Users & FieldName : Contactno.
Try this query..
db.collectionName.find().forEach( function (x) {
x.FieldName = parseInt(x.FieldName);
db.collectionName.save(x);
});
Eventually I used
db.my_collection.find({moop: {$exists: true}}).forEach(function(obj) {
obj.moop = new NumberInt(obj.moop);
db.my_collection.save(obj);
});
to turn moop from string to integer in my_collection following the example in Simone's answer MongoDB: How to change the type of a field?.
String can be converted to numbers in MongoDB v4.0 using $toInt operator. In this case
db.col.aggregate([
{
$project: {
_id: 0,
moopNumber: { $toInt: "$moop" }
}
}
])
outputs:
{ "moopNumber" : 1234 }
Here is a pure MongoDB based solution for this problem which I just wrote for fun. It's effectively a server-side string-to-number parser which supports positive and negative numbers as well as decimals:
db.collection.aggregate({
$addFields: {
"moop": {
$reduce: {
"input": {
$map: { // split string into char array so we can loop over individual characters
"input": {
$range: [ 0, { $strLenCP: "$moop" } ] // using an array of all numbers from 0 to the length of the string
},
"in":{
$substrCP: [ "$moop", "$$this", 1 ] // return the nth character as the mapped value for the current index
}
}
},
"initialValue": { // initialize the parser with a 0 value
"n": 0, // the current number
"sign": 1, // used for positive/negative numbers
"div": null, // used for shifting on the right side of the decimal separator "."
"mult": 10 // used for shifting on the left side of the decimal separator "."
}, // start with a zero
"in": {
$let: {
"vars": {
"n": {
$switch: { // char-to-number mapping
branches: [
{ "case": { $eq: [ "$$this", "1" ] }, "then": 1 },
{ "case": { $eq: [ "$$this", "2" ] }, "then": 2 },
{ "case": { $eq: [ "$$this", "3" ] }, "then": 3 },
{ "case": { $eq: [ "$$this", "4" ] }, "then": 4 },
{ "case": { $eq: [ "$$this", "5" ] }, "then": 5 },
{ "case": { $eq: [ "$$this", "6" ] }, "then": 6 },
{ "case": { $eq: [ "$$this", "7" ] }, "then": 7 },
{ "case": { $eq: [ "$$this", "8" ] }, "then": 8 },
{ "case": { $eq: [ "$$this", "9" ] }, "then": 9 },
{ "case": { $eq: [ "$$this", "0" ] }, "then": 0 },
{ "case": { $and: [ { $eq: [ "$$this", "-" ] }, { $eq: [ "$$value.n", 0 ] } ] }, "then": "-" }, // we allow a minus sign at the start
{ "case": { $eq: [ "$$this", "." ] }, "then": "." }
],
default: null // marker to skip the current character
}
}
},
"in": {
$switch: {
"branches": [
{
"case": { $eq: [ "$$n", "-" ] },
"then": { // handle negative numbers
"sign": -1, // set sign to -1, the rest stays untouched
"n": "$$value.n",
"div": "$$value.div",
"mult": "$$value.mult",
},
},
{
"case": { $eq: [ "$$n", null ] }, // null is the "ignore this character" marker
"then": "$$value" // no change to current value
},
{
"case": { $eq: [ "$$n", "." ] },
"then": { // handle decimals
"n": "$$value.n",
"sign": "$$value.sign",
"div": 10, // from the decimal separator "." onwards, we start dividing new numbers by some divisor which starts at 10 initially
"mult": 1, // and we stop multiplying the current value by ten
},
},
],
"default": {
"n": {
$add: [
{ $multiply: [ "$$value.n", "$$value.mult" ] }, // multiply the already parsed number by 10 because we're moving one step to the right or by one once we're hitting the decimals section
{ $divide: [ "$$n", { $ifNull: [ "$$value.div", 1 ] } ] } // add the respective numerical value of what we look at currently, potentially divided by a divisor
]
},
"sign": "$$value.sign",
"div": { $multiply: [ "$$value.div" , 10 ] },
"mult": "$$value.mult"
}
}
}
}
}
}
}
}
}, {
$addFields: { // fix sign
"moop": { $multiply: [ "$moop.n", "$moop.sign" ] }
}
})
I am certainly not advertising this as the bee's knees or anything and it might have severe performance implications for larger datasets over a client based solutions but there might be cases where it comes in handy...
The above pipeline will transform the following documents:
{ "moop": "12345" } --> { "moop": 12345 }
and
{ "moop": "123.45" } --> { "moop": 123.45 }
and
{ "moop": "-123.45" } --> { "moop": -123.45 }
and
{ "moop": "2018-01-03" } --> { "moop": 20180103.0 }
Three things need to care for:
parseInt() will store double data type in mongodb. Please use new NumberInt(string).
in Mongo shell command for bulk usage, yield won't work. Please DO NOT add 'yield'.
If you already change string to double by parseInt(). It looks like you have no way to change the type to int directly. The solution is a little bit wired: change double to string first and then change back to int by new NumberInt().
If you can edit all documents in aggregate :
"TimeStamp": {$toDecimal: {$toDate: "$Your Date"}}
And for the client, you set the query :
Date.parse("Your date".toISOString())
That's what makes you whole work with ISODate.
Try:
"TimeStamp":{$toDecimal: { $toDate:"$Datum"}}
Though $toInt is really useful, it was added on mongoDB 4.0, I've run into this same situation in a database running 3.2 which upgrading to use $toInt was not an option due to some other application incompatibilities, so i had to come up with something else, and actually was surprisingly simple.
If you $project and $add zero to your string, it will turn into a number
{
$project : {
'convertedField' : { $add : ["$stringField",0] },
//more fields here...
}
}
It should be saved. It should be like this :
db. my_collection.find({}).forEach(function(theCollection) {
theCollection.moop = parseInt(theCollection.moop);
db.my_collection.save(theCollection);
});
Collation is what you need:
db.collectionName.find().sort({PartnerID: 1}).collation({locale: "en_US", numericOrdering: true})
db.user.find().toArray().filter(a=>a.age>40)