I'm using mongodb with aggregation framework and I need to somehow pass in my query value with some string.
I need something like this:
{
'$project' => {
'value' => '$value',
'label' => 'Some string'
}
}
And in result I should have the following:
{
value => 'value of $value',
label => 'Some string'
}
I have to use only aggregation framework.
In shell you can use this "solution":
t = "Some text..."
db.towns.aggregate([
{'$project': {"nm" : "$name", "text": {$substr : [t,0, t.length]}}}
])
You only need to adapt for your programming language.
independently of the programming language, and before $literal is there, you can use for example $concat to assign static value to projection (how to define is as parameter depends actually on the PL/driver that you use):
{
'$project' : {
'value' : {$concat: ['$value']} ,
....
}
}
Related
I am trying to implement a search feature to MongoDB and this is the aggregate pipeline I am using:
[
{
'$search': {
'text': {
'query': 'albus',
'path': [
'first_name', 'email', 'last_name'
]
}
}
}, {
'$project': {
'_id': 1,
'first_name': 1,
'last_name': 1
}
}, {
'$limit': 5
}
]
The command returns documents that contain only exactly albus or Albus, but return nothing for queries like alb, albu, etc. In the demo video I watched here: https://www.youtube.com/watch?time_continue=8&v=kZ77X67GUfk, the instructor was able to search based on substring.
The search index I am currently using is the default dynamic one.
How would I need to change my command?
You need to use the autocomplete feature, so your query will look like this:
{
$search: {
"autocomplete": {
'query': 'albus',
'path': [
'first_name', 'email', 'last_name'
]
}
}
}
Mind you both first_name, email and last_name need to be mapped as autocomplete type so a name like albus will be indexed as a, al, alb, albu, albus. Obviously this will vastly increase your index size.
Another thing to consider is tweaking the maxGrams and tokenization parameters. this will allow very long names to still work as expected and if you want to allow substring match like lbu matching albus.
Here I have created a collection with a single document
db.getCollection('example').insert({"example":1});
I have tried to use Projection, and I get back the _id.
db.getCollection('example').find({"example":1},{"_id":1});
{
"_id" : ObjectId("562a6300bbc948a4315f3abc")
}
However, I need the below output as shown below.
id and not _id
ObjectId("562a6300bbc948a4315f3abc") vs "562a6300bbc948a4315f3abc"
{
"id" : "562a6300bbc948a4315f3abc"
}
Although I can process #1 and #2 on my app server(PHP based) to get the desired ouput, I am looking if there is a way to get the expected result on querying from mongo itself
MongoDB 4.0 adds the $convert aggregation operator and the $toString alias which allows you to do exactly that:
db.getCollection('example').aggregate([
{ "$match": { "example":1 } },
{ "$project": { "_id": { "$toString": "$_id" } } }
])
A main usage would most likely be though to use the _id value as a "key" in a document.
db.getCollection('example').insertOne({ "a": 1, "b": 2 })
db.getCollection('example').aggregate([
{ "$replaceRoot": {
"newRoot": {
"$arrayToObject": [
[{
"k": { "$toString": "$_id" },
"v": {
"$arrayToObject": {
"$filter": {
"input": { "$objectToArray": "$$ROOT" },
"cond": { "$ne": ["$$this.k", "_id"] }
}
}
}
}]
]
}
}}
])
Which would return:
{
"5b06973e7f859c325db150fd" : { "a" : 1, "b" : 2 }
}
Which clearly shows the string, as does the other example.
Generally though there is usually a way to do "transforms" on the cursor as documents are returned from the server. This is usually a good thing since an ObjectId is a 12-byte binary representation as opposed to a 24 character hex "string" which takes a lot more space.
The shell has a .map() method
db.getCollection('example').find().map(d => Object.assign(d, { _id: d._id.valueOf() }) )
And NodeJS has a Cursor.map() which can do much the same thing:
let cursor = db.collection('example').find()
.map(( _id, ...d }) => ({ _id: _id.toString(), ...d }));
while ( await cursor.hasNext() ) {
let doc = cursor.next();
// do something
})
And the same method exists in other drivers as well ( just not PHP ), or you can just iterate the cursor and transform the content as is more likely the best thing to do.
In fact, whole cursor results can be reduced into a single object with great ease by simply adding to any cursor returning statement, when working in the shell
.toArray().reduce((o,e) => {
var _id = e._id;
delete e._id;
return Object.assign(o, { [_id]: e })
},{ })
Or for full ES6 JavaScript supporting environments like nodejs:
.toArray().reduce((o,({ _id, ...e })) => ({ ...o, [_id]: e }),{ })
Really simple stuff without the complexity of what needs to process in the aggregation framework. And very possible in any language by much the same means.
You need to use the .aggregate() method.
db.getCollection('example').aggregate([ { "$project": { "_id": 0, "id": "$_id" } } ]);
Which yields:
{ "id" : ObjectId("562a67745488a8d831ce2e35") }
or using the .str property.
db.getCollection('example').find({"example":1},{"_id":1}).map(function(doc) {
return {'id': doc._id.str }
})
Which returns:
[ { "id" : "562a67745488a8d831ce2e35" } ]
Well if you are using the PHP driver you can do something like this:
$connection = new MongoClient();
$db = $connection->test;
$col = $db->example;
$cursor = $col->find([], ["_id" => 1]);
foreach($cursor as $doc) { print_r(array("id" => $doc["_id"])); }
Which yields:
Array
(
[id] => MongoId Object
(
[$id] => 562a6c60f850734c0c8b4567
)
)
Or using again the MongoCollection::aggregate method.
$result = $col->aggregate(array(["$project" => ["id" => "$_id", "_id" => 0]]))
Then using the foreach loop:
Array
(
[_id] => MongoId Object
(
[$id] => 562a6c60f850734c0c8b4567
)
)
One simple solution for traversing MongoCursor on PHP side is to use Generators as well as foreach or array_map($function, iterator_to_array($cursor)).
Example:
function map_traversable(callable $mapper, \Traversable $iterator) {
foreach($iterator as $val) {
yield $mapper($val);
}
}
You can meet more at PHP documentation about generators syntax.
So, now you can use/reuse it (or similar implementation) for any propose of "projecting" your data on PHP side with any amount of mapping (just like pipeline does in aggregate) but with fewer iterations amount. And this solution is pretty convenient for OOP in a case of reusing your map functions.
UPD:
Just for your case example below:
$cursor = $db->getCollection('example')->find(["example":1],["_id":1]);
$mapper = function($record) {
return array('id' => (string) $record['_id']); //see \MongoId::__toString()
}
$traversableWithIdAsStringApplied = map_traversable($mapper, $cursor);
//...
now you can proceed with more mappings applied to $traversableWithIdAsStringApplied or use just iterator_to_array for simple array retrieving.
I'm in this scenario right now:
I have a collection X:
{
_id:ObjectId('56edbb4d5f084a51131dd4c6'),
userRef:ObjectId('56edbb4d5f084a51131dd4c6'),
serialNumber:'A123123',
...
}
I need to aggregate all documents, grouping them by the userRef + serialNumber, so I'm trying to use concat like this:
$group: {
_id: {
'$concat': ['$userRef','-','$serialNumber']
},
...
So basically in my aggregation in MongoDB, I need to group documents by the concatenation of a ObjectId and a string. However, It seems that $concat only accepts strings as parameters:
uncaught exception: aggregate failed: {
"errmsg" : "exception: $concat only supports strings, not OID",
"code" : 16702,
"ok" : 0
}
Is there a way to convert an ObjectId to a String within an aggregation expression?
EDIT:
This question is related, but I the solution doesn't fit my problem. (Specially because I can't use ObjectId.toString() during the aggregation)
Indeed I couldn't find any ObjectId().toString() operation in Mongo's documentation, but I wonder if there's any tricky thing that can be done in this case.
Now you can try with $toString aggregation which simply
converts ObjectId to string
db.collection.aggregate([
{ "$addFields": {
"userRef": { "$toString": "$userRef" }
}},
{ "$group": {
"_id": { "$concat": ["$userRef", "-", "$serialNumber"] }
}}
])
You can check the output here
I couldn't find a way to do what I wanted, so instead, I created a MapReduce function that, in the end, generated the keys the way I wanted to (concatenating other keys).
At the end, it looked something like this:
db.collection('myCollection').mapReduce(
function() {
emit(
this.userRef.str + '-' + this.serialNumber , {
count: 1,
whateverValue1:this.value1,
whateverValue2:this.value2,
...
}
)
},
function(key, values) {
var reduce = {}
.... my reduce function....
return reduce
}, {
query: {
...filters_here....
},
out: 'name_of_output_collection'
}
);
You can simply use $toString to apply $concat in aggregation on ObjectIDs in the following way -
$group: {
'_id': {
'$concat': [
{ '$toString' : '$userRef' },
'-',
{ '$toString' : '$serialNumber'}
]
},
}
I think you may try to resolve it by using an Array which contains both fields:
{$project:{newkey:['$userRef','$serialNumber']},{$match:{newkey:{$in:filterArray}}}}
this may match the data with both fields to the filter. Please notice that the data in the newkey array should have the same data type with the filterArray elements.
You can use $substr https://docs.mongodb.com/manual/reference/operator/aggregation/substr/#exp._S_substr to cast any object to string before $concat.
This is a sample of code that's working for me.
group_id_i['_id'] = {
'$concat' => [
{ '$substr' => [ {'$year' => '$t'}, 0, -1] }, '-',
{ '$substr' => [ {'$month' => '$t'}, 0, -1] }, '-',
{ '$substr' => [ {'$dayOfMonth' => '$t'}, 0, -1] }
]
}
Where t is DateTime field, this aggregation returns data like so.
{
"_id" => "28-9-2016",
"i" => 2
}
I'm trying to use Doctrine ODM to interact with my mongo database, and I'm trying to use this mongo query that runs without a problem in MongoDB CLI:
db.products.aggregate(
[
{
$match: {highEndEmployees: {$lt: 2001 }, lowEndEmployees: {$gt: 1400} }
},
{
$project: {
lowEndFinalPrice: {
$add: [
{ $multiply: ["$priceMultiplierUser", "$lowEndUsers"] },
{ $multiply: ["$priceMultiplierEmployee", "$lowEndEmployees"] },
{ $multiply: ["$priceMultiplierJobOpenings", "$lowEndJobOpenings"] },
"$priceBase"
]
},
pricePerUser: {$multiply: ["$priceMultiplierUser", "$lowEndUsers"]},
pricePerEmployee: {$multiply: ["$priceMultiplierEmployee", "$lowEndEmployees"]},
pricePerJobOpening: {$multiply: ["$priceMultiplierJobOpenings", "$lowEndJobOpenings"]},
}
},
{
$sort: { lowEndFinalPrice: 1 }
}
]);
In Doctrine ODM, that query would translate to:
array(
"aggregate" => "products",
"pipeline" => array(
array('$match' =>
array(
"highEndEmployees" => array('$lt' => 2001),
"lowEndEmployees" => array('$gt'=> 1400)
)
),
array('$project' => array(
'lowEndFinalPrice' => array(
'$add' => '$priceBase',
'$add' => array(
'$multiply' => array('$priceMultiplierUser', '$lowEndUsers')
)
)
))
)
)
);
But if I try to run that query, I get the following error:
exception: the $add operator does not accept an object as an operand
However, if I remove this line:
'$add' => array(
'$multiply' => array('$priceMultiplierUser', '$lowEndUsers')
)
From the array, the query runs perfectly. So I'm using that that mongo complains when I'm trying to nest another array within the $add key.
What is the right way to write the query above in PHP+Doctrine ODM?
The answer was pretty straight forward. The problem was in the way I was building the BSON in PHP to send it over to MongoDB.
For example, the following part:
array(
'$add' => '$priceBase',
'$add' => array(
'$multiply' => array('$priceMultiplierUser', '$lowEndUsers')
)
the second '$add' overrides the first one, thus not only creating an invalid formula, but also creating the wrong BSON.
It should be like this:
array(
'$add' => array(
'$basePrice',
array(
'$multiply' => array('$priceMultiplierUser', '$lowEndUsers')
)
)
So that, when using json_encode on that array, I end up with something like this:
$add: [
"$priceBase",
{ $multiply: ["$priceMultiplierJobOpenings", "$lowEndJobOpenings"] }
]
Which is a valid BSON aggregation in MongoDB
I'm trying to figure out how to nest an AND and OR operation with MongoID, like so (taken from something that I used to use with MongoMapper):
{
:$and=> [
{
:$or => [
{"name"=> "joe"}, {"name" => "randy" }
{
:$or=> [
{"something" => "else" }, {"another" => "thing" }
]
}
]
}
I'm not too terribly familiar with the way union and intersection works, but the kicker is that each child within the AND is optional/not guaranteed. In other words, each query within AND is programmatic, there could be 2 items to check against, 1 item, etc.
I thought about doing something like this:
Model.or({ :name => "...." }).union.or( :something => "...." })
But, the only problem with that is I'm not sure on the best practice of constructing the query based on user input. I have a sinatra-based application that acts as an API point for my users that is connecting to my MongoID models, and I'd like for users to be able to construct queries like this (maybe not this complicated) over the API.
I'm migrating over to MongoID from MongoMapper for various reasons, but with MongoMapper these queries were a little simpler because everything, such as nested and and or operators, are supported within a where method.
Turns out that MongoID (well, more specifically Origin::Query) supports Mongo selector syntax within many of their DSL functions, like so:
Model.where( { "name" => { "$or" => [ "betsy", "charles" ] } )
So gathering from my above example, you can just do this:
Model.all_of( [
{
"$or" => [
{"name"=> "joe"}, {"name" => "randy" }
},
{
"$or" => [
{"something" => "else" }, {"another" => "thing" }
]
}
]