Json Structure:
"_id" : ObjectId("55d6cb28725f3019a5241781"),
"Number" : {
"value" : "1234567",
},
"DeviceID" : {
"value" : "01",
}
"type" : {
"value" : "ce06"}
Now i want to find only those keys document which start from /dev/.
i tried this script:
var cur = db.LIVEDATA.find({"ProductIMEIno.value":"359983007488004"});
cur.forEach(function(doc){
var keynames = Object.keys(doc);
print('the length is '+keynames.length);
for(var i=0;i<keynames.length;i++){
if(keynames[i].match(/Dev/)){
print("the name is "+keynames); }}} )
but this is not working properly.
Desired Output;
Only this document should show on the basis of key name search.
"DeviceID" : {
"value" : "01",
MongoDB isn't designed to find keys dynamically like this; it's much easier to use it to find values dynamically, so you could restructure your data structure to allow this:
"_id" : ObjectId("55d6cb28725f3019a5241781"),
"data" : [
{
"key" : "Number",
"value" : "1234567",
},
{
"key": "DeviceID",
"value" : "01",
},
{
"key" : "type",
"value" : "ce06"
}
]
Then you will be able to query it like this:
db.LIVEDATA.aggregate([
{$match: {"ProductIMEIno.value":"359983007488004"}},
{$unwind: "$data"},
{$match: {"data.key" : /^dev/i }}
]);
That will return data structured like this:
{
"_id" : ObjectId("55d6cb28725f3019a5241781"),
"data" : {
"key" : "DeviceID",
"value" : "01"
}
}
Suppose you have a data collection like this:
[
{
"Number": {
"value": "1234567"
},
"DeviceID": {
"value": "01"
},
"DeviceID2": {
"value": "01",
"name": "abc123"
},
"type": {
"value": "ce06"
}
},
{
"Number": {
"value": "1234568"
},
"DeviceID": {
"value": "02"
},
"type": {
"value": "ce07"
}
}
]
You can use following aggregation:
db.collection.aggregate([
{
"$match": {}
},
{
"$addFields": {
"root_key_value_list": {
"$objectToArray": "$$ROOT"
}
}
},
{
"$unwind": "$root_key_value_list"
},
{
"$match": {
"root_key_value_list.k": {
"$regex": "^Dev"
}
}
},
{
"$group": {
"_id": "$_id",
"root_key_value_list": {
"$push": "$root_key_value_list"
}
}
},
{
"$project": {
"root": {
"$arrayToObject": "$root_key_value_list"
}
}
},
{
"$replaceRoot": {
"newRoot": "$root"
}
}
])
the result will be:
[
{
"DeviceID": {
"value": "01"
},
"DeviceID2": {
"name": "abc123",
"value": "01"
}
},
{
"DeviceID": {
"value": "02"
}
}
]
playground:
https://mongoplayground.net/p/z5EeHALCqzy
I have some documents like this:
{
"hash": "14a076f9f6cecfc58339330eeb492e267f63062f6d5f669c7cdbfecf9eb4de32",
"started_services": [],
"deleted_files": [],
"software": {
"adobe" : {
"licenses" : [
{ "key": "2384723",
"date": "26-10-2012"
},
{ "key": "23888823",
"date": "09-11-2012"
}
]
}
}
}
How do I retrieve just the hash value and the list of "key" values?
I did the following, but, as you see, the result has the entire path which I do not want.
> db.repository.find({"$and": [{"datetime_int": {"$gte": 1451952000}},{"software.adobe.licenses.key" : { $exists : true}}]}, {hash:1, "software.adobe.licenses.key":1, _id:0}).limit(10)
{ "hash" : "a1532e0609aaf6acfa9e505e93af0bee0856a9a67398aeaa72aa6eb2fffd134e", "software" : { "adobe" : { "licenses" : [ { "key" : "2008350" }, { "key" : "2018350" }, { "key" : "2028350" }, { "key" : "2038350" }, { "key" : "2048350" }, { "key" : "2058350" }, { "key" : "2068350" }, { "key" : "2078350" }...]}}}
The result I want should look like this:
{"hash": "a1532e0609aaf6acfa9e505e93af0bee0856a9a67398aeaa72aa6eb2fffd134e",
"key": ["2008350", "2018350", "2028350", "2038350", "2048350", "2058350", "2068350", "2078350"]
}
How do I do that?
You can do this with the aggregation framework.
db.repository.aggregate([
{ "$match": {
"datetime_int": { "$gte": 1451952000 },
"software.adobe.licenses.key" : { "$exists" : true }
}},
{ "$project": {
"hash": 1,
"key": {
"$map": {
"input": "$software.adobe.licenses",
"as": "soft",
"in": "$$soft.key"
}
}
}}
])
Starting From MongoDB 3.2 you can directly project the sub-document array field.
{ "$project": { "hash": 1, "key": "$software.adobe.licenses.key"}}
db.key.aggregate((
{ "$match": {
"datetime_int": { "$gte": 1451952000 }
}},
{"$unwind":"$software.adobe.licenses"},
{"$project":{"key":"$software.adobe.licenses.key", "hash":1, "_id":0}}
))
outputs the following :
{ "hash" : "14a076f9f6cecfc58339330eeb492e267f63062f6d5f669c7cdbfecf9eb4de32", "key" : [ "2384723", "23888823" ] }
I wrote a server for mobile app analytics in which I have a sharded(!Upd) collection with events as follows:
{
"event": "install",
"userId": "a",
"time": 2014-02-09,
"data" : ...
},
{
"event": "login",
"userId": "a",
"time": 2014-02-12,
"data" : ...
},
{
"event": "install",
"userId": "b",
"time": 2014-4-29,
"data" : ...
},
{
"event": "login",
"userId": "b",
"time": 2014-4-30,
"data" : ...
}
...
I need to select users, who have the event install and not login in next day after the install event (in other words, I want to select Users who install the app, but do not login in next day). So the output for above data should be:
{
"userId": "a",
"data" : ...
}
How do carry out this task with aggregation framework or mapreduce? Or maybe another solution?
This is a bit tricky one :-)
You can do it with aggregation, if time is just a date field (without time data),
then
having collection
{
"_id" : ObjectId("57694365ef9176ec54960a66"),
"event" : "install",
"userId" : "a",
"time" : ISODate("2014-09-02T00:00:00.000Z")
},{
"_id" : ObjectId("57694365ef9176ec54960a67"),
"event" : "login",
"userId" : "a",
"time" : ISODate("2014-12-02T00:00:00.000Z")
},{
"_id" : ObjectId("57694365ef9176ec54960a68"),
"event" : "install",
"userId" : "b",
"time" : ISODate("2014-04-29T00:00:00.000Z")
},{
"_id" : ObjectId("57694365ef9176ec54960a69"),
"event" : "login",
"userId" : "b",
"time" : ISODate("2014-04-30T00:00:00.000Z")
}
we can use aggregate query:
var match = {
$match : {
"event" : "install"
}
};
var projectNextDayDate = {
$project : {
_id : 1,
event : 1,
userId : 1,
time : 1,
nextDay : {
$add : ["$time", 24 * 60 * 60 * 1000]
}
}
}
var lookup = {
$lookup : {
from : "zella",
localField : "nextDay",
foreignField : "time",
as : "mergedDocs"
}
}
var nowMatchUsers = {
$project : {
_id : 1,
event : 1,
userId : 1,
time : 1,
nextDay : 1,
mergedDocs : {
$filter : {
input : "mergedDocs",
as : "m",
cond : {
$eq : ["$$m.userId", "$userId"]
}
}
}
}
}
var findEmptyArrays = {
$match : {
mergedDocs : []
}
}
db.zella.aggregate([match, projectNextDayDate, lookup, findEmptyArrays])
with this output:
{
"_id" : ObjectId("57694365ef9176ec54960a66"),
"event" : "install",
"userId" : "a",
"time" : ISODate("2014-09-02T00:00:00.000Z"),
"nextDay" : ISODate("2014-09-03T00:00:00.000Z"),
"mergedDocs" : []
}
The assumption here is that time is a date 2014-09-02T00:00:00.000
the other way to merge collection is to use user ID, as a $lookup point, but then there will be more logic to filter result set and performance could be dropped.
You could try running the following aggregation pipeline:
db.test.aggregate([
{
"$project": {
"event": 1,
"userId": 1,
"time": 1,
"data": 1,
"dayAfter": {
"$add": [ "$time", 24 * 60 * 60 * 1000 ]
}
}
},
{ "$match": { "event": { "$in": ["install", "login"] } } },
{
"$group": {
"_id": "$userId",
"eventsTimeLine": {
"$push": {
"event": "$event",
"time": "$time",
"dayAfter": "$dayAfter"
}
},
"data": { "$push": "$data" }
}
},
{ "$unwind": "$eventsTimeLine" },
{ "$sort": { "eventsTimeLine.event": 1 } },
{
"$group": {
"_id": "$_id",
"dayAfterInstall": { "$first": "$eventsTimeLine.dayAfter" },
"loginTime": { "$last": "$eventsTimeLine.time" },
"data": { "$first": "$data" }
}
},
{
"$project": {
"isChurn": { "$ne": [ "$loginTime", "$dayAfterInstall" ] },
"userId": "$_id", "data": 1, "_id": 0
}
},
{ "$match" : { "isChurn" : true } }
])
Here another solution with mapreduce and aggregation:
var mapFunction = function() {
if (this.event != 'install' && this.event != 'login'){
return;
}
var value = {data: this.data, count: 1};
if (this.event == 'install'){
var nextDay = new Date(this.date.getTime() + 24 * 60 * 60 * 1000)
emit({userId:this.userId, nextDayAfterInstall:nextDay}, value );
} else
if (this.event == 'login'){
emit({userId:this.userId, nextDayAfterInstall:this.date}, value );
}
};
var reduceFunction = function(event, values) {
var value = { data: null, count: 1 };
for (var index = 0; index < values.length; ++index) {
value.count += values[index].count;
value.data = values[index].data;
}
return value ;
};
db.events.mapReduce(
mapFunction,
reduceFunction,
{ out: "case1_mr_out" }
)
var groupByUserId = {
$group :
{
_id : { userId: "$_id.userId" },
data : { $last: '$value.data' },
count : { $max: '$value.count' }
}
}
var filterWhereOnlyOne = {
$match : {
"count" : 1
}
};
db.case1_mr_out.aggregate([groupByUserId,filterWhereOnlyOne])
I have two aggregate operations that I'd like to combine. The first operation returns, for example:
{ "_id" : "Colors", "count" : 12 }
{ "_id" : "Animals", "count" : 6 }
and the second operation returns, for example:
{ "_id" : "Red", "count" : 10 }
{ "_id" : "Blue", "count" : 9 }
{ "_id" : "Green", "count" : 9 }
{ "_id" : "White", "count" : 7 }
{ "_id" : "Yellow", "count" : 7 }
{ "_id" : "Orange", "count" : 7 }
{ "_id" : "Black", "count" : 5 }
{ "_id" : "Goose", "count" : 4 }
{ "_id" : "Chicken", "count" : 3 }
{ "_id" : "Grey", "count" : 3 }
{ "_id" : "Cat", "count" : 3 }
{ "_id" : "Rabbit", "count" : 3 }
{ "_id" : "Duck", "count" : 3 }
{ "_id" : "Turkey", "count" : 2 }
{ "_id" : "Elephant", "count" : 2 }
{ "_id" : "Shark", "count" : 2 }
{ "_id" : "Fish", "count" : 2 }
{ "_id" : "Tiger", "count" : 2 }
{ "_id" : "Purple", "count" : 1 }
{ "_id" : "Pink", "count" : 1 }
How do I combine the 2 operations to achieve the following?
{ "_id" : "Colors", "count" : 12, "items" :
[
{ "_id" : "Red", "count" : 10 },
{ "_id" : "Blue", "count" : 9 },
{ "_id" : "Green", "count" : 9 },
{ "_id" : "White", "count" : 7 },
{ "_id" : "Yellow", "count" : 7 },
{ "_id" : "Orange", "count" : 7 },
{ "_id" : "Black", "count" : 5 },
{ "_id" : "Grey", "count" : 3 },
{ "_id" : "Purple", "count" : 1 },
{ "_id" : "Pink", "count" : 1 }
]
},
{ "_id" : "Animals", "count" : 6, "items" :
[
{ "_id" : "Goose", "count" : 4 },
{ "_id" : "Chicken", "count" : 3 },
{ "_id" : "Cat", "count" : 3 },
{ "_id" : "Rabbit", "count" : 3 },
{ "_id" : "Duck", "count" : 3 },
{ "_id" : "Turkey", "count" : 2 },
{ "_id" : "Elephant", "count" : 2 },
{ "_id" : "Shark", "count" : 2 },
{ "_id" : "Fish", "count" : 2 },
{ "_id" : "Tiger", "count" : 2 }
]
}
Schema
var ListSchema = new Schema({
created: {
type: Date,
default: Date.now
},
title: {
type: String,
default: '',
trim: true,
required: 'Title cannot be blank'
},
items: {
type: Array,
default: [String],
trim: true
},
creator: {
type: Schema.ObjectId,
ref: 'User'
}
});
Operation 1
db.lists.aggregate(
[
{ $group: { _id: "$title", count: { $sum: 1 } } },
{ $sort: { count: -1 } }
]
)
Operation 2
db.lists.aggregate(
[
{ $unwind: "$items" },
{ $group: { _id: "$items", count: { $sum: 1 } } },
{ $sort: { count: -1 } }
]
)
This really depends on the kind of results you are after in a respone. The things you are asking about seem to indicate that you are looking for "facet counts" in a result, but I'll touch on that a bit later.
For as basic result, there is nothing wrong with this as an approach:
Thing.aggregate(
[
{ "$group": {
"_id": {
"type": "$type", "name": "$name"
},
"count": { "$sum": 1 }
}},
{ "$group": {
"_id": "$_id.type",
"count": { "$sum": "$count" },
"names": {
"$push": { "name": "$_id.name", "count": "$count" }
}
}}
],
function(err,results) {
console.log(JSON.stringify(results, undefined, 2));
callback(err);
}
)
Which should give you a result like this:
[
{
"_id": "colours",
"count": 50102,
"names": [
{ "name": "Green", "count": 9906 },
{ "name": "Yellow", "count": 10093 },
{ "name": "Red", "count": 10083 },
{ "name": "Orange", "count": 9997 },
{ "name": "Blue", "count": 10023 }
]
},
{
"_id": "animals",
"count": 49898,
"names": [
{ "name": "Tiger", "count": 9710 },
{ "name": "Lion", "count": 10058 },
{ "name": "Elephant", "count": 10069 },
{ "name": "Monkey", "count": 9963 },
{ "name": "Bear", "count": 10098 }
]
}
]
Where the very basic approach here is to simply $group in two stages, where the first stage aggregates on the combination of keys down to the lowest ( most granular ) grouping level, and then process a $group again to basically "add up" the totals on the highest ( least granular ) grouping level, also thus adding the lower results to an array of items.
But this is not "separated" as it would be in "facet counts", so to do this becomes a little more complex, as well as a little more insane. But first the example:
Thing.aggregate(
[
{ "$group": {
"_id": {
"type": "$type",
"name": "$name"
},
"count": { "$sum": 1 }
}},
{ "$group": {
"_id": "$_id.type",
"count": { "$sum": "$count" },
"names": {
"$push": { "name": "$_id.name", "count": "$count" }
}
}},
{ "$group": {
"_id": null,
"types": {
"$push": {
"type": "$_id", "count": "$count"
}
},
"names": { "$push": "$names" }
}},
{ "$unwind": "$names" },
{ "$unwind": "$names" },
{ "$group": {
"_id": "$types",
"names": { "$push": "$names" }
}},
{ "$project": {
"_id": 0,
"facets": {
"types": "$_id",
"names": "$names",
},
"data": { "$literal": [] }
}}
],
function(err,results) {
console.log(JSON.stringify(results[0], undefined, 2));
callback(err);
}
);
Which will produce output like this:
{
"facets": {
"types": [
{ "type": "colours", "count": 50102 },
{ "type": "animals", "count": 49898 }
],
"names": [
{ "name": "Green", "count": 9906 },
{ "name": "Yellow", "count": 10093 },
{ "name": "Red", "count": 10083 },
{ "name": "Orange", "count": 9997 },
{ "name": "Blue", "count": 10023 },
{ "name": "Tiger", "count": 9710 },
{ "name": "Lion", "count": 10058 },
{ "name": "Elephant", "count": 10069 },
{ "name": "Monkey", "count": 9963 },
{ "name": "Bear", "count": 10098 }
]
},
"data": []
}
What should be apparent though is while "possible", the kind of "juggling" going on here in the pipeline to produce this output format is not really efficient. Compared to the first example, there is a lot of overhead in here just to simply split out the results into their own array responses and independently of the grouping keys. This notably becomes more complex with the more "facets" to generate.
Also as hinted at here in the output, what people generally ask of "facet counts" is that that the result "data" is also included in the response ( likely paged ) in addition to the aggregated facets. So the further complications should be apparent right here:
{ "$group": {
"_id": null,
(...)
Where the requirement of this type of operation is to basically "stuff" every piece of data into a single object. In most cases, and certainly where you want the actual data in results ( using 100,000 in this sample ) it becomes completely impractical to follow this approach and will almost certainly exceed the BSON document limit size of 16MB.
In such a case, where you want to produce results and the "facets" of that data in a response, then the best approach here is to run each aggregation and the output page as separate query operations and "stream" the output JSON ( or other format ) back to the receiving client.
As a self contained example:
var async = require('async'),
mongoose = require('mongoose'),
Schema = mongoose.Schema;
mongoose.connect('mongodb://localhost/things');
var data = {
"colours": [
"Red","Blue","Green","Yellow","Orange"
],
"animals": [
"Lion","Tiger","Bear","Elephant","Monkey"
]
},
dataKeys = Object.keys(data);
var thingSchema = new Schema({
"name": String,
"type": String
});
var Thing = mongoose.model( 'Thing', thingSchema );
var writer = process.stdout;
mongoose.connection.on("open",function(err) {
if (err) throw err;
async.series(
[
function(callback) {
process.stderr.write("removing\n");
Thing.remove({},callback);
},
function(callback) {
process.stderr.write("inserting\n");
var bulk = Thing.collection.initializeUnorderedBulkOp(),
count = 0;
async.whilst(
function() { return count < 100000; },
function(callback) {
var keyLen = dataKeys.length,
keyIndex = Math.floor(Math.random(keyLen)*keyLen),
type = dataKeys[keyIndex],
types = data[type],
typeLen = types.length,
nameIndex = Math.floor(Math.random(typeLen)*typeLen),
name = types[nameIndex];
var obj = { "type": type, "name": name };
bulk.insert(obj);
count++;
if ( count % 1000 == 0 ) {
process.stderr.write('insert count: ' + count + "\n");
bulk.execute(function(err,resp) {
bulk = Thing.collection.initializeUnorderedBulkOp();
callback(err);
});
} else {
callback();
}
},
callback
);
},
function(callback) {
writer.write("{ \n \"page\": 1,\n \"pageSize\": 25,\n")
writer.write(" \"facets\": {\n"); // open object response
var stream = Thing.collection.aggregate(
[
{ "$group": {
"_id": "$name",
"count": { "$sum": 1 }
}}
],
{
"cursor": {
"batchSize": 1000
}
}
);
var counter = 0;
stream.on("data",function(data) {
stream.pause();
if ( counter == 0 ) {
writer.write(" \"names\": [\n");
} else {
writer.write(",\n");
}
data = { "name": data._id, "count": data.count };
writer.write(" " + JSON.stringify(data));
counter++;
stream.resume();
});
stream.on("end",function() {
writer.write("\n ],\n");
var stream = Thing.collection.aggregate(
[
{ "$group": {
"_id": "$type",
"count": { "$sum": 1 }
}}
],
{
"cursor": {
"batchSize": 1000
}
}
);
var counter = 0;
stream.on("data",function(data) {
stream.pause();
if ( counter == 0 ) {
writer.write(" \"types\": [\n");
} else {
writer.write(",\n");
}
data = { "name": data._id, "count": data.count };
writer.write(" " + JSON.stringify(data));
counter++;
stream.resume();
});
stream.on("end",function() {
writer.write("\n ]\n },\n");
var stream = Thing.find({}).limit(25).stream();
var counter = 0;
stream.on("data",function(data) {
stream.pause();
if ( counter == 0 ) {
writer.write(" \"data\": [\n");
} else {
writer.write(",\n");
}
writer.write(" " + JSON.stringify(data));
counter++;
stream.resume();
});
stream.on("end",function() {
writer.write("\n ]\n}\n");
callback();
});
});
});
}
],
function(err) {
if (err) throw err;
process.exit();
}
);
});
With the output like:
{
"page": 1,
"pageSize": 25,
"facets": {
"names": [
{"name":"Red","count":10007},
{"name":"Tiger","count":10012},
{"name":"Yellow","count":10119},
{"name":"Monkey","count":9970},
{"name":"Elephant","count":10046},
{"name":"Bear","count":10082},
{"name":"Orange","count":9982},
{"name":"Green","count":10005},
{"name":"Blue","count":9884},
{"name":"Lion","count":9893}
],
"types": [
{"name":"colours","count":49997},
{"name":"animals","count":50003}
]
},
"data": [
{"_id":"55bf141f3edc150b6abdcc02","type":"animals","name":"Lion"},
{"_id":"55bf141f3edc150b6abdc81b","type":"colours","name":"Blue"},
{"_id":"55bf141f3edc150b6abdc81c","type":"colours","name":"Orange"},
{"_id":"55bf141f3edc150b6abdc81d","type":"animals","name":"Bear"},
{"_id":"55bf141f3edc150b6abdc81e","type":"animals","name":"Elephant"},
{"_id":"55bf141f3edc150b6abdc81f","type":"colours","name":"Orange"},
{"_id":"55bf141f3edc150b6abdc820","type":"colours","name":"Green"},
{"_id":"55bf141f3edc150b6abdc821","type":"animals","name":"Lion"},
{"_id":"55bf141f3edc150b6abdc822","type":"animals","name":"Monkey"},
{"_id":"55bf141f3edc150b6abdc823","type":"colours","name":"Yellow"},
{"_id":"55bf141f3edc150b6abdc824","type":"colours","name":"Yellow"},
{"_id":"55bf141f3edc150b6abdc825","type":"colours","name":"Orange"},
{"_id":"55bf141f3edc150b6abdc826","type":"animals","name":"Monkey"},
{"_id":"55bf141f3edc150b6abdc827","type":"colours","name":"Blue"},
{"_id":"55bf141f3edc150b6abdc828","type":"animals","name":"Tiger"},
{"_id":"55bf141f3edc150b6abdc829","type":"colours","name":"Red"},
{"_id":"55bf141f3edc150b6abdc82a","type":"animals","name":"Monkey"},
{"_id":"55bf141f3edc150b6abdc82b","type":"animals","name":"Elephant"},
{"_id":"55bf141f3edc150b6abdc82c","type":"animals","name":"Tiger"},
{"_id":"55bf141f3edc150b6abdc82d","type":"animals","name":"Bear"},
{"_id":"55bf141f3edc150b6abdc82e","type":"colours","name":"Yellow"},
{"_id":"55bf141f3edc150b6abdc82f","type":"animals","name":"Lion"},
{"_id":"55bf141f3edc150b6abdc830","type":"animals","name":"Elephant"},
{"_id":"55bf141f3edc150b6abdc831","type":"colours","name":"Orange"},
{"_id":"55bf141f3edc150b6abdc832","type":"animals","name":"Elephant"}
]
}
There are some considerations in here, notably that mongoose .aggregate() does not really directly support the standard node stream interface. There is an .each() method available from .cursor() on an aggregate method, but the "stream" implied from the core API method gives a lot more control here, so the .collection mehod here to get the underlying driver object is preferable. Hopefully a future mongoose release will consider this.
So if your end goal is such a "facet count" alongside the results as demonstrated here, then each aggregation and results make the most sense to "stream" in the way as demonstrated. Without that, the aggregation becomes both overcomplicated as well as very likely to exceed the BSON limit, just as doing otherwise in this case would.