How can I project one BsonDocument to new instance without querying a collection?
inputs: document: BsonDocument, fields: new string[] { "_id", "meta.name", "type" }
output: BsonDocument with only the above elements populated
Itch scratched
input
{
"_id" : ObjectId("58b454f40960a1788ef48ebc"),
"schema" : {
"id" : "asset",
"version" : {
"major" : 1,
"minor" : 0
}
},
"type" : "asset",
"meta" : {
"name" : "Most Amazing Product",
"type" : null,
"legacy" : {
"url" : "https://s3.amazonaws.com/bucket_name/guid"
}
},
"content" : {
"is_s3" : true,
"s3_bucket" : "bucket_name",
"s3_key" : "guid.doc",
"url" : "https://s3.amazonaws.com/guid.doc"
},
"modified-date" : ISODate("2017-08-09T15:25:57.972Z"),
"modified-by" : "api"
}
code
nuget: MongoDB.Driver 2.4.4
using MongoDB.Bson;
using MongoDB.Bson.Serialization;
using MongoDB.Bson.IO;
BsonDocument original = BsonDocument.Parse(#"{ ""_id"" : ObjectId(""58b454f40960a1788ef48ebc""), ""schema"" : { ""id"" : ""asset"", ""version"" : { ""major"" : 1, ""minor"" : 0 } }, ""type"" : ""asset"", ""meta"" : { ""name"" : ""Most Amazing Product"", ""type"" : null, ""legacy"" : { ""url"" : ""https://s3.amazonaws.com/bucket_name/guid"" } }, ""content"" : { ""is_s3"" : true, ""s3_bucket"" : ""bucket_name"", ""s3_key"" : ""guid.doc"", ""url"" : ""https://s3.amazonaws.com/guid.doc"" }, ""modified-date"" : ISODate(""2017-08-09T15:25:57.972Z""), ""modified-by"" : ""api"" }");
string[] fields = new[] { "_id", "meta.name", "type" };
BsonDocument projection = new BsonDocument();
foreach (var fieldName in fields)
{
BsonDocument source = original;
BsonDocument target = projection;
string[] parts = fieldName.Split(new[] { "." }, StringSplitOptions.RemoveEmptyEntries);
for (int i = 0; i < parts.Length; i++)
{
string currentName = parts[i];
if (i == parts.Length - 1)
{
if(source.Contains(currentName))
target[currentName] = source[currentName];
}
else
{
// Does the source have a current property at this level
if (source.Contains(currentName))
{
// first time this has been visited on target
if (target.Contains(currentName) == false)
{
target.Add(currentName, new BsonDocument());
}
source = source[currentName] as BsonDocument;
target = target[currentName] as BsonDocument;
}
else
{
// no need to go any further if the source doesn't have the property specified
break;
}
}
}
}
result
{
"_id" : ObjectId("58b454f40960a1788ef48ebc"),
"meta" : {
"name" : "Most Amazing Product"
},
"type" : "asset"
}
Related
I want to find one document and clone/copy that document and create 100 new documents with new value for few fields using shell script in mongodb.
Below is my document
{
"_id" : ObjectId("5ef59bde562c9824176e9f20"),
"productDefinition" : {
"product" : {
"companies" : {
"company" : {
"productionformation" : {
"productNumber" : "E128",
"venderNumber" : "0470",
"venderName" : "ALPHA SERVICES LLC"
}
}
}
}
},
"executionId" : "123456"
}
After executing the shell script, i want to have new 100 collection with new values for the below fields
"executionId" : "NewExecutionId" // This value will be Fixed for all new 100 documents
"productNumber" : "1" //This value will be increasing.. for first document 1, for second document 2, etc..
"venderNumber" : "1" //This value will be increasing.. for first document 1, for second document 2, etc..
My new collection will be looking like this.
First new document
{
"_id" : ObjectId("5ef59bde562c9824176e9f20"),
"productDefinition" : {
"product" : {
"companies" : {
"company" : {
"productionformation" : {
"productNumber" : "1",
"venderNumber" : "1",
"venderName" : "ALPHA SERVICES LLC"
}
}
}
}
},
"executionId" : "newExecutionId"
}
Second new document
{
"_id" : ObjectId("5ef59bde562c9824176e9f20"),
"productDefinition" : {
"product" : {
"companies" : {
"company" : {
"productionformation" : {
"productNumber" : "2",
"venderNumber" : "2",
"venderName" : "ALPHA SERVICES LLC"
}
}
}
}
},
"executionId" : "newExecutionId"
}
Third new document
{
"_id" : ObjectId("5ef59bde562c9824176e9f20"),
"productDefinition" : {
"product" : {
"companies" : {
"company" : {
"productionformation" : {
"productNumber" : "3",
"venderNumber" : "3",
"venderName" : "ALPHA SERVICES LLC"
}
}
}
}
},
"executionId" : "newExecutionId"
}
Like this fourth document , fifth document, etc... till 100th document...
I tried with this script. but its not working.
copy = db.myCollection.find({"executionId" : "123456",
"productDefinition.product.companies.company.productionformation.productNumber" : "E128" ,
"productDefinition.product.companies.company.productionformation.venderNumber" :"0470" })
for (var i = 1; i< 101; i++){
copy.executionId = "newExecutionId";
copy.productDefinition.product.companies.company.productionformation.productNumber = i;
copy.productDefinition.product.companies.company.productionformation.venderNumber" = i;
db.myCollection.insert(copy);
}
You will be needing to fix following things:
Use findOne instead of find as it will return single matching document.
Use let (instead of var) while running the loop because there are asynchronous DB operations in loop body.
Similarly, create a Deep copy of matchedDoc result / (copy variable) inside for loop body, to avoid updating same object's reference value.
Hope it helps !
My schema structure looks something like this:
{
"__v":0,
"_id":ObjectId("5708423e897db8255aaaa9dd"),
"podId":169400000005,
"env":[
{
"type":"1",
"id":3852000000035,
"_id":ObjectId("5708423e897db8255aaaa9de")
},
{
"type":"2",
"id":3852000000040,
"_id":ObjectId("5708423e897db8255aaaa9df")
}
],
"name":"Test Build",
"parameters":[
{
"name":"sound.left",
"type":"1",
"paramName":"Left Sound Control",
"paramType":"booleanParameter",
"testValue":null,
"liveValue":null,
"_id":ObjectId("5708423f897db8255aaaaa0d")
},
{
"name":"sound.right",
"type":"1",
"paramName":"Right Sound Control",
"paramType":"booleanParameter",
"testValue":null,
"liveValue":null,
"_id":ObjectId("5708423f897db8255aaaaa0d")
},
...
]
}
I have 3 known variables: podId for podId, codeName for parameters.name, envType for parameters.type. Using them I want to update the object using podId, codeName, and envType, with a new variable paramValue that will contain the testValue value.
What I've tried
db.pods.update({
podId: podId,
parameters: {
$elemMatch: {
name: codeName,
type: envType
}
}
}, {
$set: {
'parameters.$.testValue': paramValue
}
});
I'm trying to update the testValue where podId == podId, parameters.name == codeName, and parameters.type == envType, but the above did not update anything.
I also tried
db.pods.update({
podId: podId,
parameters: {
name: codeName,
type: envType
}
}, {
$push: {
parameters: {
testValue: paramValue
}
}
},
function(err) {
if(err) throw err;
});
basically taking what worked to build the object when I only had to compare the podId, and adding the extra criteria; it didn't work this time.
edit: fixed schema type Number to String
Let these are variables that you have in mongo shell:
> var podId = 169400000005
> var codeName = "sound.right"
> var envType = "1"
> var paramValue = "updatedParamValue"
This query you can try:
db.test.update({"podId":podId, "parameters.name" : codeName, "parameters.type" : envType}, {$set : {"parameters.0.testValue" : paramValue,"parameters.1.testValue" : paramValue}})
Depending upon which testValue you want to update, you can pass index to that.
Update
Following result i get after the update query:
db.test.find().pretty()
{
"_id" : ObjectId("570c850ab9477f18ac5297f9"),
"__v" : 0,
"podId" : 169400000005,
"env" : [
{
"type" : "1",
"id" : 3852000000035,
"_id" : ObjectId("5708423e897db8255aaaa9de")
},
{
"type" : "2",
"id" : 3852000000040,
"_id" : ObjectId("5708423e897db8255aaaa9df")
}
],
"name" : "Test Build",
"parameters" : [
{
"name" : "sound.left",
"type" : "1",
"paramName" : "Left Sound Control",
"paramType" : "booleanParameter",
"testValue" : "updatedParamValue",
"liveValue" : null,
"_id" : ObjectId("5708423f897db8255aaaaa0d")
},
{
"name" : "sound.right",
"type" : "1",
"paramName" : "Right Sound Control",
"paramType" : "booleanParameter",
"testValue" : "updatedParamValue",
"liveValue" : null,
"_id" : ObjectId("5708423f897db8255aaaaa0d")
}
]
}
Thanks.
We have a basic enquiry management tool that we're using to track some website enquiries in our administration suite, and we're using the ObjectId of each document in our enquiries collection to sort the enquiries by the date they were added.
{
"_id" : ObjectId("53a007db144ff47be1000003"),
"comments" : "This is a test enquiry. Please ignore. We'll delete it shortly.",
"customer" : {
"name" : "Test Enquiry",
"email" : "test#test.com",
"telephone" : "07890123456",
"mobile" : "07890123456",
"quote" : false,
"valuation" : false
},
"site" : [],
"test" : true,
"updates" : [
{
"_id" : ObjectId("53a007db144ff47be1000001"),
"status" : "New",
"status_id" : ObjectId("537de7c3a5e6e668ffc2335c"),
"status_index" : 100,
"substatus" : "New Web Enquiry",
"substatus_id" : ObjectId("5396bb9fa5e6e668ffc23388"),
"notes" : "New enquiry received from website.",
},
{
"_id" : ObjectId("53a80c977d299cfe91bacf81"),
"status" : "New",
"status_id" : ObjectId("537de7c3a5e6e668ffc2335c"),
"status_index" : 100,
"substatus" : "Attempted Contact",
"substatus_id" : ObjectId("53a80e06a5e6e668ffc2339e"),
"notes" : "In this test, we pretend that we've not managed to get hold of the customer on the first attempt.",
},
{
"_id" : ObjectId("53a80e539b966b8da5c40c36"),
"status" : "Approved",
"status_id" : ObjectId("52e77a49d85e95f00ebf6c72"),
"status_index" : 200,
"substatus" : "Enquiry Confirmed",
"substatus_id" : ObjectId("53901f1ba5e6e668ffc23372"),
"notes" : "In this test, we pretend that we've got hold of the customer after failing to contact them on the first attempt.",
}
]
}
Within each enquiry is an updates array of objects which also have an ObjectId as their main identity field. We're using an $unwind and $group aggregation to pull the first and latest updates, as well as the count of updates, making sure we only take enquiries where there have been more than one update (as one is automatically inserted when the enquiry is made):
db.enquiries.aggregate([
{
$match: {
"test": true
}
},
{
$unwind: "$updates"
},
{
$group: {
"_id": "$_id",
"latest_update_id": {
$last: "$updates._id"
},
"first_update_id": {
$first: "$updates._id"
},
"update_count": {
$sum: 1
}
}
},
{
$match: {
"update_count": {
$gt: 1
}
}
}
])
This results in the following output:
{
"result" : [
{
"_id" : ObjectId("53a295ad122ea80200000005"),
"latest_update_id" : ObjectId("53a80bdc7d299cfe91bacf7e"),
"first_update_id" : ObjectId("53a295ad122ea80200000003"),
"update_count" : 2
},
{
"_id" : ObjectId("53a007db144ff47be1000003"),
"latest_update_id" : ObjectId("53a80e539b966b8da5c40c36"),
"first_update_id" : ObjectId("53a007db144ff47be1000001"),
"update_count" : 3
}
],
"ok" : 1
}
This is then passed through to our code (node.js, in this case) where we perform a few operations on it and then present some information on our dashboard.
Ideally, I'd like to add another $group pipeline aggregation to the query which would subtract the timestamp of first_update_id from the timestamp of latest_update_id to give us a timespan, which we could then use $avg on.
Can anyone tell me if this is possible? (Thank you!)
As Neil already pointed out, you can't get to the timestamp from the ObjectId in the aggregation framework.
You said that speed is not important, so using MapReduce you can get what you want:
var map = function() {
if (this.updates.length > 1) {
var first = this.updates[0];
var last = this.updates[this.updates.length - 1];
var diff = last._id.getTimestamp() - first._id.getTimestamp();
var val = {
latest_update_id : last._id,
first_update_id : first._id,
update_count : this.updates.length,
diff: diff
}
emit(this._id, val);
}
};
var reduce = function() { };
db.runCommand(
{
mapReduce: "enquiries",
map: map,
reduce: reduce,
out: "mrresults",
query: { test : true}
}
);
This are the results:
{
"_id" : ObjectId("53a007db144ff47be1000003"),
"value" : {
"latest_update_id" : ObjectId("53a80e539b966b8da5c40c36"),
"first_update_id" : ObjectId("53a007db144ff47be1000001"),
"update_count" : 3,
"diff" : 525944000
}
}
Edit:
If you want to get the average diff for all documents you can do it like this:
var map = function() {
if (this.updates.length > 1) {
var first = this.updates[0];
var last = this.updates[this.updates.length - 1];
var diff = last._id.getTimestamp() - first._id.getTimestamp();
emit("1", {diff : diff});
}
};
var reduce = function(key, values) {
var reducedVal = { count: 0, sum: 0 };
for (var idx = 0; idx < values.length; idx++) {
reducedVal.count += 1;
reducedVal.sum += values[idx].diff;
}
return reducedVal;
};
var finalize = function (key, reducedVal) {
reducedVal.avg = reducedVal.sum/reducedVal.count;
return reducedVal;
};
db.runCommand(
{
mapReduce: "y",
map: map,
reduce: reduce,
finalize : finalize,
out: "mrtest",
query: { test : true}
}
);
And the example output:
> db.mrtest.find().pretty()
{
"_id" : "1",
"value" : {
"count" : 2,
"sum" : 1051888000,
"avg" : 525944000
}
}
This is a follow-up from this question, where I tried to solve this problem with the aggregation framework. Unfortunately, I have to wait before being able to update this particular mongodb installation to a version that includes the aggregation framework, so have had to use MapReduce for this fairly simple pivot operation.
I have input data in the format below, with multiple daily dumps:
"_id" : "daily_dump_2013-05-23",
"authors_who_sold_books" : [
{
"id" : "Charles Dickens",
"original_stock" : 253,
"customers" : [
{
"time_bought" : 1368627290,
"customer_id" : 9715923
}
]
},
{
"id" : "JRR Tolkien",
"original_stock" : 24,
"customers" : [
{
"date_bought" : 1368540890,
"customer_id" : 9872345
},
{
"date_bought" : 1368537290,
"customer_id" : 9163893
}
]
}
]
}
I'm after output in the following format, that aggregates across all instances of each (unique) author across all daily dumps:
{
"_id" : "Charles Dickens",
"original_stock" : 253,
"customers" : [
{
"date_bought" : 1368627290,
"customer_id" : 9715923
},
{
"date_bought" : 1368622358,
"customer_id" : 9876234
},
etc...
]
}
I have written this map function...
function map() {
for (var i in this.authors_who_sold_books)
{
author = this.authors_who_sold_books[i];
emit(author.id, {customers: author.customers, original_stock: author.original_stock, num_sold: 1});
}
}
...and this reduce function.
function reduce(key, values) {
sum = 0
for (i in values)
{
sum += values[i].customers.length
}
return {num_sold : sum};
}
However, this gives me the following output:
{
"_id" : "Charles Dickens",
"value" : {
"customers" : [
{
"date_bought" : 1368627290,
"customer_id" : 9715923
},
{
"date_bought" : 1368622358,
"customer_id" : 9876234
},
],
"original_stock" : 253,
"num_sold" : 1
}
}
{ "_id" : "JRR Tolkien", "value" : { "num_sold" : 3 } }
{
"_id" : "JK Rowling",
"value" : {
"customers" : [
{
"date_bought" : 1368627290,
"customer_id" : 9715923
},
{
"date_bought" : 1368622358,
"customer_id" : 9876234
},
],
"original_stock" : 183,
"num_sold" : 1
}
}
{ "_id" : "John Grisham", "value" : { "num_sold" : 2 } }
The even indexed documents have the customers and original_stock listed, but an incorrect sum of num_sold.
The odd indexed documents only have the num_sold listed, but it is the correct number.
Could anyone tell me what it is I'm missing, please?
Your problem is due to the fact that the format of the output of the reduce function should be identical to the format of the map function (see requirements for the reduce function for an explanation).
You need to change the code to something like the following to fix the problem, :
function map() {
for (var i in this.authors_who_sold_books)
{
author = this.authors_who_sold_books[i];
emit(author.id, {customers: author.customers, original_stock: author.original_stock, num_sold: author.customers.length});
}
}
function reduce(key, values) {
var result = {customers:[] , num_sold:0, original_stock: (values.length ? values[0].original_stock : 0)};
for (i in values)
{
result.num_sold += values[i].num_sold;
result.customers = result.customers.concat(values[i].customers);
}
return result;
}
I hope that helps.
Note : the change num_sold: author.customers.length in the map function. I think that's what you want
My documents look like this:
{
"_id" : "Tvq579754r",
"Status" : "passed",
"Title" : "up08c",
"ProjectID" : "Tvq5p",
"Version" : "1.0.0",
"Platform" : "platform_x",
"METRIC_A" : 11114.85,
"METRIC_B" : 68.9,
"METRIC_C" : 65.35,
},
{
"_id" : "Tvq579755r",
"Status" : "passed",
"Title" : "up09c",
"ProjectID" : "Tvq5p",
"Version" : "1.0.0",
"Platform" : "platform_x",
"METRIC_A" : 21114.85,
"METRIC_B" : 168.9,
"METRIC_C" : 165.35,
},
{
"_id" : "Tvq579756r",
"Status" : "passed",
"Title" : "up09c",
"ProjectID" : "Tvq5p",
"Version" : "1.0.0",
"Platform" : "platform_x",
"METRIC_A" : 31114.85,
"METRIC_B" : 268.9,
"METRIC_C" : 265.35,
}
Now I have no problem grouping and getting $avg and $sum of my METRIC_ fields by grouping by ProjectID, Version, Platform and Title, but what I'd like to do within the aggregation framework (if possible) is to return an object that uses the grouped values as keys, such as:
{
<Project ID> : {
<Version> : {
<Platform> : {
<Title> : {
"METRIC_A": <sum of METRIC_A>,
"METRIC_B": <sum of METRIC_B>,
"METRIC_C": <sum of METRIC_C>,
}
}
}
}
}
Or, in context of my example:
{
'Tvq5p' : {
'1.0.0' : {
'platform_x' : {
'up08c' : {
"METRIC_A": 11114.85,
"METRIC_B": 68.9,
"METRIC_C": 65.35,
},
'up09c' : {
"METRIC_A": 52229.7,
"METRIC_B": 437.8,
"METRIC_C": 430.7,
}
}
}
}
}
I am currently doing it once the query results are received by the consuming service, which isn't terribly slow or anything, but I just thought it would be nice to come that way right out of Mongo. Is this even possible?
Thanks.
In MongoDB there is the group operation.
db.records.group( {
key: { 'platform_x': 1, 'title': 1 },
reduce: function(cur, result) {
result.metric_a += cur.metric_a;
result.metric_b += cur.metric_b;
result.metric_c += cur.metric_c;
},
initial: { metric_a = 0, metric_b = 0, metric_c = 0 }
} )
If that doesn't work I'd recommend a Map Reduce.