Break Down Multiple Arrays to Documents with Comparison - mongodb

Question

I'd like to break down a record into new multiple records in mongo. How to do it?
Current data:
{"user_id": 123, "scores": [65, 71, 79, 80], "materials": ["A", "B", "C", "D"]}
And I want to create the data below from above one:
{"user_id": 123, "score_original": 65, "score_improvement": 6, "material": "A"},
{"user_id": 123, "score_original": 71, "score_improvement": 8, "material": "B"},
{"user_id": 123, "score_original": 79, "score_improvement": 1, "material": "C"}

Answer 1

The question really could do with some clarification, but if your intention is to expand each item with a comparison to each "next" array element and where the arrays are always of equal length in direct comparison, then there are a couple of approaches.
That is there is a "simple way" ( at the end ) and more complex ways, depending on your needs. So to step through them so you understand what is involved from each:
Agregate
With a modern MongoDB release you can use the aggregation framework to merge and compare the array elements and then expand into new items like so:
db.getCollection('junk').aggregate([
{ "$project": {
"user_id": 1,
"data": {
"$map": {
"input": {
"$slice": [
{ "$zip": {
"inputs": [
"$scores",
{ "$map": {
"input": {
"$reverseArray": {
"$reduce": {
"input": { "$reverseArray": "$scores" },
"initialValue": [],
"in": {
"$concatArrays": [
"$$value",
[
[
"$$this",
{ "$subtract": [
{ "$arrayElemAt": [
{ "$ifNull": [{ "$arrayElemAt": ["$$value", -1] }, [] ]},
0
]},
"$$this"
]}
]
]
]
}
}
}
},
"in": { "$arrayElemAt": ["$$this",-1] }
}},
"$materials"
]
}},
{ "$subtract": [{ "$size": "$scores" },1] }
]
},
"in": {
"score_original": { "$arrayElemAt": [ "$$this", 0 ] },
"score_improvement": { "$arrayElemAt": [ "$$this", 1 ] },
"material": { "$arrayElemAt": [ "$$this", 2 ] }
}
}
}
}},
{ "$unwind": "$data" },
{ "$replaceRoot": {
"newRoot": {
"$arrayToObject": {
"$concatArrays": [
[{ "k": "user_id", "v": "$user_id" }],
{ "$objectToArray": "$data" }
]
}
}
}}
])
Which returns the desired result:
/* 1 */
{
"user_id" : 123.0,
"score_original" : 65.0,
"score_improvement" : 6.0,
"material" : "A"
}
/* 2 */
{
"user_id" : 123.0,
"score_original" : 71.0,
"score_improvement" : 8.0,
"material" : "B"
}
/* 3 */
{
"user_id" : 123.0,
"score_original" : 79.0,
"score_improvement" : 1.0,
"material" : "C"
}
Much of the work is done with $reduce from "reversed" array content via $reverseArray since you want to compare to the "next" item. It's generally easier to do a "last" comparison than try to work with calculated index values in the aggregation framework operations, so this is why you "reverse" it.
The basic premise for the "improvement" values is to work through the "reversed" array comparing the present value to the last in the output array and calculate the difference using $subtract. Since you need to output both the "improvement" and also need the "previous" value for comparison, which is done via $arrayElemAt along with $ifNull checks to extract the value for comparison.
These are stored in "array pairs" for output before feeding to the next operation. Naturally you $reverseArray again to maintain the original order with the new output.
Since there are now essentially "three" arrays of values, one way of "combining" these into one is $zip which would make an array of arrays for each of the elements. It's not the only way, but again it's probably a bit clearer to read than juggling index values for extraction.
Then of course you use $map to get to the final "object" form for each array entry. But not before applying $slice since the "last" array element is being discarded due to no "improvement" over it's "next" item, which does not exist. At least that's following the logic you seem to present.
The final parts are simply using $unwind to turn the array construct into separate documents, and then reshaping the final output. Here this is applied using $replaceRoot as well as the $objectToArray and $arrayToObject operators to construct a new root document without explicit naming. However this may as well just be a simple $project instead:
{ "$project": {
"user_id": 1,
"score_original": "$data.score_original",
"score_improvement": "$data.score_improvement",
"material": "$data.material"
}}
So there are different ways that can be applied both there and in the "object" construction of the array as well. It's just that the newer operators such as $objectToArray require MongoDB 3.4.4 at least. All other things can be done with MongoDB 3.4.
Aggregate Alternate
You can alternately just work with the array indexes supplied using $range where available:
db.getCollection('junk').aggregate([
{ "$project": {
"_id": 0,
"user_id": 1,
"data": {
"$map": {
"input": { "$range": [ 0, { "$subtract": [{ "$size": "$scores" }, 1] } ] },
"as": "r",
"in": {
"score_original": { "$arrayElemAt": [ "$scores", "$$r" ] },
"score_improvement": {
"$subtract": [
{ "$arrayElemAt": [ "$scores", { "$add": [ "$$r", 1 ] } ] },
{ "$arrayElemAt": [ "$scores", "$$r" ] }
]
},
"material": { "$arrayElemAt": [ "$materials", "$$r" ] }
}
}
}
}},
{ "$unwind": "$data" },
{ "$replaceRoot": {
"newRoot": {
"$arrayToObject": {
"$concatArrays": [
[{ "k": "user_id", "v": "$user_id" }],
{ "$objectToArray": "$data" }
]
}
}
}}
])
That has the same output and also follows the basic logic as shown in the following approaches.
Map Reduce
If you don't have a MongoDB 3.4 supporting the operators used, then you can always apply mapReduce and simply calculate and emit for each array value:
db.getCollection('junk').mapReduce(
function() {
for( var i=0; i < this.scores.length-1; i++ ) {
var id = this._id.valueOf() + '_' + i;
emit(id, {
"user_id": this.user_id,
"score_original": this.scores[i],
"score_improvement": this.scores[i+1] - this.scores[i],
"material": this.materials[i]
});
}
},
function() {},
{ "out": { "inline": 1 } }
)
This does have it's own specific output via the rules of mapReduce, which should be evident in the construct of the "unique" _id value to emit:
"results" : [
{
"_id" : "59e4144331be3474a2f28a92_0",
"value" : {
"user_id" : 123.0,
"score_original" : 65.0,
"score_improvement" : 6.0,
"material" : "A"
}
},
{
"_id" : "59e4144331be3474a2f28a92_1",
"value" : {
"user_id" : 123.0,
"score_original" : 71.0,
"score_improvement" : 8.0,
"material" : "B"
}
},
{
"_id" : "59e4144331be3474a2f28a92_2",
"value" : {
"user_id" : 123.0,
"score_original" : 79.0,
"score_improvement" : 1.0,
"material" : "C"
}
}
],
You should note that aside from being far less complex in implementation there is also actual no "reducer" function defined at all. Which should also lead the inevitable conclusion here.
Iterate the cursor
This really is just a basic cursor iteration and expansion you are asking for, so that is all you really need to do. Which means working from the base defined in out mapper function as a simple shell abstraction:
db.getCollection('junk').find().forEach(d => {
for (var i=0; i < d.scores.length-1; i++) {
printjson({
"user_id": d.user_id,
"score_original": d.scores[i],
"score_improvement": d.scores[i+1] - d.scores[i],
"material": d.materials[i]
})
}
})
Which gives the output as desired:
{
"user_id" : 123,
"score_original" : 65,
"score_improvement" : 6,
"material" : "A"
}
{
"user_id" : 123,
"score_original" : 71,
"score_improvement" : 8,
"material" : "B"
}
{
"user_id" : 123,
"score_original" : 79,
"score_improvement" : 1,
"material" : "C"
}
And it really is that simple.
The base lesson here is that "whilst you can" ask a database to do complicated things, unless it actually results in a significant reduction in the data load returned from the server, then the usual best case is to simply process the data in native client code instead.
Even if the presented data in the question was obtained from some other aggregation operation, it would still generally be better at this stage to simply iterate the cursor result for the final transformation.
And if the transformation where required for further aggregation operations, then by all means follow the first process. However if the data presented is actually obtained by aggregation already and there is a need to transform in further aggregation, then you probably should examine the existing aggregation process you have, since you may not even need the intermediate state with multiple arrays, which is where most of the complexity comes from.