MongoDB aggregation query to split and convert JSON? - mongodb

Question

I have a JSON file with a horrific data structure
{ "#timestamp" : "20160226T065604,39Z",
"#toplevelentries" : "941",
"viewentry" : [ { "#noteid" : "161E",
"#position" : "1",
"#siblings" : "941",
"entrydata" : [
and entrydata is a list of 941 entries, each of which look like this
{ "#columnnumber" : "0",
"#name" : "$Created",
"datetime" : { "0" : "20081027T114133,55+01" }
},
{ "#columnnumber" : "1",
"#name" : "WriteLog",
"textlist" : { "text" : [ { "0" : "2008.OCT.28 12:54:39 CET # EMI" },
{ "0" : "2008.OCT.28 12:56:13 CET # EMI" },
There are many more columns. The structure is always this:
{
"#columnnumber": "17",
"#name": "PublicDocument",
"text": {
"0": "TMI-1-2005.pdf"
}
}
there's a column number which we can throw away, a #name which is the important part, then one of text, datetime or textlist fields where the value is always this weird subdocument with a 0 key and the actual value.
All 941 entries have the same number of these column entries and the column entry is always the same structure. Ie. if "#columnnumber": "13" has a #name: foo then it'll always be foo and if it has a datetime key then it always will have a datetime key, never a text or textlist. This monster was borne out of a SQL or similar database somewhere at the very far end of the pipeline but I have no access to the source beyond this. The goal is to revert the transformation and make it into something a SELECT statement would produce (except textlist, although I guess array_agg and similar could produce that too).
Is there a way to get 941 separate JSON entries out of MongoDB looking like:
{
$Created: "20081027T114133,55+01",
WriteLog: ["2008.OCT.28 12:54:39 CET # EMI", "2008.OCT.28 12:56:13 CET # EMI"],
PublicDocument: "TMI-1-2005.pdf"
}

Answer 1

is viewentry also a list?
if you do an aggregate on the collection, and $unwind on viewentry.entrydata you will get one document for every entrydata. It should be possible to the do a $project to reformat these documents to produce the output you need

Answer 2

this is a nice challenge,
to get outupt like that:
{
"_id" : "161E",
"field" : [
{
"name" : "$Created",
"datetime" : {
"0" : "20081027T114133,55+01"
}
},
{
"name" : "WriteLog",
"textlist" : {
"text" : [
{
"0" : "2008.OCT.28 12:54:39 CET# EMI"
},
{
"0" : "2008.OCT.28 12:56:13 CET# EMI"
}
] } } ]}
use this aggregation pipelines:
db.chx.aggregate([ {$unwind: "$viewentry"}
, {$unwind: "$viewentry.entrydata"}
,{$group:{
"_id":"$viewentry.#noteid", field:{ $push:{
"name": "$viewentry.entrydata.#name" ,
datetime:"$viewentry.entrydata.datetime",
textlist:"$viewentry.entrydata.textlist" }}
}}
]).pretty()
the next step should be extracting log entries, but I have no idea, as my brain is already fried tonight - so probably I can return later...