Pig load array to mongo - mongodb

Question

I would like to transform my output array:
I have the following code:
x = LOAD '$INPU'
USING PigStorage('\\u001')
AS (
product_id:chararray,
size:chararray
);
grouped = GROUP x BY (product_id);
sizes = FOREACH grouped {
sizes = DISTINCT $1.size;
GENERATE
$0 AS product_id,
sizes AS sizes;
}
output = foreach sizes generate
product_id as id,
sizes as sizes;
STORE output
INTO '$output'
USING com.mongodb.hadoop.pig.MongoInsertStorage('id');
this result the following:
"product_id" :"123",
"sizes": [
{
"size": "X"
},
{
"size": "M"
},
{
"size": "L"
}
]
It is possible to change the output to the following? :
product_id: "123",
sizes": ["X", "M", "L"]
i have tried flatten and BagToTuple but could not find a solution
thanks in advance

Answer 1

You've probably already seen it, but this page on mongodb.hadoop explains in great detail, and with examples, how to use MongoInsertStorage (and also MongoUpdateStorage).
I have to admit, I didn't see an option there to do what you'd like; indeed, in their example they get a similar result to yours.
However, on thing that might work is to use MongoUpdateStorage to do upserts. I'm not sure it will work, but if you use a general query with no parameters on a new or empty collection, it can do the job. If you look at the bottom part of the link I provided, they explain how to get output that looks like this
{ "_id" : ObjectId("..."), "gender":"male", "age" : 19, "cars" : ["a", "b", "c"], "first" : "Daniel", "last" : "Alabi" }
instead of this
{ "_id" : ObjectId("..."), "gender":"male", "age" : 19, "cars" : [{"car": "a"}, {"car":"b"}, {"car":"c"}], "first" : "Daniel", "last" : "Alabi" }
(I'm referring to the change in the cars field).
One last question - in your example, you change the name of product_id to id in your last foreach ... but in the output you showed, it still has the name product_id. Could it be you've been sending the wrong relation to MongoInsertStorage?
And, finally, another option is to save your collection as BSON and then use mongorestore on it - this option is also explained on that page.