Removing empty Columns in Pentaho Kettle before inserting on MongoDB - mongodb

Question

I am using Pentaho Kettle as a tool to process several CSV files before inserting them in MongoDB for the first time.
Since MongoDB is schemaless I don't seen the point in keeping the null column values of the CSV rows. I want to do receive something like this from the CSV
+------------+----------+---------+
| _id | VALUE_1 | VALUE_2 |
+------------+----------+---------+
| 1 | 1 | 1 |
| 2 | 2 | null |
| 3 | null | 2 |
+------------+----------+---------+
And insert it onto mongodb in a way that I get this in there:
{ "_id" : 1, "VALUE_1" : 1, "VALUE_2" : 1 }
{ "_id" : 2, "VALUE_1" : 2 }
{ "_id" : 3, "VALUE_2" : 2}
How would I do such a thing in Kettle? I just can't seem to find the right option there, there is a filter rows but it doesn't seem what I want.

Answer 1

I'm having the same problem. One work around I found from Matt Casters and Diethard Steiner is to un pivot the data and then remove Null Rows. Then you could pivot back and write up the JSON with a javascript step or JSON output perhaps. Similar to this:
http://diethardsteiner.blogspot.com/2010/11/pentaho-kettle-data-input-pivoted-data.html
This worked fine for small files, but I have large csv's with 30-100 columns and hundreds of thousands of rows, millions in some cases. So pivoting is very slow.. but maybe you can come up with another idea, I'd be glad to hear it! =)