Combining different columns powershell - powershell

i need to combine csv-entries which is in this format:
ID, "Name", "Sex", "Age"
1, "John", "m", "24"
2, "John", "m", "28"
2, "John", "m", "30"
3, "Melissa", "f", "24"
3, "Melissa", "f", "26"
to:
ID, "Name", "Sex", "Age", "Age" ,"Age"
1, "John", "m", "24", "28", "30"
3, "Melissa", "f", "24", "26"
What i got so far:
$csvData = Import-Csv 'test.csv'
$csvData | Group-Object "ID" | ForEach-Object {
[PsCustomObject]#{
"ID" = $_.Name
"Sex" = $_.Group."Sex" -join '","'
"Age" = $_.Group."Age" -join '","'
}
} | Export-Csv 'test_output.csv' -NoTypeInformation
which returns me this:
"ID","Name", "Sex", "Age"
"1", "John", "John", "John", "m", "m", "m", "24", "28", "30"
"2", "Melissa", "Melissa", , "f", "f", "24", "26"
How can i solve this?

If I understand you right, then this is only meant to be an exercise in PowerShell foo and not in meaningful data manipulation.
Under that premise: One way you could create your desired output (and end up with a valid csv), is to add numbered properties for each "Age" value that you want to include in the consolidated objects.
$csvData = Import-Csv 'test.csv'
$csvData | Group-Object Name | ForEach-Object {
$obj = [ordered]#{
ID = $_.Group[0].ID
Name = $_.Name
Sex = $_.Group[0].Sex
}
for ($i = 0; $i -lt $_.Group.Count; $i++) {
$obj["Age$($i+1)"] = $_.Group[$i].Age
}
[PsCustomObject]$obj
} | Export-Csv 'test_output.csv' -NoTypeInformation

Related

Converting a MongoDB String attribute to an Object which can be queried on

I have the following data in mongoDB
{ "_id" : "ee477d7a-7a0c-4420-b476-c402012c74a9",
"_class" : "com.TrackingData",
"modified" : ISODate("2014-07-09T20:23:33.117Z"),
"eventtype" : "Test",
"eventdata" : "{\"QueryDate\":\"01-APR-2014\",
\"SearchQuery\":{\
"keyword\":\"Java\",\
"location\":\"Santa Clara, CA\",
\"Facet\":\"skill~java~perl|workAuth~USC\",
\"SearchAgentId\":\"4299\"
},
\"Viewed\":[
{\"ViewedID\":\"8992ade400a\",
\"Dockey\":\"3323aba3233\",
\"PID\":\"32399a\",
\"actionsTaken\":\"email|direct message|report seeker\",
\"viewDate\":\"01-APR-2014\",
\"MessageSent\":\"true\",
\"Message\":[
{\"MessageID\":\"123aca323\",
\"Delivered\":\"True\",
\"Opened\":\"True\",
\"ClickThroughRate\":\"NotBad\",
\"MessageDate\":\"02-APR-2014\",
\"Response\":[{\"ResponseId\":\"a323a9da\",\"ResponseDate\":\"23-APR-2014\"}]}]}]}",
"eventsource" : "API-Dev Test - JMachine",
"sourceip" : "myIp",
"entityid" : "Test",
"groupid" : "ice",
"datecreated" : ISODate("2014-07-09T20:23:33.112Z") }
evendata is stored as a String I have tried to convert this to an object through the mongo shell:
db.mydb.find( { 'eventdata' : { $type : 2 } } ).forEach( function (x) { x.eventdata = new Object(x.eventdata); db.mydb.save(x);});
However is simply seemed to split these into a map:
"eventdata" : { "0" : "{", "1" : "\"", "2" : "V", "3" : "i", "4" : "e", "5" : "w", "6" : "e", "7" : "d", "8" : "\"", "9" : ":", "10" : "[", "11" : "{", "12" : "\"", "13" : "V", "14" : "i", "15" : "e", "16" : "w", "17" : "e", "18" : "d", "19" : "I", "20" : "D", "21" : "\"", "22" : ":", "23" : "\"", "24" : "8", "25" : "9", "26" : "9", "27" : "2", "28" : "a", "29" : "d", "30" : "e", "31" : "4", "32" : "0", "33" : "0", "34" : "a", "35" : "\"", "36" : ",", "37" : "\"", "38" : "D", "39" : "o", "40" : "c", "41" : "k", "42" : "e", "43" : "y", "44" : "\"", "45" : ":", "46" : "\"", "47" : "1", "48" : "7", "49" : "2", "50" : "9", "51" : "f", "52" : "7", "53" : "a", "54" : "f", "55" : "c", "56" : "d", "57" : "1", "58" : "f", "59" : "7", "60" : "6", "61" : "3", "62" : "9", "63" : "3", "64" : "a", "65" : "b", "66" : "6", "67" : "6", "68" : "d", "69" : "5", "70" : "c", "71" : "6", "72" : "4", "73" : "8", "74" : "a", "75" : "f", "76" : "3", "77" : "7", "78" : "b", "79" : "\"", "80" : ",", "81" : "\"", "82" : "P", "83" : "I", "84" : "D", "85" : "\"", "86" : ":", "87" : "\"", "88" : "\"", "89" : ",", "90" : "\"", "91" : "a", "92" : "c", "93" : "t", "94" : "i", "95" : "o", "96" : "n", "97" : "s", "98" : "T", "99" : "a", "100" : "k", "101" : "e", "102" : "n", "103" : "\"", "104" : ":", "105" : "\"", "106" : "\"", "107" : ",", "108" : "\"", "109" : "v", "110" : "i", "111" : "e", "112" : "w", "113" : "D", "114" : "a", "115" : "t", "116" : "e", "117" : "\"", "118" : ":", "119" : "\"", "120" : "0", "121" : "9", "122" : "-", "123" : "J", "124" : "U", "125" : "L", "126" : "-", "127" : "2", "128" : "0", "129" : "1", "130" : "4", "131" : " ", "132" : "2", "133" : "0", "134" : ":", "135" : "3", "136" : "1", "137" : ":", "138" : "2", "139" : "3", "140" : "\"", "141" : "}", "142" : "]", "143" : "}" },
Which still does not recongised my nested queries such as:
db.mydb.find({'eventdata.SearchQuery.keyword' :'keywordValue' }).skip(0).limit(20)
So given my original data structure. What need to be done in terms of transformation to allow me to drill down into this eventdata attribute.
In your example with forEach use JSON.parse(x.eventdata) instead of new Object(x.eventdata).
Of course you cannot query it directly with .find() - it's just string from the MongoDB point of view. You should have store it as BSON if you wanted to query it... All you can use is documented here: http://docs.mongodb.org/manual/reference/operator/query/ After the transformation from JSON to native BSON (the forEach example) it will be possible to query it.

Count and group by with mongo db

I m actually facing a problem with mongoDB.
I need to display some statistics :
- A treatment is an information that contain a date, the user who treated, a list of anomalies
Can you help me with the request to get :
"The numbers of anomalies by users ?"
Thanks for all :D
db.treatment.aggregate(
{
$group : {_id : "$anomalies", totalUser : { $sum : 1 }}
}
);
Note : change your collection and document key name if I put wrong.
Source : http://www.mkyong.com/mongodb/mongodb-aggregate-and-group-example/
So, if your collection had the following documents:
> db.treatments.find()
{ "_id" : 1, "date" : ISODate("2014-08-29T15:44:45.843Z"), "user" : "A", "anomalies" : [ "a", "b", "c" ] }
{ "_id" : 2, "date" : ISODate("2014-08-29T15:45:01.782Z"), "user" : "A", "anomalies" : [ "e", "f", "g" ] }
{ "_id" : 3, "date" : ISODate("2014-08-29T15:45:34.889Z"), "user" : "B", "anomalies" : [ "a", "b", "c", "e", "f", "g" ] }
{ "_id" : 4, "date" : ISODate("2014-08-29T15:48:01.860Z"), "user" : "B", "anomalies" : [ "a", "b", "c", "e", "f", "g" ] }
{ "_id" : 5, "date" : ISODate("2014-08-29T15:48:28.937Z"), "user" : "A", "anomalies" : [ "x", "y", "z" ] }
You can use $group stage to $sum the $size of the anomalies array
> db.treatments.aggregate([ { $group: { _id: "$user", allAnomalies: { $sum: { $size: "$anomalies" } } } } ] )
{ "_id" : "B", "allAnomalies" : 12 }
{ "_id" : "A", "allAnomalies" : 9 }

how do I insert an array of objects to mongodb using meteor?

How do I get a clean insert without extraneous characters being added when trying to insert an array of object. If I manually do an insert from mongodb shell I get the expected results, otherwise it doesn't seem to work.
What I'm trying to achieve is the results from mongodb shell:
db.test.insert([{name:"john"},{name:"jane"}]);
which yields:
db.test.find()
{ "_id" : ObjectId("53bb0768dc2469c1f440a3c2"), "name" : "john" }
{ "_id" : ObjectId("53bb0768dc2469c1f440a3c3"), "name" : "jane" }
But I don't get that, so I used the code snippet below to test several ways to insert the array of objects hoping to find the right combination:
test = new Meteor.Collection("test");
a = new Array();
a.push({name:"john"});
a.push({name:"jane"});
console.log(a);
test.insert(a);
console.log(a.toString());
test.insert(a.toString());
console.log(JSON.stringify(a));
test.insert(JSON.stringify(a));
test.insert([{name:"john"},{name:"jane"}]);
test.insert([{"name":"john"},{"name":"jane"}]);
What I get in the console:
[ { name: 'john' }, { name: 'jane' } ]
[object Object],[object Object]
[{"name":"john"},{"name":"jane"}]
What I get in the database:
db.test.find()
{ "0" : { "name" : "john" }, "1" : { "name" : "jane" }, "_id" : "SYkv79XLNQsWgkYmw" }
{ "0" : "[", "1" : "o", "2" : "b", "3" : "j", "4" : "e", "5" : "c", "6" : "t", "7" : " ", "8" : "O", "9" : "b", "10" : "j", "11" : "e", "12" : "c", "13" : "t", "14" : "]", "15" : ",", "16" : "[", "17" : "o", "18" : "b", "19" : "j", "20" : "e", "21" : "c", "22" : "t", "23" : " ", "24" : "O", "25" : "b", "26" : "j", "27" : "e", "28" : "c", "29" : "t", "30" : "]", "_id" : "SiQ3ZpGfeBqj4mXB2" }
{ "0" : "[", "1" : "{", "2" : "\"", "3" : "n", "4" : "a", "5" : "m", "6" : "e", "7" : "\"", "8" : ":", "9" : "\"", "10" : "j", "11" : "o", "12" : "h", "13" : "n", "14" : "\"", "15" : "}", "16" : ",", "17" : "{", "18" : "\"", "19" : "n", "20" : "a", "21" : "m", "22" : "e", "23" : "\"", "24" : ":", "25" : "\"", "26" : "j", "27" : "a", "28" : "n", "29" : "e", "30" : "\"", "31" : "}", "32" : "]", "_id" : "kKRiR8NjNJefBYRya" }
{ "0" : { "name" : "john" }, "1" : { "name" : "jane" }, "_id" : "RBrvkrw5xZaEGdczF" }
{ "0" : { "name" : "john" }, "1" : { "name" : "jane" }, "_id" : "2cfWJqHY4aJ6yF68s" }
I expected a simple 'test.insert(a)' to give me what I want, but it includes the array indexes. How do I build an array of objects to insert into mongodb from meteor without the array indexes? Stringify seemed to build a clean looking serialization of the array, but apparently I just don't know how to do this? The purpose of this is so I can build a complex array of objects in memory and do a bulk insert.
Meteor only lets you store root level documents as objects, if you give it an array it will try to convert it to an object. This is why you're getting this weird result. You would have to adjust your document to store arrays as part of the root document
test = new Meteor.Collection("test");
a = new Array();
a.push({name:"john"});
a.push({name:"jane"});
var doc = {
names: a
}
test.insert(a);
It won't be possible to store a document as [].
Template.first.rendered=function(){
var a=[];
a.push({name:"rahul"});
a.push({name:"vidu"});
Meteor.call("array", a , function(error,result){
});
};

How do you compute max and unique in Mongo?

I have a bunch of documents:
{ "name" : "A", "value" : 10},
{ "name" : "B", "value" : 12},
{ "name" : "B", "value" : 14},
{ "name" : "A", "value" : 16},
{ "name" : "C", "value" : 11}
I want to find the minimum value for each name, so the result would be
{ "name" : "A", "value" : 10},
{ "name" : "B", "value" : 12},
{ "name" : "C", "value" : 11}
What is the correct way to do this query in Mongo? I keep thinking in terms of SQL.
For the minimum, you can do this by using Mongo's Aggregation Framework with "$group" and "$min". Assuming that you had a collection named "myCollection," you could try something like this:
db.myCollection.aggregate({$group:{_id:"$name", minimumValue: {$min : "$value"}}});

Scalacheck json and case classes

I'm writing a service that takes a case class and serializes it to json, that I will then send to an instance running Elastic Search.
I'd like scalacheck to generate several case classes, with random missing data, like this:
val searchDescAndBrand = SearchEntry("", "Ac Adapters", "Sony", "", "", "", "", "", "", "", "", "", "", "", "","", "", "", "", "", "", 0L)
val searchBrand = SearchEntry("", ", "Sony", "", "", "", "", "", "", "", "", "", "", "", "","", "", "", "", "", "", 0L)
val searchPartNumberAndBrand = SearchEntry("02DUYT", "", "Sony", "", "", "", "", "", "", "", "", "", "", "", "","", "", "", "", "", "", 0L)
you get the idea, either fill in the values or leave them empty (the last one is a Long type.
This is the easy part, the problem is that the generated json doesn't just omit the "filed", but omits a whole section, for example:
"""
|{
| "from" : 0,
| "size" : 10,
| "query" : {
| "bool" : {
| "must" : [
| {"match" : {
| "description" : {
| "query" : "Ac Adapters",
| "type" : "phrase"
| }
| }},
| {"match" : {
| "brand" : {
| "query" : "Sony",
| "type" : "phrase"
| }
| }}
| ]
| }
| }
|}
|
""".stripMargin)
if I had a case class with the first 3 fields with data, the json would be:
"""
|{
| "from" : 0,
| "size" : 10,
| "query" : {
| "bool" : {
| "must" : [
| {"match" : {
| "part_number" : {
| "query" : "02D875",
| "type" : "phrase"
| }
| }},
| {"match" : {
| "description" : {
| "query" : "Ac Adapters",
| "type" : "phrase"
| }
| }},
| {"match" : {
| "brand" : {
| "query" : "Sony",
| "type" : "phrase"
| }
| }}
| ]
| }
| }
|}
|
""".stripMargin)
So, in short, having a value means adding
{"match" : {
| "<specific name here, based on which value we have>" : {
| "query" : "<value from scalacheck>",
| "type" : "phrase"
| }
| }}
to the result.
How would you handle such a use case?