I want to extract columns by header name from multiple CSVs into one 'master file'. I have tried multiple things but each time there is something else that occurs.
I have this code which is working now, but extremely tedious typing out which header name I need for every column, as there will be upwards of 20 of columns I need from each CSV.
Anyone have any suggestions? I'm currently using PS, but am open to other methods.
Import-Csv -Path $inputFilePath1 | ForEach-Object {
$File1Values.Add($_.HBA_ID, $_)
}
Import-Csv -Path $inputFilePath2 | Select-Object *,
| Export-Csv -Path $outputFilePath -NoTypeInformation
#{ N = "TOPF_IQ_Zscore"; E = { $File1Values[$_.HBA_ID].TOPF_IQ_Zscore } },
#{ N = "TMT_A_Zscore"; E = { $File1Values[$_.HBA_ID].TOPF_IQ_Zscore } },
#{ N = "TOPF_IQ_Zscore"; E = { $File1Values[$_.HBA_ID].TOPF_IQ_Zscore } },
#{ N = "TOPF_IQ_Zscore"; E = { $File1Values[$_.HBA_ID].TOPF_IQ_Zscore } },
#{ N = "TOPF_IQ_Zscore"; E = { $File1Values[$_.HBA_ID].TOPF_IQ_Zscore } },
#{ N = "TOPF_IQ_Zscore"; E = { $File1Values[$_.HBA_ID].TOPF_IQ_Zscore } },
#{ N = "TOPF_IQ_Zscore"; E = { $File1Values[$_.HBA_ID].TOPF_IQ_Zscore } },
#{ N = "TOPF_IQ_Zscore"; E = { $File1Values[$_.HBA_ID].TOPF_IQ_Zscore } },
#{ N = "TOPF_IQ_Zscore"; E = { $File1Values[$_.HBA_ID].TOPF_IQ_Zscore } } |
Export-Csv -Path $outputFilePath -NoTypeInformation
Thanks in advance!
Related
I have a Powershell-script which includes a lot of PsCustomObjects like this (names do not have a specific pattern):
$myObject1 = [PSCustomObject]#{
Name = 'Kevin'
State = 'Texas'
}
$myObject2 = [PSCustomObject]#{
Name = 'Peter'
Lastname = 'Fonda'
State = 'Florida'
}
Now I need to convert this programmatically into the following object-format to have a global hashtable:
$result = #{
'myObject1.Name' = 'Kevin'
'myObject1.State' = 'Texas'
'myObject2.Name' = 'Peter'
'myObject2.Lastname' = 'Fonda'
'myObject2.Sate' = 'Florida'
}
For this task I need a loop in which I can either read the name of each PsCustomOject or I need to specify the names of all object as a string-array and lookup the object-properties with the matching name.
The loop-constructor could look something like this:
$result = #{}
foreach($name in #('myObject1','myObject2')) {
$obj = myMissingFunction1 $name
foreach($p in $obj.PsObject.Properties) {
$result["$name.$($p.Name)"] = $p.Value
}
}
or this
$result = #{}
foreach($obj in #($myObject1, $myObject2)) {
$name = myMissingFunction2 $obj
foreach($p in $obj.PsObject.Properties) {
$result["$name.$($p.Name)"] = $p.Value
}
}
Unfortunately I cannot bring any of the two approaches to life. Can someone please help?
Here is how you could do it .
$Result = [Ordered]#{}
$index = 0
Foreach ($obj in $ArrayOfObject) {
foreach ($prop in ($obj | Get-Member -MemberType NoteProperty).Name) {
$Result."SomeObject$Index.$Prop" = $obj.$prop
}
$index += 1
}
Result
Name Value
---- -----
SomeObject0.Name Kevin
SomeObject0.State Texas
SomeObject1.Lastname Fonda
SomeObject1.Name Peter
SomeObject1.State Florida
Dataset used for this example
$ArrayOfObject = #(
[PSCustomObject]#{
Name = 'Kevin'
State = 'Texas'
}
[PSCustomObject]#{
Name = 'Peter'
Lastname = 'Fonda'
State = 'Florida'
}
)
Solved it finally. I completely forgot the "Get-Variable" function:
$result = #{}
foreach($name in #('myObject1','myObject2')) {
$obj = (Get-Variable $name).Value
foreach($p in $obj.PsObject.Properties) {
$result["$name.$($p.Name)"] = $p.Value
}
}
I have a source array like this
$myArray = (
#{ id = "1"; name = "first item";
subArray = (
#{ id = "A"; name = "first subitem" },
#{ id = "B"; name = "second subitem" } )
},
#{ id = "2"; name = "second item";
subArray = (
#{ id = "C"; name = "third subitem" },
#{ id = "D"; name = "fourth subitem" } )
}
)
I need to extract the relations between the parent and child arrays like following:
source target
-----------------
1 A
1 B
2 C
2 D
I have come up with a following code to achieve that
$myArray | ForEach-Object {
$id = $_.id
$_.subArray | ForEach-Object {
#{
source = $id
target = $_.id
}
}
}
I wonder if there is some more straight forward solution.
Edit:
Based on Marsze answer - slightly modified solution
$myArray| ForEach-Object
{$a=$_; $a.subArray | Select-Object #{n="source";e={$a.id}},#{n="target";e={$_.id}}
}
Looks pretty straightforward to me. You could use foreach loops instead of the pipeline cmdlet, which is faster and has the advantage, that you can reference variables on each level directly.
Also I recommended converting to PSCustomObject for the proper output format:
foreach ($a in $myArray) {
foreach ($b in $a.subArray) {
[PSCustomObject]#{source = $a.id; target = $b.id}
}
}
Alternatively with New-Object:
foreach ($a in $myArray) {
foreach ($b in $a.subArray) {
New-Object PSObject -Property #{source = $a.id; target = $b.id}
}
}
Or the Select-Object version:
foreach ($a in $myArray) {
$a.subArray | select #{n="source";e={$a.id}},#{n="target";e={$_.id}}
}
I have an array, with a nested hashtable containing data:
tid : 1
token : 12345
participant_info : #{firstname=bob; lastname=smith; email=bob.smith#email.com}
completed : y
tid : 2
token : 67890
participant_info : #{firstname=Alice; lastname=Jones; email=alice.jones#email.com}
completed : n
I would like to output this as a CSV with the nested items of firstname, surname and email pulled out of the inner hashtable - eg
tid,token,firstname,surname,email,completed
1,12345,bob,smith,bob.smith#email.com,y
2,67890,alice,jones,alice.jones#email.com,n
I'm going to guess that the answer is looping through with foreach and creating a custom ps object, but because my nested items aren't named, I can't work out how to do this using the other examples on here.
Any assistance is appreciated! Thanks!
Given your sample:
#(
[pscustomobject]#{
tid = 1
token = 12345
participant_info = #{
firstname = 'bob'
lastname = 'smith'
email = 'bob.smith#email.com'
}
completed = 'N'
}
...
)
And desired output:
id,token,firstname,surname,email,completed
1,12345,bob,smith,bob.smith#email.com,y
2,67890,alice,jones,alice.jones#email.com,n
You can do something like this:
$JsonList = #( ... )
$Path = "$Env:UserProfile\file.csv"
ForEach ($Sample in $JsonList)
{
$Sample | Select-Object -Property #(
#{N = 'id'; E = { $Sample.tid }}
#{N = 'token'; E = { $Sample.token }}
#{N = 'firstname'; E = { $Sample.participant_info.firstname }}
#{N = 'surname'; E = { $Sample.participant_info.lastname }}
#{N = 'email'; E = { $Sample.participant_info.email }}
#{N = 'completed'; E = { $Sample.completed }}
) | Export-Csv -Path $Path -Append
}
Edit: you're dealing with PSCustomObject, not Hashtable, so my previous syntax won't work for that. Here's what I'm presuming your code to look like now (I've updated my above example):
#"
[
{
"tid": 1,
"token": 12345,
"participant_info": {
"firstname": "bob",
"lastname": "smith",
"email": "bob.smith#email.com"
},
"completed": "N"
}
...
]
"# | ConvertFrom-Json
Say I have a CSV file that looks like:
arch,osversion,kb
32,6.1,KB1,http://kb1
32,6.2,KB2,http://kb2
64,6.1,KB3,http://kb3
64,6.2,KB4,http://kb4
How would this CSV get imported into structured hash table that looks like this?
32 -> 6.1 -> KB1 -> http://kb1
-> 6.2 -> KB2 -> http://kb2
64 -> 6.1 -> KB3 -> http://kb3
-> 6.2 -> KB4 -> http://kb4
The command below yields http://kb1:
$data['32'].'6.1'.'KB1'
Probably Group-Object is what you want.
$csv = #'
arch,osversion,kb,link
32,6.1,KB1,http://kb1
32,6.2,KB2,http://kb2
64,6.1,KB3,http://kb3
64,6.2,KB4,http://kb4
'#
$data = ConvertFrom-Csv $csv
$data | Group-Object -Property arch
Or maybe closer to what you want to query:
$groups = $data | Group-Object -Property arch, osversion, kb
($groups | ? Name -eq '32, 6.1, KB1').Group.link
You could even use variables...
$a = '32'
$o = '6.1'
$k = 'KB1'
($groups | ? Name -eq "$a, $o, $k").Group.link
From this, you can determine if such a pattern works for you.
Interesting task. The following code snippet could help (solves arch duplicates):
Remove-Variable data*, aux* -ErrorAction SilentlyContinue ### clear for debugging purposes
$datacsv = #'
arch,osversion,kb,link
32,6.1,KB1,http://kb1
32,6.2,KB2,http://kb2
64,6.1,KB3,http://kb3
64,6.2,KB4,http://kb4
'#
$datac = ConvertFrom-Csv $datacsv
$datag = #{}
$datac | ForEach-Object {
$auxLeaf = #{ $_.kb = $_.link }
$auxParent = #{ $_.osversion = $auxLeaf }
if ( $datag.ContainsKey( $_.arch) ) {
$auxParent += $datag[ $_.arch]
}
$datag.Set_Item( $_.arch, $auxParent )
}
Then, $datag['32']['6.1']['KB1'] returns desired value http://kb1
Another interesting problem: solve osversion duplicates in a particular arch:
Remove-Variable data*, aux* -ErrorAction SilentlyContinue ### clear for debugging purposes
$datacsv = #'
arch,osversion,kb,link
32,6.1,KB1,http://kb1
32,6.1,KB5,http://kb5
32,6.1,KB7,http://kb7
32,6.2,KB2,http://kb2
64,6.1,KB3,http://kb3
64,6.2,KB4,http://kb4
'#
$datac = ConvertFrom-Csv $datacsv
$datag = #{}
$datac | ForEach-Object {
$auxLeaf = #{ $_.kb = $_.link }
$auxParent = #{ $_.osversion = $auxLeaf }
if ( $datag.ContainsKey( $_.arch) ) {
if ( $datag[$_.arch].ContainsKey($_.osversion) ) {
$auxLeaf += $datag[$_.arch][$_.osversion]
$auxParent = #{ $_.osversion = $auxLeaf }
} else {
$auxParent += $datag[ $_.arch]
}
}
$datag.Set_Item( $_.arch, $auxParent )
}
The latter code snippet is roughly equivalent to
$datag =
#{
'32' = #{ '6.1' = #{ 'KB1'='http://kb1';
'KB5'='http://kb5';
'KB7'='http://kb7' };
'6.2' = #{ 'KB2'='http://kb2' }
};
'64' = #{ '6.1' = #{ 'KB3'='http://kb3' };
'6.2' = #{ 'KB4'='http://kb4' }
}
}
Why is this coming up blank for all machines in the passwordlastset attribute, when I export it in the csv file? Everything else works perfectly.
$Searcher = New-ObjectSystem.DirectoryServices.DirectorySearcher([ADSI]"LDAP://dc=amers,dc=jhe,dc=domain,dc=com")
$Searcher.Filter = "(&(objectCategory=computer)(objectClass=computer)(!UserAccountControl:1.2.840.113556.1.4.803:=2)(operatingSystem=Windows XP*))"
$Searcher.PageSize = 100000
$results = $Searcher.Findall()
$results | ForEach-Object { $_.GetDirectoryEntry() } |
select #{ n = 'CN'; e = { ($_.CN) } },
#{ n = 'DistinguishedName'; e = { $_.DistinguishedName } },
#{ n = 'extensionattribute7'; e = { $_.extensionattribute7 } },
#{ n = 'LastLogon'; e = { [DateTime]::FromFileTime($_.PasswordLastSet) } },
#{ n = 'OperatingSystem'; e = { $_.OperatingSystem } } |
Export-Csv 'C:\temp\WindowsXP_Only.csv' -NoType -Force
By default not all properties are returned, so you need to specify the additional properties you want.
Also, if you're looking for the last logon date (per your output), you should be using lastLogonTimestamp and not PasswordLastSet.
Here's an example using Get-ADComputer, which I greatly prefer over using older methods of searching AD. Just add your Export-CSV when you're happy with the results.
$results = get-adcomputer -Filter "operatingSystem -like 'Windows XP*'" -properties cn,lastlogontimestamp,operatingsystem,extensionattribute7,PasswordLastSet -searchbase "dc=amers,dc=jhe,dc=domain,dc=com";
$results |
select #{ n = 'CN'; e = { ($_.cn) } },
#{ n = 'DistinguishedName'; e = { $_.DistinguishedName } },
#{ n = 'extensionattribute7'; e = { $_.extensionattribute7 } },
#{ n = 'LastLogon'; e = { [DateTime]::FromFileTime($_.lastLogonTimestamp) } },
#{ n = 'PasswordLastSet'; e = { [DateTime]::FromFileTime($_.PasswordLastSet) } },
#{ n = 'OperatingSystem'; e = { $_.OperatingSystem } }
You might also find this script useful