I am trying to convert a 80K row data csv with 30 columns to sorted and filtered CSV based on specific column data from orignal CSV.
For Example My Data is in below format:
PatchName MachineName IPAddress DefaultIPGateway Domain Name USERID UNKNOWN NOTAPPLICABLE INSTALLED APPLICABLE REBOOTREQUIRED FAILED
KB456982 XXX1002 xx.yy.65.148 xx.yy.64.1 XYZ.NET XYZ\ayzuser YES
KB589631 XXX1003 xx.yy.65.176 xx.yy.64.1 XYZ.NET XYZ\cdfuser YES
KB456982 ABC1004 xx.zz.83.56 xx.zz.83.1 XYZ.NET XYZ\mnguser YES
KB456982 8797XCV xx.yy.143.187 xx.yy.143.184 XYZ.NET WPX\abcuser YES
Here MachineName would be filtered to Uniq and PatchName would transpose to Last Columns headers with holding "UNKNOWN, NOAPPLICABLE, INSTALLED, FAILED, REBOOTREQUIRED columns Values if YES occurred -
Expected Result:
MachineName IPAddress DefaultIPGateway Domain Name USERID KB456982 KB589631
XXX1002 xx.yy.65.148 xx.yy.64.1 XYZ.NET XYZ\ayzuser UNKNOWN
XXX1003 xx.yy.65.176 xx.yy.64.1 XYZ.NET XYZ\cdfuser NOTAPPLICATBLE
ABC1004 xx.zz.83.56 xx.zz.83.1 XYZ.NET XYZ\mnguser UNKNOWN
8797XCV xx.yy.143.187 xx.yy.143.184 XYZ.NET WPX\abcuser FAILED
Looking for help to achieve this, so far I am able to transpose PathcName rows to columns but not able to include all the columns along with and apply the condition. [It takes 40 Minutes to process this]
$b = #()
foreach ($Property in $a.MachineName | Select -Unique) {
$Props = [ordered]#{ MachineName = $Property }
foreach ($Server in $a.PatchName | Select -Unique){
$Value = ($a.where({ $_.PatchName -eq $Server -and $_.MachineName -eq $Property })).NOTAPPLICABALE
$Props += #{ $Server = $Value }
}
$b += New-Object -TypeName PSObject -Property $Props
}
This is what I came up with:
$data = Import-Csv -LiteralPath 'C:\path\to\data.csv'
$lookup = #{}
$allPatches = $data.PatchName | Select-Object -Unique
# Make 1 lookup entry for each computer, to keep the username and IP and so on.
# Add the patch details from the current row (might add more than one patch per computer)
foreach ($row in $data)
{
if (-not $lookup.ContainsKey($row.MachineName))
{
$lookup[$row.MachineName] = ($row | Select-Object -Property MachineName, IPAddress, DefaultIPGateway, DomainName, UserID)
}
$patchStatus = $row.psobject.properties |
Where-Object {
$_.name -in #('applicable', 'notapplicable', 'installed', 'rebootrequired', 'failed', 'unknown') -and
-not [string]::IsNullOrWhiteSpace($_.value)
} |
Select-Object -ExpandProperty Name
$lookup[$row.MachineName] | Add-Member -NotePropertyName $row.PatchName -NotePropertyValue $patchStatus
}
# Pull the computer details out of the lookup, and add all the remaining patches
# so they will convert to CSV properly, then export to CSV
$lookup.Values | ForEach-Object {
$computer = $_
foreach ($patch in $allPatches | where-object {$_ -notin $computer.psobject.properties.name})
{
$computer | Add-Member -NotePropertyName $patch -NotePropertyValue ''
}
$computer
} | Export-Csv -LiteralPath 'c:\path\to\output.csv' -NoTypeInformation
Related
I am trying to read group membership of computers from a particular OU and write to a CSV file. The input criteria for the group membership is like if the computer is part of say "admin" and i need the csv file in the below format
---------------------------------------------------------
Computer Group1 Group2 Group3 Group4
ABCD admin1 admin2 admin3 admin4
EFGH admin2 admin3
XYZZ admin1 admin4
--------------------------------------------------------------
but end up like this.
---------------------------------------------------------
Computer Group1 Group2 Group3 Group4
ABCD admin1 admin2 admin3 admin4
EFGH admin2 admin3
XYZZ admin1 admin4
--------------------------------------------------------------
The code is like this
$All_computer = Get-ADComputer -Filter * -Property * -SearchBase $ou -Server $server | Select-object Name,DNSHostName,Description,memberof
$computerExport = $All_computer |
ForEach-Object {
$ComputerName = $_.Name
$Description = $_.description
$DNSHostname = $_.DNSHostName
$memberof = $_.memberof
$groups = $memberof.where{$_ -like "*$unput_group*"}
$Group_member = [pscustomobject]#{
Workstation = $ComputerName
Comp_Description = $Description
DNS_Hostname = $DNSHostname
}
$i = 0
foreach($group in $Groups)
{
$i++
$member = $group.split('=')[1].split(',')[0]
$Group_member | add-member -MemberType NoteProperty -Name "Group$i" -Value $member
}
$Group_member
}
}
$computerExport | export-csv .\Membership_status.csv -NoTypeInformation
What do i need to do to get the group membership to populate to proper column.
Well of course it does. I mean it's doing what you're asking it to do.
You're only adding the number of properties to the custom object that is found from the where object query. I am really struggling to understand what you're trying to do this for but I THINK what you really want is for each object to have all the possible properties but to have null values for those that don't match that particular computer or better yet to use a boolean.
So... Maybe like this:
[string]$GroupSearch = "admin"
$All_computer = Get-ADComputer -Filter * -Property DNSHostName, Description, memberof -SearchBase $ou -Server $server | Select-Object Name, DNSHostName, Description, memberof
$MatchedGroups = $All_Computer.MemberOf | Sort -Unique | ?{$_.Name -match $GroupSearch}
$computerExport = ForEach ($C in $All_computer) {
$Group_member = [pscustomobject]#{
Workstation = $($C.Name)
Comp_Description = $($C.Description)
DNS_Hostname = $($C.DNSHostName)
}
ForEach ($group in $MatchedGroups) {
[string]$GrpName = $($group.split('=')[1].split(',')[0])
If ($C.MemberOf -contains $group) {
$Group_member | Add-Member -MemberType NoteProperty -Name $GrpName -Value $true
} else {
$Group_member | Add-Member -MemberType NoteProperty -Name $GrpName -Value $false
}
}
$Group_member
}
$computerExport | Export-Csv .\Membership_status.csv -NoTypeInformation
If I understand the question, you need to get all computers from a certain OU that are member of group(s) with a similar partial name.
To do that, I would suggest creating an array of computer objects at first with a temporary extra property called 'Groups' in which the group names that match the partial name are stored.
Later, we'll put these in the correct order as new properties called 'Group1', 'Group2' etc.
# the partial groupname to search for
$unput_group = 'admin'
# Get-ADComputer by default already returns these properties:
# DistinguishedName, DNSHostName, Enabled, Name, ObjectClass, ObjectGUID, SamAccountName, SID, UserPrincipalName
# get an array of computer objects that are member of 'admin*' group(s) with the desired properties
# one extra temporary property is added which contains an array of 'admin*' group names
$All_computer = Get-ADComputer -Filter * -Property Description, MemberOf -SearchBase $ou -Server $server |
Where-Object { $_.MemberOf -match $unput_group} |
Select-Object #{Name = 'Workstation'; Expression = {$_.Name}},
#{Name = 'Comp_Description'; Expression = {$_.Description}},
#{Name = 'DNS_Hostname'; Expression = {$_.DNSHostName}},
#{Name = 'Groups'; Expression = { #($_.MemberOf |
Where-Object { $_ -match "CN=($unput_group[^,]+)" } |
ForEach-Object { $matches[1] }) }}
# get all the group names from the computers we have collected and sort unique
$All_Groups = $All_computer.Groups | Sort-Object -Unique
# build a lookup hashtable with property names ('Group1', 'Group2' etc)
$hash = [ordered]#{}
for ($i = 1; $i -le $All_Groups.Count; $i++) {
$hash["Group$i"] = $All_Groups[$i - 1]
}
# now loop through the collection and add the group properties in order
$result = foreach ($computer in $All_computer) {
foreach ($propertyName in $hash.Keys) {
$group = if ($computer.Groups -contains $hash[$propertyName]) { $hash[$propertyName] }
$computer | Add-Member -MemberType NoteProperty -Name $propertyName -Value $group
}
# output the updated object and remove the temporary 'Groups' property
$computer | Select-Object * -ExcludeProperty Groups
}
# finally, save the results to disk
$result | Export-Csv -Path .\Membership_status.csv -NoTypeInformation
I have 2 .csv files, they have matching columns, I am trying to compare the two and have a final output .csv that only has the differences
corpold.csv is a previous imported file.
corpnew.csv is the new import file.
I need to export a CSV that includes all items that are not in corpold.csv, only changed items that exist in both CSVs and exclude any rows that exist in corpold.csv but not in corpnew.csv.
$reference = Import-Csv -Path D:\corpold.csv
$lookup = $reference | Group-Object -AsHashTable -AsString -Property EMPID
$results = Import-Csv -Path D:\corpnew.csv | foreach {
$email = $_.EMAIL_ADDRESS
$status = $_.ACTIVE
$fs = $_.FIRST_NAME
$ls = $_.LAST_NAME
$id = $_.EMPID
$title = $_.JOB_TITLE
$code = $_.JOB_CODE
$type = $_.USER_TYPE
$designee = $_.DESIGNEE
$stores = $_.STORES
$hiredate = $_.HIRE_DATE
$dept = $_.DEPARTMENT
$grp = $_.GROUP
if ($lookup.ContainsKey($id)) {
# if exists in yesterdays file
# trying to figure out how to compare and only provide results into
# the Export-Csv that have changed while excluding any items in
# corpold that do not exist in corpnew
} else {
# if it does not exist update all fields
[PSCustomObject]#{
ACTIVE = $status
EMAIL_ADDRESS = $email
FIRST_NAME = $fs
LAST_NAME = $ls
EMPID = $id
JOB_TITLE = $title
JOB_CODE = $code
USER_TYPE = $type
DESIGNEE = $designee
STORES = $stores
HIRE_DATE = $hiredate
DEPARTMENT = $dept
GROUP = $grp
}
}
}
# Sample outputs
$results
$results | Export-Csv -Path D:\delta.csv -NoTypeInformation
There are two operations to be done here: find differences and compare objects which exist in both files.
Compare objects and find new/deleted entries
To compare object you can utilize Compare-Object cmdlet like this:
Compare-Object -ReferenceObject $reference -DifferenceObject $results -Property EMPID -IncludeEqual
This will give you the list of EMPID and SideIndicator showing whether object exist only in first (<=), only in second (=>) or in both (==). You can filter by SideIndicatorand then process it accordingly.
Alternative way is to use Where-Object like this:
$reference | Where-Object empid -NotIn $results.empid
$reference | Where-Object empid -In $results.empid
$results | Where-Object empid -NotIn $reference.empid
First one will give you entries only in first file, second one results existing in both, last one will give you entries only existing in second file.
Find edited entries
What you basically have to do is to iterate all the entries and then check if any of the columns has been changed. If yes, add it to $changedEntries.
Example of script:
$IDsInBoth = $results | Where-Object empid -In $reference.empid | Select-Object -ExpandProperty EMPID
$AllProperties = $results | Get-Member | Where-Object MemberType -eq "NoteProperty" | Select-Object -ExpandProperty Name
$changedEntries = #()
$IDsInBoth | ForEach-Object {
$changed = $false
$newEntry = $results | Where-Object EMPID -eq $_
$oldEntry = $reference | Where-Object EMPID -eq $_
foreach ($p in $AllProperties) {
if ($oldEntry."$p" -ne $newEntry."$p") {
$changed = $true
}
}
if ($changed) {
$changedEntries += $newEntry
}
}
I'm trying to create a PowerShell script that transpose a CSV file from column to rows.
I found examples of doing the opposite (converting row based CSV to column) but I found nothing on column to rows. My problem being that I don't know exactly how many column I'll have. I tried adapting the row to column to column to rows but unsuccessfully.
$a = Import-Csv "input.csv"
$a | FT -AutoSize
$b = #()
foreach ($Property in $a.Property | Select -Unique) {
$Props = [ordered]#{ Property = $Property }
foreach ($Server in $a.Server | Select -Unique){
$Value = ($a.where({ $_.Server -eq $Server -and
$_.Property -eq $Property })).Value
$Props += #{ $Server = $Value }
}
$b += New-Object -TypeName PSObject -Property $Props
}
$b | FT -AutoSize
$b | Out-GridView
$b | Export-Csv "output.csv" -NoTypeInformation
For example my CSV can look like this:
"ID","DATA1"
"12345","11111"
"54321","11111"
"23456","44444"
or this (number of column can vary):
"ID","DATA1","DATA2","DATA3"
"12345","11111","22222","33333"
"54321","11111",,
"23456","44444","55555",
and I would like the script to convert it like this:
"ID","DATA"
"12345","11111"
"12345","22222"
"12345","33333"
"54321","11111"
"23456","44444"
"23456","55555"
The trick is to query the members of the table to get the column names. Once you do that then the rest is straightforward:
function Flip-Table ($Table) {
Process {
$Row = $_
# Get all the columns names, excluding the ID field.
$Columns = ($Row | Get-Member -Type NoteProperty | Where-Object Name -ne ID).Name
foreach ($Column in $Columns) {
if ($Row.$Column) {
$Properties = [Ordered] #{
"ID" = $Row.ID
"DATA" = $Row.$Column
}
New-Object PSObject -Property $Properties
}
}
# Garbage collection won't kick in until the end of the script, so
# invoke it every 100 input rows.
$Count++;
if (($Count % 100) -eq 0) {
[System.GC]::GetTotalMemory('forceFullCollection') | out-null
}
}
}
Import-Csv input.csv | Flip-Table | Export-Csv -NoTypeInformation output.csv
Well, here is mine. I'm not as fancy as the rest:
$in = Get-Content input.csv | Select -Skip 1
$out = New-Object System.Collections.ArrayList
foreach($row in $in){
$parts = $row.Split(',')
$id = $parts[0]
foreach($data in $parts[1..$parts.Count]){
if($data -ne '' -AND $data -ne $null){
$temp = New-Object PSCustomObject -Property #{'ID' = $id;
'Data' = $data}
$out.Add($temp) | Out-Null
}
}
}
$out | Export-CSV output.csv -NoTypeInformation
You can do something like this
# Convert csv to object
$csv = ConvertFrom-Csv #"
"ID","DATA1","DATA2","DATA3"
"12345","11111","22222","33333"
"54321","11111",,
"23456","44444","55555"
"#
# Ignore common members and the ID property
$excludedMembers = #(
'GetHashCode',
'GetType',
'ToString',
'Equals',
'ID'
)
$results = #()
# Iterate around each csv row
foreach ($row in $csv) {
$members = $row | Get-Member
# Iterate around each member from the 'row object' apart from our
# exclusions and empty values
foreach ($member in $members |
Where { $excludedMembers -notcontains $_.Name -and $row.($_.Name)}) {
# add to array of objects
$results += #{ ID=$row.ID; DATA=$row.($member.Name)}
}
}
# Write the csv string
$outstring = "ID,DATA"
$results | foreach { $outstring += "`n$($_.ID),$($_.DATA)" }
# New csv object
$csv = $outstring | ConvertFrom-Csv
Probably not the most elegant solution, but should do what you need
I left some comments explaining what it does
If you only want to accept a limited number DATA columns (e.g. 5), you could do:
ForEach ($i in 1..5) {$CSV | ? {$_."Data$i"} | Select ID, #{N='Data'; E={$_."Data$i"}}}
And if you have a potential unlimited number of DATA columns:
ForEach ($Data in ($CSV | Select "Data*" -First 1).PSObject.Properties.Name) {
$CSV | ? {$_.$Data} | Select ID, #{N='Data'; E={$_.$Data}}
}
I am trying to select property from events I am able to create with script
$events = Get-WinEvent -FilterHashtable #{logname='Security'; ID=4728; } -MaxEvents 1
$event = $events
[xml]$eventXML = [xml]$Event.ToXml()
$eventXML.Event.EventData.Data
if I run this it outputs, I need to select targetname,targetusername,subjectusername, I am not able to get the output, anyhelp will be very much appreciated.
As Swonkie already pointed out, Data is an array, and the values you're looking for are actually XML nodes in that array, hence you can't easily select them like you would with the properties of an object.
I would probably create a hashtable for each Data array, filter the array for the nodes you want selected, then build a custom object from each hashtable.
$names = 'TargetName', 'TargetUserName', 'SubjectUserName'
$events | ForEach-Object {
([xml]$_.ToXml()).Event.EventData | ForEach-Object {
$props = #{}
$_.Data |
Where-Object { $names -contains $_.Name } |
ForEach-Object { $props[$_.Name] = $_.'#text' }
New-Object -Type PSObject -Property $props
}
}
$eventXML.Event.EventData.Data is an array.
$eventXML.Event.EventData.Data | where { $_.Name -eq 'SubjectUserName' } | select -ExpandProperty '#text'
Or for PowerShell 3 and up:
$eventXML.Event.EventData.Data | where Name -eq 'SubjectUserName' | select -ExpandProperty '#text'
I have a txt files (semicolon separated) containing over 3 million records where columns 1 to 4 have some general information. Columns 5 and 6 have detailed information. There can be up to 4 different detailed information for the same general information in columns 1 to 4.
My sample input:
Server;Owner;Company;Username;Property;Value
Srv1;Dave;Sandbox;kwus91;Memory;4GB
Srv1;Dave;Sandbox;kwus91;Processes;135
Srv1;Dave;Sandbox;kwus91;Storage;120GB
Srv1;Dave;Sandbox;kwus91;Variant;16
Srv2;Pete;GWZ;aiwq71;Memory;8GB
Srv2;Pete;GWZ;aiwq71;Processes;234
Srv3;Micael;P12;mxuq01;Memory;16GB
Srv3;Micael;P12;mxuq01;Processes;239
Srv3;Micael;P12;mxuq01;Storage;160GB
Srv4;Stefan;MTC;spq61ep;Storage;120GB
Desired output:
Server;Owner;Company;Username;Memory;Processes;Storage;Variant
Srv1;Dave;Sandbox;kwus91;4GB;135;120GB;16
Srv2;Pete;GWZ;aiwq71;8GB;234;;
Srv3;Micael;P12;mxuq01;16GB;239;160GB;
Srv4;Stefan;MTC;spq61ep;;;120GB;
If a values doesn't exist for general information (Columns 1-4) it has to stay blank.
My current code:
$a = Import-csv .\Input.txt -Delimiter ";"
$a | FT -AutoSize
$b = #()
foreach ($Server in $a.Server | Select -Unique) {
$Props = [ordered]#{ Server = $Server }
$Owner = ($a.where({ $_.Server -eq $Server})).Owner | Select -Unique
$Company = ($a.where({ $_.Server -eq $Server})).Company | Select -Unique
$Username = ($a.where({ $_.Server -eq $Server})).Username | Select -Unique
$Props += #{Owner = $Owner}
$Props += #{Company = $Company}
$Props += #{Username = $Username}
foreach ($Property in $a.Property | Select -Unique){
$Value = ($a.where({ $_.Server -eq $Server -and
$_.Property -eq $Property})).Value
$Props += #{ $Property = $Value }
}
$b += New-Object -TypeName PSObject -Property $Props
}
$b | FT -AutoSize
$b | Export-Csv .\Output.txt -NoTypeInformation -Delimiter ";"
After a lot of trying and getting errors: My script works.
But it takes a lot of time.
Is there a possibility to make performance better for around 3 Million lines in txt file? I'm calculating with more or less 2.5 Million unique values for $Server.
I'm running Windows 7 64bit with PowerShell 4.0.
try Something like this:
#Import Data and create empty columns
$List=import-csv "C:\temp\file.csv" -Delimiter ";"
#get all properties name with value not empty
$ListProperty=($List | where Value -ne '' | select property -Unique).Property
#group by server
$Groups=$List | group Server
#loop every rows and store data by group and Property Name
$List | %{
$Current=$_
#Take value not empty and group by Property Name
$Group=($Groups | where Name -eq $Current.Server).Group | where Value -ne '' | group Property
#Add all property and first value not empty
$ListProperty | %{
$PropertyName=$_
$PropertyValue=($Group | where Name -eq $PropertyName | select -first 1).Group.Value
$Current | Add-Member -Name $PropertyName -MemberType NoteProperty -Value $PropertyValue
}
$Current
} | select * -ExcludeProperty Property, Value -unique | export-csv "c:\temp\result.csv" -notype -Delimiter ";"