Speeding powershell script with large csv file up - powershell

Question

I am kinda new to powershell. Only be toying with it for a few days now and have written the below script to help search for multiple conditions in a csv file. I wrote something similar in VB and it takes 2 days to process the csv file. This powershell script takes about 6 hours to process 6500 machines and 9 policies. What i am trying to do is look in Policy.csv for a computer from computers.csv and a policy from a list and report if the computer has it or not.
Policy.csv has 6 fields in the table that need to be in the final report with an additional field added for status of the policy.
Computers.csv has 2 fields in the table that are the computer name and the OU it is in.
Packlist.txt is just the list of the applications(policies) that are being looked for.
Edit:
Samples of the csv files are as follows
Policy.csv
Device,Device DN,Group,Group DN,Policy Domain,Policy
Comp1,OU=Here,Domain_app,OU=Here,Ou=Apps,Server1,Policy1
Comp1,OU=Here,Domain_app,OU=Here,Ou=Apps,Server1,Policy2
Comp1,OU=Here,Domain_app,OU=Here,Ou=Apps,Server1,Policy3
Comp2,OU=There,Domain_app,OU=Here,Ou=Apps,Server1,Policy1
Comp2,OU=There,Domain_app,OU=Here,Ou=Apps,Server1,Policy2
Comp2,OU=There,Domain_app,OU=Here,Ou=Apps,Server1,Policy3
Computer.csv
Device,Device DN
Comp1,OU=Here
Comp2,OU=There
Comp3,OU=AnyWhere
Packlist.txt
Policy1
Policy3
Result.csv
Device,Device DN,Group,Group DN,Policy Domain,Policy,Status
Comp1,OU=Here,Domain_app,OU=Here,Ou=Apps,Server1,Policy1,Entitled
Comp2,OU=There,Domain_app,OU=Here,Ou=Apps,Server1,Policy1,Entitled
Comp3,OU=AnyWhere,,,,Policy1,Notentitled
Comp1,OU=Here,Domain_app,OU=Here,Ou=Apps,Server1,Policy3,Entitled
Comp2,OU=There,Domain_app,OU=Here,Ou=Apps,Server1,Policy3,Entitled
Comp3,OU=AnyWhere,,,,Policy3,Notentitled
The code is:
$data=import-csv -path c:\packagestatus\policy.csv
$computers=import-csv -path c:\packagestatus\computers.csv
$policylist= (Get-content -path c:\packagestatus\packlist.txt)
$policycount = $Policylist.count
$computercount = $computers.count
$Policycounter = 1
foreach ($policy in $policylist)
{
$Policy
$host.ui.RawUI.WindowTitle = "Processing $policyCounter of $policycount"
$Data_temp = $data|where-object{$_."Policy Instance" -eq $policy}
$computercounter = 1
foreach ($Computer in $computers)
{
$host.ui.RawUI.WindowTitle = "Processing Policy $policyCounter of $policycount and Computer $computercounter of $computercount"
if ($data_temp|Where-Object{$_.Device -eq $computer.device})
{
$result = $data_temp|where-object{$_.Device -eq $computer.device}|Where-Object{$_."Policy Instance" -eq $policy}
$result|Add-member -membertype Noteproperty -name Status -value Entitled
$result|export-csv -path c:\packagestatus\result1.csv -NoTypeInformation -append
}
Else
{
$result1 = New-Object -TypeName PSObject
$result1|add-member -membertype noteproperty -name "Device" -value $computer.device
$result1|add-member -membertype noteproperty -name "Device DN" -value $computer."Device DN"
$result1|add-member -membertype noteproperty -name "Group" -value $null
$result1|add-member -membertype noteproperty -name "Group DN" -value $null
$result1|add-member -membertype noteproperty -name "Policy Domain" -value $null
$result1|add-member -membertype noteproperty -name "Policy Instance" -value $Policy
$result1|add-member -membertype noteproperty -name "Status" -value NotEntitled
$result1|export-csv -path c:\packagestatus\result1.csv -force -NoTypeInformation -append
}
$computercounter++
}
$policycounter++
}
$host.ui.RawUI.WindowTitle = "Completed"

Answer 1

Ok, I think this should run faster for you...
I start by making a function to create objects for a given computer name, DN, and policy that it's missing. Then I load up the data from the files. Next I make a regex string to match against for all of the policies that are in the $Policylist. I do the same for the computer list. Then I filter down the $Data array for only entries that are in the policy list and also in the computer list.
This will, hopefully, limit the data that we're dealing with, and I think that will be faster in general. Next I group it by device, and for each grouping I look for any missing policies, and run that list against the function, and any matching policies I add the 'Status' property and output that entry. This is all collected in the $Results array.
Once we process all the computers we have records for, we look for the computers that weren't in the list, and create a NotEntitled object for all policies, and all all those to the $Results.
Lastly we sort and output $Results. It would be faster to not sort it I suppose, but probably harder to read as well. Here's the code, let me know how it works out for you:
Function NotEntitled{
[CmdletBinding()]
Param(
[String]$Device,
[String]$DeviceDN,
[String[]]$Pack
)
Process{
ForEach($Item in $Pack){
[PSCustomObject]#{
'Device' = $Device
'Device DN' = $DeviceDN
'Group' = $null
'Group DN' = $null
'Policy Domain' = $null
'Policy' = $Item
'Status' = 'NotEntitled'
}
}
}
}
$Data = import-csv -path c:\packagestatus\policy.csv
$Computers = import-csv -path c:\packagestatus\computers.csv
$Policylist = ,(Get-content -path c:\packagestatus\packlist.txt)
$PolicyReg = ($Policylist|%{[regex]::Escape($_)}) -join '|'
$ComputerReg = ($Computers.Device|%{[regex]::Escape($_)}) -join '|'
$FilteredData = $Data | Where{$_.Policy -match $PolicyReg -and $_.device -match $ComputerReg}
$Results = $FilteredData | Group Device | ForEach{
$Device = $_.group
$MissingPolicies = ,($Policylist | Where{$_ -notin $Device.Policy})
If(![string]::IsNullOrEmpty($MissingPolicies)){NotEntitled $Device[0].Device $Device[0].'Device DN' $MissingPolicies}
$Device | ForEach{Add-Member -InputObject $_ -NotePropertyName 'Status' -NotePropertyValue 'Entitled' -PassThru}
}
$CompList = $FilteredData | Select -ExpandProperty Device -Unique
$Results += $Computers | Where{$_.Device -notin $CompList} | ForEach{NotEntitled $_.Device $_.'Device DN' $Policylist}
$Results | Sort Device,Policy | Export-Csv c:\packagestatus\Result.csv -NoTypeInformation
I took your sample data, changed Comp1 Policy3 to Comp1 Policy4 (so that I could have a computer with only a partial policy set), and ran it and got these results output:
"Device","Device DN","Group","Group DN","Policy Domain","Policy","Status"
"Comp1","OU=Here","Domain_app","OU=Here,Ou=Apps","Server1","Policy1","Entitled"
"Comp1","OU=Here",,,,"Policy3","NotEntitled"
"Comp2","OU=There","Domain_app","OU=Here,Ou=Apps","Server1","Policy1","Entitled"
"Comp2","OU=There","Domain_app","OU=Here,Ou=Apps","Server1","Policy3","Entitled"
"Comp3","OU=AnyWhere",,,,"Policy1","NotEntitled"
"Comp3","OU=AnyWhere",,,,"Policy3","NotEntitled"