Powershell script very slow when running with very big array of data - powershell

Question

ive been dabbling with powershell for a while now and ive been trying to modify some data in an array.
Problem is that my source array is very large and this script takes hours to run. Maybe someone can help my optimize my script.
With a small source array the script runs just fine btw.
$array_metric_hour = #()
$array_metric_hour =
foreach ($resource in $resources) {
Write-Progress -Id 0 "Step $resource"
foreach ($hour in $Time_Array) {
Write-Progress -Id 1 -ParentId 0 "Step $resource - Substep" ($hour.timestamp+":00")
[pscustomobject] #{
resourceID = $resource
resourceName = $array_bill.resources.($resource).name
time = $hour.timestamp+":00"
Poweredon = ((($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp}).poweredon | Measure-Object -Maximum).Maximum)
#Cpu_On = if (($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp -and $_.poweredOn -eq "0,0"}).poweredon) {0} else {(($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp -and $_.poweredOn -ne "0,0"}).provisionedCpu | Measure-Object -Maximum).Maximum}
Mem_GB_On = if (($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp -and $_.poweredOn -eq "0,0"}).poweredon) {0} else {(($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp -and $_.poweredOn -ne "0,0"}).provisionedMem_GB | Measure-Object -Maximum).Maximum}
hardware_Diskspace_GB = ((($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp}).hardware_Diskspace_GB | Measure-Object -Maximum).Maximum)
#used_Diskspace_GB = ((($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp}).used_Diskspace_GB | Measure-Object -Maximum).Maximum)
}
}
}
Some extra information that is required:
$Time_array has every full hour in a month, so 745 values in this case.
$array_combined exist of 98131 lines (5 minute interval with metrics during a month.
this array has the folowing items per interval.
resourceID
resourceName
timestamps
human_timestamp
hour
date
poweredOn
provisionedMem_GB
hardware_Diskspace_GB
used_Diskspace_GB

Answer 1

thanks for all the comments, next time ill try to supply all and correct information.
The suggestions of creating an extra filter was the winner for me. the scripts is 50 times faster in the current state and for now quick enough.
Added $filter1 and $filter2.
$array_metric_hour = #()
$array_metric_hour =
foreach ($resource in $resources) {
$filter1 = $Array_combined | Where-Object {$_.resourceID -eq $resource}
Write-Progress -Id 0 "Step $resource"
foreach ($hour in $Time_Array) {
$filter2 = $filter1 | Where-Object {$_.hour -eq $hour.timestamp}
Write-Progress -Id 1 -ParentId 0 "Step $resource - Substep" ($hour.timestamp+":00")
[pscustomobject] #{
resourceID = $resource
resourceName = $array_bill.resources.($resource).name
time = $hour.timestamp+":00"
Poweredon = if ($filter2 | where poweredOn -eq "0,0") {"0"} else {($filter2.poweredOn | Measure-Object -Maximum).Maximum}
Mem_GB_On = if ($filter2 | where poweredOn -eq "0,0") {0} else {(($filter2).provisionedMem_GB | Measure-Object -Average).Average}
hardware_Diskspace_GB = ((($filter2).hardware_Diskspace_GB | Measure-Object -Average).Average)
}
}
}

Answer 2

Since you're still not have provided much more of your code (e.g. where does "Array_combined" come from?), here are some important notes:
Don't use "Write-Progress" on every iteration! It has a very very huge impact on performace when using PS <=5.1, 6 and 7.
With the current "7.1" build I am using it works like a charm ("7.1.0-preview.7"). Have to look when they fixed it.
Avoid pipe'ing as much as you can when you want to have the best performance. Streaming data from one command to the other is compared to statements like "foreach {}" (NOT "Foreach-Object"!) really bad.
Here is an example for your template, even when there are some important steps missing:
# Progress bar definition
$progressActivity1 = 'Processing items'
$progressCounter1 = -1
$progressMax1 = #($resources).Count
$progressInterval1 = [math]::Ceiling($progressMax1 * 0.1) # each 10%
$progressId1 = 1
$progressParentId1 = 0
# *** use a list if your script adds objects several times.
# *** Note: "Arrays" are immutable and will be re-created each time you add something
$array_metric_hour = [System.Collections.Generic.List[psobject]]::new()
# *** good approach to add the result of a forEach-statement directly to variable. Performance is similar compared to adding objects to a list.
$array_metric_hour = foreach ($resource in $resources) {
# Progress bar counter & drawing (each 10%)
$progressCounter1++
If ($progressCounter1 % $progressInterval1 -eq 0) {
Write-Progress -Activity $progressActivity1 -PercentComplete($progressCounter1 / $progressMax1 * 100) -Id $progressId1 -ParentId $progressParentId1
}
# *** "Array_combined" is unknwon.... but according to the usage:
# !!! try to create a dictionary/hashtable of "Array_combined" with "resourceID" as key.
# !!! hash tables/dictionaries are much faster to access a particular item than arrays
# !!! access would be: $filter1 = $Array_combined[$resource]
$filter1 = $Array_combined | Where-Object { $_.resourceID -eq $resource }
# Progress bar definition
$progressActivity2 = 'Processing items'
$progressCounter2 = -1
$progressMax2 = #($Time_Array).Count
$progressInterval2 = [math]::Ceiling($progressMax2 * 0.1) # each 10%
$progressId2 = 2
$progressParentId2 = $progressId1
foreach ($hour in $Time_Array) {
# ??? don't know what $filter1 is about ...
# !!! replace that; use a hastable/dictionary
# !!! alternatively: use "foreach"-statement OR method ".where{}" which was introduced in PS 4.0
$filter2 = $filter1 | Where-Object { $_.hour -eq $hour.timestamp }
# Progress bar counter & drawing (each 10%)
$progressCounter2++
If ($progressCounter2 % $progressInterval2 -eq 0) {
Write-Progress -Activity $progressActivity2 -PercentComplete($progressCounter2 / $progressMax2 * 100) -Id $progressId2 -ParentId $progressParentId2
}
[pscustomobject] #{
resourceID = $resource
resourceName = $array_bill.resources.($resource).name
time = $hour.timestamp + ':00'
# ??? "Where-Object" could be replaced ... but don't know the background or data ....
# !!! replace "Measure-Object" with "[Linq.Enumerable]" methods if possible
Poweredon = if ($filter2 | Where-Object poweredOn -EQ '0,0') { '0' } else { ($filter2.poweredOn | Measure-Object -Maximum).Maximum }
# !!! same as above
Mem_GB_On = if ($filter2 | Where-Object poweredOn -EQ '0,0') { 0 } else { (($filter2).provisionedMem_GB | Measure-Object -Average).Average }
# !!! same as above
hardware_Diskspace_GB = ((($filter2).hardware_Diskspace_GB | Measure-Object -Average).Average)
}
}
}
# Progress completed
Write-Progress -Activity $progressActivity -Completed -Id $progressId1
Write-Progress -Activity $progressActivity -Completed -Id $progressId2