Powershell extraction memory consuming - powershell

Question

I have worked on a little project which is to extract some information out of a file server. To perform that projet I have created a script that outputs all the information in a .csv file. The problem is that Powershell eats up all my computer's RAM during the process because there is like hundreds Gb of data to parse.
Hereunder is my script.
$folder = Get-ChildItem -Recurse 'Complete_Path' | select FullName, #{Name="Owner";Expression={(Get-Acl $_.FullName).Owner}}, CreationTime, LastWriteTime, LastAccessTime, PSIsContainer | sort FullName
$output = #()
$folder | foreach {
$type =
if ($_.PSIsContainer -eq "True") {
Write-Output "Folder"
}
else {
Write-Output "File"
}
$size =
if ($_.PSIsContainer -eq "True") {
Get-ChildItem -Recurse $_.FullName | measure -Property Length -Sum -ErrorAction SilentlyContinue | select -ExpandProperty Sum
}
else {
Get-Item $_.FullName | measure -Property Length -Sum -ErrorAction SilentlyContinue | select -ExpandProperty Sum
}
$hash = #{
FullName = $_.FullName
Owner = $_.Owner
CreationTime = $_.CreationTime
LastWriteTime = $_.LastWriteTime
LastAccessTime = $_.LastAccessTime
Type = $type
'Size in MB' = [math]::Round($($size/1Mb),2)
}
$output += New-Object PSObject -Property $hash
}
$output | select FullName, Owner, CreationTime, LastWriteTime, LastAccessTime, Type, 'Size in MB' | Export-Csv C:\myDOCS.csv -Delimiter ";" -NoTypeInformation -Encoding UTF8
Have you guys any idea how can I get the job done faster and less ram consuming? It can take days to get the extraction.
Thank you in advance.

Answer 1

Replace your Powershell array $output=#() with A .Net PSObject list $output = [System.Collections.Generic.List[psobject]]::new() and use the .Add method of that object to add your items.
For small list, you won't notice but using Powershell array and += operator is a big performance sink. Each time you do +=, array is recreated entirely with one more item.
Include Length in your initial Get-ChildItem statement. Later on, you can measure the sum without going through Get-ChildItem again all the time
Pipeline play nice on memory but slower overall. I tend to prefer not using the pipeline when performance become an issue.
Something like that should already be significantly faster
$folder = Get-ChildItem -Recurse "$($env:USERPROFILE)\Downloads" | select FullName, #{Name = "Owner"; Expression = { (Get-Acl $_.FullName).Owner } }, CreationTime, LastWriteTime, LastAccessTime, PSIsContainer, Length | sort FullName
$output = [System.Collections.Generic.List[psobject]]::new()
foreach ($Item in $folder) {
if ($Item.PSIsContainer) {
$Type = 'Folder'
$size = $folder.Where( { $_.FullName -like $item.FullName }).FullName | measure -Property Length -Sum -ErrorAction SilentlyContinue | select -ExpandProperty Sum
}
else {
$Type = 'File'
$size = $Item.Length
}
$size = [math]::Round($($size / 1Mb), 2)
$hash = #{
FullName = $Item.FullName
Owner = $Item.Owner
CreationTime = $Item.CreationTime
LastWriteTime = $Item.LastWriteTime
LastAccessTime = $Item.LastAccessTime
Type = $Type
'Size in MB' = $size
}
[void]($output.Add((New-Object PSObject -Property $hash)))
}
$output | select FullName, Owner, CreationTime, LastWriteTime, LastAccessTime, Type, 'Size in MB' | Export-Csv C:\myDOCS.csv -Delimiter ";" -NoTypeInformation -Encoding UTF8
You could still improve on the size calculation so the deepest of folders size is calculated first, then parent folder can grab the value and sum up the children folder instead of recalculating the files
Another thought would be to not do the Get-ACl immediately (I suspect this one is slow to perform ) and get your items, do the rest, then parallelize the Get-ACL so you can get the values on a number of parallel threads and add the the value as you get it to your list.
Think about testing your code on smaller batches and use the Measure-Command to determine where are the slowest operation in your code.
I recommend you taking a look at some more advanced topic on the subject.
Here's a good article to get your started : Slow Code: Top 5 ways to make your Powershell scripts run faster

Answer 2

Is this better with the whole thing in one pipeline?
Get-ChildItem -Recurse |
select FullName, #{Name="Owner";Expression={(Get-Acl $_.FullName).Owner}},
CreationTime, LastWriteTime, LastAccessTime, PSIsContainer | sort FullName |
foreach {
$type =
if ($_.PSIsContainer -eq "True") {
Write-Output "Folder"
}
else {
Write-Output "File"
}
$size =
if ($_.PSIsContainer -eq "True") {
Get-ChildItem -Recurse $_.FullName |
measure -Property Length -Sum -ErrorAction SilentlyContinue |
select -ExpandProperty Sum
}
else {
Get-Item $_.FullName |
measure -Property Length -Sum -ErrorAction SilentlyContinue |
select -ExpandProperty Sum
}
$hash = #{
FullName = $_.FullName
Owner = $_.Owner
CreationTime = $_.CreationTime
LastWriteTime = $_.LastWriteTime
LastAccessTime = $_.LastAccessTime
Type = $type
'Size in MB' = [math]::Round($($size/1Mb),2)
}
New-Object PSObject -Property $hash
} | select FullName, Owner, CreationTime, LastWriteTime, LastAccessTime,
Type, 'Size in MB' |
Export-Csv myDOCS.csv -Delimiter ";" -NoTypeInformation -Encoding UTF8