optimize full path extraction of 1 million files +/- and iteration from file - powershell

Question

I am a programming enthusiast and novice, I am using Powershell to try to solve the following need:
I need to extract the full path of files with extension .img. inside a folder with +/- 900 thousand folders and +/- million files. -/+ 900,000 img files.
Each img file must be processed in an exe. that is read from a file.
Which is better to store the result of the GetChildItem in a variable or a file?
I would greatly appreciate your guidance and support to optimize and / or find the best way to speed up processes vs. resource consumption.
Thank you un advance!!
This is the code I am currently using:
$PSDefaultParameterValues['*:Encoding'] = 'Ascii'
$host.ui.RawUI.WindowTitle = “DICOM IMPORT IN PROGRESS”
#region SET WINDOW FIXED WIDTH
$pshost = get-host
$pswindow = $pshost.ui.rawui
$newsize = $pswindow.buffersize
$newsize.height = 3000
$newsize.width = 150
$pswindow.buffersize = $newsize
$newsize = $pswindow.windowsize
$newsize.height = 50
$newsize.width = 150
$pswindow.windowsize = $newsize
#endregion
#
$out = ("$pwd\log_{0:yyyyMMdd_HH.mm.ss}_import.txt" -f (Get-Date))
cls
"`n" | tee -FilePath $out -Append
"*****************" | tee -FilePath $out -Append
"**IMPORT SCRIPT**" | tee -FilePath $out -Append
"*****************" | tee -FilePath $out -Append
"`n" | tee -FilePath $out -Append
#
# SET SEARCH FOLDERS #
"Working Folder" | tee -FilePath $out -Append
$path1 = Read-Host "Enter folder location" | tee -FilePath $out -Append
"`n" | tee -FilePath $out -Append
#
#
# SET & SHOW HOSTNAME
"SERVER NAME" | tee -FilePath $out -Append
$ht = hostname | tee -FilePath $out -Append
Write-Host $ht
Start-Sleep -Seconds 3
"`n" | tee -FilePath $out -Append
#
#
# GET FILES
"`n" | tee -FilePath $out -Append
#"SEARCHING IMG FILES, PLEASE WAIT..." | tee -FilePath $out -Append
$files = $path1 | Get-ChildItem -recurse -file -filter *.img | ForEach-Object { $_.FullName }
# SHOW Get-ChildItem PROCESS ON CONSOLE
Out-host -InputObject $files
"`n" | tee -FilePath $out -Append
Write-Output ($files | Measure).Count "IMG FILES FOUND TO PUSH" | tee -FilePath $out -Append
# DUMP Get-ChildIte into a file
$files > $pwd\pf
Start-Sleep -Seconds 5
# TIMESTAMP
"`n" | tee -FilePath $out -Append
"IMPORT START" | tee -FilePath $out -Append
("{0:yyyy/MM/dd HH:mm:ss}" -f (Get-Date)) | tee -FilePath $out -Append
"********************************" | tee -FilePath $out -Append
"`n" | tee -FilePath $out -Append
#
#
#SET TOOL
$ir = $Env:folder_tool
$pt = "utils\tool.exe"
#
#PROCESSING FILES
$n = 1
$pe = foreach ($file in Get-Content $pwd\pf ) {
$tb = (Get-Date -f HH:mm:ss) | tee -FilePath $out -Append
$fp = "$n. $file" | tee -FilePath $out -Append
#
$ep = & $ir$pt -c $ht"FIR" -i $file | tee -FilePath $out -Append
$as = "`n" | tee -FilePath $out -Append
# PRINT CONSOLE IMG FILES PROCESS
Write-Host $tb
Write-Host $fp
Out-host -InputObject $ep
Write-Host $as
$n++
}
#
#TIMESTAMP
"********************************" | tee -FilePath $out -Append
"IMPORT END" | tee -FilePath $out -Append
("{0:yyyy/MM/dd HH:mm:ss}" -f (Get-Date)) | tee -FilePath $out -Append
"`n" | tee -FilePath $out -Append

Answer 1

Try using parallel with PoshRSJob.
Replace Start-Process in Process-File with your code and note that there is no access to console. Process-File must return string.
Adjust $JobCount and $inData.
The main idea is to load all file list into ConcurrentQueue, start 20 background jobs and wait them to exit. Each job will take value from queue and pass to Process-File, then repeat until queue is empty.
NOTE: If you stop script, RS Jobs will continue to run until they finished or powershell closed. Use Get-RSJob | Stop-RSJob and Get-RSJob | Remove-RSJob to stop background work
Import-Module PoshRSJob
Function Process-File
{
Param(
[String]$FilePath
)
$process = Start-Process -FilePath 'ping.exe' -ArgumentList '-n 5 127.0.0.1' -PassThru -WindowStyle Hidden
$process.WaitForExit();
return "Processed $FilePath"
}
$JobCount = [Environment]::ProcessorCount - 2
$inData = [System.Collections.Concurrent.ConcurrentQueue[string]]::new(
[System.IO.Directory]::EnumerateFiles('S:\SCRIPTS\FileTest', '*.img')
)
$JobScript = [scriptblock]{
$inQueue = [System.Collections.Concurrent.ConcurrentQueue[string]]$args[0]
$outBag = [System.Collections.Concurrent.ConcurrentBag[string]]$args[1]
$currentItem = $null
while($inQueue.TryDequeue([ref] $currentItem) -eq $true)
{
try
{
# Add result to OutBag
$result = Process-File -FilePath $currentItem -EA Stop
$outBag.Add( $result )
}
catch
{
# Catch error
Write-Output $_.Exception.ToString()
}
}
}
$resultData = [System.Collections.Concurrent.ConcurrentBag[string]]::new()
$i_cur = $inData.Count
$i_max = $i_cur
# Start jobs
$jobs = #(1..$JobCount) | % { Start-RSJob -ScriptBlock $JobScript -ArgumentList #($inData, $resultData) -FunctionsToImport #('Process-File') }
# Wait queue to empty
while($i_cur -gt 0)
{
Write-Progress -Activity 'Doing job' -Status "$($i_cur) left of $($i_max)" -PercentComplete (100 - ($i_cur / $i_max * 100))
Start-Sleep -Seconds 3 # Update frequency
$i_cur = $inData.Count
}
# Wait jobs to complete
$logs = $jobs | % { Wait-RSJob -Job $_ } | % { Receive-RSJob -Job $_ }
$jobs | % { Remove-RSJob -Job $_ }
$Global:resultData = $resultData
$Global:logs = $logs
$Global:resultData is array of Process-File return strings

Answer 2

Which is better to store the result of the GetChildItem in a variable or a file?
If you're hoping to keep memory utilization low, the best solution is to not store them at all - simply consume the output from Get-ChildItem directly:
$pe = Get-ChildItem -Recurse -File -filter *.img |ForEach-Object {
$file = $_.FullName
$tb = (Get-Date -f HH:mm:ss) | tee -FilePath $out -Append
$fp = "$n. $file" | tee -FilePath $out -Append
#
$ep = & $ir$pt -c $ht"FIR" -i $file | tee -FilePath $out -Append
$as = "`n" | tee -FilePath $out -Append
# PRINT CONSOLE IMG FILES PROCESS
Write-Host $tb
Write-Host $fp
Out-host -InputObject $ep
Write-Host $as
$n++
}