Efficient way to find and replace many strings in a large text file - powershell

Question

The Text file contains a software output on a time domain analysis. 10800 seconds simulation and 50 nodes being considered. We have 540,000 strings to be replaced in 540 MB text file with 4.5 million lines.
Which is currently projected to take more than 4 days. Something is going wrong. Don't know what. Please suggest me a better efficient approach.
Below is the function which does the find and replace.
To replace the string the script goes through the original text file line by line at the same time it generates a duplicate file with replaced strings. So another 540 MB file with 4.5 million lines will be generated at the end of the script.
Function ReplaceStringsInTextFile
{
$OutputfilebyLine = New-Object -typename System.IO.StreamReader $inputFilePathFull
$uPreviousValue = 0
$time = 60
$u = 0; $LastStringWithoutFindResult = 0
$lineNumber = 0
while ($null -ne ($line = $OutputfilebyLine.ReadLine())) {
$lineNumber = $lineNumber + 1
if ($time -le $SimulationTimeSeconds) # time simulation start and end checks
{
# 10800 strings corresponds to one node
# there are 50 nodes.. Thus 540,000 values
# $StringsToFindFileContent contains strings to find 540,000 strings
# $StringsToReplaceFileContent contains strings to replace 540,000 strings
$StringToFindLineSplit = -split $StringsToFindFileContent[$time-60]
$StringToReplaceLineSplit = -split $StringsToReplaceFileContent[$time-60]
if($u -le $NumberofNodes-1)
{
$theNode = $Nodes_Ar[$u]
$StringToFindvalue = $StringToFindLineSplit[$u]
$StringToReplacevalue = $StringToReplaceLineSplit[$u]
if (($line -match $theNode) -And ($line -match $StringToFindvalue)){
$replacedLine = $line.replace($StringToFindvalue,$StringToReplacevalue)
add-content -path $WriteOutputfilePathFull -value "$replacedLine"
$uPreviousValue = $u
$checkLineMatched = 1
if (($line -match $LastNodeInArray)) {
$time = $time + 1
$LastStringWithoutFindResult = 0
}
} elseIf (($line -match $LastNodeInArray) -And ($checkLineMatched -eq 0)) {
$LastStringWithoutFindResult = $LastStringWithoutFindResult + 1
} else {
#"Printing lines without match"
add-content -path $WriteOutputfilePathFull -value "$line"
$checkLineMatched = 0
}
}
if ($checkLineMatched -eq 1) {
# incrementing the value of node index to next one in case the last node is found
$u = $uPreviousValue + 1
if ($u -eq $Nodes_Ar.count) {
$u = 0
$timeElapsed = (get-date -displayhint time) - $startTime
"$($timeElapsed.Hours) Hours $($timeElapsed.Minutes) Minutes $($timeElapsed.Seconds) Seconds"
}
}
}
# Checking if the search has failed for more than three cycles
if ($LastStringWithoutFindResult -ge 5) { # showing error dialog in case of search error
[System.Windows.Forms.MessageBox]::Show("StringToFind Search Fail. Please correct StringToFind values. Aborting now" , "Status" , 0)
$OutputfilebyLine.close()
}
}
$OutputfilebyLine.close()
}
The above function is the last part of the script. Which is taking the most time.
I had run the script in under 10 hours 1 year ago.
Update The script sped up running after 4 hours and suddenly time to complete projection reduced from 4 days to under 3 hours. The script finished running in 7 hours and 9 minutes. However i am not sure what made the sudden change in speed other than asking the question on stack overflow :)

Answer 1

As per the suggestion by https://stackoverflow.com/users/478656/tessellatingheckler
I have avoided writing one line at a time using
add-content -path $WriteOutputfilePathFull -value "$replacedLine"
Instead i am now writing ten thousand lines at a time using add-content
$tenThousandLines = $tenThousandLines + "`n" + $replacedLine
And at the appropriate time I am using add-content to write 10,000 lines at one go like below. The if block follows my methods logic
if ($lineNumber/10000 -gt $tenThousandCounter){
clear-host
add-content -path $WriteOffpipeOutputfilePathFull -value "$tenThousandLines"
$tenThousandLines = ""
$tenThousandCounter = $tenThousandCounter + 1
}
I have encountered system out of memmory exception error when trying to add 15,000 or 25,000 lines at a time. After using this the time required for the operation has reduced from 7 hours to 5 hours. And at another time to 2 hours and 36 minutes.