Synchronising PowerShell background jobs that use Add-Content - powershell

Question

I have a small PowerShell program that starts a few threads to do parallel calculations and then when they are finished they append a line with the results to a text file and proceed to do some more. This worked fine in development and testing, but occasionally in production it hangs, and it seems the file is "jammed open". I have the writes wrapped in "try" blocks, but that does not help. I have written a toy application to illustrate the problem, it hangs after about 10-15 minutes usually (and writing about 3000 lines).
It seems to me I would have been better off with a Python solution using mutexs or something, but I am pretty far down this road now. Looking for ideas how I can easily fix this. I really thought Add-Content would have been atomic...
Parentjob.ps1
# Start a bunch of jobs
$curdir = "c:\transfer\filecollide"
$tokens = "tok00","tok01","tok02",
"tok03","tok04","tok05",
"tok06","tok07","tok08"
$jobs = #()
foreach ($tok in $tokens)
{
$job = Start-Job -FilePath ".\childjob.ps1" -ArgumentList "${curdir}",$tok,2,1000
Start-Sleep -s 3 # stagger things a bit
Write-Output " Starting:${tok} job"
$jobs += ,$job
}
foreach ($job in $jobs)
{
wait-job $job
$out = receive-job $job
Write-Output($out)
}
childjob.ps1
param(
[string]$curdir = ".",
[string]$tok = "tok?",
[int]$interval = 10,
[int]$ntodo = 1
)
$nwritefails = 0
$nwritesuccess = 0
$nwrite2fails = 0
function singleLine
{
param(
[string]$tok,
[string]$fileappendout = "",
[int]$timeout = 3
)
$curdatetime = (Get-Date)
$sout = "${curdatetime},${tok},${global:nwritesuccess},${global:nwritefails},${global:nwrite2fails}"
$global:nwritesuccess++
try
{
Add-Content -Path $fileappendout -Value "${sout}"
}
catch
{
$global:nwritefails++
try
{
Start-Sleep -s 1
Add-Content -Path $fileappendout -Value "${sout}"
}
catch
{
$global:nwrite2fails++
Write-Output "Failed to write to ${fileappendout}"
}
}
}
Write-Output "Starting to process ${tok}"
#Start of main code
cd "${curdir}"
$ndone = 0
while ($true)
{
singleLine $tok "outfile.txt"
$ndone++
if ($ndone -gt $ntodo){ break }
Start-Sleep -s $interval
}
Write-Output "Successful ${tok} appends:${nwritesuccess} failed:${nwritefails} failed2:${nwrite2fails}"

Answer 1

Why not have the jobs write the results to the output stream, and use Receive-Job in the main thread to collect the results and update the file? You can do this while the jobs are still running. What you're writing to the out stream now looks like it might be more appropriately written to the Progress stream.