Optimize PowerShell Script for Parallel Processing - powershell

Question

I have a PowerShell script on Windows 2012 R2 that is used to export data from a database into a CSV file. I have a check in there to escape double quotes and text qualify necessary fields. I am looking to increase the performance of the script because it runs a little slower than I would like (exporting 20GB/20 million rows) and it only utilizes about 10% of the CPU. Does anyone have any suggestions for improvement?
$ConnectionString = "Data Source=server1; Database=Development; Trusted_Connection=True;";
$streamWriter = New-Object System.IO.StreamWriter ".\output.csv"
$sqlConn = New-Object System.Data.SqlClient.SqlConnection $ConnectionString
$sqlCmd = New-Object System.Data.SqlClient.SqlCommand
$sqlCmd.Connection = $sqlConn
$sqlCmd.CommandText = "SELECT * FROM Development.dbo.All_Opportunities WITH(NOLOCK)"
$sqlConn.Open();
$reader = $sqlCmd.ExecuteReader();
# Initialze the array the hold the values
$array = #()
for ( $i = 0 ; $i -lt $reader.FieldCount; $i++ )
{ $array += #($i) }
# Write Header
$streamWriter.Write($reader.GetName(0))
for ( $i = 1; $i -lt $reader.FieldCount; $i ++)
{ $streamWriter.Write($("," + $reader.GetName($i))) }
$streamWriter.WriteLine("") # Close the header line
while ($reader.Read())
{
# get the values;
$fieldCount = $reader.GetValues($array);
# add quotes if the values have a comma or double quote
for ($i = 0; $i -lt $array.Length; $i++)
{
if ($array[$i] -match "`"|,")
{
$array[$i] = '"' + $array[$i].Replace("`"", "`"`"").ToString() + '"';
}
}
$newRow = [string]::Join(",", $array);
$streamWriter.WriteLine($newRow)
}
$reader.Close();
$sqlConn.Close();
$streamWriter.Close();

Answer 1

You can split it to jobs and start them in the background
Try :
https://learn.microsoft.com/en-us/powershell/module/Microsoft.PowerShell.Core/Start-Job?view=powershell-5.1
Hope it help

Answer 2

So, I had a similar issue about a year ago, albeit with a slightly smaller table (~1 GB). Initially I just used:
Import-Module -Name SqlServer -Cmdlet Read-SqlTableData;
Read-SqlTableData -ServerInstance $SqlServer -DatabaseName $Database -SchemaName $Schema -TableName $Table |
Export-Csv -Path $OutputFilePath -NoTypeInformation
It worked, but it used a ton of memory (5+ GB out of 16 GB) and took about 7-9 minutes to run. All of these tests were with a spinning metal disk in a laptop, so bear that in mind with what follows as well.
I wondered if I could get it to go faster. I initially wrote it like this, which took about half the time, and about 100 MB of RAM:
$SqlServer = '...';
$SqlDatabase = '...';
$OutputFilePath = '...';
$SqlQuery = '...';
$SqlConnectionString = 'Data Source={0};Initial Catalog={1};Integrated Security=SSPI' -f $SqlServer, $SqlDatabase;
$Utf8NoBOM = New-Object -TypeName System.Text.UTF8Encoding -ArgumentList $false;
$StreamWriter = New-Object -TypeName System.IO.StreamWriter -ArgumentList $OutputFilePath, $Utf8NoBOM;
$CsvDelimiter = '"';
$CsvDelimiterEscape = '""';
$CsvSeparator = ',';
$SQLConnection = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $SqlConnectionString;
$SqlCommand = $SQLConnection.CreateCommand();
$SqlCommand.CommandText = $SqlQuery;
$SQLConnection.Open();
$SqlDataReader = $SqlCommand.ExecuteReader();
for ($Field = 0; $Field -lt $SqlDataReader.FieldCount; $Field++) {
if ($Field -gt 0) { $StreamWriter.Write($CsvSeparator); }
$StreamWriter.Write($CsvDelimiter);
$StreamWriter.Write($SqlDataReader.GetName($Field).Replace($CsvDelimiter, $CsvDelimiterEscape));
$StreamWriter.Write($CsvDelimiter);
}
$StreamWriter.WriteLine();
while ($SqlDataReader.Read()) {
for ($Field = 0; $Field -lt $SqlDataReader.FieldCount; $Field++) {
if ($Field -gt 0) { $StreamWriter.Write($CsvSeparator); }
$StreamWriter.Write($CsvDelimiter);
$StreamWriter.Write($SqlDataReader.GetValue($Field).ToString().Replace($CsvDelimiter, $CsvDelimiterEscape));
$StreamWriter.Write($CsvDelimiter);
}
$StreamWriter.WriteLine();
}
$SqlDataReader.Close();
$SqlDataReader.Dispose();
$SQLConnection.Close();
$SQLConnection.Dispose();
$StreamWriter.Close();
$StreamWriter.Dispose();
As you can see, it's basically the same pattern as yours.
I wondered if I could improve it even more, so I tried adding a StringBuilder since I'd had success doing that with other projects. I still have the code, but I found that it didn't work any faster, and took about 200 MB of RAM:
$SqlServer = '...'
$SqlDatabase = '...'
$OutputFilePath = '...'
$SqlQuery = '...';
$SqlConnectionString = 'Data Source={0};Initial Catalog={1};Integrated Security=SSPI' -f $SqlServer, $SqlDatabase;
$StringBuilderBufferSize = 50MB;
$StringBuilder = New-Object -TypeName System.Text.StringBuilder -ArgumentList ($StringBuilderBufferSize + 1MB);
$Utf8NoBOM = New-Object -TypeName System.Text.UTF8Encoding -ArgumentList $false;
$StreamWriter = New-Object -TypeName System.IO.StreamWriter -ArgumentList $OutputFilePath, $Utf8NoBOM;
$CsvDelimiter = '"';
$CsvDelimiterEscape = '""';
$CsvSeparator = ',';
$SQLConnection = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $SqlConnectionString;
$SqlCommand = $SQLConnection.CreateCommand();
$SqlCommand.CommandText = $SqlQuery;
$SQLConnection.Open();
$SqlDataReader = $SqlCommand.ExecuteReader();
for ($Field = 0; $Field -lt $SqlDataReader.FieldCount; $Field++) {
if ($Field -gt 0) { [void]$StringBuilder.Append($CsvSeparator); }
[void]$StringBuilder.Append($CsvDelimiter);
[void]$StringBuilder.Append($SqlDataReader.GetName($Field).Replace($CsvDelimiter, $CsvDelimiterEscape));
[void]$StringBuilder.Append($CsvDelimiter);
}
[void]$StringBuilder.AppendLine();
while ($SqlDataReader.Read()) {
for ($Field = 0; $Field -lt $SqlDataReader.FieldCount; $Field++) {
if ($Field -gt 0) { [void]$StringBuilder.Append($CsvSeparator); }
[void]$StringBuilder.Append($CsvDelimiter);
[void]$StringBuilder.Append($SqlDataReader.GetValue($Field).ToString().Replace($CsvDelimiter, $CsvDelimiterEscape));
[void]$StringBuilder.Append($CsvDelimiter);
}
[void]$StringBuilder.AppendLine();
if ($StringBuilder.Length -ge $StringBuilderBufferSize) {
$StreamWriter.Write($StringBuilder.ToString());
[void]$StringBuilder.Clear();
}
}
$SqlDataReader.Close();
$SqlDataReader.Dispose();
$SQLConnection.Close();
$SQLConnection.Dispose();
$StreamWriter.Write($StringBuilder.ToString());
$StreamWriter.Close();
$StreamWriter.Dispose();
No matter what I tried, I couldn't seem to get it under ~4:30 for about 1 GB of data.
I never considered parallelism because you'd have to break your query up into 4 equal pieces such that you could be certain that you'd get the complete data set, or otherwise do some pretty difficult process management with Runspace Pools. Even then you'd have to write to four different files, and eventually combine the files back together. Maybe it would work out, but it wasn't an interesting problem to me anymore at that point.
Eventually I just created a package with the Import Export Wizard, saved it as a package, and ran it with DTExec.exe. This takes about 45-60 seconds or so for 1 GB of data. The only drawbacks are that you need to specify the table when you build the package, it doesn't dynamically determine the columns, and it's an unreasonable pain to get the output file to be UTF8.
I did find that bcp.exe and sqlcmd.exe were faster. BCP was extremely fast, and took 20-30 seconds. However, the output formats are extremely limited, and BCP in particular is needlessly difficult to use.