Powershell FTP Send large file System.OutOfMemoryException - powershell

Question

I have a script partially based on the one here: Upload files with FTP using PowerShell
It all works absolutely fine with tiny files but I am trying to use it to make the process we use for exporting access mdb files to clients that only have ftp more robust.
My first test involved a 10MB file and I ran into a System.OutOfMemoryException at the Get-Content stage
The powershell ISE was running to nearly 2GIG usage during the get attempt.
Here is a full script sample (Be gentle. I am fairly new to it):
#####
# User variables to control the script
#####
# How many times connection will be re-tried
$connectionTries = 5
#time between tries in seconds
$connectionTryInterval = 300
#Where to log the output
$logFile = "D:\MyPath\ftplog.txt"
#maximum log file size in KB before it is archived
$logFileMaxSize = 500
#formatted date part for the specific file to transfer
#This is appended to the filename base. Leave as "" for none
$datePart = ""
#base part of the file name
$fileNameBase = "Myfile"
#file extension
$fileExtension = ".mdb"
#location of the source file (please include trailing backslash)
$sourceLocation = "D:\MyPath\"
#location and credentials of the target ftp server
$userName = "iamafish"
$password = "ihavenofingers"
$ftpServer = "10.0.1.100"
######
# Main Script
#####
#If there is a log file and it is longer than the declared limit then archive it with the current timestamp
if (test-path $logfile)
{
if( $((get-item $logFile).Length/1kb) -gt $logFileMaxSize)
{
write-host $("archiving log to ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt")
rename-item $logFile $("ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt")
}
}
#start new log entry
#Add-Content $logFile "___________________________________________________________"
#write-host $logEntry
#contruct source file and destination uri
$fileName = $fileNameBase + $datePart + $fileExtension
$sourceFile = $sourceLocation + $fileName
$sourceuri = "ftp://" + $ftpServer + "/" + $fileName
# Create a FTPWebRequest object to handle the connection to the ftp server
$ftprequest = [System.Net.FtpWebRequest]::create($sourceuri)
# set the request's network credentials for an authenticated connection
$ftprequest.Credentials = New-Object System.Net.NetworkCredential($username,$password)
$ftprequest.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftprequest.UseBinary = $true
$ftprequest.KeepAlive = $false
$succeeded = $true
$errorMessage = ""
# read in the file to upload as a byte array
trap [exception]{
$script:succeeded = $false
$script:errorMessage = $_.Exception.Message
Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|" + $_.Exception.Message)
#write-host $logEntry
#write-host $("TRAPPED: " + $_.Exception.GetType().FullName)
#write-host $("TRAPPED: " + $_.Exception.Message)
exit
}
#The -ea 1 forces the error to be trappable
$content = gc -en byte $sourceFile -ea 1
$try = 0
do{
trap [System.Net.WebException]{
$script:succeeded = $false
$script:errorMessage = $_.Exception.Message
Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|" + $_.Exception.Message)
#write-host $logEntry
#write-host $("TRAPPED: " + $_.Exception.GetType().FullName)
$script:try++
start-sleep -s $connectionTryInterval
continue
}
$ftpresponse = $ftprequest.GetResponse()
} while(($try -le $connectionTries) -and (-not $succeeded))
if ($succeeded) {
Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|0|" + "Starting file transfer.")
# get the request stream, and write the bytes into it
$rs = $ftprequest.GetRequestStream()
$rs.Write($content, 0, $content.Length)
# be sure to clean up after ourselves
$rs.Close()
$rs.Dispose()
$content.Close()
$content.Dispose()
Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|0|" + "Transfer complete.")
#write-host $logEntry
}
I can't put code in comments so, thanks to pointers from keith I have moved the file acces bit down to the bottom to link it with the other like so..
trap [Exception]{
$script:succeeded = $false
$script:errorMessage = $_.Exception.Message
Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|Check File Connection|" + $_.Exception.Message)
$sourceStream.Close()
$sourceStream.Dispose()
#write-host $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|Attempt to open file|" + $_.Exception.Message)
#write-host $("TRAPPED: " + $_.Exception.GetType().FullName)
exit
}
$sourceStream = New-Object IO.FileStream ($(New-Object System.IO.FileInfo $sourceFile),[IO.FileMode]::Open)
[byte[]]$readbuffer = New-Object byte[] 1024
# get the request stream, and write the bytes into it
$rs = $ftprequest.GetRequestStream()
do{
$readlength = $sourceStream.Read($readbuffer,0,1024)
$rs.Write($readbuffer,0,$readlength)
} while ($readlength -ne 0)
I just need to work out why I get: Exception calling "GetResponse" with "0" argument(s): "Cannot access a disposed object.
every other time I run it. Is this a quirk of running it in the ISE or am I doing somethign drasically wrong with either initial declaration or final disposing?
I'll post the full final script when done since I think it will make a nice sturdy ftp export example with error trapping and logging.
OK, here is the full script. Dispose is edited out but with or without it runnign the script within 5 minutes will either get me a message that I cannot use a disposed opject or tell me that the getResponse() has produced an error (226) File transfered (running in ISE). Whilst this will not be a problem during normal opperation I would like to correctly log oout of the FTP session and clean the resources at the end of the script and ensure I am correctly declaring them as needed.
#####
# User variables to control the script
#####
# How many times connection will be re-tried
$connectionTries = 5
#time between tries in seconds
$connectionTryInterval = 1
#Where to log the output
$logFile = "D:\MyPath\ftplog.txt"
#maximum log file size in KB before it is archived
$logFileMaxSize = 500
#log to file or console - #true=log to file, #false = log to console
$logToFile=$false
#formatted date part for the specific file to transfer
#This is appended to the filename base. Leave as "" for none
$datePart = ""
#base part of the file name
$fileNameBase = "MyFile"
#file extension
$fileExtension = ".mdb"
#location of the source file (please include trailing backslash)
$sourceLocation = "D:\MyPath\"
#location and credentials of the target ftp server
$userName = "iamafish"
$password = "ihavenofingers"
$ftpServer = "10.0.1.100"
######
# Main Script
#####
function logEntry($entryType, $section, $message)
{
#just to make a one point switch for logging to console for testing
# $entryType: 0 = success, 1 = Error
# $section: The section of the script the log entry was generated from
# $message: the log message
#This is pipe separated to fit in with my standard MSSQL linked flat file schema for easy querying
$logString = "$(get-Date -format "yyyy-MM-dd hh:mm:ss")|$entryType|$section|$message"
if($script:logtoFile)
{
Add-Content $logFile $logString
}
else
{
write-host $logString
}
}
#If there is a log file and it is longer than the declared limit then archive it with the current timestamp
if (test-path $logfile)
{
if( $((get-item $logFile).Length/1kb) -gt $logFileMaxSize)
{
write-host $("archiving log to ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt")
rename-item $logFile $("ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt")
New-Item $logFile -type file
}
}
else
{
New-Item $logFile -type file
}
#contruct source file and destination uri
$fileName = $fileNameBase + $datePart + $fileExtension
$sourceFile = $sourceLocation + $fileName
$destination = "ftp://" + $ftpServer + "/" + $fileName
#Check if the source file exists
if ((test-path $sourceFile) -eq $false)
{
logEntry 1 "Check Source File" $("File not found: " + $sourceFile)
Exit
}
# Create a FTPWebRequest object to handle the connection to the ftp server
$ftpRequest = [System.Net.FtpWebRequest]::create($destination)
# set the request's network credentials for an authenticated connection
$ftpRequest.Credentials = New-Object System.Net.NetworkCredential($username,$password)
$ftpRequest.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftpRequest.UseBinary = $true
$ftpRequest.KeepAlive = $false
$succeeded = $true
$try = 1
do{
trap [Exception]{
$script:succeeded = $false
logEntry 1 "Check FTP Connection" $_.Exception.Message
$script:try++
start-sleep -s $connectionTryInterval
continue
}
$ftpResponse = $ftpRequest.GetResponse()
} while(($try -le $connectionTries) -and (-not $succeeded))
if ($succeeded) {
logEntry 0 "Connection to FTP" "Success"
# Open a filestream to the source file
trap [Exception]{
logEntry 1 "Check File Connection" $_.Exception.Message
$sourceStream.Close()
$ftpResponse.Close()
exit
}
$sourceStream = New-Object IO.FileStream ($(New-Object System.IO.FileInfo $sourceFile),[IO.FileMode]::Open)
[byte[]]$readbuffer = New-Object byte[] 1024
logEntry 0 "Starting file transfer" "Success"
# get the request stream, and write the bytes into it
$rs = $ftpRequest.GetRequestStream()
do{
$readlength = $sourceStream.Read($readbuffer,0,1024)
$rs.Write($readbuffer,0,$readlength)
} while ($readlength -ne 0)
logEntry 0 "Transfer complete" "Success"
# be sure to clean up after ourselves
$rs.Close()
#$rs.Dispose()
$sourceStream.Close()
#$sourceStream.Dispose()
}
$ftpResponse.Close()
Example of trying to trap the Transfer OK response at the end:
logEntry 0 "Starting file transfer" "Success"
# get the request stream, and write the bytes into it
$rs = $ftpRequest.GetRequestStream()
do{
$readlength = $sourceStream.Read($readbuffer,0,1024)
$rs.Write($readbuffer,0,$readlength)
} while ($readlength -ne 0)
$rs.Close()
#start-sleep -s 2
trap [Exception]{
$script:succeeded = $false
logEntry 1 "Check FTP Connection" $_.Exception.Message
continue
}
$ftpResponse = $ftpRequest.GetResponse()

Answer 1

Having hit a similar issue myself with RAM usage hitting the GB's uploading a 3MB file, I found that replacing:
$content = gc -en byte $sourceFile
With:
$content = [System.IO.File]::ReadAllBytes($sourceFile)
Gives much better performance. As mentioned elsewhere, chunking would be a better solution for really large files, as then you're not holding the whole file in memory at once, but the code above at least only consumes ~(size of file) bytes of RAM, which means it should be good up to the ~10s of MB kind of range.

Answer 2

Rather than read the whole file into memory using Get-Content, try reading it in a chunk at a time and writing it to the FTP request stream. I would use one of the lower level .NET file stream APIs to do the reading. Admittedly, you wouldn't think a 10MB would pose a memory problem though.
Also, make sure you get the response after geting the request stream and writing to it. The get of the response stream is what uploads the data. From the docs:
When using an FtpWebRequest object to
upload a file to a server, you must
write the file content to the request
stream obtained by calling the
GetRequestStream method or its
asynchronous counterparts, the
BeginGetRequestStream and
EndGetRequestStream methods. You must
write to the stream and close the
stream before sending the request.
Requests are sent to the server by
calling the GetResponse method or its
asynchronous counterparts, the
BeginGetResponse and EndGetResponse
methods. When the requested operation
completes, an FtpWebResponse object is
returned. The FtpWebResponse object
provides the status of the operation
and any data downloaded from the
server.