Powershell read file in chunks - powershell

Question

I've had a script written in Powershell which transferred a file via FTP which worked fine by using:
$content = [System.IO.File]::ReadAllBytes($backup_app_data)
But this stopped working once the file size reached 2Gb, by throwing an error saying that this method is limited to reading files up to this size.
Exception calling "ReadAllBytes" with "1" argument(s): "The file is
too long. This operation is currently limited to supporting files less
than 2 gigabytes in size.
Now, I'm trying to modify my script and use an alternate approach with Read, which from what I understand will allow me to read the file in chunks and then write those chunks and so on.
Unfortunately although the script I've modified seems to be working, since a file is created on my FTP location, there is no data being transferred and no error is thrown by the script.
There is just a 0kb file created in the destination folder and the script ends.
I've tried to do some debugging, but I can't seem to find the issue. Below is the code I'm currently using.
$file = 'G:\Backups\DATA_backup_2016_09_05.zip'
$dest_name = 'DATA_backup_2016_09_05.zip'
$ftp = [System.Net.FtpWebRequest]::Create($ftp+$dest_name)
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user, $pass)
$ftp.UseBinary = $true
$ftp.UsePassive = $true
# determine the size of the file
$file_size = (Get-Item $file).length
$chunk_size = 512mb
$bytes_to_read = $file_size
$iterations = $file_size / $chunk_size
$iter = 0
$fstream = [System.IO.FileStream]
[byte[]] $byte_array
while ($bytes_to_read > 0){
if($iterations > 1) {
$content = $fstream.Read($byte_array, $iter, $chunk_size)
}
else {
$content = $fstream.Read($byte_array, $iter, $bytes_to_read)
}
$ftp.ContentLength = $content.Length
$rs = $ftp.GetRequestStream()
$rs.Write($content, 0, $content.Length)
# keep the loop going
$iter = $iter + 1
$iterations = $iterations - 1
$bytes_to_read = $bytes_to_read - $chunk_size
}
$rs.Close()
$rs.Dispose()
Any help is appreciated.

Answer 1

So, based on the suggestions in the comment section and using as an example the answer from the possible duplicate question, I managed to find the solution myself.
Below is the code I used to transfer via FTP a .zip archive which was 3.74Gb.
$ftp_addr = "ftp://ftp.somerandomwebsite.com/folder1/"
$user = "usr"
$pass = "passwrd"
$bufSize = 256mb
# ... missing code where I identify the file I want to FTP ... #
# Initialize connection to FTP
$ftp = [System.Net.FtpWebRequest]::Create($ftp_addr + $backup_file + ".zip")
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user, $pass)
$ftp.Timeout = -1 #infinite timeout
$ftp.ReadWriteTimeout = -1 #infinite timeout
$ftp.UseBinary = $true
$ftp.UsePassive = $true
$requestStream = $ftp.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file_to_ftp)
$chunk = New-Object byte[] $bufSize
while ( $bytesRead = $fileStream.Read($chunk, 0, $bufSize) ){
$requestStream.write($chunk, 0, $bytesRead)
$requestStream.Flush()
}
$fileStream.Close()
$requestStream.Close()
So far, this code has worked flawlessly for about multiple (20+) attempts. I hope it helps others!