[System.IO.File]::ReadAlltext($File) OutOfMemory error - powershell

Question

I'm trying to upload a zip file to Nexus repository after converting the zipe file (1.6gb) to bytes.
The problem is when it readallbytes of the zip file, if the file size is big it throw system.outofmemory error. it doesn't throw the error if it's just 3-600mb, is there a way to read bytes and avoid the oufofmemory error?
Exception calling "ReadAllText" with "1" argument(s): "Exception of type 'System.OutOfMemoryException' was thrown."
$fileBin = [System.IO.File]::ReadAlltext($File)
Function Upload-File-To-Nexus {
[CmdletBinding()]
Param(
[Parameter(Mandatory = $True )][ValidateNotNullOrEmpty()][string]$File, # aka File
[parameter(Mandatory = $True )][ValidateNotNullOrEmpty()][string]$Repository,
[parameter(Mandatory = $True )][ValidateNotNullOrEmpty()][string]$Directory,
[Parameter(Mandatory = $False)][ValidateNotNullOrEmpty()][string]$FileName # destination filename, which can be derived from uploaded file
)
Begin {
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("fahq-ra-build:tEDlCI=0m9CES8l*lk?b"))
$header = #{authorization = "Basic $token" }
$Uri = "https://nexus-arps.corp.firstam.com/service/rest/v1/components?repository=${Repository}"
$boundary = [System.Guid]::NewGuid().ToString()
$LF = "`r`n"
}
Process {
If (!($FileName -ne $Null -And $FileName -ne "")) {
# if filename is null, then just use the filename from the input file
$FileName = Split-Path $File -Leaf
}
Try { $fileBin = [System.IO.File]::ReadAlltext($File) }
Catch {
throw $_.exception
Throw "Unable to read file $File. Aborted."
}
$bodyLines = (
"--${boundary}",
"Content-Disposition: form-data; name=`"raw.directory`"",
"",
"${Directory}",
"--${boundary}",
"Content-Disposition: form-data; name=`"raw.asset1`"; filename=`"${FileName}`"",
"Content-Type: application/octet-stream",
"",
$file,
"",
"--${boundary}",
"Content-Disposition: form-data; name=`"raw.asset1.filename`"",
"",
"${FileName}",
"--${boundary}--",
""
) -join $LF
$Response = Invoke-WebRequest -Uri $Uri -Method "POST" -Headers $header -ContentType "multipart/form-data; boundary=`"$boundary`"" -Body $bodyLines
If ($Response.StatusCode -ge 200 -And $Response.StatusCode -lt 300) {
$Output = "https://${Server}/repository/${repository}/${directory}/${FileName}" -Replace "//", "/"
Write-Output $Output
}
Else {
Write-Output $Response
}
}
}

Answer 1

You probably meant .ReadAllBytes(), given that your'e not dealing with a text-based files.
However, you don't have to load the file into memory - use Invoke-WebRequest's -InFile parameter to specify a file to upload.

Answer 2

IIRC, the way ReadAllBytes() works is it doesn't know or lookup the file size up front. Instead, it guesses a buffer size for the resulting byte array. When the guess fails, it doubles the guess, allocates a new array, and copies what was read so far. This repeats until the file is loaded.
You get OutOfMemory, then, not only from actually running out of memory but also from running out address space in the current process from the extra buffer, where individual processes are limited by default to only 2GB.
If that sounds bad and inefficient, you're right, and RealAllBytes() is really only useful for small files.
When dealing with larger files, you need to use the streaming APIs, such as File.OpenRead(). Here you can check the actual size of the file, use that to allocate the exact size byte array, and then read in a small chunk (say, 4K) at a time.
Even better, since you're already using Invoke-WebRequest did you know it supports a -InFile argument, which can be used to upload a file without loading the entire thing into memory?