Powershell unzip stream - powershell

Question

Is there a built-in cmdlet or some composition thereof that would allow me to start unzipping a file stream as each chunk is downloaded? I have a PowerShell script that needs to download a large (10 GB) file, and I have to wait until it is done right now before it starts expanding...
$wc = New-Object net.webclient
$wc.Downloadfile($appDataSnapshotUri, "%DataSnapshotFileName%.zip") # this can take some time
Expand-Archive -Path "%DataSnapshotFileName%.zip" -DestinationPath Run # so can this

Answer 1

OK, turns out zip file doesn't need to be fully downloaded to be decompressed, you can compress/decompress streams. There is some built in capabilities in .Net for stream compression, but it will not work with zip archives. You can use SharpZipLib library for that:
Download .nupckg from https://www.nuget.org/packages/SharpZipLib/
Extract files to any folder. You'll need ICSharpCode.SharpZipLib.dll from lib/net45
Below is my simplified translation of their example:
https://github.com/icsharpcode/SharpZipLib/wiki/Zip-Samples#unpack-a-zip-using-zipinputstream-eg-for-unseekable-input-streams
Add-Type -Path ".\ICSharpCode.SharpZipLib.dll"
$outFolder = ".\unzip"
$wc = [System.Net.WebClient]::new()
$zipStream = $wc.OpenRead("http://gitlab/test/test1/raw/master/sample.zip")
$zipInputStream = [ICSharpCode.SharpZipLib.Zip.ZipInputStream]::New($zipStream)
$zipEntry = $zipInputStream.GetNextEntry()
$fileName = $zipEntry.Name
$buffer = New-Object byte[] 4096
$sw = [System.IO.File]::Create("$outFolder\$fileName")
[ICSharpCode.SharpZipLib.Core.StreamUtils]::Copy($zipInputStream, $sw, $buffer)
$sw.Close()
It will only extract first entry, you can add a while loop it this sample works.
Here is a snippet with while loop to extract multiple files (put it after $zipEntry = $zipInputStream.GetNextEntry() on the example above):
While($zipEntry) {
$fileName = $zipEntry.Name
Write-Host $fileName
$buffer = New-Object byte[] 4096
$sw = [System.IO.File]::Create("$outFolder\$fileName")
[ICSharpCode.SharpZipLib.Core.StreamUtils]::Copy($zipInputStream, $sw, $buffer)
$sw.Close()
$zipEntry = $zipInputStream.GetNextEntry()
}
Edit
Here is what I found to work...
Add-Type -Path ".\ICSharpCode.SharpZipLib.dll"
$outFolder = "unzip"
$wc = [System.Net.WebClient]::new()
$zipStream = $wc.OpenRead("https://github.com/Esri/file-geodatabase-api/raw/master/FileGDB_API_1.5/FileGDB_API_1_5_VS2015.zip")
$zipInputStream = [ICSharpCode.SharpZipLib.Zip.ZipInputStream]::New($zipStream)
$zipEntry = $zipInputStream.GetNextEntry()
while($zipEntry) {
if (-Not($zipEntry.IsDirectory)) {
$fileName = $zipEntry.Name
$buffer = New-Object byte[] 4096
$filePath = "$pwd\$outFolder\$fileName"
$parentPath = "$filePath\.."
Write-Host $parentPath
if (-Not (Test-Path $parentPath)) {
New-Item -ItemType Directory $parentPath
}
$sw = [System.IO.File]::Create("$pwd\$outFolder\$fileName")
[ICSharpCode.SharpZipLib.Core.StreamUtils]::Copy($zipInputStream, $sw, $buffer)
$sw.Close()
}
$zipEntry = $zipInputStream.GetNextEntry()
}

Answer 2

To expand on Mike Twc's answer, a script to do it with and without stream, and compare how long it takes:
$url = "yoururlhere"
function UnzipStream () {
Write-Host "unzipping via stream"
$stopwatch1 = [system.diagnostics.stopwatch]::StartNew()
Add-Type -Path ".\ICSharpCode.SharpZipLib.dll"
$outFolder = "unzip-stream"
$wc = [System.Net.WebClient]::new()
$zipStream = $wc.OpenRead($url)
$zipInputStream = [ICSharpCode.SharpZipLib.Zip.ZipInputStream]::New($zipStream)
$zipEntry = $zipInputStream.GetNextEntry()
while($zipEntry) {
if (-Not($zipEntry.IsDirectory)) {
$fileName = $zipEntry.Name
$buffer = New-Object byte[] 4096
$filePath = "$pwd\$outFolder\$fileName"
$parentPath = "$filePath\.."
Write-Host $parentPath
if (-Not (Test-Path $parentPath)) {
New-Item -ItemType Directory $parentPath
}
$sw = [System.IO.File]::Create("$pwd\$outFolder\$fileName")
[ICSharpCode.SharpZipLib.Core.StreamUtils]::Copy($zipInputStream, $sw, $buffer)
$sw.Close()
}
$zipEntry = $zipInputStream.GetNextEntry()
}
$stopwatch1.Stop()
Write-Host "extraction took $($stopWatch1.ElapsedMilliseconds) millis with stream"
}
function UnzipWithoutStream() {
Write-Host "Extracting without stream"
$stopwatch2 = [system.diagnostics.stopwatch]::StartNew()
$outFolder2 = "unzip-normal"
$wc2 = New-Object System.Net.WebClient
$wc2.DownloadFile($url, "$pwd\download.zip")
$of2 = New-Item -ItemType Directory $outFolder2
Expand-Archive -Path "download.zip" -DestinationPath $of2.FullName
$stopwatch2.Stop()
Write-Host "extraction took $($stopWatch2.ElapsedMilliseconds) millis without stream"
}
UnzipStream
UnzipWithoutStream