How can I decode the body of a jdownloader *.DLC container file with Powershell?
I have managed to write a small Powershell snippet to decrypt a DLC container file based on the PyLoad code reference.
It prompts for a given file, decrypts the content and puts the URLs into the clipboard.
For the archives here the working sample code:
# script for decoding a DLC-file:
Remove-Variable * -ea 0
$ErrorActionPreference = 'stop'
$utf8 = [System.Text.Encoding]::UTF8
# file selector:
Add-Type -AssemblyName 'System.Windows.Forms'
$browser = [System.Windows.Forms.OpenFileDialog]::new()
$browser.Filter = 'DLC files (*.dlc)|*.dlc'
$browser.InitialDirectory = "$env:USERPROFILE\Downloads"
$null = $browser.ShowDialog()
$fileName = $browser.FileName
if (![System.IO.File]::Exists($fileName)) {break}
$dlc = [System.IO.File]::ReadAllText($fileName)
$len = $dlc.Length
$key = $dlc.Substring($len-88)
$data = $dlc.Substring(0,$len-88)
$bytes = [System.Convert]::FromBase64String($data)
$aesKey = 'cb99b5cbc24db398'
$aesIV = '9bc24cb995cb8db3'
$url = "http://service.jdownloader.org/dlcrypt/service.php?srcType=dlc&destType=pylo&data=$key"
$result = Invoke-WebRequest $url
$rc64 = ([xml]$result.Content).rc
$rc = [System.Convert]::FromBase64String($rc64)
$aes = [System.Security.Cryptography.Aes]::Create()
$aes.Key = $utf8.GetBytes($aeskey)
$aes.IV = $utf8.GetBytes($aesIV)
$aes.Padding = [System.Security.Cryptography.PaddingMode]::None
$dec = $aes.CreateDecryptor()
$result = $dec.TransformFinalBlock($rc, 0, $rc.Length)
$dec.Dispose()
$aes.key = $result
$aes.IV = $result
$dec = $aes.CreateDecryptor()
$enc = $dec.TransformFinalBlock($bytes, 0, $bytes.Length)
$dec.Dispose()
$b64 = $utf8.GetString($enc).Trim([char]0)
$byte = [System.Convert]::FromBase64String($b64)
$xml = [xml]$utf8.GetString($byte)
$urlList = foreach($url64 in $xml.dlc.content.package.file.url) {
$urlb = [System.Convert]::FromBase64String($url64)
$utf8.GetString($urlb)
}
cls
$urlList | Set-Clipboard
$urlList
Here is also a short demo how to encode any text into a DLC-format:
# DLC-encryption:
# get a random encryption key (RCP):
$bytes = [System.Security.Cryptography.Rfc2898DeriveBytes]::new($null,8)
$hexbin = [System.Runtime.Remoting.Metadata.W3cXsd2001.SoapHexBinary]::new($bytes.Salt)
$rcp = $hexbin.ToString().ToLower()
# let jdownloader generate an RC-value for this key:
$url = "http://service.jdownloader.org/dlcrypt/service.php?jd=1&srcType=plain&data=$rcp"
$result = Invoke-WebRequest $url
$rc = ([xml]"<xml>$($result.Content)</xml>").xml.rc
# if we store the rcp/rc-pair, then we could decode a DLC in offline mode
# encode any information with our initial key:
$data = 'something to encode'
$bytes = $utf8.GetBytes($data)
$aes = [System.Security.Cryptography.Aes]::Create()
$rcb = $utf8.GetBytes($rcp)
$aes.Padding = [System.Security.Cryptography.PaddingMode]::Zeros
$enc = $aes.CreateEncryptor($rcb, $rcb)
$out = $enc.TransformFinalBlock($bytes, 0, $bytes.Length)
$enc.Dispose()
# the DLC-content is a concatenation of the base64-string of the encrypted data
# plus the base64-string of the RC which corresponds to our initial RCP-key.
$data = [System.Convert]::ToBase64String($out)
write-host "$data$rc"
And here a correct XML-format of a decrypted DLC container. All(!) shown values here (strings/numbers/dates) must be in base64-format. I just decoded them all to make it a bit easier to understand them. Some elements or values might be optional, some may be subject of change. I have not tested that in details. It is just a readable format of a single DLC-file I created with JDownloader:
<dlc>
<header>
<generator>
<app>JDownloader</app>
<version>43307</version>
<url>http://jdownloader.org</url>
</generator>
<tribute/>
<dlcxmlversion>20_02_2008</dlcxmlversion>
</header>
<content>
<package category="various" comment="" name="Packagename">
<file>
<url>http://appwork.org/projects/DLCAPI/dlcapi_v1.0.rar</url>
<filename>dlcapi_v1.0.rar</filename>
<size>5838</size>
</file>
</package>
</content>
</dlc>
Related
I have a 7zip-compressed string (LZMA2) and I want to unzip that into memory via Powershell without using the file-system, but with no luck. I am able to unzip regular zip-strings, but no 7z-strings. Here a sample code to show my attempt so far:
# define settings for http-client:
Add-Type -AssemblyName System.Net.Http
$handler = [System.Net.Http.HttpClientHandler]::new()
$client = [System.Net.Http.HttpClient]::new($handler)
# get the 7z-compressed EPG-file:
$url = [string]::Concat('http://www.', 'vuplus', '-', 'community.net', '/ryt', 'ec/ryt', 'ecDE_Basic', '.xz')
$result = $client.GetAsync($url).Result
$data = $result.Content.ReadAsStringAsync().Result
# unzip:
$enc = [System.Text.Encoding]::GetEncoding(28591)
$ziparr = $enc.GetBytes($data)
$in = [System.IO.MemoryStream]::new($ziparr)
$mem = [System.IO.MemoryStream]::new()
$mode = [System.IO.Compression.CompressionMode]::Decompress
$zip = [System.IO.Compression.GzipStream]::new($in, $mode)
$zip.CopyTo($mem)
$arr = $mem.ToArray()
$unzippedText = $enc.GetString($arr)
If that is not possible via OS/.net-functions, is there any way to use 7za.dll for such without using the file-system?
I have found a solution by using the dll from the 7zipSharp-Package and the 7z64.dll. Here the code-snippet:
# define settings for http-client:
Add-Type -AssemblyName System.Net.Http
$handler = [System.Net.Http.HttpClientHandler]::new()
$client = [System.Net.Http.HttpClient]::new($handler)
# get the 7z-compressed EPG-file:
$url = [string]::Concat('http://', 'www.vuplus', '-', 'community.net', '/ryt', 'ec/ryt', 'ecDE_Basic', '.xz')
$result = $client.GetAsync($url).Result
$data = $result.Content.ReadAsStreamAsync().Result
# references:
# https://globalcdn.nuget.org/packages/sevenzipsharp.net45.1.0.19.nupkg
# https://github.com/squid-box/SevenZipSharp/raw/dev/SevenZip/7z64.dll
Add-Type -Path "C:\temp\SevenZipSharp.dll"
[SevenZip.SevenZipExtractor]::SetLibraryPath('C:\temp\7z64.dll')
$ex = [SevenZip.SevenZipExtractor]::new($data)
$mem = [System.IO.MemoryStream]::new()
$ex.ExtractFile(0, $mem)
$arr = $mem.ToArray()
$enc = [System.Text.Encoding]::GetEncoding(28591)
$unzippedText = $enc.GetString($arr)
I want to download only a single file from an online ZIP archive via Powershell.
For this I created a demo-code which is already working, but I am still struggling to get the correct parsing logic on the ZIP-directory. Here is the code I have so far:
# demo code downloading a single DLL file from an online ZIP archive
# and extracting the DLL into memory for mounting it to the main process.
cls
Remove-Variable * -ea 0
# definition for the ZIP archive, the file to be extracted and the checksum:
$url = 'https://github.com/sshnet/SSH.NET/releases/download/2020.0.1/SSH.NET-2020.0.1-bin.zip'
$sub = 'net40/Renci.SshNet.dll'
$md5 = '5B1AF51340F333CD8A49376B13AFCF9C'
# prepare HTTP client:
Add-Type -AssemblyName System.Net.Http
$handler = [System.Net.Http.HttpClientHandler]::new()
$client = [System.Net.Http.HttpClient]::new($handler)
# get the length of the ZIP archive:
$req = [System.Net.HttpWebRequest]::Create($url)
$req.Method = 'HEAD'
$length = $req.GetResponse().ContentLength
$zip = [byte[]]::new($length)
# get the last 10k:
# how to get the correct length of the central ZIP directory here?
$start = $length-10kb
$end = $length-1
$client.DefaultRequestHeaders.Add('Range', "bytes=$start-$end")
$result = $client.GetAsync($url).Result
$last10kb = $result.content.ReadAsByteArrayAsync().Result
$last10kb.CopyTo($zip, $start)
# get the block containing the DLL file:
# how to get the exact file-offset from the ZIP directory?
$start = $length-3537kb
$end = $length-3201kb
$client.DefaultRequestHeaders.Clear()
$client.DefaultRequestHeaders.Add('Range', "bytes=$start-$end")
$result = $client.GetAsync($url).Result
$block = $result.content.ReadAsByteArrayAsync().Result
$block.CopyTo($zip, $start)
# extract the DLL file from archive:
Add-Type -AssemblyName System.IO.Compression
$stream = [System.IO.Memorystream]::new()
$stream.Write($zip,0,$zip.Length)
$archive = [System.IO.Compression.ZipArchive]::new($stream)
$entry = $archive.GetEntry($sub)
$bytes = [byte[]]::new($entry.Length)
[void]$entry.Open().Read($bytes, 0, $bytes.Length)
# check MD5:
$prov = [Security.Cryptography.MD5CryptoServiceProvider]::new().ComputeHash($bytes)
$hash = [string]::Concat($prov.foreach{$_.ToString("x2")})
if ($hash -ne $md5) {write-host 'dll has wrong checksum.' -f y ;break}
# load the DLL:
[void][System.Reflection.Assembly]::Load($bytes)
# use the single demo-call from the DLL:
$test = [Renci.SshNet.NoneAuthenticationMethod]::new('test')
'done.'
Only open point in this code is the correct method to identify the length of the central directory at the end of the ZIP archive and how to get the correct file-offset for the single file to be extracted (in my code I just found the ranges by pure try&error).
I already checked this wiki https://en.wikipedia.org/wiki/ZIP_(file_format)#Structure and also the PKWARE definitions https://gist.github.com/steakknife/820b73ebf25146180198febdb6f0e183 but beside the block definitions I could not find a programmatical approach to get the offset for ethe EOCD and the individual file. Can someone help here, please?
After a couple of additional tests I came to this solution:
# demo code downloading a single DLL file from an online ZIP archive
# and extracting the DLL into memory to mount it finally to the main process.
cls
Remove-Variable * -ea 0
# definition for the ZIP archive, the file to be extracted and the checksum:
$url = 'https://github.com/sshnet/SSH.NET/releases/download/2020.0.1/SSH.NET-2020.0.1-bin.zip'
$sub = 'net40/Renci.SshNet.dll'
$md5 = '5B1AF51340F333CD8A49376B13AFCF9C'
'prepare HTTP client:'
Add-Type -AssemblyName System.Net.Http
$handler = [System.Net.Http.HttpClientHandler]::new()
$client = [System.Net.Http.HttpClient]::new($handler)
'get the length of the ZIP archive:'
# dont use System.Web.HttpRequest, it is frequently hanging:
$req = [System.Net.Http.HttpRequestMessage]::new('HEAD', $url)
$result = $client.SendAsync($req).Result
$zipLength = $result.Content.Headers.ContentLength
$zip = [byte[]]::new($zipLength)
$req.Dispose()
'get the last 10k:'
$start = $zipLength-10kb
$end = $zipLength-1
$client.DefaultRequestHeaders.Add('Range', "bytes=$start-$end")
$result = $client.GetAsync($url).Result
$last10kb = $result.content.ReadAsByteArrayAsync().Result
$last10kb.CopyTo($zip, $start)
"get the 'End of CD' block:"
$enc = [System.Text.Encoding]::GetEncoding(28591)
$end = $enc.GetString($last10kb, $last10kb.Length-256, 256)
$eocd = [regex]::Match($end, 'PK\x05\x06.*').value
$eocd = $enc.GetBytes($eocd)
'get the central directory:'
$cdLength = [bitconverter]::ToUInt32($eocd, 12)
$cdStart = [bitconverter]::ToUInt32($eocd, 16)
$cd = [byte[]]::new($cdLength)
[array]::Copy($zip, $cdStart, $cd, 0, $cdLength)
'search all file headers for correct file name:'
$fileHeaders = [regex]::Split($enc.GetString($cd),'PK\x01\x02')
foreach ($header in $fileHeaders) {
$len = $header.Length
if ($len -ge 42) {
$bytes = $enc.GetBytes($header)
$nameLength = [bitconverter]::ToUInt16($bytes, 24)
if ($nameLength -eq $sub.length -and ($nameLength + 42) -le $len) {
$name = $header.Substring(42, $nameLength)
if ($name -eq $sub) {
$size = [bitconverter]::ToUInt32($bytes, 16) + 256
$start = [bitconverter]::ToUInt32($bytes, 38)
break
}
}
}
}
if (!$start) {write-host 'we could not find file in the ZIP archive' -f y ;break}
'get the block containing the file:'
$end = $start+$size
$client.DefaultRequestHeaders.Clear()
$client.DefaultRequestHeaders.Add('Range', "bytes=$start-$end")
$result = $client.GetAsync($url).Result
$block = $result.content.ReadAsByteArrayAsync().Result
$block.CopyTo($zip, $start)
$client.dispose()
'extract the DLL file from archive:'
Add-Type -AssemblyName System.IO.Compression
$stream = [System.IO.Memorystream]::new()
$stream.Write($zip,0,$zip.Length)
$archive = [System.IO.Compression.ZipArchive]::new($stream)
$entry = $archive.GetEntry($sub)
$bytes = [byte[]]::new($entry.Length)
[void]$entry.Open().Read($bytes, 0, $bytes.Length)
'check MD5:'
$prov = [Security.Cryptography.MD5CryptoServiceProvider]::new().ComputeHash($bytes)
$hash = [string]::Concat($prov.foreach{$_.ToString("x2")})
if ($hash -ne $md5) {write-host 'dll has wrong checksum.' -f y ;break}
'load the DLL:'
[void][System.Reflection.Assembly]::Load($bytes)
'use the single demo-call from the DLL:'
$test = [Renci.SshNet.NoneAuthenticationMethod]::new('test')
'done.'
I was successfully able to open a port on my computer (using only PowerShell) and know when HTTP requests are done to that port. I came up with this simple code:
$listener = [System.Net.Sockets.TcpListener]5566;
$listener.Start();
while ($true) {
$client = $Listener.AcceptTcpClient();
Write-Host "Connected!";
$client.Close();
}
If I open my browser and type http://localhost:5566 in the PowerShell interface it will show a message that a user got connected.
What I need to do is to get the GET parameters of this HTTP request. For example, if instead I had opened my browser and typed http://localhost:5566/test.html?parameter1=xxx¶meter2=yyy.
How can I grab the GET parameters (parameter1 and parameter2) name and values using my simplified code above?
If you are comfortable using the HttpListener instead of the TcpListener. It's easier to do the job.
Below script will output in a browser
Path is /test.html
parameter2 is equal to yyy
parameter1 is equal to xxx
Quick and dirty script
$listener = New-Object System.Net.HttpListener
$listener.Prefixes.Add("http://localhost:5566/")
try {
$listener.Start();
while ($true) {
$context = $listener.GetContext()
$request = $context.Request
# Output the request to host
Write-Host $request | fl * | Out-String
# Parse Parameters from url
$rawUrl = $request.RawUrl
$Parameters = #{}
$rawUrl = $rawUrl.Split("?")
$Path = $rawUrl[0]
$rawParameters = $rawUrl[1]
if ($rawParameters) {
$rawParameters = $rawParameters.Split("&")
foreach ($rawParameter in $rawParameters) {
$Parameter = $rawParameter.Split("=")
$Parameters.Add($Parameter[0], $Parameter[1])
}
}
# Create output string (dirty html)
$output = "<html><body><p>"
$output = $output + "Path is $Path" + "<br />"
foreach ($Parameter in $Parameters.GetEnumerator()) {
$output = $output + "$($Parameter.Name) is equal to $($Parameter.Value)" + "<br />"
}
$output = $output + "</p></body></html>"
# Send response
$statusCode = 200
$response = $context.Response
$response.StatusCode = $statusCode
$buffer = [System.Text.Encoding]::UTF8.GetBytes($output)
$response.ContentLength64 = $buffer.Length
$output = $response.OutputStream
$output.Write($buffer,0,$buffer.Length)
$output.Close()
}
} finally {
$listener.Stop()
}
Cheers
Glenn
I've had a script written in Powershell which transferred a file via FTP which worked fine by using:
$content = [System.IO.File]::ReadAllBytes($backup_app_data)
But this stopped working once the file size reached 2Gb, by throwing an error saying that this method is limited to reading files up to this size.
Exception calling "ReadAllBytes" with "1" argument(s): "The file is
too long. This operation is currently limited to supporting files less
than 2 gigabytes in size.
Now, I'm trying to modify my script and use an alternate approach with Read, which from what I understand will allow me to read the file in chunks and then write those chunks and so on.
Unfortunately although the script I've modified seems to be working, since a file is created on my FTP location, there is no data being transferred and no error is thrown by the script.
There is just a 0kb file created in the destination folder and the script ends.
I've tried to do some debugging, but I can't seem to find the issue. Below is the code I'm currently using.
$file = 'G:\Backups\DATA_backup_2016_09_05.zip'
$dest_name = 'DATA_backup_2016_09_05.zip'
$ftp = [System.Net.FtpWebRequest]::Create($ftp+$dest_name)
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user, $pass)
$ftp.UseBinary = $true
$ftp.UsePassive = $true
# determine the size of the file
$file_size = (Get-Item $file).length
$chunk_size = 512mb
$bytes_to_read = $file_size
$iterations = $file_size / $chunk_size
$iter = 0
$fstream = [System.IO.FileStream]
[byte[]] $byte_array
while ($bytes_to_read > 0){
if($iterations > 1) {
$content = $fstream.Read($byte_array, $iter, $chunk_size)
}
else {
$content = $fstream.Read($byte_array, $iter, $bytes_to_read)
}
$ftp.ContentLength = $content.Length
$rs = $ftp.GetRequestStream()
$rs.Write($content, 0, $content.Length)
# keep the loop going
$iter = $iter + 1
$iterations = $iterations - 1
$bytes_to_read = $bytes_to_read - $chunk_size
}
$rs.Close()
$rs.Dispose()
Any help is appreciated.
So, based on the suggestions in the comment section and using as an example the answer from the possible duplicate question, I managed to find the solution myself.
Below is the code I used to transfer via FTP a .zip archive which was 3.74Gb.
$ftp_addr = "ftp://ftp.somerandomwebsite.com/folder1/"
$user = "usr"
$pass = "passwrd"
$bufSize = 256mb
# ... missing code where I identify the file I want to FTP ... #
# Initialize connection to FTP
$ftp = [System.Net.FtpWebRequest]::Create($ftp_addr + $backup_file + ".zip")
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user, $pass)
$ftp.Timeout = -1 #infinite timeout
$ftp.ReadWriteTimeout = -1 #infinite timeout
$ftp.UseBinary = $true
$ftp.UsePassive = $true
$requestStream = $ftp.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file_to_ftp)
$chunk = New-Object byte[] $bufSize
while ( $bytesRead = $fileStream.Read($chunk, 0, $bufSize) ){
$requestStream.write($chunk, 0, $bytesRead)
$requestStream.Flush()
}
$fileStream.Close()
$requestStream.Close()
So far, this code has worked flawlessly for about multiple (20+) attempts. I hope it helps others!
I want to know how to catch url vars with powershell system.net.HttpListener
Thanks
$listener = New-Object system.net.HttpListener
$listener.Prefixes.Add('http://127.0.0.1:8080')
$listener.Start()
$context = $listener.GetContext() # block
$request = $context.Request
$response = $context.Response
# $var = read post/get var
$page = Get-Content -Path C:\play.html -Raw
$page = $page.Replace('%VAR%',$var)
$buffer = [System.Text.Encoding]::UTF8.GetBytes($page)
$response.ContentLength64 = $buffer.Length
$output = $response.OutputStream
$output.Write($buffer,0,$buffer.Length)
$output.Close()
$listener.Stop()
If the method header is GET then use the QueryString property to get the query parameters. If the method header is POST then check HasEntityBody property and if that is true, read the POST data from the body using the InputSteam property.