I need to be able to download a folder with its contents from a blob storage account using the Azure Storage REST API only.
I have created a function (New-StorageAccountAthorisationHeader) that creates the (authentication) header that I can download a single file, but I cannot find any reference on how I might go about downloading the whole folder.
If I pass the folder as the $blob parameter, I get a BlobNotFound error.
The URL of the said folder is: https://mystorageaccount.blob.core.windows.net/acontainer/somefolder. The contents of "somefolder" looks like:
Folder1
FolderA
FileA.txt
FolderB
FileB.txt
FileC.txt
New-StorageAccountAthorisationHeader:
function New-StorageAccountAuthorizationHeader
{
[cmdletbinding()]
param
(
[string]$StorageAccountName,
[string]$Container,
[string]$Blob,
[string]$accesskey ,
[string]$ResourceUri,
[string]$xmsversion = "2017-04-17"
)
$xmsdate = Get-Date
$xmsdate = $xmsdate.ToUniversalTime()
$xmsdate = $xmsdate.toString('r')
function GetRestApiParameters
{
[cmdletbinding()]
param
(
[Parameter(Mandatory=$true)]
[string]$Uri
)
if($Uri.Contains("?"))
{
Write-Verbose "URI to extract REST parameters: $uri"
return ($Uri.Split("?")[1]).Split("&")
}
}
Write-Verbose "Generating string for signature encryption..."
$partUrl = "/$StorageAccountName/"
if($Container)
{
$partUrl = $partUrl + "$Container/"
}
if($Blob)
{
$parturl = $partUrl + "$Blob"
}
######Don't change the line count or indentation of the here-string#####
$hereString = #"
GET
x-ms-date:$xmsdate
x-ms-version:$xmsversion
$partUrl
"#
$hereString =$hereString -replace "$([char]13)$([char]10)","$([char]10)" #Change `r`n to just `n
$empty = $oSignature = New-Object System.Text.StringBuilder
$empty = $oSignature.Append($hereString)
Write-Verbose "Appending parameters from URI into authorisation string..."
$restParameters = GetRestApiParameters -Uri $ResourceUri -Verbose
if ($restParameters -ne $null)
{
foreach ($param in $restParameters)
{
$empty = $oSignature.Append("$([char]10)$($param.Replace('=',':'))")
}
}
#$oSignature.toString()
Write-Verbose "Encrypting string..."
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = [Convert]::FromBase64String($accesskey)
$signature = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($oSignature.ToString()))
$signature = [Convert]::ToBase64String($signature)
Write-Verbose "Building header..."
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("x-ms-version", $xmsversion)
$headers.Add("x-ms-date", $xmsdate)
$headers.Add("Authorization", "SharedKey " + $StorageAccountName + ":" + $signature)
#$headers.Add("x-ms-blob-type","BlockBlob")
#$headers.Add("Content-Type", "application\xml")
Write-Verbose ("Header: $($headers | Out-String)")
Return $headers
}
And I would call it:
$StorageAccountName = "mystorageaccount"
$container = "acontainer"
$blob = "somefile.txt"
$uriToDownloadBlobs = "https://" + $StorageAccountName + ".blob.core.windows.net/$container/$blob"
$header = $null
$header = New-StorageAccountAuthorizationHeader -StorageAccountName $StorageAccountName -ResourceUri $uriToDownloadBlobs -Verbose -Container $container -Blob $blob
$result = Invoke-WebRequest -Headers $header -Uri $uriToDownloadBlobs -OutFile C:\Temp\$blob -PassThru
$result
So this works, but as I said, I'm after any hints to help with downloading the whole folder.
It looks like this is not possible? Although I'd be interested to see how it's done with the likes of Azure Storage Explorer.
My solution was to zip the files up and then use the above to download the single ZIP file. A few extra lines of code to compress and extract at either end, but it was the quickest way at the time and it works well with VSTS tasks.
Related
I try to write a PowerShell Script and test it using https://api.binance.com/api/v3/order/test REST link but it doesn't work. I can't understand what should I use as message, what as body and what as header. It seems that here everything is clear and when I see a linux example I should have the same link in Output:
https://github.com/binance-exchange/binance-official-api-docs/blob/master/rest-api.md
Could someone who understands REST Post method help me to figure out what should I change here.
My output is:
POST https://api.binance.com/api/v3/order/test?symbol=LTCBTC&side=BUY&type=LIMIT&timeInForce=GTC&quantity=1&price=0.1&recvWindow=5000×tamp=1515586306172&signature=LxHZUfC5MiTUfMPyEtgaVShlV1j4ITo3QxvtPAzPkwQ=
Many thanks in advance.
$apiKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$unixEpochStart = Get-Date -Date "01/01/1970"
$now = Get-Date
$timestamp = (New-TimeSpan -Start $unixEpochStart -End $now.ToUniversalTime()).TotalMilliseconds
$timestamp = ([math]::Round($timestamp, 0)).ToString()
$apimessage = "symbol=LTCBTC&side=BUY&type=LIMIT&timeInForce=GTC&quantity=1&price=0.1&recvWindow=5000×tamp=$timestamp"
$apisecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = [Convert]::FromBase64String($apisecret)
$signature = $hmacsha.ComputeHash([Text.Encoding]::ASCII.GetBytes($apimessage))
$signature = [Convert]::ToBase64String($signature)
$uri = "https://api.binance.com/api/v3/order/test?$apimessage&signature=$signature"
$header = #{
"X-MBX-APIKEY" = $apiKey
}
Invoke-RestMethod -Method Post -Uri $uri -Headers $header -Verbose
The issue is this line of code:
$signature = [Convert]::ToBase64String($signature)
Binance expects the HMAC SHA256 transmitted in hex form. Replace the above with this and the issue should be resolved.
$signature = [System.BitConverter]::ToString($signature).Replace('-', '').ToLower()
Below is my working code.
APIKey and Secret are from the example of the Binance API doc
https://github.com/binance-exchange/binance-official-api-docs/blob/master/rest-api.md#signed-endpoint-examples-for-post-apiv1order
Result of $Signature should be the same as in the Binance example: c8db56825ae71d6d79447849e617115f4a920fa2acdcab2b053c4b2838bd6b71
If you use your own APIKey and Secret it should be working
$APIKey = "vmPUZE6mv9SD5VNHk4HlWFsOr6aKE2zvsw0MuIgwCIPy6utIco14y7Ju91duEh8A"
$APISecret = "NhqPtmdSJYdKjVHjA7PZj4Mge3R5YNiP1e3UZjInClVN65XAbvqqM6A7H5fATj0j"
$TimeStamp = (Get-Date (Get-Date).ToUniversalTime() -UFormat %s).replace(',', '').replace('.', '').SubString(0,13)
$QueryString = "symbol=LTCBTC&side=BUY&type=LIMIT&timeInForce=GTC&quantity=1&price=0.1&recvWindow=5000×tamp=1499827319559"
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = [Text.Encoding]::ASCII.GetBytes($APISecret)
$signature = $hmacsha.ComputeHash([Text.Encoding]::ASCII.GetBytes($QueryString))
$signature = [System.BitConverter]::ToString($signature).Replace('-', '').ToLower()
$uri = "https://api.binance.com/api/v3/account?$QueryString&signature=$signature"
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("X-MBX-APIKEY",$APIKey)
try {
Invoke-RestMethod -Uri $uri -Headers $headers -Method Get
}
Catch {
$streamReader = [System.IO.StreamReader]::new($_.Exception.Response.GetResponseStream())
$ErrResp = $streamReader.ReadToEnd() | ConvertFrom-Json
$streamReader.Close()
$ErrResp
}
below all the functions you need to request the Binance API.
Reference: https://blog.p-difm.com/interact-with-your-binance-account-using-the-api-and-powershell/
function Get-UnixTimeStamp{
<#
.SYNOPSIS
Return the timestamp in millisecond of the Unix Epoch
.DESCRIPTION
Unix Epoch started the Thursday, January 1, 1970 12:00:00 AM. The function return the number of second from that time.
.EXAMPLE
Get-UnixTimeStamp
#>
param(
)
return $(Get-Date (Get-Date).ToUniversalTime() -UFormat %s).replace(',', '').replace('.', '').SubString(0,13)
}
function Get-BinanceAPISignature{
<#
.SYNOPSIS
Prepare the signature that will be sent with the API request
.DESCRIPTION
Endpoint requires sending a valid API-Key and signature
.PARAMETER QueryString
The queryString must contains the symobol, timestamp and a recvWindow
.PARAMETER EndPoint
The EndPoint you want to request
.EXAMPLE
$URI = Get-BinanceAPISignature -QueryString $QueryString -EndPoint "/api/v3/openOrders"
#>
param(
[Parameter(Mandatory=$true)]$QueryString,
[Parameter(Mandatory=$true)]$EndPoint
)
$APISecret = "ASDHFASUHDFIOUSAHlUGLULUHALUHliuhalushduauauhIUH"
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = [Text.Encoding]::ASCII.GetBytes($APISecret)
$signature = $hmacsha.ComputeHash([Text.Encoding]::ASCII.GetBytes($QueryString))
$signature = [System.BitConverter]::ToString($signature).Replace('-', '').ToLower()
$URI = "https://api.binance.com$($EndPoint)?$QueryString&signature=$signature"
return $URI
}
function Get-BinanceAPIHeader{
<#
.SYNOPSIS
Prepare the header that will be sent with the API request
.DESCRIPTION
The header include your APIKey
.PARAMETER
#APIKey
.EXAMPLE
Get-BinanceAPIHeader
#>
param(
)
$APIKey = "HDAUSHF3989898hiuHGhuhI987898HiuahsduhaiusduhUIH"
$Headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$Headers.Add("X-MBX-APIKEY",$APIKey)
return $Headers
}
function Request-API{
<#
.SYNOPSIS
Run the CURL command with defined parameters
.DESCRIPTION
Call the API and error handling. Return the result of the request
.PARAMETER Method
Choose a method according to the EndPoint
.PARAMETER URI
This parameter needs to be obtained with Get-BinanceAPISignature
.PARAMETER Headers
This parameter needs to be obtained with Get-BinanceAPIHeaderx
.EXAMPLE
$ObjResults = Request-API -Method Get -URI $URI -Headers $Headers
#>
[cmdletbinding()]
param(
[Parameter(Mandatory=$true)][ValidateSet("POST","GET","DELETE")]$Method,
[Parameter(Mandatory=$true)]$URI,
[Parameter(Mandatory=$true)]$Headers
)
try{
$ArrayJsonResult = Curl $URI -Method $Method -Headers $Headers #-Verbose
$ObjCustomResult = $ArrayJsonResult.Content | ConvertFrom-Json
}
catch{
$streamReader = [System.IO.StreamReader]::new($_.Exception.Response.GetResponseStream())
$ErrResp = $streamReader.ReadToEnd() | ConvertFrom-Json
$streamReader.Close()
$LastError = $Error[0].ErrorDetails.Message | ConvertFrom-Json
write-host "1: " $ErrResp -b Red
write-host "2: " $LastError.code -b Red
write-host "3: " $LastError.msg -b Red
switch ($LastError.code){
("-1105") {write-host "TimeStamp issue"}
("-1003") {write-host "Too much request, IP Banned"; break}
("-2010") {write-host "Stop price would trigger immediately or Account has too many open stop loss and/or take profit orders on the symbol."}
("-1013") {write-host "The amount is not enough for this currency or not following the step size rule for the symbol."}
("-1111") {write-host "Too many decimal check the precision required with Get-ExchangeInfo"}
}
}
return $ObjCustomResult
}
And here an example how to use it
Reference: https://blog.p-difm.com/new-order-with-api-binance-and-powershell/
function New-Order{
<#
.SYNOPSIS
Place an order to buy or sell crypto
.PARAMETER Symbol
The crypto you want to buy or sell
.PARAMETER Side
ValidateSet "BUY" or "SELL"
.PARAMETER OrderType
ValidateSet "OrderMarket" or "OrderLimit" or "OrderStopLimit"
.PARAMETER Quantity
Specifies the amount you want to spend (when buying) or receive (when selling)
.PARAMETER FiatAmount
specifies the amount you want to spend in USDT
.EXAMPLE
New-Order -Symbol BTCUSDT -Side BUY -OrderType OrderMarket -FiatAmount 20 -Verbose
.EXAMPLE
New-Order -Symbol BTCUSDT -Side BUY -OrderType OrderLimit -FiatAmount 1000 -Price 33000
.EXAMPLE
New-Order -Symbol BTCUSDT -Side BUY -OrderType OrderStopLimit -Quantity 0.002 -Price 33000 -StopPrice 36000
.EXAMPLE
New-Order -Symbol XRPUSDT -Side SELL -OrderType OrderLimit -Quantity 100 -Price 0.5
.EXAMPLE
New-Order -Symbol XRPUSDT -Side SELL -OrderType OrderStopLimit -Quantity 100 -Price 0.55 -StopPrice 0.5
#>
[cmdletbinding()]
param(
[Parameter(Mandatory=$true)]
[string]$Symbol,
[Parameter(Mandatory=$true)]
[ValidateSet("BUY", "SELL")]
[string]$Side,
[Parameter(Mandatory=$true)]
[ValidateSet("OrderMarket", "OrderLimit", "OrderStopLimit")]
[string]$OrderType,
[Parameter(Mandatory=$true, ParameterSetName = 'quantity')]
[double]$Quantity,
[Parameter(Mandatory=$true, ParameterSetName = 'quantity2')]
[double]$FiatAmount
)
DynamicParam{
$paramDictionary = New-Object -Type System.Management.Automation.RuntimeDefinedParameterDictionary
if ($OrderType -ne "OrderMarket"){
# price param
$attributes = New-Object -Type System.Management.Automation.ParameterAttribute
$attributes.Mandatory = $true
$attributeCollection = New-Object -Type System.Collections.ObjectModel.Collection[System.Attribute]
$attributeCollection.Add($attributes)
$dynParam1 = New-Object -Type System.Management.Automation.RuntimeDefinedParameter("Price", [double], $attributeCollection)
$paramDictionary.Add("Price", $dynParam1)
}
if ($OrderType -eq "OrderStopLimit"){
# StopPrice param
$attributes2 = New-Object -Type System.Management.Automation.ParameterAttribute
$attributes2.Mandatory = $true
$attributeCollection2 = New-Object -Type System.Collections.ObjectModel.Collection[System.Attribute]
$attributeCollection2.Add($attributes2)
$dynParam2 = New-Object -Type System.Management.Automation.RuntimeDefinedParameter("StopPrice", [double], $attributeCollection2)
$paramDictionary.Add("StopPrice", $dynParam2)
}
return $paramDictionary
}
BEGIN{
# Check prerequisit
try{
Get-Command -Name Get-UnixTimeStamp -ErrorAction Stop | out-null
Get-Command -name Get-BinanceAPISignature -ErrorAction Stop | Out-Null
Get-Command -Name Get-BinanceAPIHeader -ErrorAction Stop | Out-Null
Get-Command -Name Request-API -ErrorAction Stop | Out-Null
}
catch{
Write-Host "Load Get-UnixTimeStamp, Get-BinanceAPISignature, Get-BinanceAPIHeader, Request-API first prior to laod the current script" -b red
Break
}
$TimeStamp = Get-UnixTimeStamp
# retrieve value from dyn param
if ($OrderType -ne "OrderMarket"){
$Price = $paramDictionary.values[0].value[0]
$StopPrice = $paramDictionary.values[0].value[1]
}
switch($OrderType){
"OrderMarket"{
if($PSBoundParameters.ContainsKey('Quantity')){
$QueryString = "symbol=$Symbol&side=$Side&type=MARKET&quantity=$Quantity×tamp=$TimeStamp&recvWindow=5000"
}
else{
$QueryString = "symbol=$Symbol&side=$Side&type=MARKET"eOrderQty=$FiatAmount×tamp=$TimeStamp&recvWindow=5000"
}
}
"OrderLimit"{
if($PSBoundParameters.ContainsKey('FiatAmount')){
$CurrentPrice = Get-Price -Symbol $Symbol -Decimal 8
$Quantity = [math]::Round($FiatAmount / $CurrentPrice, 6)
}
$QueryString = "symbol=$Symbol&side=$Side&type=LIMIT&price=$Price&timeInForce=GTC&quantity=$Quantity×tamp=$TimeStamp&recvWindow=5000"
}
"OrderStopLimit"{
if($PSBoundParameters.ContainsKey('FiatAmount')){
$CurrentPrice = Get-Price -Symbol $Symbol -Decimal 0
$Quantity = [math]::Round($FiatAmount / $CurrentPrice, 6)
}
$QueryString = "symbol=$Symbol&side=$Side&type=TAKE_PROFIT_LIMIT&stopPrice=$StopPrice&price=$Price&timeInForce=GTC&quantity=$Quantity×tamp=$TimeStamp&recvWindow=5000"
}
}
}
PROCESS{
$URI = Get-BinanceAPISignature -QueryString $QueryString -EndPoint "/api/v3/order"
$Headers = Get-BinanceAPIHeader
$ObjResults = $null # need to do this?
$ObjResults = Request-API -Method POST -URI $URI -Headers $Headers
}
END{
return $ObjResults
}
}
I want to run a PS script when I want to publish to FTP server. I took this script as structure : structure script.
I have very simple folder :
C:\Uploadftp\Files\doc.txt
C:\Uploadftp\Files\Files2
C:\Uploadftp\Files\Files2\doc2.txt
nothing fancy there.
Here is my script :
cd C:\Uploadftp
$location = Get-Location
"We are here: $location"
$user = "test" # Change
$pass = "test" # Change
## Get files
$files = Get-ChildItem -recurse
## Get ftp object
$ftp_client = New-Object System.Net.WebClient
$ftp_client.Credentials = New-Object System.Net.NetworkCredential($user,$pass)
$ftp_address = "ftp://test/TestFolder"
## Make uploads
foreach($file in $files)
{
$directory = "";
$source = $($file.DirectoryName + "/" + $file);
if ($file.DirectoryName.Length -gt 0)
{
$directory = $file.DirectoryName.Replace($location,"")
}
$directory = $directory.Replace("\","/")
$source = $source.Replace("\","/")
$directory += "/";
$ftp_command = $($ftp_address + $directory + $file)
# Write-Host $source
$uri = New-Object System.Uri($ftp_command)
"Command is " + $uri + " file is $source"
$ftp_client.UploadFile($uri, $source)
}
I keep getting this error :
Exception calling "UploadFile" with "2" argument(s): "An exception occurred during a WebClient request."
If I hardcode specific folder for $uri and tell source to be some specific folder on my computer, this script doesn't create directory, it creates a file. What am I doing wrong?
P.S. dont hit me too hard, its my fist time ever doing something in power shell.
Try the "Create-FtpDirectory" function from https://github.com/stej/PoshSupport/blob/master/Ftp.psm1
function Create-FtpDirectory {
param(
[Parameter(Mandatory=$true)]
[string]
$sourceuri,
[Parameter(Mandatory=$true)]
[string]
$username,
[Parameter(Mandatory=$true)]
[string]
$password
)
if ($sourceUri -match '\\$|\\\w+$') { throw 'sourceuri should end with a file name' }
$ftprequest = [System.Net.FtpWebRequest]::Create($sourceuri);
$ftprequest.Method = [System.Net.WebRequestMethods+Ftp]::MakeDirectory
$ftprequest.UseBinary = $true
$ftprequest.Credentials = New-Object System.Net.NetworkCredential($username,$password)
$response = $ftprequest.GetResponse();
Write-Host Upload File Complete, status $response.StatusDescription
$response.Close();
}
I'm trying to make PowerShell send a web request to our SSRS server and capture the results. I've hit a wall using the rs:FORMAT=EXCEL parameter in the SSRS url string. I have the following:
First, init the credentials:
$User = "MYDOMAIN\MyUser"
$PWord = ConvertTo-SecureString -String "WooHooStringP$W0rd" -AsPlainText -Force
$c = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $User, $PWord
Now, request a report:
Invoke-WebRequest `
-UserAgent ([Microsoft.PowerShell.Commands.PSUserAgent]::InternetExplorer) `
-Credential $c `
-Uri "http://myserver/ReportServer_DEV/Pages/ReportViewer.aspx?/folder+path/report+name"
This works fine. I can even grab the results (enclosing this request and using ().Content).
Then, specify a format instead of plain rendering:
Invoke-WebRequest `
-UserAgent ([Microsoft.PowerShell.Commands.PSUserAgent]::InternetExplorer) `
-Credential $c `
-Uri "http://myserver/ReportServer_DEV/Pages/ReportViewer.aspx?/folder+path/report+name&rs:format=HTML4.0"
Note the rs:Format specification? Works like a charm.
Then, for the grande finale, give me an Excel file:
Invoke-WebRequest `
-UserAgent ([Microsoft.PowerShell.Commands.PSUserAgent]::InternetExplorer) `
-Credential $c `
-Uri "http://myserver/ReportServer_DEV/Pages/ReportViewer.aspx?/folder+path/report+name&rs:format=EXCEL"
No can do, bud:
Invoke-WebRequest : The remote server returned an error: (401) Unauthorized.
At line:1 char:11
+ $bytez = (Invoke-WebRequest `
+ ~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
Why does the rs:format=EXCEL option throw an Unauthorised exception where all the other URLs are served by SSRS?
I've figured it out! I went about this the wrong way: SSRS offers access through a webservice that PowerShell can consume without the need to hack the URL and capture a response. I found a script that did this and modified it to suit my purpose:
function GetRSConnection($server, $instance)
{
# Create a proxy to the SSRS server and give it the namespace of 'RS' to use for
# instantiating objects later. This class will also be used to create a report
# object.
$User = "DOMAIN\Username"
$PWord = ConvertTo-SecureString -String "Pa$$w0rd" -AsPlainText -Force
$c = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $User, $PWord
$reportServerURI = "http://" + $server + "/" + $instance + "/ReportExecution2005.asmx?WSDL"
$RS = New-WebServiceProxy -Class 'RS' -NameSpace 'RS' -Uri $reportServerURI -Credential $c
$RS.Url = $reportServerURI
return $RS
}
function GetReport($RS, $reportPath)
{
# Next we need to load the report. Since Powershell cannot pass a null string
# (it instead just passses ""), we have to use GetMethod / Invoke to call the
# function that returns the report object. This will load the report in the
# report server object, as well as create a report object that can be used to
# discover information about the report. It's not used in this code, but it can
# be used to discover information about what parameters are needed to execute
# the report.
$reportPath = "/" + $reportPath
$Report = $RS.GetType().GetMethod("LoadReport").Invoke($RS, #($reportPath, $null))
# initialise empty parameter holder
$parameters = #()
$RS.SetExecutionParameters($parameters, "nl-nl") > $null
return $report
}
function AddParameter($params, $name, $val)
{
$par = New-Object RS.ParameterValue
$par.Name = $name
$par.Value = $val
$params += $par
return ,$params
}
function GetReportInFormat($RS, $report, $params, $outputpath, $format)
{
# Set up some variables to hold referenced results from Render
$deviceInfo = "<DeviceInfo><NoHeader>True</NoHeader></DeviceInfo>"
$extension = ""
$mimeType = ""
$encoding = ""
$warnings = $null
$streamIDs = $null
# Report parameters are handled by creating an array of ParameterValue objects.
# Add the parameter array to the service. Note that this returns some
# information about the report that is about to be executed.
# $RS.SetExecutionParameters($parameters, "en-us") > $null
$RS.SetExecutionParameters($params, "nl-nl") > $null
# Render the report to a byte array. The first argument is the report format.
# The formats I've tested are: PDF, XML, CSV, WORD (.doc), EXCEL (.xls),
# IMAGE (.tif), MHTML (.mhtml).
$RenderOutput = $RS.Render($format,
$deviceInfo,
[ref] $extension,
[ref] $mimeType,
[ref] $encoding,
[ref] $warnings,
[ref] $streamIDs
)
# Determine file name
$parts = $report.ReportPath.Split("/")
$filename = $parts[-1] + "."
switch($format)
{
"EXCEL" { $filename = $filename + "xls" }
"WORD" { $filename = $filename + "doc" }
"IMAGE" { $filename = $filename + "tif" }
default { $filename = $filename + $format }
}
if($outputpath.EndsWith("\\"))
{
$filename = $outputpath + $filename
} else
{
$filename = $outputpath + "\" + $filename
}
$filename
# Convert array bytes to file and write
$Stream = New-Object System.IO.FileStream($filename), Create, Write
$Stream.Write($RenderOutput, 0, $RenderOutput.Length)
$Stream.Close()
}
$RS = GetRSConnection -server "DEVBOX" -instance "ReportServer_DEV"
$report = GetReport -RS $RS -reportPath "folder name/report name"
$params = #()
$params = AddParameter -params $params -name "Month" -val "201311"
GetReportInformat -RS $RS -report $report -params $params -outputpath "i:\test" -format "EXCEL"
Using web request:
[string]$Domain = "DomainUsername"
[string]$Username = "Username"
[string]$Password = "Password"
[string]$ReportServer = "http://ssrsreportserver/ReportServer/ReportExecution2005.asmx" #Report Server
[string]$ReportLocation = "/Report Location/Report Name" #Report Location ON SSRS
$ReportLocation = $ReportLocation.Replace("/", "%2f")
$ReportLocation = $ReportLocation.Replace(" ", "+")
[string]$outputFile = $PSScriptRoot + '\Report.xlsx' #Save location for the file
#If the report has any parameters
[string]$ParamString = "";
$ParamString += "¶m1=paramvalue"
$ParamString += "¶m2=paramvalue"
[string]$URL = $ReportServer + "?" + $ReportLocation + "&rs:Command=Render&rs:Format=" + "EXCELOPENXML" + "&rs:ParameterLanguage=en-GB" + $ParamString
Write-Host $URL
$Req = [System.Net.WebRequest]::Create($URL);
$Req.Credentials = new-object System.Net.NetworkCredential($Username, $Password, $Domain)
$Req.Timeout = 30000;
$WebStream = $Req.GetResponse().GetResponseStream();
$MemStream = New-Object System.IO.MemoryStream
$WebStream.CopyTo($MemStream);
[long]$Len = $MemStream.Length;
[byte[]]$outBytes = [System.Byte[]]::CreateInstance([System.Byte], $Len)
$MemStream.Seek(0, [System.IO.SeekOrigin]::Begin);
$MemStream.Read($outBytes, 0, [int]$Len);
$WebStream.Close();
$MemStream.Close();
$MemStream.Dispose();
$Stream = New-Object System.IO.FileStream($outputFile), Create, Write
$Stream.Write($outBytes, 0, $outBytes.Length)
$Stream.Close()
Invoke-Item $outputFile
I'm learning powershell to create a script that uploads files to a sub-site library inside SharePoint. I managed to get it working and uploading .doc files to my library, however I also want fill in any metadata specified in additional columns with the file at the same time. I'm not using the SharePoint snap-in and instead using the webclient functions. Here is the simple powershell script
# create the Variable Path and Pass the source folder path
$path = "THE FILE"
# create the Variable destination and pass the URL of the SharePoint List
$destination = "SharePoint SubSite"
# Store the current user default credentials in the Variable Credentials
$credentials = [System.Net.CredentialCache]::DefaultCredentials;
# Create the object of the Webclient
$webclient = New-Object System.Net.WebClient;
$webclient.Credentials = $credentials;
$webclient.UploadFile($destination + “/” + "Filename", “PUT”, TheFile)
This code works however I don't know how to use the webclient functions to pass metadata to custom columns. When the file is uploaded I noticed the only first two columns are updated but the custom fields are left blank.
If this is even possible let me know.
Much thanks for your help
Basically three options are available here (it is assumed SharePoint 2010 is used):
consume SharePoint 2010 REST interface
consume SharePoint Web Services
consume SharePoint 2010 Client SDK
How to consume SharePoint 2010 REST Interface via PowerShell
The following function demonstrates how to perform CRUD operations using SharePoint 2010 REST interface:
<#
.Synopsis
Sends an HTTP or HTTPS request to a SharePoint 2010 REST-compliant web service.
.DESCRIPTION
This function sends an HTTP or HTTPS request to a Representational State
Transfer (REST)-compliant ("RESTful") SharePoint Online web service.
.EXAMPLE
Invoke-SPRestMethod -Url "https://contoso.sharepoint.com/_vti_bin/ListData.svc/Projects"
#>
Function Invoke-SPRestRequest()
{
Param(
[Parameter(Mandatory=$True)]
[String]$WebUrl,
[Parameter(Mandatory=$True)]
[String]$ListName,
[Parameter(Mandatory=$False)]
[int]$ItemId,
[Parameter(Mandatory=$False)]
[String]$QueryOptions,
[Parameter(Mandatory=$False)]
[Microsoft.PowerShell.Commands.WebRequestMethod]$Method = [Microsoft.PowerShell.Commands.WebRequestMethod]::Get,
[Parameter(Mandatory=$False)]
[System.Net.ICredentials]$Credentials,
[Parameter(Mandatory=$False)]
[String]$Payload,
[Parameter(Mandatory=$False)]
[String]$ETag,
[Parameter(Mandatory=$False)]
[String]$XHTTPMethod,
[Parameter(Mandatory=$False)]
[System.String]$Accept = "application/json;odata=verbose",
[Parameter(Mandatory=$False)]
[String]$ContentType = "application/json;odata=verbose"
)
#Construct Endpoint URL
$endpointUrl = $WebUrl + "/_vti_bin/listdata.svc/" + $ListName
if($ItemId){
$endpointUrl = $endpointUrl + "(" + $ItemId + ")"
}
if($QueryOptions){
$endpointUrl = $endpointUrl + $QueryOptions
}
$client = New-Object System.Net.WebClient
if($Credentials) {
$client.Credentials = $Credentials
}
$client.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED", "f")
$client.Headers.Add("Content-Type",$ContentType)
$client.Headers.Add("Accept",$Accept)
if($Method -eq "Get") {
$result = $client.DownloadString($endpointUrl) | ConvertFrom-Json
}
elseif ($Method -eq "Post") {
if($ETag) {
$client.Headers.Add("If-Match", $ETag)
}
if($XHTTPMethod) {
$client.Headers.Add("X-HTTP-Method", $XHTTPMethod)
}
if($Payload) {
$client.UploadString($endpointUrl,$Method,$Payload)
}
else {
$client.UploadString($endpointUrl,$Method)
}
}
$client.Dispose()
return $result
}
Gist: Invoke-SPRestRequest.ps1
Example
The following example demonstrates how to upload a file into SharePoint 2010 and set metadata properties. It consists of:
Upload a file operation
Find the uploaded list item associated with uploaded file
Update list item properties
Code:
$UserName = "username"
$Password = Read-Host -Prompt "Enter the password"
$WebUrl = "https://contoso.sharepoint.com/project"
$FolderUrl = "/project/Shared Documents/Archive"
Function Find-ListItem([string]$WebUrl,[System.Net.ICredentials]$Credentials,[string]$ListName,[string]$QueryOptions)
{
$result = Invoke-SPRestRequest -WebUrl $WebUrl -ListName $ListName -QueryOptions $QueryOptions -Credentials $Credentials
return $result.d.results
}
Function Update-ListItem([string]$WebUrl,[System.Net.ICredentials]$Credentials,[string]$ListName,[int]$ItemId)
{
$itemPayload = #{
"DocumentStatusValue" = "Close";
} | ConvertTo-Json
Invoke-SPRestRequest -WebUrl $WebUrl -ListName $ListName -ItemId $ItemId -Credentials $Credentials -Method Post -Payload $itemPayload -ETag "*" -XHTTPMethod "MERGE"
}
Function Upload-File([string]$WebUrl,[System.Net.ICredentials]$Credentials,[string]$FolderUrl, [string]$FileName)
{
$client = New-Object System.Net.WebClient
#$client.Credentials = $Credentials
$client.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED", "f")
$siteUri = New-Object Uri($WebUrl)
$fileUri = New-Object Uri($siteUri, ($FolderUrl + "/" + [System.IO.Path]::GetFileName($FileName)))
$result = $client.UploadFile($fileUri, "PUT", $FileName)
$client.Dispose()
}
#1.Upload a File
Upload-File -WebUrl $WebUrl -FolderUrl $FolderUrl -FileName "D:\tmp\SharePoint User Guide.docx"
#2.Find an associated List Item for a File
$query = "?`$filter=Path eq '" + $FolderUrl + "'"
$ListItem = Find-ListItem -WebUrl $WebUrl -ListName "Documents" -QueryOptions $query
#3.Update List Item properties
Update-ListItem -WebUrl $WebUrl -ListName "Documents" -ItemId $ListItem.Id
References
SharePoint Foundation REST Interface
I want to use PowerShell to transfer files with FTP to an anonymous FTP server. I would not use any extra packages. How?
I am not sure you can 100% bullet proof the script from not hanging or crashing, as there are things outside your control (what if the server loses power mid-upload?) - but this should provide a solid foundation for getting you started:
# create the FtpWebRequest and configure it
$ftp = [System.Net.FtpWebRequest]::Create("ftp://localhost/me.png")
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential("anonymous","anonymous#localhost")
$ftp.UseBinary = $true
$ftp.UsePassive = $true
# read in the file to upload as a byte array
$content = [System.IO.File]::ReadAllBytes("C:\me.png")
$ftp.ContentLength = $content.Length
# get the request stream, and write the bytes into it
$rs = $ftp.GetRequestStream()
$rs.Write($content, 0, $content.Length)
# be sure to clean up after ourselves
$rs.Close()
$rs.Dispose()
There are some other ways too. I have used the following script:
$File = "D:\Dev\somefilename.zip";
$ftp = "ftp://username:password#example.com/pub/incoming/somefilename.zip";
Write-Host -Object "ftp url: $ftp";
$webclient = New-Object -TypeName System.Net.WebClient;
$uri = New-Object -TypeName System.Uri -ArgumentList $ftp;
Write-Host -Object "Uploading $File...";
$webclient.UploadFile($uri, $File);
And you could run a script against the windows FTP command line utility using the following command
ftp -s:script.txt
(Check out this article)
The following question on SO also answers this: How to script FTP upload and download?
I'm not gonna claim that this is more elegant than the highest-voted solution...but this is cool (well, at least in my mind LOL) in its own way:
$server = "ftp.lolcats.com"
$filelist = "file1.txt file2.txt"
"open $server
user $user $password
binary
cd $dir
" +
($filelist.split(' ') | %{ "put ""$_""`n" }) | ftp -i -in
As you can see, it uses that dinky built-in windows FTP client. Much shorter and straightforward, too. Yes, I've actually used this and it works!
Easiest way
The most trivial way to upload a binary file to an FTP server using PowerShell is using WebClient.UploadFile:
$client = New-Object System.Net.WebClient
$client.Credentials =
New-Object System.Net.NetworkCredential("username", "password")
$client.UploadFile(
"ftp://ftp.example.com/remote/path/file.zip", "C:\local\path\file.zip")
Advanced options
If you need a greater control, that WebClient does not offer (like TLS/SSL encryption, etc), use FtpWebRequest. Easy way is to just copy a FileStream to FTP stream using Stream.CopyTo:
$request = [Net.WebRequest]::Create("ftp://ftp.example.com/remote/path/file.zip")
$request.Credentials =
New-Object System.Net.NetworkCredential("username", "password")
$request.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$fileStream = [System.IO.File]::OpenRead("C:\local\path\file.zip")
$ftpStream = $request.GetRequestStream()
$fileStream.CopyTo($ftpStream)
$ftpStream.Dispose()
$fileStream.Dispose()
Progress monitoring
If you need to monitor an upload progress, you have to copy the contents by chunks yourself:
$request = [Net.WebRequest]::Create("ftp://ftp.example.com/remote/path/file.zip")
$request.Credentials =
New-Object System.Net.NetworkCredential("username", "password")
$request.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$fileStream = [System.IO.File]::OpenRead("C:\local\path\file.zip")
$ftpStream = $request.GetRequestStream()
$buffer = New-Object Byte[] 10240
while (($read = $fileStream.Read($buffer, 0, $buffer.Length)) -gt 0)
{
$ftpStream.Write($buffer, 0, $read)
$pct = ($fileStream.Position / $fileStream.Length)
Write-Progress `
-Activity "Uploading" -Status ("{0:P0} complete:" -f $pct) `
-PercentComplete ($pct * 100)
}
$ftpStream.Dispose()
$fileStream.Dispose()
Uploading folder
If you want to upload all files from a folder, see
PowerShell Script to upload an entire folder to FTP
I recently wrote for powershell several functions for communicating with FTP, see https://github.com/AstralisSomnium/PowerShell-No-Library-Just-Functions/blob/master/FTPModule.ps1. The second function below, you can send a whole local folder to FTP. In the module are even functions for removing / adding / reading folders and files recursively.
#Add-FtpFile -ftpFilePath "ftp://myHost.com/folder/somewhere/uploaded.txt" -localFile "C:\temp\file.txt" -userName "User" -password "pw"
function Add-FtpFile($ftpFilePath, $localFile, $username, $password) {
$ftprequest = New-FtpRequest -sourceUri $ftpFilePath -method ([System.Net.WebRequestMethods+Ftp]::UploadFile) -username $username -password $password
Write-Host "$($ftpRequest.Method) for '$($ftpRequest.RequestUri)' complete'"
$content = $content = [System.IO.File]::ReadAllBytes($localFile)
$ftprequest.ContentLength = $content.Length
$requestStream = $ftprequest.GetRequestStream()
$requestStream.Write($content, 0, $content.Length)
$requestStream.Close()
$requestStream.Dispose()
}
#Add-FtpFolderWithFiles -sourceFolder "C:\temp\" -destinationFolder "ftp://myHost.com/folder/somewhere/" -userName "User" -password "pw"
function Add-FtpFolderWithFiles($sourceFolder, $destinationFolder, $userName, $password) {
Add-FtpDirectory $destinationFolder $userName $password
$files = Get-ChildItem $sourceFolder -File
foreach($file in $files) {
$uploadUrl ="$destinationFolder/$($file.Name)"
Add-FtpFile -ftpFilePath $uploadUrl -localFile $file.FullName -username $userName -password $password
}
}
#Add-FtpFolderWithFilesRecursive -sourceFolder "C:\temp\" -destinationFolder "ftp://myHost.com/folder/" -userName "User" -password "pw"
function Add-FtpFolderWithFilesRecursive($sourceFolder, $destinationFolder, $userName, $password) {
Add-FtpFolderWithFiles -sourceFolder $sourceFolder -destinationFolder $destinationFolder -userName $userName -password $password
$subDirectories = Get-ChildItem $sourceFolder -Directory
$fromUri = new-object System.Uri($sourceFolder)
foreach($subDirectory in $subDirectories) {
$toUri = new-object System.Uri($subDirectory.FullName)
$relativeUrl = $fromUri.MakeRelativeUri($toUri)
$relativePath = [System.Uri]::UnescapeDataString($relativeUrl.ToString())
$lastFolder = $relativePath.Substring($relativePath.LastIndexOf("/")+1)
Add-FtpFolderWithFilesRecursive -sourceFolder $subDirectory.FullName -destinationFolder "$destinationFolder/$lastFolder" -userName $userName -password $password
}
}
Here's my super cool version BECAUSE IT HAS A PROGRESS BAR :-)
Which is a completely useless feature, I know, but it still looks cool \m/ \m/
$webclient = New-Object System.Net.WebClient
Register-ObjectEvent -InputObject $webclient -EventName "UploadProgressChanged" -Action { Write-Progress -Activity "Upload progress..." -Status "Uploading" -PercentComplete $EventArgs.ProgressPercentage } > $null
$File = "filename.zip"
$ftp = "ftp://user:password#server/filename.zip"
$uri = New-Object System.Uri($ftp)
try{
$webclient.UploadFileAsync($uri, $File)
}
catch [Net.WebException]
{
Write-Host $_.Exception.ToString() -foregroundcolor red
}
while ($webclient.IsBusy) { continue }
PS. Helps a lot, when I'm wondering "did it stop working, or is it just my slow ASDL connection?"
You can simply handle file uploads through PowerShell, like this.
Complete project is available on Github here https://github.com/edouardkombo/PowerShellFtp
#Directory where to find pictures to upload
$Dir= 'c:\fff\medias\'
#Directory where to save uploaded pictures
$saveDir = 'c:\fff\save\'
#ftp server params
$ftp = 'ftp://10.0.1.11:21/'
$user = 'user'
$pass = 'pass'
#Connect to ftp webclient
$webclient = New-Object System.Net.WebClient
$webclient.Credentials = New-Object System.Net.NetworkCredential($user,$pass)
#Initialize var for infinite loop
$i=0
#Infinite loop
while($i -eq 0){
#Pause 1 seconde before continue
Start-Sleep -sec 1
#Search for pictures in directory
foreach($item in (dir $Dir "*.jpg"))
{
#Set default network status to 1
$onNetwork = "1"
#Get picture creation dateTime...
$pictureDateTime = (Get-ChildItem $item.fullName).CreationTime
#Convert dateTime to timeStamp
$pictureTimeStamp = (Get-Date $pictureDateTime).ToFileTime()
#Get actual timeStamp
$timeStamp = (Get-Date).ToFileTime()
#Get picture lifeTime
$pictureLifeTime = $timeStamp - $pictureTimeStamp
#We only treat pictures that are fully written on the disk
#So, we put a 2 second delay to ensure even big pictures have been fully wirtten in the disk
if($pictureLifeTime -gt "2") {
#If upload fails, we set network status at 0
try{
$uri = New-Object System.Uri($ftp+$item.Name)
$webclient.UploadFile($uri, $item.FullName)
} catch [Exception] {
$onNetwork = "0"
write-host $_.Exception.Message;
}
#If upload succeeded, we do further actions
if($onNetwork -eq "1"){
"Copying $item..."
Copy-Item -path $item.fullName -destination $saveDir$item
"Deleting $item..."
Remove-Item $item.fullName
}
}
}
}
You can use this function :
function SendByFTP {
param (
$userFTP = "anonymous",
$passFTP = "anonymous",
[Parameter(Mandatory=$True)]$serverFTP,
[Parameter(Mandatory=$True)]$localFile,
[Parameter(Mandatory=$True)]$remotePath
)
if(Test-Path $localFile){
$remoteFile = $localFile.Split("\")[-1]
$remotePath = Join-Path -Path $remotePath -ChildPath $remoteFile
$ftpAddr = "ftp://${userFTP}:${passFTP}#${serverFTP}/$remotePath"
$browser = New-Object System.Net.WebClient
$url = New-Object System.Uri($ftpAddr)
$browser.UploadFile($url, $localFile)
}
else{
Return "Unable to find $localFile"
}
}
This function send specified file by FTP.
You must call the function with these parameters :
userFTP = "anonymous" by default or your username
passFTP = "anonymous" by default or your password
serverFTP = IP address of the FTP server
localFile = File to send
remotePath = the path on the FTP server
For example :
SendByFTP -userFTP "USERNAME" -passFTP "PASSWORD" -serverFTP "MYSERVER" -localFile "toto.zip" -remotePath "path/on/the/FTP/"
Goyuix's solution works great, but as presented it gives me this error: "The requested FTP command is not supported when using HTTP proxy."
Adding this line after $ftp.UsePassive = $true fixed the problem for me:
$ftp.Proxy = $null;
Simple solution if you can install curl.
curl.exe -p --insecure "ftp://<ftp_server>" --user "user:password" -T "local_file_full_path"