powershell and FTP, script not waiting for transfer to complete - powershell

I have a script that, in a nutshell, does the following:
copies required files to a temporary folder
compresses the files in the temporary folder to a .zip file
FTPs the .zip file to our FTP server
tidies up and deletes the temporary folder and .zip file
I have pinched the FTP code from a previous post:
Upload files with FTP using PowerShell
and modified it where necessary (keeping the basics in tact - I think).
The issue I have is that while the .zip file is being FTP'd the script doesn't wait until it is complete. It gets part way through, anywhere from 20Mb to 60Mb before it continues executing, tidies up and deletes the file it is transferring.
The temporary folder is always the same but the .zip filename varies depending on the date so I can't really reverse the order of operations.
Can anyone suggest how I might get the script to wait until the FTP process has completed, success or fail, before it moves on?
Cheers,
Andrew.
Edit: For those that asked....
function FTPtoServer ()
{
<#
What this function has to/should do:
- accept the right number of parameters,
minimum/mandatory: username, password, file
optional: proxy server address/port, proxy username and password
- check that the source file exists, then extract the filename.
- if a proxy is specified, set the appropriate parameters
- transmit the file
- if any errors occur, throw and return
#>
param(
[string]$sourcefile=$(throw 'A sourcefile is required, -sourcefile'), <#fully qualified zip file name#>
[string]$FTPUser =$(throw 'An FTP username is required, -ftpuser'),
[string]$FTPPass =$(throw 'An FTP password is required, -ftppass'),
#
[string]$proxyServer, #proxySocket?? it is an address and port
[string]$proxyUser,
[string]$proxyPass
)
#local variables
$FTPserver = "ftp://ftp.servername.com.au"
#check if the sourcefile exists, if not return/throw an error
# The sourcefile should contain the full path to the file.
if (-not (test-path $sourcefile)){
throw "the source file could not be located: $sourcefile"
}
# extract the filename from the sourcefile.
$filename = split-path -path $sourcefile -leaf
# create the FtpWebRequest and configure it
$ftp = [System.Net.FtpWebRequest]::Create("$FTPserver/$filename")
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($FTPUser,$FTPPass)
$ftp.UseBinary = $true
$ftp.UsePassive = $false
#proxy info
# ******** DANGER Will Robinson - this proxy config has not been
# tested and may not work.
if ($proxyServer){
$proxy = New-Object System.Net.WebProxy $proxyServer
if ($proxyUser -and $proxyPass){
$proxy.Credentials = new-object System.Net.NetworkCredential($proxyUser,$proxyPass)
}
$ftp.Proxy = $proxy
$ftp.UsePassive = $true #apparently, must usePassive if using proxy
}
#now we have checked and prepared everything, lets try and send the file.
# read in the file to upload as a byte array
try{
#work out how much we are sending
$content = [System.IO.File]::ReadAllBytes("$sourceFile")
$ftp.ContentLength = $content.Length
try {
# get the request stream, and write the bytes into it
$rs = $ftp.GetRequestStream()
$rs.Write($content, 0, $content.Length)
# be sure to clean up after ourselves
$rs.Close()
$rs.Dispose()
}
catch {
$errorMessage = "FTP failed. " + $_.exception.message
throw $errormessage
}
}
catch {
$errorMessage = "Unable to transmit file " + $sourceFile + "`r`n" + $_.exception.message
throw $errormessage
}
}
The above is in a separate file, but is called by the following:
try {
FTPtoServer -sourcefile $sourcefile -ftpuser $FTPUser -ftppass $FTPPass
}
catch {
$errorMessage = "FTPtoServer function failed with error: $_"
finishFail -failmessage $errorMessage
}
Cheers.

Found it.
I executed the FTP code above in isolation using a large file (~140Mb) and it threw the error; "The underlying connection was closed: An unexpected error occured on a receive."
I rebooted the FTP server, checked the user account etc etc.
I also tested the M$ FTP client with the same file and it transferred completely and correctly.
Anyway, I found this article: https://www.codeproject.com/Questions/597175/FileplusUploadplustoplusFTPplusserver which also has the error I received.
As it turns out, the Timeout value of FTPWebRequest is NOT -1 as in the doco but 100 seconds.
I checked my FTP logs and sure enough, time between logon and logoff was about 100 seconds.
I added the line: $ftp.Timeout = -1 to my code and first attempt transferred the entire file completely without error.
Previous transfers had worked as they fell below the 100 second timeout.
Many thanks for the posts and help.

I use an alternative oldschool method myself, it should work for you, and doesn't need any extra components on the server.
$ftp_user = "username"
$ftp_password = "password"
$ftp_address = "ftp.someserver.com"
$ftp_commands = #"
open $ftp_address
$ftp_user
$ftp_password
lcd c:\jobs\output
put estate_data_current.xml
bye
"#
set-content -encoding "ASCII" -path ftp_commands.txt -value $ftp_commands
ftp -s:ftp_commands.txt

Related

Create Receive Location and Send Port in Existing BizTalk Application using Powershell

I am needing to create receive locations and send ports in an already existing BizTalk application using Powershell. I have only seen some documentation on how to create an application but not to call upon one. Any suggestions would be beneficial. There are some things that are commented out, and that is because I cannot disclose that information. I added at the last part on what I have learned of how to create an application, but that is not something that I want for my script. The program below is what I have so far:
#===Create a receive port and location function===#
Function CreateRPandRL ()
{
#Creating Receive Port
$myReceivePort = $catalog.AddNewReceivePort($false)
$myReceivePort.Name = "My Receive Port"
#Creating Receive Location
$myReceiveLocation = $myReceivePort.AddNewReceiveLocation()
foreach ($handler in $catalog.ReceiveHandlers)
{
if ($handler.TransportType.Name -eq "FILE")
{
$myReceiveLocation.ReceiveHandler = $handler
break
}
}
#Associate a transport protocol and file location with receive location
$myReceiveLocation.TransportType = $catalog.ProtocolTypes["FILE"]
$myReceiveLocation.Address = #pick-up file location
#Assign the first receive pipeline found to process the message
foreach ($pipeline in $catalog.Pipelines)
{
if ($pipeline.Type -eq [Microsoft.BizTalk.ExplorerOM.PipelineType] "File_Receive")
{
$myReceiveLocation.ReceivePipeline = $pipeline
break
}
#Enable the receive location
$myReceiveLocation.Enable = $true
}
#Try to commit the changes made so far. If the commit fails, roll back changes
$catalog.SaveChanges()
}
Function CreateSendPorts($Catalog)
{
#=== Register a trap handler to discard changes on exceptions ===#
$ErrorActionPreference="silentlycontinue"
trap { "Exception encountered:`r`n"; $_; "`r`nDiscarding Changes.`r`n";$Catalog.DiscardChanges();exit; }
#=== create a new static one-way send port using FILE transport ===#
$mySendPort = $Catalog.AddNewSendPort($false,$false)
$mySendPort.Name = "My Send Port"
$mySendPort.PrimaryTransport.TransportType = $catalog.ProtocolTypes["FILE"]
$mySendPort.PrimaryTransport.Address = #drop-off file location
$mySendPort.SendPipeline = $Catalog.Pipelines["Microsoft.BizTalk.DefaultPipelines.EdiSend"]
#=== Persist new ports to BizTalk configuration database ===#
Write-Host "Adding $mySendPort.Name..."
$catalog.SaveChanges();
Write-Host "`r`n $mySendPort.Name has been created."
#=== specify filters for content-based routing ===#
Write-Host $mySendPort.Name: Adding a filter
$mySendPort.Filter = "<Filter><Group>" +
"<Statement Property='EDI.ISA06' Operator='0' Value='9999999999'/>" +
"<Statement Property='EDI.ST01' Operator='0' Value='999'/>" +
"<Statement Property='EDI.IsSystemGeneratedAck' Operator='0' Value='true'/>" +
"<Statement Property='BTS.ReceivePortName' Operator='0' Value= $myReceivePort.Name/>" +
"</Group></Filter>"
#=== Persist all changes to BizTalk configuration database ===#
Write-Host $mySendPort.Name + ": Saving changes"
$catalog.SaveChanges();
Write-Host "`r`nFilters for $mySendPort.Name created"
#===========Changing Send Port status===========#
#Register a trap handler to discard changes on exceptions
$ErrorActionPreference="silentlycontinue"
trap { "Exception encountered:`r`n"; $_; "`r`nDiscarding Changes.`r`n";$Catalog.DiscardChanges();exit; }
#start the send port to begin processing messages
$mySendPort.Status = [Microsoft.BizTalk.ExplorerOM.PortStatus] "Started"
Write-Host "Changing" + $mySendPort.Name + "status to ($mySendPort.Status)..."
$catalog.SaveChanges()
Write-Host "Complete."
}
#===Main Script===#
#make sure the ExplorerOM assembly is loaded
[void][System.reflection.Assembly]::LoadwithPartialName("Microsoft.BizTalk.ExplorerOM")
#Connect to the BizTalk management database
$Catalog = New-Object Microsoft.BizTalk.ExplorerOM.BtsCatalogExplorer
$Catalog.ConnectionString = "SERVER = #server_address; DATABASE=BizTalkMgmtDB; Integrated Security=SSPI"
#Implementing functions
CreateRPandRL
CreateSendPorts
Start-Sleep -Seconds 60
<#
#===BizTalk Application===#
$app = $Catalog.AddNewApplication()
$app.Name = "TestingPowershellScript"
$app.CreateRPandRL()
$app.CreateSendPorts()
#>
Hoo boy, this takes me back a few years, I'm glad I'm not the only one to struggle with this. You want to leave that alone and switch to the BizTalk PowerShell Extensions (information on this is sketchy), they are sooooooo much easier to work with in PowerShell.
I cobbled this together from some scripts I used, and left out some of the fancy stuff, but what you want is basically:
$InitializeDefaultBTSDrive = $false
Import-Module "$env:BTSINSTALLPATH\SDK\Utilities\PowerShell\BizTalkFactory.PowerShell.Extensions.dll" -WarningAction Ignore
New-PSDrive -Name BizTalk -PSProvider BizTalk -Root BizTalk:\ -Instance $DatabaseName -Database $BizTalkMgmtDb
This opens up a whole world of goodies, because it's loaded as a PSDrive, you can navigate round it, create things, delete things, use it all as native as any other drive/filesystem, such as:
Get-ChildItem "BizTalk:\All Artifacts\Receive Locations"
Get-ChildItem "BizTalk:\All Artifacts\Receive Locations" | Disable-ReceiveLocation
Get-ChildItem "BizTalk:\Platform Settings\Host Instances" | Stop-HostInstance
Get-ChildItem "BizTalk:\Platform Settings\Host Instances" | Where-Object { $_.IsDisabled -eq $false } | Start-HostInstance
Get-ChildItem "BizTalk:\All Artifacts\Receive Locations" | Enable-ReceiveLocation
Get-ChildItem -Path "BizTalk:\Health and Activity\Service Instances"
There's so much more than the above, and none of this is what you really asked for, what you actually want is:
Import-Bindings -Path "BizTalk:" -Source $bindings
Where $bindings is your XML bindings file.
My advice, don't even try this. Most of the useful settings for Adapters are not exposed by any API so this will get you maybe half way at most.
Instead, script the import of a binding file which does support all settings for all Adapters.

query on powershell script that downloads the Microsoft Ebook Giveaway books

Got a whatsapp message that Microsoft is giving away free ebooks from the below url.
URL : Microsoft Ebook Giveaway
To download all the books in one go, the following powershell script was used, which is available in the same url.
Now my problem is, if I run the powershell script as a whole, it is not throwing any error. All the books from the url gets downloaded to a single location in my computer.
But if I try to run the script line by line to understand what each statement does, it is giving the following error when the , $bookList = Invoke-WebRequest $downLoadList gets executed,
Now to resolve this error, there are many others posts in stack overflow, that passes the username and password to overcome this error. Those scripts / solutions are not working at my end.
More than the error, why is it, that the script runs without any errors/issues when I execute the full script, but throws an error when I execute line by line ?
Any inputs on the nature of execution or helpful tips in overcoming the error will be useful... Thank you.
Error :
Invoke-WebRequest : (my ip number )
Credentials are missing.
Make sure to specify a domain with your username
This website has been blocked by a cyber security policy
and SecureWeb does not currently support web exceptions
If you have an exception, copy the link below into a new tab
http://ligman.me/2tk1D2V
At line:1 char:13
+ $bookList = Invoke-WebRequest $downLoadList
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest)
[Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.
InvokeWebRequestCommand
PowerShell Script :
###############################################################
# Eric Ligmans Amazing Free Microsoft eBook Giveaway
# https://blogs.msdn.microsoft.com/mssmallbiz/2017/07/11/largest-free-microsoft-ebook-giveaway-im-giving-away-millions-of-free-microsoft-ebooks-again-including-windows-10-office-365-office-2016-power-bi-azure-windows-8-1-office-2013-sharepo/
# Link to download list of eBooks
# http://ligman.me/2tk1D2V
# Thanks David Crosby for the template (https://social.technet.microsoft.com/profile/david%20crosby/)
#
# Modified by Robert Cain (http://arcanecode.me)
# Added code to check to see if a book was already downloaded,
# and if so was it the correct file size. If so, the book
# download is skipped. This allows users to simply rerun the
# script if their download process is interrupted.
###############################################################
# Set the folder where you want to save the books to
$dest = "I:\new_microsoft\" # Make sure the file path ends in a \
# Download the source list of books
$downLoadList = "http://ligman.me/2tk1D2V"
$bookList = Invoke-WebRequest $downLoadList
# Convert the list to an array
[string[]]$books = ""
$books = $bookList.Content.Split("`n")
# Remove the first line - it's not a book
$books = $books[1..($books.Length -1)]
$books # Here's the list
# Get the total number of books we need to download
$bookCount = $($books).Count
# Set a simple counter to let the user know what book
# number we're currently downloading
$currentBook = 0
# As an option, we can have it log progress to a file
$log = $true
if ($log -eq $true)
{
# Construct a log file name based on the date that
# we can save progress to
$dlStart = Get-Date
$dlStartDate = "$($dlStart.Year)-$($dlStart.Month)-$($dlStart.Day)"
$dlStartTime = "$($dlStart.Hour)-$($dlStart.Minute)-$($dlStart.Second)"
$logFile = "$($dest)BookDlLog-$dlStartDate-$dlStartTime.txt"
}
# Download the books
foreach ($book in $books)
{
# Increment current book number
$currentBook++
try
{
# Grab the header with the books full info
$hdr = Invoke-WebRequest $book -Method Head
# Get the title of the book from the header then
# make it a safe string (remove special characters)
$title = $hdr.BaseResponse.ResponseUri.Segments[-1]
$title = [uri]::UnescapeDataString($title)
# Construct the path to save the file to
$saveTo = $dest + $title
# If the file doesn't exist, download it
if ($(Test-Path $saveTo) -eq $false)
{
$msg = "Downloading book $currentBook of $bookCount - $title"
$msg
if ($log -eq $true) { "`n$($msg)" | Add-Content $logFile }
Invoke-WebRequest $book -OutFile $saveTo
}
else
{
# If it does exist, we need to make sure it wasn't
# a partial download. If the file size on the server
# and the file size on local disk don't match,
# redownload it
# Get the size of the file from the download site
$dlSize = $hdr.BaseResponse.ContentLength
# Get the size of the file on disk
$fileSize = $(Get-ChildItem $saveTo).Length
if ($dlSize -ne $fileSize)
{
# If not equal we need to download the book again
$msg = "Redownloading book $currentBook of $bookCount - $title"
$msg
if ($log -eq $true) { "`n$($msg)" | Add-Content $logFile }
Invoke-WebRequest $book -OutFile $saveTo
}
else
{
# Otherwise we have a good copy of the book, just
# let the user know we're skipping it.
$msg = "Book $currentBook of $bookCount ($title) already exists, skipping it"
$msg
if ($log -eq $true) { "`n$($msg)" | Add-Content $logFile }
}
}
} # end try
catch
{
$msg = "There was an error downloading $title. You may wish to try to download this book manually."
Write-Host $msg -ForegroundColor Red
if ($log -eq $true) { "`n$($msg)" | Add-Content $logFile }
} # end catch
} # end foreach
# Let user know we're done, and give a happy little beep
# in case they aren't looking at the screen.
#"Done downloading all books"
#[Console]::Beep(500,300)

Error Message from Powershell Script mail to EmailAddress

I have constructed an script for our Dataloader in Salesforce:
**# The Function for the Check Error File**
function CheckErrorFile {
Param ([string]$perrorfilepath)
$csvlines = (Get-Content $perrorfilepath | Measure-Object -line).Lines
# header is always 1 row so we look for > 1 rows
if ($csvlines -gt 1)
{
$errorCount = $csvLines - 1
$errorMessage = [string]::Format("** errors ** {0} failed rows in {1}", $errorCount, $perrorfilepath)
# this will print red error message but not stop execution
write-error $errorMessage
}
}
**# set up locations/variables for data loader**
$dataloader = "C:\Program Files (x86)\salesforce.com\Data Loader\bin\process.bat"
$dlconfig = "U:\Projekte\Salesforce\Dataloader-Test-Infor-Files\Example-Migration-SF-Test\DataLoaderConfig"
set-location "C:\Program Files (x86)\salesforce.com\Data Loader\bin"
$dlLogFolder = "U:\Projekte\Salesforce\Dataloader-Test-Infor-Files\Example-Migration-SF-Test\Logs\"
**# execute the data loader**
& $dataloader $dlconfig testsf-delete
**# check for errors in the error.csv file**
CheckErrorFile ("{0}error_testsf.csv" -f $dlLogFolder)
So far so good it think
But what i want to is that the Error Code which comes Back from CheckErrorFile Command should sent to an Email Address.
So i had think about 2 Considerations:
when if error sent to mail
pack the error output in a variable and sent to mail adress
But i'm not so good at Powershell that i know how i have to integrate this in my script.
I found a site with an send trigger when a eventlog has changed but I'm not sure if i can use this for my purposes and how i have to integrate this:
Powershell Email when Eventlog is Changed
May someone could help
thnks

Unable to delete a directory with square brackets in name using WinSCP and PowerShell

I am trying to write a PowerShell script that automatically deletes empty directories on our FTP server. I don't have any direct access to the server on which the directories reside - I can only access them via FTP. For this reason, I have written a PowerShell script that uses WinSCP .NET assembly to (try to) delete the empty directories. However, I have a problem, in that many of the directories on the server have square brackets in the directory name.
For example, the directory name might be called: [a]153432
There are lots of these directories, hence my desire to write a script to delete them. This occurs because they've been created by a program that uses a number to create the directories it requires. In order to work on this program, I created an empty directory
/latest/[b]test
My program goes like this:
# program to test deletion of directories with square-brackets in the name.
$HOSTNAME="myftpservername"
$USERNAME="myftpusername"
$PASSWORD="myftppassword"
$DLL_LOCATION="C:\Program Files (x86)\WinSCP\WinSCPnet.dll"
$LOCAL_DIRECTORY="C:\testdir\extended\baseroot"
# Load WinSCP .NET assembly
[Reflection.Assembly]::LoadFrom($DLL_LOCATION) | Out-Null
# Session.FileTransferred event handler
try
{
$sessionOptions = New-Object WinSCP.SessionOptions
$sessionOptions.Protocol = [WinSCP.Protocol]::ftp
$sessionOptions.HostName = $HOSTNAME
$sessionOptions.UserName = $USERNAME
$sessionOptions.Password = $PASSWORD
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
$remoteFileWithUnixPath = "/latest/[b]test"
$removalResult = $session.RemoveFiles($remoteFileWithUnixPath)
if ($removalResult.IsSuccess)
{
Write-Host ("Removal of remote file {0} succeeded" -f $remoteFileWithUnixPath)
}
else
{
Write-Host ("Removal of remote file {0} failed" -f $remoteFileWithUnixPath)
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
exit 0
}
catch [Exception]
{
Write-Host $_.Exception.Message
exit 1
}
When I run it, it displays the following message:
PS C:\Users\dbuddrige\Documents\dps> .\delete-squarebracket-dir.ps1
Removal of remote file /latest/[b]test succeeded
However, the directory is not deleted.
I have also tried specifying the directory name with delimiters such as:
$remoteFileWithUnixPath = "/latest/\[b\]test"
But this also fails (but at least it says so):
PS C:\Users\dbuddrige\Documents\dps> .\delete-squarebracket-dir.ps1
Removal of remote file /latest/\[b\]test failed
BUT, if I change the directory name [and the variable $remoteFileWithUnixPath] to something like
/latest/foo
And then re-run the program, it deletes the directory /latest/foo just fine.
Does anyone have any ideas what I need to do to get this to work?
The answer to this question was pointed out by Martin Prikryl [Thanks!]
The fully working code follows:
# program to test deletion of directories with square-brackets in the name.
$HOSTNAME="myftpservername"
$USERNAME="myftpusername"
$PASSWORD="myftppassword"
$DLL_LOCATION="C:\Program Files (x86)\WinSCP\WinSCPnet.dll"
$LOCAL_DIRECTORY="C:\testdir\extended\baseroot"
# Load WinSCP .NET assembly
[Reflection.Assembly]::LoadFrom($DLL_LOCATION) | Out-Null
# Session.FileTransferred event handler
try
{
$sessionOptions = New-Object WinSCP.SessionOptions
$sessionOptions.Protocol = [WinSCP.Protocol]::ftp
$sessionOptions.HostName = $HOSTNAME
$sessionOptions.UserName = $USERNAME
$sessionOptions.Password = $PASSWORD
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
$remoteFileWithUnixPath = "/latest/[b]test"
$removalResult =
$session.RemoveFiles($session.EscapeFileMask($remoteFileWithUnixPath))
if ($removalResult.IsSuccess)
{
Write-Host ("Removal of remote file {0} succeeded" -f $remoteFileWithUnixPath)
}
else
{
Write-Host ("Removal of remote file {0} failed" -f $remoteFileWithUnixPath)
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
exit 0
}
catch [Exception]
{
Write-Host $_.Exception.Message
exit 1
}
Some methods of the WinSCP .NET assembly, including the the Session.RemoveFiles, accept a file mask, not a simple path.
Square brackets have special meaning in the file mask.
You should always use the RemotePath.EscapeFileMask method on file paths, before passing them to assembly methods.
I'd use the escape sequence in the commandline:
Remove-Item 'C:\Scripts\Test`[1`].txt'
Or use the -LiteralPath parameter.
Remove-Item -LiteralPath C:\scripts\test[1].txt
The -LiteralPath parameter does not try to convert the string.

Converting byte array to true/false

Through various different posts on StackOverflow and other places I was able to put together a powershell script that FTP uploads files and it works great. However I wanted to add a bit more verbosity to it. See code below:
foreach ($file in $uploadfiles)
{
# create the full path to the file on remote server (odd but okay!)
$ftp_command = $ftp + $file
# for debugging
#$ftp_command
# create a new URI object for the full path of the file
$uri = New-Object System.URI($ftp_command)
#for debugging
#$uri
# finally do our upload to the remote server - URI object, full path to local file
#$responseArray = $ftpclient.UploadFile($uri,$file.Fullname)
$result = $ftpclient.UploadFile($uri,$file.Fullname)
if ($result) {
$file.Fullname + " uploaded successfully"
} else {
$file.Fullname + " not uploaded successfully"
}
}
Basically after the file is uploaded I wanted to check and see if it was successful or not. Upload file is supposed to return a byte array ( http://msdn.microsoft.com/en-us/library/36s52zhs(v=vs.80).aspx ; A Byte array containing the body of the response from the resource.). I'm new to powershell so that's probably where my problem is, but for the life of me I can't get anything to come out of $result so I can test. Is it possible the server isn't returning anything or I'm just not accessing/setting the the byte array correctly? I've tried a variety of different things, but haven't yet figured anything out.
Thanks,
I would not use the $result in your case begining PowerShell 2.0 you can use the try/catch sections.
try
{
$ftpclient.UploadFile($uri,$file.Fullname)
}
catch [System.Net.WebException]
{
# here $_ gives you the exception details
}