Uploading a File via FTP using PowerShell - powershell

I am writing a Powershell script that watches a directory, and when a file (or multiple) is uploaded into the directory, it takes those files, copies them to another folder, sends them to an FTP server, and then deletes the file from the original directory.
I am having problems connecting to the FTP server. I am not sure if the problem is the way I am configuring the Web Client, or if the problem is that the ftp URI has spaces in it and I am not escaping them properly.
Here is the code:
$source = "c:/testFtp"
$ftpdestination = "ftp://username:password#ftp.ftpsite.com/folder with space/folder with space"
$webclient = New-Object -TypeName System.Net.WebClient
$files = Get-ChildItem $source
foreach ($file in $files)
{
Write-Host "Uploading $file"
try {
$ftp = "$ftpdestination/$file"
$uri = New-Object -TypeName System.Uri -ArgumentList $ftp
$webclient.UploadFile($uri, $source/$file)
} catch {
Add-content "$logs" -value "There was an error uploading to ftp"
}
}
$webclient.Dispose()
I have tried escaping the folder spaces multiple ways, so I am beginning to think that is not the problem and that I am not configuring the web client properly.
It is also not catching errors very often, so I don't believe it throws an error when the webclient has failure on the upload. Any help is appreciated!

ANSWER:
It turns out the WebClient was connecting properly, but SSL was blocking the files from being sent. I found this out by using C# to compile and run the script, and it gave me better error handling, as I am new to Powershell scripts and cannot seem to get good error handling.
After researching, I could not find a way to enable SSL with WebClient, so I switched over to FtpWebRequest. Here is the successful code (This try catch block does not seem to log errors as I would like, but the tool will successfully send files to the ftp server now:
try {
$ftp = [System.Net.FtpWebRequest]::Create("$ftpsite/$filename")
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential("$username","$password")
$ftp.UseBinary = $true
$ftp.UsePassive = $true
$ftp.EnableSSL = $true #<-----------------This was the line that made this work
$ftp.KeepAlive = $false
# read in the file to upload as a byte array
$content = [System.IO.File]::ReadAllBytes("$source/$file")
$ftp.ContentLength = $content.Length
# get the request stream, and write the bytes into it
$rs = $ftp.GetRequestStream()
$rs.Write($content, 0, $content.Length)
# be sure to clean up after ourselves
$rs.Close()
$rs.Dispose()
Write-Host "Successfully uploaded: $source/$file"
Copy-Item "$source/$file" -Destination "$copydestination"
Remove-Item "$source/$file"
$logline = "$(Get-Date), Added File: $file to $copydestination"
Add-content "$logs" -value $logline
} catch {
$res = $ftp.GetResponse()
Add-content "$logs" -value "There was an error: $res.StatusCode"
Write-Error -Message "There was an error." -ErrorAction Stop
}

Related

WinSCP Script to Loop through Files

I am using the following slightly modified script (from https://winscp.net/eng/docs/script_local_move_after_successful_upload) to upload files to an SFTP site on AWS...and in a PRD env I will have to have this run through approx 0.5M small files...
param (
$localPath = "C:\FTP\*.DAT",
$remotePath = "/",
$backupPath = "C:\FTP\Complete\"
)
try
{
# Load WinSCP .NET assembly
#Add-Type -Path "WinSCPnet.dll"
$ScriptPath = $(Split-Path -Parent $MyInvocation.MyCommand.Definition)
[Reflection.Assembly]::UnsafeLoadFrom( $(Join-Path $ScriptPath "WinSCPnet.dll") ) | Out-Null
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "somewhere.com"
UserName = "user"
SshHostKeyFingerprint = "ssh-rsa 2048 XXXXXXXI"
SshPrivateKeyPath = "C:\blahblah.ppk"
}
$session = New-Object WinSCP.Session
$transferOptions = New-Object WinSCP.TransferOptions
# Look to ignore any file property change errors
$transferOptions.FilePermissions = $Null # This is default
$transferOptions.PreserveTimestamp = $False
try
{
# Connect
$session.Open($sessionOptions)
# Upload files, collect results
$transferResult = $session.PutFiles($localPath, $remotePath)
# Iterate over every transfer
foreach ($transfer in $transferResult.Transfers)
{
# Success or error?
if ($Null -eq $transfer.Error)
{
Write-Host "Upload of $($transfer.FileName) succeeded, moving to backup"
# Upload succeeded, move source file to backup
Move-Item $transfer.FileName $backupPath
}
else
{
Write-Host "Upload of $($transfer.FileName) failed: $($transfer.Error.Message)"
}
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
exit 0
}
catch
{
Write-Host "Error: $($_.Exception.Message)"
exit 1
}
The scripts works great, but unfortunately only loads one file before disconnecting. The script fails if there is an error loading any of the files, so it is working as expected.
But the error I am getting, is
"Upload of file 'somefile.DAT' was successful, but error occurred while setting the permissions and/or timestamp.
If the problem persists, turn off setting permissions or preserving timestamp. Alternatively you can turn on 'Ignore permission errors' option.
The server does not support the operation.
Error code: 8
Error message from server (US-ASCII): SETSTAT unsupported"
I think I have the following settings possibly configured incorrectly, but I'm not sure what I am doing wrong here....thoughts?
$transferOptions.FilePermissions = $Null # This is default
$transferOptions.PreserveTimestamp = $False
I've actually managed to get this to work by modifying the session and transfer options..
$session.Open($sessionOptions)
$transferOptions = New-Object WinSCP.TransferOptions
# Look to ignore any file property change errors
$transferOptions.PreserveTimeStamp = $false
$transferOptions.FilePermissions = $Null
$transferOptions.AddRawSettings("IgnorePermErrors", "1")
# Upload files, collect results
$transferResult = $session.PutFiles($localPath, $remotePath, $False, $transferOptions)

Need help for ftp files status

i have been looking for a way to know the status of my ftp files, there are few log files are being uploading on my ftp server after every 15mints, but few times it fails to upload i just want an alert when ever a file fails to upload.
following code has been tired
function update {
$ftprequest = [System.Net.FtpWebRequest]::Create("ftp://ftpsite.com/Script_Apps/install_firefox.exe")
$ftprequest.Method = [System.Net.WebRequestMethods+Ftp]::GetDateTimestamp
$response = $ftprequest.GetResponse().StatusDescription
$tokens = $response.Split(" ")
$code = $tokens[0]
$localfile = (Get-Item "$dir\Apps\install_firefox.exe").LastWriteTimeUtc
if ($tokens -gt $localfile) {
write-host "Updating Firefox Installer..."
$File = "$dir\Apps\install_firefox.exe"
$ftp = "ftp://ftpsite.com/Script_Apps/install_firefox.exe"
$webclient = New-Object System.Net.WebClient
$uri = New-Object System.Uri($ftp)
$webclient.DownloadFile($uri, $File)
"Updated Firefox" >> $global:logfile
mainmenu
}
else {
Write-Host "Local Copy is Newer."
sleep 3
mainmenu
}
}
I'm going to assume this line deals with downloading/uploading files for your FTP site.
$ftp = "ftp://ftpsite.com/Script_Apps/install_firefox.exe"
$webclient = New-Object System.Net.WebClient
$uri = New-Object System.Uri($ftp)
$webclient.DownloadFile($uri, $File)
If so, then we can lookout for errors and do something when one occurs with try{}catch{} blocks.
You can put any code inside a try block, and when it hits a terminating error it will capture the error as $Error[0] which we can reference in a catch block.
try{
$ftp = "ftp://ftpsite.com/Script_Apps/install_firefox.exe"
$webclient = New-Object System.Net.WebClient
$uri = New-Object System.Uri($ftp)
$webclient.DownloadFile($uri, $File)
}
catch{
#Write out an error to the screen
"failed to download file $($ftp). `n--Ran into error $($Error[0].Exception.Message)"
#Todo - send a notification
}
This will give a result like this, when it fails to download.
Failed to download file ftp://ftpsite.com/Script_Apps/install_firefox.exe.
---Ran into error Exception calling "DownloadFile" with "2" argument(s):
"Unable to connect to the remote server"
Then, you can determine the best way to send a notification to suit your needs. There's way's to send e-mail from PowerShell, or Tweets, or even using a Push message to the Pushbullet app on your phone!

Session.getfiles() not working correctly - Powershell

Trying to run the following script. Connect to a ftp / sftp server and get the files from a remotepath to a localpath
Since there are a list of sources and destinations, I decided to create a csv file and pass the value using $.Source and $.Destination
Session.getfiles () seems to not be working when I import my csv file.
But if I hard code the path, it seems to work.
$session.GetFiles("c:\source\test", "c:\destination\", $False, $transferOptions)
Goal: Script reads the list of Sources and move files according to its destination. Also go back to a # amount of days and downdload the latest file -
Error: "No such File" or "cant get attributes of file"
try{
Add-Type -Path "WinSCPnet.dll"
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::sftp
HostName = "xxxxx"
UserName = "xxxxr"
Password = "xxxxxxx"
PortNumber="xx"
FTPMode="Active"
GiveUpSecurityAndAcceptAnySshHostKey = $true
}
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Download files
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.TransferMode = [WinSCP.TransferMode]::Binary
Import-Csv -Path "C:\movefiles.csv" -ErrorAction Stop
$transferResult =
$session.GetFiles($_.Source, $_.Destination, $False, $transferOptions)
# Throw on any error
$transferResult.Check()
# Print results
foreach ($transfer in $transferResult.Transfers)
{
Write-Host "Download of $($transfer.FileName) succeeded"
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
#exit 0
}
catch
{
Write-Host "Error: $($_.Exception.Message)"
#exit 1
}
By commenting the line out
$transferresult.check()
Script seems to work. But still missing how to get only new files

PowerShell Script to download an entire folder to FTP [duplicate]

I like to write a PowerShell script to download all files and subfolders from my FTP server. I found a script to download all files from one specific folder, but I also like to download the subfolders and their files.
#FTP Server Information - SET VARIABLES
$ftp = "ftp://ftp.abc.ch/"
$user = 'abc'
$pass = 'abc'
$folder = '/'
$target = "C:\LocalData\Powershell\"
#SET CREDENTIALS
$credentials = new-object System.Net.NetworkCredential($user, $pass)
function Get-FtpDir ($url,$credentials) {
$request = [Net.WebRequest]::Create($url)
$request.Method = [System.Net.WebRequestMethods+FTP]::ListDirectory
if ($credentials) { $request.Credentials = $credentials }
$response = $request.GetResponse()
$reader = New-Object IO.StreamReader $response.GetResponseStream()
$reader.ReadToEnd()
$reader.Close()
$response.Close()
}
#SET FOLDER PATH
$folderPath= $ftp + "/" + $folder + "/"
$Allfiles=Get-FTPDir -url $folderPath -credentials $credentials
$files = ($Allfiles -split "`r`n")
$files
$webclient = New-Object System.Net.WebClient
$webclient.Credentials = New-Object System.Net.NetworkCredential($user,$pass)
$counter = 0
foreach ($file in ($files | where {$_ -like "*.*"})){
$source=$folderPath + $file
$destination = $target + $file
$webclient.DownloadFile($source, $target+$file)
#PRINT FILE NAME AND COUNTER
$counter++
$counter
$source
}
Thanks for your help (:
The .NET framework or PowerShell do not have any explicit support for recursive file operations (including downloads). You have to implement the recursion yourself:
List the remote directory
Iterate the entries, downloading files and recursing into subdirectories (listing them again, etc.)
Tricky part is to identify files from subdirectories. There's no way to do that in a portable way with the .NET framework (FtpWebRequest or WebClient). The .NET framework unfortunately does not support the MLSD command, which is the only portable way to retrieve directory listing with file attributes in FTP protocol. See also Checking if object on FTP server is file or directory.
Your options are:
Do an operation on a file name that is certain to fail for file and succeeds for directories (or vice versa). I.e. you can try to download the "name". If that succeeds, it's a file, if that fails, it a directory.
You may be lucky and in your specific case, you can tell a file from a directory by a file name (i.e. all your files have an extension, while subdirectories do not)
You use a long directory listing (LIST command = ListDirectoryDetails method) and try to parse a server-specific listing. Many FTP servers use *nix-style listing, where you identify a directory by the d at the very beginning of the entry. But many servers use a different format. The following example uses this approach (assuming the *nix format)
function DownloadFtpDirectory($url, $credentials, $localPath)
{
$listRequest = [Net.WebRequest]::Create($url)
$listRequest.Method =
[System.Net.WebRequestMethods+Ftp]::ListDirectoryDetails
$listRequest.Credentials = $credentials
$lines = New-Object System.Collections.ArrayList
$listResponse = $listRequest.GetResponse()
$listStream = $listResponse.GetResponseStream()
$listReader = New-Object System.IO.StreamReader($listStream)
while (!$listReader.EndOfStream)
{
$line = $listReader.ReadLine()
$lines.Add($line) | Out-Null
}
$listReader.Dispose()
$listStream.Dispose()
$listResponse.Dispose()
foreach ($line in $lines)
{
$tokens = $line.Split(" ", 9, [StringSplitOptions]::RemoveEmptyEntries)
$name = $tokens[8]
$permissions = $tokens[0]
$localFilePath = Join-Path $localPath $name
$fileUrl = ($url + $name)
if ($permissions[0] -eq 'd')
{
if (!(Test-Path $localFilePath -PathType container))
{
Write-Host "Creating directory $localFilePath"
New-Item $localFilePath -Type directory | Out-Null
}
DownloadFtpDirectory ($fileUrl + "/") $credentials $localFilePath
}
else
{
Write-Host "Downloading $fileUrl to $localFilePath"
$downloadRequest = [Net.WebRequest]::Create($fileUrl)
$downloadRequest.Method =
[System.Net.WebRequestMethods+Ftp]::DownloadFile
$downloadRequest.Credentials = $credentials
$downloadResponse = $downloadRequest.GetResponse()
$sourceStream = $downloadResponse.GetResponseStream()
$targetStream = [System.IO.File]::Create($localFilePath)
$buffer = New-Object byte[] 10240
while (($read = $sourceStream.Read($buffer, 0, $buffer.Length)) -gt 0)
{
$targetStream.Write($buffer, 0, $read);
}
$targetStream.Dispose()
$sourceStream.Dispose()
$downloadResponse.Dispose()
}
}
}
Use the function like:
$credentials = New-Object System.Net.NetworkCredential("user", "mypassword")
$url = "ftp://ftp.example.com/directory/to/download/"
DownloadFtpDirectory $url $credentials "C:\target\directory"
The code is translated from my C# example in C# Download all files and subdirectories through FTP.
Though Microsoft does not recommend FtpWebRequest for a new development.
If you want to avoid troubles with parsing the server-specific directory listing formats, use a 3rd party library that supports the MLSD command and/or parsing various LIST listing formats; and recursive downloads.
For example with WinSCP .NET assembly you can download whole directory with a single call to Session.GetFiles:
# Load WinSCP .NET assembly
Add-Type -Path "WinSCPnet.dll"
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::Ftp
HostName = "ftp.example.com"
UserName = "user"
Password = "mypassword"
}
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Download files
$session.GetFiles("/directory/to/download/*", "C:\target\directory\*").Check()
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
Internally, WinSCP uses the MLSD command, if supported by the server. If not, it uses the LIST command and supports dozens of different listing formats.
The Session.GetFiles method is recursive by default.
(I'm the author of WinSCP)
For retrieving files /folder from FTP via powerShell I wrote some functions, you can get even hidden stuff from FTP.
Example for getting all files and subfolders (even hidden ones) in a specific folder:
Get-FtpChildItem -ftpFolderPath "ftp://myHost.com/root/leaf/" -userName "User" -password "pw" -Directory -File
You can just copy the functions from the following module without needing any 3rd library installed:
https://github.com/AstralisSomnium/PowerShell-No-Library-Just-Functions/blob/master/FTPModule.ps1

PowerShell FTP download files and subfolders

I like to write a PowerShell script to download all files and subfolders from my FTP server. I found a script to download all files from one specific folder, but I also like to download the subfolders and their files.
#FTP Server Information - SET VARIABLES
$ftp = "ftp://ftp.abc.ch/"
$user = 'abc'
$pass = 'abc'
$folder = '/'
$target = "C:\LocalData\Powershell\"
#SET CREDENTIALS
$credentials = new-object System.Net.NetworkCredential($user, $pass)
function Get-FtpDir ($url,$credentials) {
$request = [Net.WebRequest]::Create($url)
$request.Method = [System.Net.WebRequestMethods+FTP]::ListDirectory
if ($credentials) { $request.Credentials = $credentials }
$response = $request.GetResponse()
$reader = New-Object IO.StreamReader $response.GetResponseStream()
$reader.ReadToEnd()
$reader.Close()
$response.Close()
}
#SET FOLDER PATH
$folderPath= $ftp + "/" + $folder + "/"
$Allfiles=Get-FTPDir -url $folderPath -credentials $credentials
$files = ($Allfiles -split "`r`n")
$files
$webclient = New-Object System.Net.WebClient
$webclient.Credentials = New-Object System.Net.NetworkCredential($user,$pass)
$counter = 0
foreach ($file in ($files | where {$_ -like "*.*"})){
$source=$folderPath + $file
$destination = $target + $file
$webclient.DownloadFile($source, $target+$file)
#PRINT FILE NAME AND COUNTER
$counter++
$counter
$source
}
Thanks for your help (:
The .NET framework or PowerShell do not have any explicit support for recursive file operations (including downloads). You have to implement the recursion yourself:
List the remote directory
Iterate the entries, downloading files and recursing into subdirectories (listing them again, etc.)
Tricky part is to identify files from subdirectories. There's no way to do that in a portable way with the .NET framework (FtpWebRequest or WebClient). The .NET framework unfortunately does not support the MLSD command, which is the only portable way to retrieve directory listing with file attributes in FTP protocol. See also Checking if object on FTP server is file or directory.
Your options are:
Do an operation on a file name that is certain to fail for file and succeeds for directories (or vice versa). I.e. you can try to download the "name". If that succeeds, it's a file, if that fails, it a directory.
You may be lucky and in your specific case, you can tell a file from a directory by a file name (i.e. all your files have an extension, while subdirectories do not)
You use a long directory listing (LIST command = ListDirectoryDetails method) and try to parse a server-specific listing. Many FTP servers use *nix-style listing, where you identify a directory by the d at the very beginning of the entry. But many servers use a different format. The following example uses this approach (assuming the *nix format)
function DownloadFtpDirectory($url, $credentials, $localPath)
{
$listRequest = [Net.WebRequest]::Create($url)
$listRequest.Method =
[System.Net.WebRequestMethods+Ftp]::ListDirectoryDetails
$listRequest.Credentials = $credentials
$lines = New-Object System.Collections.ArrayList
$listResponse = $listRequest.GetResponse()
$listStream = $listResponse.GetResponseStream()
$listReader = New-Object System.IO.StreamReader($listStream)
while (!$listReader.EndOfStream)
{
$line = $listReader.ReadLine()
$lines.Add($line) | Out-Null
}
$listReader.Dispose()
$listStream.Dispose()
$listResponse.Dispose()
foreach ($line in $lines)
{
$tokens = $line.Split(" ", 9, [StringSplitOptions]::RemoveEmptyEntries)
$name = $tokens[8]
$permissions = $tokens[0]
$localFilePath = Join-Path $localPath $name
$fileUrl = ($url + $name)
if ($permissions[0] -eq 'd')
{
if (!(Test-Path $localFilePath -PathType container))
{
Write-Host "Creating directory $localFilePath"
New-Item $localFilePath -Type directory | Out-Null
}
DownloadFtpDirectory ($fileUrl + "/") $credentials $localFilePath
}
else
{
Write-Host "Downloading $fileUrl to $localFilePath"
$downloadRequest = [Net.WebRequest]::Create($fileUrl)
$downloadRequest.Method =
[System.Net.WebRequestMethods+Ftp]::DownloadFile
$downloadRequest.Credentials = $credentials
$downloadResponse = $downloadRequest.GetResponse()
$sourceStream = $downloadResponse.GetResponseStream()
$targetStream = [System.IO.File]::Create($localFilePath)
$buffer = New-Object byte[] 10240
while (($read = $sourceStream.Read($buffer, 0, $buffer.Length)) -gt 0)
{
$targetStream.Write($buffer, 0, $read);
}
$targetStream.Dispose()
$sourceStream.Dispose()
$downloadResponse.Dispose()
}
}
}
Use the function like:
$credentials = New-Object System.Net.NetworkCredential("user", "mypassword")
$url = "ftp://ftp.example.com/directory/to/download/"
DownloadFtpDirectory $url $credentials "C:\target\directory"
The code is translated from my C# example in C# Download all files and subdirectories through FTP.
Though Microsoft does not recommend FtpWebRequest for a new development.
If you want to avoid troubles with parsing the server-specific directory listing formats, use a 3rd party library that supports the MLSD command and/or parsing various LIST listing formats; and recursive downloads.
For example with WinSCP .NET assembly you can download whole directory with a single call to Session.GetFiles:
# Load WinSCP .NET assembly
Add-Type -Path "WinSCPnet.dll"
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::Ftp
HostName = "ftp.example.com"
UserName = "user"
Password = "mypassword"
}
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Download files
$session.GetFiles("/directory/to/download/*", "C:\target\directory\*").Check()
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
Internally, WinSCP uses the MLSD command, if supported by the server. If not, it uses the LIST command and supports dozens of different listing formats.
The Session.GetFiles method is recursive by default.
(I'm the author of WinSCP)
For retrieving files /folder from FTP via powerShell I wrote some functions, you can get even hidden stuff from FTP.
Example for getting all files and subfolders (even hidden ones) in a specific folder:
Get-FtpChildItem -ftpFolderPath "ftp://myHost.com/root/leaf/" -userName "User" -password "pw" -Directory -File
You can just copy the functions from the following module without needing any 3rd library installed:
https://github.com/AstralisSomnium/PowerShell-No-Library-Just-Functions/blob/master/FTPModule.ps1