Trying to run the following script. Connect to a ftp / sftp server and get the files from a remotepath to a localpath
Since there are a list of sources and destinations, I decided to create a csv file and pass the value using $.Source and $.Destination
Session.getfiles () seems to not be working when I import my csv file.
But if I hard code the path, it seems to work.
$session.GetFiles("c:\source\test", "c:\destination\", $False, $transferOptions)
Goal: Script reads the list of Sources and move files according to its destination. Also go back to a # amount of days and downdload the latest file -
Error: "No such File" or "cant get attributes of file"
try{
Add-Type -Path "WinSCPnet.dll"
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::sftp
HostName = "xxxxx"
UserName = "xxxxr"
Password = "xxxxxxx"
PortNumber="xx"
FTPMode="Active"
GiveUpSecurityAndAcceptAnySshHostKey = $true
}
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Download files
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.TransferMode = [WinSCP.TransferMode]::Binary
Import-Csv -Path "C:\movefiles.csv" -ErrorAction Stop
$transferResult =
$session.GetFiles($_.Source, $_.Destination, $False, $transferOptions)
# Throw on any error
$transferResult.Check()
# Print results
foreach ($transfer in $transferResult.Transfers)
{
Write-Host "Download of $($transfer.FileName) succeeded"
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
#exit 0
}
catch
{
Write-Host "Error: $($_.Exception.Message)"
#exit 1
}
By commenting the line out
$transferresult.check()
Script seems to work. But still missing how to get only new files
Related
I am using the following slightly modified script (from https://winscp.net/eng/docs/script_local_move_after_successful_upload) to upload files to an SFTP site on AWS...and in a PRD env I will have to have this run through approx 0.5M small files...
param (
$localPath = "C:\FTP\*.DAT",
$remotePath = "/",
$backupPath = "C:\FTP\Complete\"
)
try
{
# Load WinSCP .NET assembly
#Add-Type -Path "WinSCPnet.dll"
$ScriptPath = $(Split-Path -Parent $MyInvocation.MyCommand.Definition)
[Reflection.Assembly]::UnsafeLoadFrom( $(Join-Path $ScriptPath "WinSCPnet.dll") ) | Out-Null
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "somewhere.com"
UserName = "user"
SshHostKeyFingerprint = "ssh-rsa 2048 XXXXXXXI"
SshPrivateKeyPath = "C:\blahblah.ppk"
}
$session = New-Object WinSCP.Session
$transferOptions = New-Object WinSCP.TransferOptions
# Look to ignore any file property change errors
$transferOptions.FilePermissions = $Null # This is default
$transferOptions.PreserveTimestamp = $False
try
{
# Connect
$session.Open($sessionOptions)
# Upload files, collect results
$transferResult = $session.PutFiles($localPath, $remotePath)
# Iterate over every transfer
foreach ($transfer in $transferResult.Transfers)
{
# Success or error?
if ($Null -eq $transfer.Error)
{
Write-Host "Upload of $($transfer.FileName) succeeded, moving to backup"
# Upload succeeded, move source file to backup
Move-Item $transfer.FileName $backupPath
}
else
{
Write-Host "Upload of $($transfer.FileName) failed: $($transfer.Error.Message)"
}
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
exit 0
}
catch
{
Write-Host "Error: $($_.Exception.Message)"
exit 1
}
The scripts works great, but unfortunately only loads one file before disconnecting. The script fails if there is an error loading any of the files, so it is working as expected.
But the error I am getting, is
"Upload of file 'somefile.DAT' was successful, but error occurred while setting the permissions and/or timestamp.
If the problem persists, turn off setting permissions or preserving timestamp. Alternatively you can turn on 'Ignore permission errors' option.
The server does not support the operation.
Error code: 8
Error message from server (US-ASCII): SETSTAT unsupported"
I think I have the following settings possibly configured incorrectly, but I'm not sure what I am doing wrong here....thoughts?
$transferOptions.FilePermissions = $Null # This is default
$transferOptions.PreserveTimestamp = $False
I've actually managed to get this to work by modifying the session and transfer options..
$session.Open($sessionOptions)
$transferOptions = New-Object WinSCP.TransferOptions
# Look to ignore any file property change errors
$transferOptions.PreserveTimeStamp = $false
$transferOptions.FilePermissions = $Null
$transferOptions.AddRawSettings("IgnorePermErrors", "1")
# Upload files, collect results
$transferResult = $session.PutFiles($localPath, $remotePath, $False, $transferOptions)
I currently have a working script that upload files to a SFTP remote sirectory. The problem I am having is that there will be 3 files and they should be uploaded in sequence at different intervals. I have already thought about using Windows Task Scheduler as to take care of the frequency of the upload but there is another issue. I have identified that the files differ in naming based on one keyword. Is there a way I can modify my code to look for the files in the directory by a particular name? For example it searches in the directory for a file with "customer" in its name. Based on that keyword/name it then uploads that particular file. Please see current working script:
# Load WinSCP .NET assembly
Add-Type -Path "WinSCPnet.dll"
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "server"
UserName = "username"
Password = "password"
SshHostKeyFingerprint = "key"
}
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Upload files
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.TransferMode = [WinSCP.TransferMode]::Binary
$transferResult =
$session.PutFiles("E:\CMBPAID", "/NESAMSCARIMED", $False, $transferOptions)
# Throw on any error
$transferResult.Check()
# Print results
foreach ($transfer in $transferResult.Transfers)
{
Write-Host "Upload of $($transfer.FileName) succeeded"
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
Use file mask *customer*:
$transferResult =
$session.PutFiles("E:\CMBPAID\*customer*", "/NESAMSCARIMED/", $False, $transferOptions)
(note the slash added to the end of the target path)
Linux user here trying to write a script in PowerShell to automate SFTP transfers from a Windows server to a remote server. I have everything working for the most part (though I still have some testing to do). My current issue is after the script runs one time it is done. I need the FileSystemWatcher to run every time it finds a new file creation. I realize this is because of the exit but when I remove that portion and a user drops in a directory it loops through every file in the directory and uploads every file in the directory over and over. What I need it to do is run through the process once and then remain active to listen for another file creation event. My issue may be my limited workings with coding in Windows so any help is appreciated!
And yes I realize the duplication of code in the handling of my files vs my folders is redundant. I will most likely make that portion a function that accepts parameters depending on if it is given a file or folder.
I am posting my entire code in case it can help someone later
###Watches a directory for new files and fires off the batch file to push to Connexxion
$folder = "C:\pathTo\watchfolder"
$watcher = New-Object System.IO.FileSystemWatcher
$watcher.Path = $folder
$watcher.Filter = "*.csv"
$watcher.IncludeSubdirectories = $true
$watcher.EnableRaisingEvents = $true
### LISTEN FOR CREATE
Register-ObjectEvent $watcher Created -SourceIdentifier FileCreated -Action {
$folderpath = $(split-path -LiteralPath $event.SourceEventArgs.FullPath)
$filename = $event.SourceEventArgs.Name
$filepath = (Join-Path $folderpath $filename.split("\")[-1])
Write-Host folderpath: $folderpath
Write-Host filename: $filename
Write-Host filepath: $filepath
$remotePath = "/data/"
If($filename.contains("\")){
Write-Host "Here is directory"
try
{
#Set path to winscp
$assemblyPath = "C:\Program Files (x86)\WinSCP"
Add-Type -Path (Join-Path $assemblyPath "WinSCPnet.dll")
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "hostname"
UserName = "username"
Password = "passwd"
SshHostKeyFingerprint = "ssh-rsa 2048 xx:xx"
}
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Upload files, collect results
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.TransferMode = [WinSCP.TransferMode]::Binary
$transferResult =
$session.PutFiles($folderpath, $remotePath, $False, $transferOptions)
# Throw on any error
$transferResult.Check()
# Print results
foreach ($transfer in $transferResult.Transfers)
{
Write-Host "Upload of $($transfer.FileName) succeeded"
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
exit 0
}#end of first try
catch
{
Write-Host "Error: $($_.Exception.Message)"
exit 1
}
}
Else{
Write-Host "Here is a file"
try
{
#Set path to winscp
$assemblyPath = "C:\Program Files (x86)\WinSCP"
Add-Type -Path (Join-Path $assemblyPath "WinSCPnet.dll")
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "hostname"
UserName = "username"
Password = "passwd"
SshHostKeyFingerprint = "ssh-rsa 2048 xx:xx"
}
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Upload files, collect results
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.TransferMode = [WinSCP.TransferMode]::Binary
$transferResult =
$session.PutFiles($filepath, $remotePath, $False, $transferOptions)
# Throw on any error
$transferResult.Check()
# Print results
foreach ($transfer in $transferResult.Transfers)
{
Write-Host "Upload of $($transfer.FileName) succeeded"
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
exit 0
}#end of first try
catch
{
Write-Host "Error: $($_.Exception.Message)"
exit 1
}
}
}#end of action
while ($true) {sleep 5}
Just take the path to the new file and use RemotePath.TranslateLocalPathToRemote to map it to a server path, and upload the file there.
The code will be a lot simpler. All you need to do is to make sure new folders are re-created on the server.
# Watches a directory for new files and
# fires off the batch file to push to connection
$remotePath = "/data"
$localPath = "C:\pathTo\watchfolder"
$watcher = New-Object System.IO.FileSystemWatcher
$watcher.Path = $localPath
$watcher.Filter = "*.csv"
$watcher.IncludeSubdirectories = $True
$watcher.EnableRaisingEvents = $True
### LISTEN FOR CREATE
Register-ObjectEvent $watcher Created -SourceIdentifier FileCreated -Action {
try
{
$localFilePath = $event.SourceEventArgs.FullPath
Write-Host "Local path: $localFilePath"
$assemblyPath = "C:\Program Files (x86)\WinSCP"
Add-Type -Path (Join-Path $assemblyPath "WinSCPnet.dll")
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "example.com"
UserName = "username"
Password = "password"
SshHostKeyFingerprint = "ssh-rsa 2048 xxxxxxxxxxx...="
}
$session = New-Object WinSCP.Session
try
{
$remoteFilePath =
[WinSCP.RemotePath]::TranslateLocalPathToRemote(
$localFilePath, $localPath, $remotePath)
Write-Host "Remote path: $remoteFilePath"
# Connect
$session.Open($sessionOptions)
# Check if corresponding remote directory exists, if not, create it
$i = $remoteFilePath.LastIndexOf("/")
$remoteDirPath = $remoteFilePath.SubString(0, $i)
if (($remoteDirPath.Length -gt 0) -and
!$session.FileExists($remoteDirPath))
{
Write-Host "New subdirectory, creating $remoteDirPath on server"
$session.CreateDirectory($remoteDirPath)
}
$session.PutFiles($localFilePath, $remoteFilePath).Check()
Write-Host "Upload of $localFilePath succeeded"
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
} #end of first try
catch
{
Write-Host "Error: $($_.Exception.Message)"
}
} #end of action
while ($True) { sleep 5 }
I need to create a script that does the following:
Copies all files in a folder to an FTP site.
If the copy was successful move the files to an archive.
The archive should be a freshly created folder with today's date (so we know when they were transmitted).
I've tried to cannibalise other scripts to get something to work but I'm not getting anywhere so I need some help I've been working on this for hours.
I'm using the WinSCP DLL only because my other (working) script uses SFTP which needs it. I know normal FTP doesn't but I couldn't find any easily transferrable code so trying to modify that instead.
So, here's the code I have, which doesn't even run, never mind run properly, can someone help me get it right? Sorry it's a bit of a mess.
param (
$localPath = "c:\test\source",
$remotePath = "/upload",
$folder = ($_.CreationTime | Get-Date -Format yyyyMMdd)
# not sure this works but don't see how to point the destination
# to the newly created folder
$backupPath = "c:\test\destination\$folder"
)
try
{
# Load WinSCP .NET assembly
Add-Type -Path "C:\Windows\System32\WindowsPowerShell\v1.0\WinSCPnet.dll"
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::ftp
HostName = "xxxxxxxx"
UserName = "xxxxxxxx"
Password = "xxxxxxxx"
}
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Upload files, collect results
$transferResult = $session.PutFiles($localPath, $remotePath)
# Iterate over every transfer
foreach ($transfer in $transferResult.Transfers)
{
# Success or error?
if ($transfer.Error -eq $Null)
{
# If today's folder doesn't exist, create it
if (!(Test-Path $BackupPath))
{
New-Item -ItemType Directory -Force -Path $BackupPath
}
Write-Host ("Upload of {0} succeeded, moving to Uploaded folder" -f
$transfer.FileName)
# Upload succeeded, move source file to backup
Move-Item $transfer.FileName $backupPath
}
else
{
Write-Host ("Upload of {0} failed: {1}" -f
$transfer.FileName, $transfer.Error.Message)
}
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
exit 0
}
catch [Exception]
{
Write-Host ("Error: {0}" -f $_.Exception.Message)
exit 1
}
So there's the code. I'm happy to use built in PowerShell for the FTP side to simplify it, I just want it to work.
I'm not sure what's your concern with the code. It looks pretty much ok, except for a syntax error, when setting $folder:
Why are you even trying to use $_.CreationTime as folder timestamp? Just use current date:
$folder = (Get-Date -Format "yyyyMMdd")
See Formatting timestamps in PowerShell in WinSCP documentation.
Also I do not see a point of setting $folder and $backupPath in the params block. Move it after the params block. If you want this anyway, you are missing a comma after the $folder assignment.
Other than that, your code should work.
You cannot simplify it by using the built-in PowerShell (or rather .NET) FTP functionality, as it does not have as powerful commands as WinSCP .NET assembly.
I'd write the code as:
$localPath = "C:\source\local\path\*"
$remotePath = "/dest/remote/path/"
$folder = (Get-Date -Format "yyyyMMdd")
$backupPath = "C:\local\backup\path\$folder"
# If today's folder doesn't exist, create it
if (!(Test-Path $BackupPath))
{
New-Item -ItemType Directory -Force -Path $BackupPath | Out-Null
}
try
{
# Load WinSCP .NET assembly
Add-Type -Path "WinSCPnet.dll"
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::ftp
HostName = "ftp.example.com"
UserName = "username"
Password = "password"
}
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Upload files, collect results
$transferResult = $session.PutFiles($localPath, $remotePath)
# Iterate over every transfer
foreach ($transfer in $transferResult.Transfers)
{
# Success or error?
if ($transfer.Error -eq $Null)
{
Write-Host ("Upload of $($transfer.FileName) succeeded, " +
"moving to backup")
# Upload succeeded, move source file to backup
Move-Item $transfer.FileName $backupPath
}
else
{
Write-Host ("Upload of $($transfer.FileName) failed: " +
"$($transfer.Error.Message)")
}
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
exit 0
}
catch [Exception]
{
Write-Host "Error: $($_.Exception.Message)"
exit 1
}
Based on Moving local files to different location after successful upload.
I have previously using .bat scripts for this process and am relatively new to writing powershell scripts. I am conducting a transfer from an FTP location and need to delete each file from the FTP server after transfer. What is the correct syntax for this? Code is as follows:-
$a=Get-Date -format MMMM
try
{
# Load WinSCP .NET assembly
Add-Type -Path "C:\Program Files (x86)\WinSCP\WinSCPnet.dll"
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "xxx.xxx.xxx.xxx"
UserName = "joe"
Password = "joebloggs"
SshHostKeyFingerprint = "ssh-rsa 1024 01:07:da:11:22:33:44:55:66"
FtpMode = [WinSCP.FtpMode]::Active
}
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Download files
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.TransferMode = [WinSCP.TransferMode]::Binary
$transferResult = $session.GetFiles("/*", "C:\Users\User\Dropbox\Rockar\Data\System Data\Raw Feeds\Stock\"+$a+"\", $False, $transferOptions)
# Throw on any error
$transferResult.Check()
# Print results
foreach ($transfer in $transferResult.Transfers)
{
Write-Host ("Download of {0} succeeded" -f $transfer.FileName)
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
Discovered the answer with a bit of further reading on the WinSCP site about getfiles parameters winscp.net/eng/docs/library_session_getfiles#parameters the remove parameter just needs setting to $true from $false
You should call $session.RemoveFiles, here's the method description:
https://winscp.net/eng/docs/library_session_removefiles