I'm trying to automate uploading and downloading from an ftp site using cURL inside MAtlab, but I'm having difficulties. Essentially I want one computer continuously uploading new files to an ftp, yet since there is a disk quota on the ftp, I want another computer continuously downloading and removing those same files from the ftp.
Easy enough, but my problem arises from wanting to make sure that I don't download a file that is still being uploaded, thereby resulting in an incomplete file.
First off, is there a way in cURL to make it so that the file wouldn't be available for download from the ftp site until the entire file has been uploaded?
One way around this is that I could upload files to one directory, and once they are finished uploading, then I could transfer them to a "Finished" directory on the ftp site. Then the download program would only look for files inside that "Finished" directory. However, I don't know how to transfer files within an ftp site using cURL.
Is it possible to transfer files between directories on an ftp site using cURL without having to download the file first?
And if anyone else has better ideas on how to perform this task, I'd love to hear em!
Thanks!
You can upload the files using a special name and then rename it when done, and have the download client only download files with that special "upload completed" name style.
Or you move them between directories just as you say (which is essentially a rename as well, just changing the directory too).
With the command line curl, you can perform "raw" commands after the upload with the -Q option and you can even find a tiny example in the curl FAQ: http://curl.haxx.se/docs/faq.html#Can_I_use_curl_to_delete_rename
Related
I'm struggling to find how I would be able to download a file that's accessible via a URL, but don't keep downloading it if the file on the URL is newer than the file in existence.
Using Power shell this command downloads the file, but it will just download it again regardless if it's there already.
Invoke-WebRequest http://www.domain.co.uk/downloads/File.zip -OutFile C:\Temp\File.zip
I know the command line for copying local files with "XCOPY /D" to check for the date/time stamp but I am wondering if something similar can be done downloading a file from the internet?
Many thanks in Advance
We have a customer with their SFTP site, and I would like to copy files from specific folder, by using any automated process.
One of the example which I found, is winscp.net, but I have not managed how to use it, for my purpose.
http://www.itworld.com/article/2928599/windows/how-to-automate-sftp-file-transfers-in-microsoft-windows.html
QUESTION: All I will need is not run script and the file should be copied from their directory to my local folder. Is it possible at all?
I found the way of using PSFTP(Putty) to connect to the server, but how to make it automatically, do not know.
I think its a good solution for your problem.
You have to install winscp and the code some files and it will do automaticaly. I used 3 or 4 times. Also you need the key for your SFTP to connect throught it.
Here a link to the guide step by step.
https://winscp.net/eng/docs/guides
Here is a link to the scripting webpage
https://winscp.net/eng/docs/scripting
I recommend you to create an ini file to load all of the characteristics of your ftp connection and then execute an script over this.
It can be launched like that
WinSCP.com /ini=[your ini file] /script=[Your script file(what you want to do when it is connect)]
Hope this helps!.
I use the concept of uploading and downloading files from FTP server. I have been successful in both for a single file.
Now, for downloading, there may be more than one files on ftp. So, multiple files to be downloaded at local side.
I have referred this too, but couldn't get exactly what I want.
I want to download multiple files from ftp programatically. And for that I have to check the existence of file for particular URL.
Ftp protocal have the list cmd , you can list the files on the server and then download it.
I'm trying to think of a good solution for automating the deployment of my .NET website to the live server via FTP.
The problem with using a simple FTP deployment tool is that FTPing the files takes some time. If I FTP directly into the website application's folder, the website has to be taken down whilst I wait for the files to all be transferred. What I do instead is manually FTP to a seperate folder, then once the transfer is completed, manually copy and paste the files into the real website folder.
To automate this process I am faced with a number of challenges:
I don't want to FTP all the files - I only want to FTP those files that have been modified since the last deployment. So I need a program that can manage this.
The files should be FTPed to a seperate directory, then copy+pasted into the correct destination onces complete.
Correct security permissions need to be retained on the directories. If a directory is copied over, I need to be sure that the permissions will be retained (this could probably be solved by rerunning a script that applies the correct permissions).
So basically I think that the tool that I'm looking for would do a FTP sync via a temporary directory.
Are there any tools that can manage these requirements in a reliable way?
I would prefer to use rsync for this purpose. But seems you are using windows OS here, some more effort is needed, cygwin stuff or something alike.
I am using CGI.pm version 3.10 for file upload using Perl. I have a Perl script which uploads the file and one of my application keeps track of different revisions of the uploaded document with check-in check-out facility.
Re-creational steps:
I have done a checkout(download a file) using my application (which is web based uses apache).
Logout from current user session.
Login again with same credentials and then check-in (upload) a new file.
Output:
Upload successful
Perl upload script shows the correct uploaded data
New revision of the file created
Output is correct and expected except the one case which is the issue
Issue:
The content of the newly uploaded file are same as the content of the last uploaded revision in DB.
I am using a temp folder for copying the new content and if I print the new content in upload script then it comes correct. I have no limit on CGI upload size. It seems somewhere in CGI environment it fails might be the version i am using. I am not using taint mode.
Can anybody helps me to understand what might be the possible reason?
Sounds like you're getting the old file name stuck in the file upload field. Not sure if that can happen for filefield but this is a feature for other field types.
Try adding the -nosticky pragma, eg, use CGI qw(-nosticky :all);. Another pragma to try is -private_tempfiles, which should prevent the user from "eavesdropping" even on their own uploads.
Of course, it could be that you need to localize (my) some variable or add -force to the filefield.
I found the issue. The reason was destination path of the copied file was not correct, this was because my application one of event maps the path of copied file to different directory and this path is storing in user session. This happens only when I run the event just before staring upload script. This was the reason that it was hard to catch. As upload script is designed to pick the new copied file from same path so it always end up uploading the same file in DB with another revision. The new copied file lying in new path.
Solved by mapping correct path before upload.
Thanks