I'm struggling to find how I would be able to download a file that's accessible via a URL, but don't keep downloading it if the file on the URL is newer than the file in existence.
Using Power shell this command downloads the file, but it will just download it again regardless if it's there already.
Invoke-WebRequest http://www.domain.co.uk/downloads/File.zip -OutFile C:\Temp\File.zip
I know the command line for copying local files with "XCOPY /D" to check for the date/time stamp but I am wondering if something similar can be done downloading a file from the internet?
Many thanks in Advance
Related
I was using Invoke-WebRequest for some time in my script and today first time I got problem with it - once I run script, it starts to download ZIP file and finishes without warnings, but when I am trying to open file, WinRar gives me an error and I can see that file size is less than it should be. I tried several time and got same result. Then I tried to download file with Google Chrome just to ensure that file is not corrupted, but it downloaded (with correct file size) and opened without any issues.
After couple of hours I ran script again and it downloaded file as well. So now I am not sure that I can trust Invoke-WebRequest since it seems that sometimes it can make incomplete download.
Myabe someone can point me on how I can avoid this situation? Or maybe better to use another method?
I came across this post:
Upload files with FTP using PowerShell
But I'm looking how i can upload a directory with files in it to a FTP server.
Thanks in Advance
You should write a script that loop into directory and then upload files one by one. I don't remember that ftp command on windows can upload an entire directory.
You should find example here :
http://www.dostips.com/DtTipsFtpBatchScript.php
I'm trying to automate uploading and downloading from an ftp site using cURL inside MAtlab, but I'm having difficulties. Essentially I want one computer continuously uploading new files to an ftp, yet since there is a disk quota on the ftp, I want another computer continuously downloading and removing those same files from the ftp.
Easy enough, but my problem arises from wanting to make sure that I don't download a file that is still being uploaded, thereby resulting in an incomplete file.
First off, is there a way in cURL to make it so that the file wouldn't be available for download from the ftp site until the entire file has been uploaded?
One way around this is that I could upload files to one directory, and once they are finished uploading, then I could transfer them to a "Finished" directory on the ftp site. Then the download program would only look for files inside that "Finished" directory. However, I don't know how to transfer files within an ftp site using cURL.
Is it possible to transfer files between directories on an ftp site using cURL without having to download the file first?
And if anyone else has better ideas on how to perform this task, I'd love to hear em!
Thanks!
You can upload the files using a special name and then rename it when done, and have the download client only download files with that special "upload completed" name style.
Or you move them between directories just as you say (which is essentially a rename as well, just changing the directory too).
With the command line curl, you can perform "raw" commands after the upload with the -Q option and you can even find a tiny example in the curl FAQ: http://curl.haxx.se/docs/faq.html#Can_I_use_curl_to_delete_rename
I am using CGI.pm version 3.10 for file upload using Perl. I have a Perl script which uploads the file and one of my application keeps track of different revisions of the uploaded document with check-in check-out facility.
Re-creational steps:
I have done a checkout(download a file) using my application (which is web based uses apache).
Logout from current user session.
Login again with same credentials and then check-in (upload) a new file.
Output:
Upload successful
Perl upload script shows the correct uploaded data
New revision of the file created
Output is correct and expected except the one case which is the issue
Issue:
The content of the newly uploaded file are same as the content of the last uploaded revision in DB.
I am using a temp folder for copying the new content and if I print the new content in upload script then it comes correct. I have no limit on CGI upload size. It seems somewhere in CGI environment it fails might be the version i am using. I am not using taint mode.
Can anybody helps me to understand what might be the possible reason?
Sounds like you're getting the old file name stuck in the file upload field. Not sure if that can happen for filefield but this is a feature for other field types.
Try adding the -nosticky pragma, eg, use CGI qw(-nosticky :all);. Another pragma to try is -private_tempfiles, which should prevent the user from "eavesdropping" even on their own uploads.
Of course, it could be that you need to localize (my) some variable or add -force to the filefield.
I found the issue. The reason was destination path of the copied file was not correct, this was because my application one of event maps the path of copied file to different directory and this path is storing in user session. This happens only when I run the event just before staring upload script. This was the reason that it was hard to catch. As upload script is designed to pick the new copied file from same path so it always end up uploading the same file in DB with another revision. The new copied file lying in new path.
Solved by mapping correct path before upload.
Thanks
I am having a lot of issues trying to automate downloading from an ftp site. I know the folder the file will be in, and I know that it will be a .zip file. However I do not know what the files will be named.
So I have code that works if I know the file name...for example:
$sourceuri = "ftp://myFtpSite/test/myZipFile.zip"
I would like to be able to use wildcards in this string so it will recongize any zip file. So I could write something like
$sourceuri = "ftp://myFtpSite/test/_.zip"
and it would download any zip file in that folder.
I know this question is ancient, but have you considered just using the console app ftp.exe? You can build a text file with commands (such as "mget *.zip" to retrieve all .zip files) and automate the process.
ftp -s:filename