I am writing a script that polls an FTP site for files and downloads them locally as and when available. The files are deposited to the FTP site randomly by various source parties. I need a way to be able to detect if the file on the FTP site has been transferred over completely by the source party, before downloading them. Any thoughts on how to go about this?
If you have control over the client, a much safer, cleaner and efficient way is to have the client do the following:
Upload the file to ..../partial/somefile
Rename ..../partial/somefile to ..../complete/somefile
This causes the file to appear in the latter directory all at once, so all you have to do is scan that directory. You could even ask the OS to be notified of additions to that directory if you wanted a non-polling solution.
If you cannot manipulate the FTP server itself the only way of checking that comes to my mind is polling the filesize and if the filesize doesn't change for a longer time you can be quite sure the upload has finished. But nobody can guarantee. Ideally you can adapt the FTP server and make it execute some script after finishing some upload.
Some pseudo-code:
my %filesizes;
my %processed;
sub poll {
foreach (#files_on_ftp) {
if($_->filesize == $filesizes{$_->filename} and not $processed{$_->filename}) {
process($_);
$processed{$_->filename)++;
}
}
}
Like ikegami's solution depends mine of client side:
first is file uploaded
if it is completed, client uploads empty flag-file (like file.name.txt.finished)
When you see finished-file, you know file is ready.
Related
I'm hoping someone can help. I've started using the Community TFS Build Extensions, in particular the FTP activity. I followed the documentation here and got to grips with the it pretty easily. I'm encountering one major problem though.
My Web app has a basic enough structure:
I start by creating the FindMatchingFile activity which places the files in the drop location into an IEnumberable variable called FilesToFTP :
String.Format("{0}\**\*.*", BuildDetail.DropLocation)
When I iterate through the variable and print out the results, all seems correct:
G:\builds\Build.1203\CredentialManagement\bin\BusLogic.dll
G:\builds\Build.1203\CredentialManagement\css\style.css
G:\builds\Build.1203\CredentialManagement\AppError.aspx
......
G:\builds\Build.1203\CredentialManagement\Web.config
etc etc.
The problem is, when I pass that IEnumerable to the Ftp activity (converting it to a string array), it FTP uploads all the files on the server however it doesn't keep the directory structure of my Web app. It just piles all the output (dlls, aspx etc) into one directory. See the following two screenshots.
Is there any way I can use the FTP activity to upload all the output from the drop location recursively? I feel like I'm doing something simple wrong.
The FTP activity in TFS Build Extensions doesn't upload files recursively.
I think it would be a good value addition to the activity. Please create a request for the project and we will add in it. For now, you can go around it by calling the Ftp activity recursively for each directory and setting the RemoteDirectory for each.
I am writing a web service using play 2.0 in scala.
After getting client's configuration parameters, then the web service will take a while to generate some files. (Note: some files can be short, some other files can take long).
I figured out how to get client's configurations and trigger my computation,
now I need to present the generated files to client side at the end of the session.
So questions are:
How to present the files to client?
I'm thinking about returning a static folder link to client, so that they can go into the folder to
see further what files are there, and so when clicking on the files, the details of the files can be shown, either jpg or text.
But how?
How to not to block client during the generation of the files?
For example, client can still click on the files that are available at that moment.
Then here comes the third question:
How to let the client side know that the long-time computation file is available and listed in the folder?
I'm pretty new to web application, thanks for any suggestion, advice, or little examples are greatly appreciated too!
I would create a web interface to show the created files to your clients and create a background job to for the long computation process. You can create a wait page where a javascript check if the file is created.
I am developing a simple iPad application that should submit some text from a form to a .csv file. I could manage to update the .csv file which is saved locally in the documents folder on my computer. However, I need to keep the file on a server, probably download the file, append data, and upload it again (Export a bulk of data to the file on server). Any idea how I could do something like that?
I guess ftp might be the easiest way, see this question
your other options likely involve writing a server-side service to post data to.
Could anybody explain to me the process of uploading to and downloading form and ftp server with the iPhone SDK. If you could just point me in the right direction (e.g. documentation etc.). How difficult is a task like this?
Thanks in advance.
You can use this. It support all the basic ftp operations:
Download file
Upload file
Delete file
Delete directory
Create directory
List directory contents
[DISCLAIMER] I am the developer of the library, I needed a ftp library too in the past and came over this answer. However, I decided to write one myself because s7ftprequest didn't support at that point several operations that I needed.(like download or list directory)
Try this Simple FTP Download
The Apple documentation will provide far more info in general than I could. Have a look at:
http://developer.apple.com/iphone/library/documentation/Networking/Conceptual/CFNetwork/CFFTPTasks/CFFTPTasks.html#//apple_ref/doc/uid/TP30001132-CH9-SW1
which details the FTP information you need. If you prefer a PDF with all the networking info in it, have a look at:
http://developer.apple.com/iphone/library/documentation/Networking/Conceptual/CFNetwork/CFNetwork.pdf
In this, you'll be particularly interested in Chapter 5. Both detail working with FTP sites, including uploading, downloading, retrieving directory listings, etc.
s7ftprequest only for uploading files to FTP.
The below is sample code from apple
http://developer.apple.com/library/ios/#samplecode/SimpleFTPSample/Introduction/Intro.html
Limitations:
FTPS (that is, FTP over TLS)
deleting items
renaming items
other less common FTP commands
custom FTP commands
I ended up using GoldRacoon. It turns out that in iOS/objc land, there's BlackRaccoon as the original FTP client lib, then WhiteRaccoon was forked from that, and then GoldRacoon was forked from WhiteRaccoon (I think)
pod search GoldRaccoon
... will show you.
I ended up making a few changes (in my own fork) so that you can pass in a successBlock & failBlock into any request, so that block-y callers (like my classes) don't have extra work to manage the delegate callbacks. Github link (my fork): https://github.com/xaphod/GoldRaccoon
I'm currently hosting an Eclipse plugin update site on sourceforge.net . SF.net does not allow access to server logs but I'd still like to know how many downloads the plugin gets.
Is there an alternative way of gathering them?
I'm not going to have any sort of 'call home' feature in the plugin, so please don't suggest that.
I wrote a blog about how to track downloads of an Eclipse plug-in update site. What you can do is specify a url to your server and every time a download is initiated the update site will send an HTTP HEAD request to that url, which you can then use to count the number of times the plug-in was downloaded. If you want to track some information about who is downloading the plug-in you can pass information, like the package name, version, os, and and store it in a database.
http://programmingfortherestofus.blogspot.com/2014/08/tracking-downloads-to-your-eclipse.html
I hope it helps!
It is possible to host the plugin jars in the file release service, and then get your site.xml file to point to them. You need to point at a specific mirror to make it work.
This will tell you how many times people download each file as with a normal file release.
Unfortunately, in practice this is a lot of work to maintain, and tends to be unreliable (I kept getting bug reports saying the update site wasn't working).
You could write a very simple php script which just serves up the relevant file, and logs the download to a file or DB. Make sure it double checks the URL is a valid one to download to the user of course :)
Once that's in place, you can update the site.xml to point to the correct thing, or you could probably use URL rewriting to intercept requests to your jar file and pass them through the script. I've never tried that on the SF servers, but it might work.
EDIT:
Even better, just have a php script which sends a redirect like this:
<?php
$file = $_GET('file');
// Now log the access to file
header('Location: ' . $file);
?>
Just a thought: AFAIK, SourceForge does tell you how much data you served. You know the size of your plugin JARs. Divide the data served by the size of your plugin and you get a rough estimate of how many downloads you had.