We have a site that offers very large files for download to clients. These files can be from 2GB up to about 14GB. What is the best way to handle download of these files? The site is written in php and the link is provided with download.php?hash=[somehash].
Are there any programs that works great for very large files sent over http? Or should we look at a way to do this with torrent-files or something else?
This is 100 % legal for the septics out there.
you can do this with php, but you should support continueing of paused download by supporting the byte-range header
have a look at this page: http://www.coneural.org/florian/papers/04_byteserving.php
Edit: also have a look at this question: Downloading large files reliably in PHP
We decided to go for a http-downloadclient and configured our Linux-server to handle http-download resume.
I recommend this to everyone who reads this.
Note that this is based on HUGE downloads, like 10GB >
Related
Need some help please with web related matters since I don't know much about web (more on the software side of things).
Basically, I am developing an iphone app and would like to send data to a local server once in a while (for simplicity, let's just say I want to send this info to my personal computer which will act as the server). This is just some simple data, and I dont care about the format (actually .txt is the best, but I am open to any format which will make it easier - I am just transferring numbers).
What would be the best way to go about this process? A quick step by step explanation would be highly appreciated. From my very basic knowledge I assume that I will need to:
setup my Mac as a server (which I think should be done from settings?)
Create a URL connection on my app and send the file?
I am probably missing 50 other steps here...
Thanks!
One path is to set up a webDAV server -- you'll have to Google that up, it's far too big a topic to cover here.
To the specific questions you asked:
1) Your mac can become a web server by turning on the WebSharing in preferences, or a file server by turing on fileSharing. Be sure to set permissions the way you want them.
2) If your mac is a web server, you could write a simple CGI script (perl, ruby, or the like -- this is simple tutorial stuff that's all over the www) that accepts your text as a parameter. From your iPhone app, you'd make an NSURLRequest to a URL similar to:
http://192.168.10.1/webPage.html?this+is+the+text+I+want+to+send
Of course, you can get fancier ans use POST requests (the above example is a GET request), but that's going to require more reading.
If you want to transfer files via file sharing, that's a bit more complicted.
What would REALLY help us answer is if you could specify the question a bit more tightly. As it is, you've asked about a very broad area that covers quite a bit of ground.
My customer has a need to be able to receive large video files his site members(1 - 5 GB, maybe larger if possible). Hopefully I can get some good ideas from the overflow community on this
My origonal thoughts are to some how set up a file send site where only members can log in and e-mail their video's in. Would this be tough to create? Perhaps there are already some services out there that would authenticate my user's and allow them to email large files(The cost maybe worth it).
Would some sort of FTP Solution also be an available option? I would have to know which user is uploading files. I've never done an FTP upload site before.
Any thought or helpful links would be great! Thanks!
One option would be to build some kind of client side uploader which would allow you to measure progress, resume uploads etc
Possible technologies could include Silverlight, Java Applet or Flash
See silverlight example on codeplex
I have an iOS app which stores data in the local directory. I'd like to be able to sync this data between multiple devices running my app. Currently this is using core data, but I'd expect to have to change to some text-based file storage system to make syncing easier.
I was expecting the DropBox API would make this nice and easy, and that I could tell the API to simply sync the contents of my data folder on startup/save. However it seems the DropBox API is nothing more than a glorified way of uploading and downloading files.
Am I wrong on this assumption? Can the DropBox API actually make it easy to keep a folder full of text files in sync? If not, is there some other service or even advice you can give? Syncing is hard - I was hoping DropBox would make it easier.
Using /metadata, getting information about the contents of a folder couldn't be easier. All you need to do is check the modified date, and if it's different than yours locally, perform the appropriate action. It will also give you the metadata for any file contents, so you can pick out which files need to be uploaded / downloaded / added / removed as necessary.
Any more functionality than this would be very application-dependent; you can decide for yourself when and how you want to deal with differing files.
Dropbox API Documentation
Won't repeat since Andy has answered your question, thought you may also find this tutorial helpful:
http://www.nanaimostudio.com/blog/2011/1/20/how-to-synchronize-your-app-data-using-dropbox-api.html
Check out the new DropBox Sync API.
As ohho mentioned, there's the DropBox Sync API available now which I think will do what you mentioned (I haven't tried it personally, as I integrated support for Dropbox in my app using the earlier SDK, before this came along). This will probably suffice if you're just working with dropbox.
In my case I've been starting to add support for additional online storage ("Cloud") services to my app, and am finding that it's somewhat challenging to deal with the differences between them in how they handle various concepts like authentication, version, naming etc. For example Dropbox's API references files according to their path, whereas Box's API references files according to their file id (which has an associated path, but I believe remains the same if the file is moved). Then there's also all the issues of conflict resolution (aka merging) which come about when multiple clients try and sync conflicting changes to the server.
I've actually been recently considering starting an open source project to develop a library which can connect to all the major services, and provide an abstraction layer that papers over their differences, and also handles the syncing process for you. I've actually got some code already (which I haven't publicly released to date) but could use that to get the project started.
Would this be of use to you? Would you (or any others) be willing to participate in such an effort?
(note to mods: I realise this may be deviating somewhat off-topic - sorry about that. If you can recommend a more suitable forum to discuss this I'd be happy to take the conversation there).
There is a RSS feed that can be limited to one of the shared folders but it kind of filters events to keep volumes low.
I am administrating a small, private website with 100% trusted users (about 60 people, i know them all personally).
I am having many problems with the PHP based upload system i have in place currently, mainly with users encountering timeout errors and other varying issues due to the way the upload is handled (not to mention the complete deadzone in the UI created by making the user stare blindly at the page until the upload finishes
Anyways, i have been tossing around alternative forms of file uploading i could offer. FTP accounts were nixed due to the level of tech savvy required. Flash/Java uploaders were nixed because i don't really want proprietary third party applets running on my site.
The other idea i came up with that i think would be perfect would be to offer the ability to EMAIL the files to the server. Emailing attachments is a simple enough task, and better yet it provides the user with some tangible feedback to the uploading process.
My question is, how could i go about implementing such a system?
The server is running Gentoo Linux with Apache and i have full root access. Mail dameons can be installed to my needs.
If you have a better way to upload files, perhaps you could offer that instead?
Stick with PHP. It's certainly not perfect but the problems you're describing can probably be handled. max-execution-time and upload_max_filesize are configurable values. I would at least try tweaking those numbers (no php code changes required) before trying to implement an email based solution.
There are several file upload libraries with progress bars using pure javascript. Keep it in PHP.
I am working on the developement of a application that will perform online backup of the files and folder in the PC, automatically or manually. Currently, I was keeping only the latest version of the file at the server.Now, I have to implement the versioning so that only the changes can be transfered to the online server and user must be able to download any of the available version of the file at Backup Server.
I need to perform Deduplication for this. Guys, though I am able to perform it using the fixed block size but facing an overhead of transferring the file having CRC information with each version backup.
I have never worked on such technology , so lacks in experience. I am eager to know is there any feasible method to embedd this functionality in the application without much pain. Is any third party tool would help to perform same thing? Please let me know?
Note: I am using FTP protocol to transfer the data.
There's a program called dump that does something similar, but it operates on filesystem blocks rather than files. rsync also may be of interest.
You will need to keep track of a large number of blocks with multiple versions and how they fit into the various versions of the original files, so you will need some kind of database to track this information, and an efficient way to query it to determine which blocks in a given file need to be transferred. Also note that adding something to the beginning of a file will cause all your blocks to be "new" if you use a naive blocking and diff scheme.
To do this well will be very complex. I highly recommend you thoroughly research already-available solutions, and if you decide you need to write your own, consider the benefits of their designs carefully.