On a remote machine, I have a number of BMP images that I want to transfer to my computer as fast as possible. These images are very similar (frames from a single scene in a video). I do not want to lose any information.
I tried zipping but zipping just takes ~30-40% of the size and takes time too. I had the idea to convert images to video using some lossless video format, transfer the file and than decode but it seems like the size gets to ~40-50% of original and the process takes time.
I believe this problem has to have better solution (because of image similarity) but I am still not able to find it. Please share your ideas. Thanks :)
I would suggest using video. Here is an SO answer that does a good job in describing MPEG4: How does MPEG4 compression work? . Video codecs that compress the footage usually convert to YUV (also stated in the referenced answer), and the only store the difference, thus greatly reducing file size overall. H.264 is a great example: people use it more, because it saves on (drumrolls) bandwidth, because of its good compression algorithm.
Related
I am looking into making a system for work where you can download huge video files, (Im talking 4k full length videos which have a file size of sometimes 500GB) and I'm looking into the best way of doing this.
Would it simply need a file manager to split the download? or could I use bittorrent?
any suggestions?
Bittorrent can be used to to 1:1 transfers and has the benefit of hash-verifying the contents and being able to resume the transfer when it has been interrupted. But that can also achieved with other tools such as rsync.
Bittorrent's strength is making the transfer between many nodes scalable and being able to work in a decentralized manner.
I need help with downloading from webserver...
What i currently do is get XML file from web servers that contains image locations, parse XML, download each image, store image on iphone, and store image name to sql database.
This takes lots of time because there is large amount of images to be downloaded and i am downloading one by one.
My app update just got rejected because reviewer decieded that downloading is too long... What is funny, last two updates passed without problems..
I was thinking about zipping those images on server and sending zip file to iphone, unzipping it there, or packing images together with binary and sending it to apple.
Any advice on how to make download faster, would be appreciated. Thanks.
BTW, zip won't help with images. They are already compressed, so it will just add overhead. Make sure your images are not any larger than you need for display and I'd do what Mario suggested above and download them in multiple async calls (at least make the one big call asynchronous.)
A key principle of UI design is to display partial results (unless they are invalid or misleading) so that the user understands that progress is being made.
If you really need all the images to make it valid, you can download a few and display them grayed out (alpha = 0.4) or something so that it's clear that this is a partial result, but that progress is being made. The reviewer probably felt that it was taking too long to startup.
Do you change those images often? Or only once per release if at all? If they change with each release only I'd package them. If they're almost never changed, go with the one huge download (so people don't have to redownload when updating) and if they're change often, download them file by file but try to do 2-3 files at once using asynchronous download (if supported).
1) I would use something like an NSOperationQueue to download around three images at a time in the background. Much more than that and the UI starts getting choppy.
2) Also display some kind of loading indicator while this is going on.
3) What format are your images in? If you are transferring over the network you should use JPG, and consider setting the quality level to something smaller (say 6 even 5). To offset the loss of quality you could send down larger images, even with the larger number of pixels you can easily be better off with a lower quality compression.
4) If you have to use PNG to preserve transparency, consider using PNGCrush on the images before sending. As noted, zip will do pretty much nothing.
One way to speed up download of those images is to put them on a CDN. Some CDNs, like Limelight have special network optimizations for sending data to mobile devices. They also just do a better job of routing content, and have higher capacity for transmitting content. What's nice about this approach is that you might not have to change your app. However CDNs can be pricy.
Likely, your images are just way too large. You said you're worried about the 20MB app limit, but I think at that point, your images are just way too large for the phone.
Rather than zipping the files, I'm pretty sure you need to downsample the size of the images. Not only that, but you should only download the ones that you need, when you need them.
If you still want to have bulk downloads, why not have it as a side option rather than the default implementation?
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
What's the best way of making a video as an output of a Perl program?
Video what? You can always use a simple graphics library like GD and a whole lot of ffmpeg to do what you want.
If you can figure out how to produce a data stream that ffmpeg's yuvmpeg4pipe input module can handle, then you could send your data into a fifo to avoid hitting the disk with with intermediate data. Being that the yuvmpeg4pipe seems to just be a header-less data stream it should be fairly easy to replicate.
This link might give you some ideas: http://kylecordes.com/2007/pipe-ffmpeg
You could also try setting up either a memory mapped file or ramdisk of sorts to write into. But even a system with 16 gigs of ram is going to fill up very quickly when working with uncompressed video.
In general it is usually better to just write out the uncompressed files (probably an image sequence in your case) and then compress it after its exported. The reason being, if you are doing anything interesting in the video, it will probably take many times longer to render the uncompressed frames than to compress the video. By saving the uncompressed copy, you are free to compress to different targets, or fine tune your compression settings...
In addition, working with image sequences opens the door to parallel processing on multiple cores or even multiple computers. This is how many commercial video rendering systems achieve greater speeds.
DOES NOT WORK!!!
UPDATE: Please ignore the below answer - upon reading through FFmpeg's source code, the URL input is not streamed - it's merely downloaded whole into a file and then regular file processing is done :(
I'm still leaving the answer up in case someone looking later find the FFMpeg's existance a useful info for Perl video processing, even though it doesn't help in this specific case.
ORIGINAL ANSWER
FFmpeg does not (based on POD) seem to allow an in-RAM sources, but it does allow URL based ones. So at the very least you can hack around needing to do disk IO by streaming your raw data via Apache or some smaller web server, and using the FFmpeg's URL input to retrieve that data from http://localhost:yourport. The raw data would natually come to the web server from a Perl program running under mod_perl/FCGI
I have a script that generates wave files, based on user input.
I want to be able to stream those wave files online(not necessarily as wave files, they can be converted on the fly to mp3 or whatever). Preferably through a embedded flash streamer, but a html5 version would be good too.
The files are generally small, around 5 seconds long, and I'd like then to be stream multiple files in one session.
Does anyone know how I should go about implementing this?
With such short audio clips I would not bother with a 'real' streaming technology, but just serve them up via HTTP as static files as quickly as the network connection will allow. A quick look at my iTunes library indicates that a 5s 128kpbs 44kHz stereo file is between 120-250KB. Almost small. If you are talking about 32kbps mono, then maybe the sizes will be a mere 15-30KB.
Encoding on-the-fly may result in undesirable issues, like scaling (CPU load from all those encoding jobs, some of which will be duplicate), latency (setting up the encoding, the actual encoding), and you won't know the end file size which can cause problems. So, setting up a caching system may make more sense.
I use wpaudioplayer to stream MP3s from my website (Example). It was originally made as a wordpress plugin but can be used as a standalone javascript.
I believe that it can play wave files as well as MP3s. If you do end up converting them before serving them I would suggest that you would
I'm working on a streaming server that will be capable of broadcasting targetted ads. Basically listeners hear the same music, but every, say, 30 minutes comes a block of ads and every listener has his/her own block. Implementing such streaming server poses various problems and this question is about one of them.
The server will work in a manner similar to Icecast, i.e. it will read the stream over the network from some stream generator and relay it to every listener. When it's time to broadcast ads, the server stops fetching the stream from the generator, reads ads from files and inserts them into each listener's buffer, transmits them and resumes on relaying stream from the generator.
When the server switches from relaying stream to broadcasting ads, it has to concatenate two MP3 streams (we broadcast in MP3). My concern is that simply appending one piece of data after another may produce some audible artifacts. Can it be done seamlessly?
I've already figured out this:
- I can make the server be aware of MP3 frames to avoid sync errors.
- I'm thinking about appending MP3 frames from the ad file after MP3 frames from the stream.
- Since ad is loaded from properly encoded MP3 file, I circumvent the problem of byte reservoir, because the first frame from the file can't use it.
But my concern is the way MDCT works. Listeners have no idea of what my server will do, so their MP3 decoders may produce some artifacts because incorrect MDCT data will be placed one after another in the stream they download. Will zero-padding at the beginning of the file with the ad compensate for this?
Do you know any libraries/tools (open source if possible) that can seamlessly join two MP3 files without decompressing them?
Can you point any good resources describing MP3 format? I searched Internet a lot, found lots of information, but I still miss the overall picture.
Maybe you know that this would be easier if I used another codec like OGG/Vorbis, AAC?
PS. This question is not a duplicate of What is the best way to merge mp3 files?. mp3wrap and tools alike are not an option for me.
I believe MP3s can be merged by simply concatenating the files. In some quick testing (cat file1.mp3 file2.mp3 > merged.mp3; mplayer merged.mp3) it seems to work as expected. Streaming from a web server probably will work just as well.
How are you going to handle switching the current input file? You can simply treat the advertisements as short tracks to play.
You should be able to concatenate mp3 files of both CBR and VBR formats.
MP3 files do not have a main header (disregarding ID3 and Xing). The audio data is stored as chunks where every chunk includes its own header. The header contains the necessary information (bitrate, sample frequency, stereo, etc) for the decoding of the audio data in that chunk.
This is one of the reasons why it is difficult to determine the duration of a mp3 file.
Another way of looking at it is, if you concatenate a CBR MP3 file with a VBR file, the end result is the same as one long VBR file with the first section of Audio at a constant bitrate.
The issue is that some MP3 players may be strict and expect a Xing header for a VBR MP3 file. This however was never the specification for the MP3 format but it is now assumed to be true.
If you're on Windows, the Microsoft DirectShow API may be the way to go. You should find that is is capable of doing things with audio and video both statically and streaming, in a variety of formats (you only need the necessary codecs, and the interface is virtually the same for all).
Saying this, DirectShow is unfortunately designed in a horribly intricate way, and has a steep learning curve, but the power it offers in unparallel if you're going to be doing audio/video manipulation on Windows. There are however a great number of samples and tutorials on how to use it, so it may not be so painful in the end. Also, if you're using the .NET Framework, there is a managed wrapped by the name of DirectShow.NET. It's not going to be an easy task whatever you do, unless there's something out there than I'm not aware of. Good luck with it anyway!
I approached a very similar problem, and after asking the right questions at various sources came up with the following...
Any worthy decoder will skip "bad" data until it hits a valid frame header. This is what ID3v2 relies upon to inject additional information into mp3 data. At the server, I'd go with analysis of source MP3 files to only serve valid MP3 frames. If you serve a few silent frames (about 7 should do it), the decoder should have time to settle before ramping up for the next load of (unassociated) MP3 data, avoiding the artefacts you (correctly) assume when concatenating frames from different encoding sessions.
More problematic is the possible switching of MP3 attributes (1/2 channels, output sample rate etc) between one frame to the next. Some decoders get quite upset when confronted with such a stream, resulting in 1/2 speed playback and the like. So, you need to ensure that all your source material is encoded to the same output attributes otherwise you may come unstuck.
You may have seen this already, but if not:
http://www.devhood.com/tutorials/tutorial_details.aspx?tutorial_id=79&printer=t
I don't see why you would want to concatenate the files. Why don't you use some sort of play list system and just change which file your sending. I would think this would allow more flexibility in the long run, and you wouldn't end up with large MP3 files.