I am trying to come up with a website with minimum bandwidth cost:
Imagine that a user is tight in bandwidth, even loading a webpage is "expensive" for him. What if the webpages are saved in his computer. However, he needs to transmit and receive data like text and compressed media files.
Is there a way to do this?
You can design an web page with Ajax, which will lower the loading time and cost. User can fetch multiple windows and content with must reloading.
Related
does anyone know of best practices or common strategies in backend design for serving dynamic images and videos to client applications?
Background: I'm currently building an application that allows users to upload their own images and videos. I'm not really sure about how to serve these media files back to the client in the most efficient way. Do I store the files on the same VPS that my application server is running on? Do I need to save the files in different qualities / densities to better adjust for the clients' screen resolution? (I'll have mostly mobile clients)
I tried googling these questions but apparently I'm asking the wrong questions :-)
I would really appreciate maybe a reference or professional vocabulary on these topics.
Thanks in advance.
1) You need to split web server and application server.
First of all do not try to stream media files from your backend unless you can offload low-level stuff to OS - most likely you will do it wrong.
Use proxy server as an web server to serve such files.
nginx will do.
Also you need to have backup of your media files the same way as you do backup of your database.
Storing static huge media files along with application server is wrong move - it will not scale at all.
You can add cron task to move files to some CDN server - when your move is complete you replace URL in database to match new location.
So by using nginx you will save precious CPU and RAM while file is getting moved to external server.
And CDN will help you to dedicate bandwidth and CPU/RAM resources to application server.
2) Regarding image resolution and downsampling:
Screens of modern handsets have the same or even better resolution compared to typical office workstation.
Link speeds have much bigger impact on UX.
If client has smartphone with huge screen but with slow link you still have to deliver image or video as fast as possible even if quality of media will not be match the resolution of handset.
It makes sense to downsample images on demand and store result on disk for nginx/CDN to serve it again.
In case of videos it makes sense to make "bad" version with big compression(quality loss) for the cases of slow link - device will downsample it itself during playback.
And you can keep client statistics (screen sizes/downlink speeds) and generate optimized versions of such video file later when you see that it is "popular".
FYI: Several years ago some social meda giant dropped idea to prepare all possible versions of the same media file in favour of FPGA on-the-fly resampler.
I do not remember the name of the company and URL to the article. It was probably instagram.
Some cloud providers have offers with FPGA or CUDA on board to do heavy lifting.
So in some cases you could exchange storage for heave horsepower to do conversion on the fly.
If I am using Varnish to cache my entire documents, by what mechanism would you advise I increment a page view count as well.
For example, lets supose that I have an auction listing, such as ebay, and I would like to cache the entire page since I know it is never going to change.
How would you then increase the page view count of this listing.
Lets say that my application is running from Zend Framework.
Would it be correct to make an ESI (Edge Side Include) to a node.js server which increments a page view count in Redis?
I'm looking for something that wil be 100% supported and will yielf accurate page view request numbers. (I'm not concerned about duplicate requests either, I"ll handle that in my application logic to prevent one IP from nuking the page view count).
I would separate your statistics logic from your application. Use a small piece of javascript that requests a resource with a unique timestamp (e.g. an image like /statistics?pageId=3&ts=234234249). You can cache your complete page (no need to bother with ESI) and have the statistics handled by a fast (multiplexing) server like node.js, netty, tornado.
If you need the pageCount in your page, request a small piece of javascript/json data instead of an image and update the DOM in javascript.
This way, you can log better statistics (e.g. dimensions of the page), you minimize traffic and keep statistics a separate concern.
I'm developing an offline web-app for a client of ours, designed to run on an iPad in airplane mode, mounted on a stand. It has no server-side dynamic pages, only a static HTML page, many JavaScript components to handle navigation and interactivity, and a bunch of small graphics assets. The whole website (static html + css + js + graphics) weighs exactly 8.3Mb.
I'm caching the whole site via an offline.manifest declared in my single HTML file, this manifest references absolutely all the files under the root directory, so that all files needed are cached.
I'm not using localStorage, IndexedDB or other offline-storage techs in my JS code. Apart from the "automatic" caching, I don't store anything on-device.
So atfer checking my webserver logs, when my client installs the webapp on its iPad homescreen, it downloads all the files once, then never downloads anything from my server afterwards. That's fine, exactly what he wanted in the first place : a full offline webapp.
Then, how comes that after several minutes of testing from my client, his iPad asks him to “increase local storage from 10Mb to 25Mb” ???
FYI, the app consists of a kind of quizz: one welcome screen, 19 question screens, one result screen ; the user can navigate backwards/forwards in the questions sequence, but they're created and nullified on-the-fly so as to minimize memory footprint. Anyway I don't believe this problem has to do with RAM access, only with "hard", permanent, cached storage.
I've noticed that with all apps, it's like the iPad has to realize it has everything, and waits a few seconds to realize that it's going to go over it's app limit.
it would be nice to have it default to a larger amount, or let you set it up with a larger amount to begin with.
Seems like my client doesn't have this problem anymore. As I'm not in direct, physical contact with him, I can't tell what he did to get rid of it.
I'm having issues with audio files on the iPhone web-app. Seems as each time an audio file is played, it's loaded first then played, even if repeating the same audio on a page that hasn't refreshed (done via javascript). From what I've research manifest files would be great but they are for offline application. I'm now researching HTML5 databases.
Does anyone know if HTML5 databases can store audio files such as mp3? The end result it then to pull the mp3 from the database. It might still have to load the file each time from the database but I'm hoping it's quicker than retrieving it from a server.
Thank you.
I think what you are after is possible, however you have a significant hurdle in that the implementation of HTML5 databases on most browsers is limited to 5mb as per w3c recommendations:
A mostly arbitrary limit of five
megabytes per origin is recommended.
Having said that the way its implemented in iPhone Safari is that databases can grow until they reach 5MB in size at which point the browser will ask the user if they wish to allow for the extra size, asking again at 10, 50, 100 and 500MB (see section "Estimated Database Size" in this post by html5doctor).
There is no limit on the number of databases you can build per domain in safari, however according to this post by Cantina Consulting you can have a total of 50MB across all databases in a single domain.
Given these parameters, a possible work-around for this implementation is to split your mp3 blobs across multiple databases, creating a new database each time your reach 4.9MB, however even if you follow this design it may not be ideal as you will still experience the following:
50MB is not a lot of audio files, a typical 5/6min song is about 5MB at 128Khz, so that only gives you space for about 1CD (60 min) of mp3 songs, after this you will need user cooperation to use additional database space.
You will still have significant security issues trying to play the mp3 blobs from the javascript runtime, it may be possible to bypass these tricking flash into thinking they are mp3 stream but I'm not sure how you'd go about it.
Feel free to have a play around with this iPhone HTML5 SQL Client I put together, you may want to use something similar for experimenting with your local mp3 Database.
My problem is that my application size is very high,
is there any idea to reduce size of application?
if i make application without content and content is uploaded my server then how i sync the application with content put on my server?
i want to know that once user download application after that when he use application then we stream the content and save his document folder.
once user stream then never required for streaming.
is it possible????
Thanks,
Reducing the size of your application depends on the TYPE of contents of your application. I highly doubt that the application code is the cause, and since you did not mention what they are I am assuming they are some kind of resource.
If your resources are images, try to use image compression programs. Or convert them to smaller sized images or optimize the images.
If your resources are documents / text files / files that have a high compression ratio when zipped. Then you can try to zip your resources and access them inside the compressed file (this will mean additional coding, and probably slower in performance).
These are just examples.
It is not advisable to stream large contents because it uses the network bandwidth which, depending on the user's plan, can cause a big spike in phone bills.
Yes it is possible that you can download your content and can save to application's document folder, when user runs your application for the first time. Thought it may affect the first impression to your user as it will take time to download.