I looking for the best way to upload an image from mobile phone to my server. I am currently using html5 to open the camera and take the picture, then I convert the file into a base64 string, then I send to the server, then save it in MongoDB.
I am expecting around 1000 to 1500 user request per day ( upload image ) , so I have the following question :
Is it a good way to do it?
Should I compress the base64, if yes how?
Should use a specific server to handle this task?
My backend is node express and the front end is ReactJS.
Thanks
It all depends on your situation. Reading and writing images from a cdn via i.e. streams is usually faster than reading and writing binary representations of images i.e. base64 from a database. However, your speed if reading from a cdn will obviously be effected by what service you use. Today, companies like Amazon can offer storage to a very cheap price so if you are not building a hobby app for like a student project you can usually afford it. Storing binary representation of images actually end up a little bit bigger in size than storing the image itself. You don't compress the base64, you compress the image before converting it. However, if you can't afford a storage account and if you know your users won't upload that many images it is usually enough to store binary representations of the images in a database. Mongo Atlas, for example, offers 512 mb for free on their database clusters. Dividing tasks of your app such as database requests and cdn services from your main application is usually a good choice if possible. This way you will divide the cpu, memory, etc. of your hardware and it will lead to faster reading and writing tasks for the user.
There are a lot of different modules for doing this in node. JIMP is a pretty nice one with loads of built in functions like resizing images and converting them to binary, either as Buffer or base64.
Related
does anyone know of best practices or common strategies in backend design for serving dynamic images and videos to client applications?
Background: I'm currently building an application that allows users to upload their own images and videos. I'm not really sure about how to serve these media files back to the client in the most efficient way. Do I store the files on the same VPS that my application server is running on? Do I need to save the files in different qualities / densities to better adjust for the clients' screen resolution? (I'll have mostly mobile clients)
I tried googling these questions but apparently I'm asking the wrong questions :-)
I would really appreciate maybe a reference or professional vocabulary on these topics.
Thanks in advance.
1) You need to split web server and application server.
First of all do not try to stream media files from your backend unless you can offload low-level stuff to OS - most likely you will do it wrong.
Use proxy server as an web server to serve such files.
nginx will do.
Also you need to have backup of your media files the same way as you do backup of your database.
Storing static huge media files along with application server is wrong move - it will not scale at all.
You can add cron task to move files to some CDN server - when your move is complete you replace URL in database to match new location.
So by using nginx you will save precious CPU and RAM while file is getting moved to external server.
And CDN will help you to dedicate bandwidth and CPU/RAM resources to application server.
2) Regarding image resolution and downsampling:
Screens of modern handsets have the same or even better resolution compared to typical office workstation.
Link speeds have much bigger impact on UX.
If client has smartphone with huge screen but with slow link you still have to deliver image or video as fast as possible even if quality of media will not be match the resolution of handset.
It makes sense to downsample images on demand and store result on disk for nginx/CDN to serve it again.
In case of videos it makes sense to make "bad" version with big compression(quality loss) for the cases of slow link - device will downsample it itself during playback.
And you can keep client statistics (screen sizes/downlink speeds) and generate optimized versions of such video file later when you see that it is "popular".
FYI: Several years ago some social meda giant dropped idea to prepare all possible versions of the same media file in favour of FPGA on-the-fly resampler.
I do not remember the name of the company and URL to the article. It was probably instagram.
Some cloud providers have offers with FPGA or CUDA on board to do heavy lifting.
So in some cases you could exchange storage for heave horsepower to do conversion on the fly.
Assuming i have a working facial recognition algorithm working for the iphone (comparison images would be stored on my machine). How can i expand this to compare image 'A' against images stored on a remote server?
Could someone give me an abstract definition? (I could download and temporarily store all images on my iphone, and then compare them against image 'A' however if i have hundreds of photos then it will take too long to process and be useless..).
Think like google goggles does - upload your image to the server and let the server crunch it. You can optimize the upload by sending a pre-processed part of the image first.
What is the input to your comparison algorithm, is it the subwindow containing the face detection? You could, just like peterept said, upload THAT smaller subwindow containing the face to the server and let the server do the work. If your input to the comparison algorithm is a set of features extracted from the face (statistics, etc.), then if it doesn't take too much CPU power to process the subwindow, you could extract those features on the phone and send them to the server for processing.
The main idea here is that you let the remote server do all the crunching, even if it is hundreds of photos (will still be faster than the phone, especially if you multithread). Then the trade-off is between sending the subwindow image or the feature set extracted from that subwindow, whichever's smaller to send.
I'm having issues with audio files on the iPhone web-app. Seems as each time an audio file is played, it's loaded first then played, even if repeating the same audio on a page that hasn't refreshed (done via javascript). From what I've research manifest files would be great but they are for offline application. I'm now researching HTML5 databases.
Does anyone know if HTML5 databases can store audio files such as mp3? The end result it then to pull the mp3 from the database. It might still have to load the file each time from the database but I'm hoping it's quicker than retrieving it from a server.
Thank you.
I think what you are after is possible, however you have a significant hurdle in that the implementation of HTML5 databases on most browsers is limited to 5mb as per w3c recommendations:
A mostly arbitrary limit of five
megabytes per origin is recommended.
Having said that the way its implemented in iPhone Safari is that databases can grow until they reach 5MB in size at which point the browser will ask the user if they wish to allow for the extra size, asking again at 10, 50, 100 and 500MB (see section "Estimated Database Size" in this post by html5doctor).
There is no limit on the number of databases you can build per domain in safari, however according to this post by Cantina Consulting you can have a total of 50MB across all databases in a single domain.
Given these parameters, a possible work-around for this implementation is to split your mp3 blobs across multiple databases, creating a new database each time your reach 4.9MB, however even if you follow this design it may not be ideal as you will still experience the following:
50MB is not a lot of audio files, a typical 5/6min song is about 5MB at 128Khz, so that only gives you space for about 1CD (60 min) of mp3 songs, after this you will need user cooperation to use additional database space.
You will still have significant security issues trying to play the mp3 blobs from the javascript runtime, it may be possible to bypass these tricking flash into thinking they are mp3 stream but I'm not sure how you'd go about it.
Feel free to have a play around with this iPhone HTML5 SQL Client I put together, you may want to use something similar for experimenting with your local mp3 Database.
I don't know which is the most efficient way of organizing images downloaded from server. I will be downloading around 200 images on to my iPhone on request for download. Which is the most efficient way of organizing ? just dropping it as a file on the phone's memory or having it in sqlite (via coredata) after download ? which one is most efficient and easy to handle ? which access is faster ?
The rule of thumb is to put them (or any bigger binary data) onto disk directly, and if the whole app organizes its data with a database / CoreData, then put the paths of the images in there.
AFAIK , Iphone has minimum 8GB of memory. That will be enough for images. Also It depends upon the frequency of image downloads. If you download 200images daily then u need some application that will push it in your sqlite db. Advantage of this will be your all image files will be inside a single db. No scattered images. But if you want to store only 200 images then i would recommend it store on your phone memory with some image managing tool like ACDSEE in windows, that will help you viewing images in slide show or what ever manner you want.
What approach is considered to be the best to store and manage video files? As databases are used for small textual data, are databases good enough to handle huge amounts of video/audio data? Are databases, the formidable solution?
Apart from size of hard disk space required for centrally managing video/audio/image content, what are the requirements of hosting such a server?
I would not store big files, like videos, in the database ; instead, I would :
store the files on disk, in the filesystem
and only store the name of the file (plus some metadata like content-type, size) in the database.
You have to consider, at least, those few points :
database is generally harder to scale than disks :
having a DB that has a size of several dozens/hundreds/more GB because of videos will make many things (backup, for example) really hard.
do you want to put more read-load on your DB servers, to serve... files ?
samer thing when writting "files" to your DB, btw
serving files (like, say, videos) from the filesystem is something that webservers do pretty well -- you can even use something lighter (like lighttpd, nginx, ...) than the webserver used to run your application (Apache, IIS, ...), if needed
it allows your application and/or some background scripts to do some tasks (like generating previews/thumbnails, for example) without involving the DB server
Using plain old files will also probably make things much easier the day you want to use some kind of CDN to distribute your videos to users.
And here are a couple of other questions/answers that might interest you :
Storing Images in DB - Yea or Nay?
Storing Images : DB or File System -
Store images(jpg,gif,png) in filesystem or DB?
store image in database or in a system file ?
Those questions/answers are about images ; but the idea will be exactly the same for videos -- except that videos will be much bigger than images !