I would like to host my images in a special CDN server which could server images with preprocessing. For example images loaded by clients may be in jpeg, png or gif format, they may present the images at a different resolution in the final document, and when the document is viewed the images will be served in webp format and exactly at the resolution set on the final documents.
so the original images loaded is as it is on the server but the image served is in webp format and in the size required for the final document.
A Chinese Cloud service provider Qiniu.com provides such services . They provide an API called imageMogr2 and with this API installed and the images hosted on their special servers can be pulled in any format, any size with other alterations.
If any one find similar services with AMAZON or other service providers will be glad to know more.
Their API documentations are available at developer.qiniu.com, but I am still relectuant to host on Chinese servers due to the great fire wall and access issues to the outside world.
Related
does anyone know of best practices or common strategies in backend design for serving dynamic images and videos to client applications?
Background: I'm currently building an application that allows users to upload their own images and videos. I'm not really sure about how to serve these media files back to the client in the most efficient way. Do I store the files on the same VPS that my application server is running on? Do I need to save the files in different qualities / densities to better adjust for the clients' screen resolution? (I'll have mostly mobile clients)
I tried googling these questions but apparently I'm asking the wrong questions :-)
I would really appreciate maybe a reference or professional vocabulary on these topics.
Thanks in advance.
1) You need to split web server and application server.
First of all do not try to stream media files from your backend unless you can offload low-level stuff to OS - most likely you will do it wrong.
Use proxy server as an web server to serve such files.
nginx will do.
Also you need to have backup of your media files the same way as you do backup of your database.
Storing static huge media files along with application server is wrong move - it will not scale at all.
You can add cron task to move files to some CDN server - when your move is complete you replace URL in database to match new location.
So by using nginx you will save precious CPU and RAM while file is getting moved to external server.
And CDN will help you to dedicate bandwidth and CPU/RAM resources to application server.
2) Regarding image resolution and downsampling:
Screens of modern handsets have the same or even better resolution compared to typical office workstation.
Link speeds have much bigger impact on UX.
If client has smartphone with huge screen but with slow link you still have to deliver image or video as fast as possible even if quality of media will not be match the resolution of handset.
It makes sense to downsample images on demand and store result on disk for nginx/CDN to serve it again.
In case of videos it makes sense to make "bad" version with big compression(quality loss) for the cases of slow link - device will downsample it itself during playback.
And you can keep client statistics (screen sizes/downlink speeds) and generate optimized versions of such video file later when you see that it is "popular".
FYI: Several years ago some social meda giant dropped idea to prepare all possible versions of the same media file in favour of FPGA on-the-fly resampler.
I do not remember the name of the company and URL to the article. It was probably instagram.
Some cloud providers have offers with FPGA or CUDA on board to do heavy lifting.
So in some cases you could exchange storage for heave horsepower to do conversion on the fly.
I use Potoswipe in my project. Potoswipe requires the definition of the images size. In a time when I used PHP that was a simple, nowadays in the serverless time it looks like an issue.
How can I get an image size on a client? I’ve checked few cloud storages providers they all are offering resizing but it is not exactly what I’m looking for, I need a full image property before upload image when I initialize the App.
The easiest thing is to find a hosting service with an API that provides the metadata you need.
Failing that, you can work out the dimensions of an image by looking at just the first few dozen bytes. All modern image formats contain dimensions in a file header. Examples in three languages are linked to in the [Photoswipe FAQ][1].
So, you need to download enough bytes (this varies per file format). To do this you need to use the Range header in the HTTP request. For example:
curl http://i.imgur.com/z4d4kWk.jpg -i -H "Range: bytes=0-1023"
... will get the first kilobyte of that image. Whatever HTTP client you use, it will have a way to set a request header.
I looking for the best way to upload an image from mobile phone to my server. I am currently using html5 to open the camera and take the picture, then I convert the file into a base64 string, then I send to the server, then save it in MongoDB.
I am expecting around 1000 to 1500 user request per day ( upload image ) , so I have the following question :
Is it a good way to do it?
Should I compress the base64, if yes how?
Should use a specific server to handle this task?
My backend is node express and the front end is ReactJS.
Thanks
It all depends on your situation. Reading and writing images from a cdn via i.e. streams is usually faster than reading and writing binary representations of images i.e. base64 from a database. However, your speed if reading from a cdn will obviously be effected by what service you use. Today, companies like Amazon can offer storage to a very cheap price so if you are not building a hobby app for like a student project you can usually afford it. Storing binary representation of images actually end up a little bit bigger in size than storing the image itself. You don't compress the base64, you compress the image before converting it. However, if you can't afford a storage account and if you know your users won't upload that many images it is usually enough to store binary representations of the images in a database. Mongo Atlas, for example, offers 512 mb for free on their database clusters. Dividing tasks of your app such as database requests and cdn services from your main application is usually a good choice if possible. This way you will divide the cpu, memory, etc. of your hardware and it will lead to faster reading and writing tasks for the user.
There are a lot of different modules for doing this in node. JIMP is a pretty nice one with loads of built in functions like resizing images and converting them to binary, either as Buffer or base64.
I am using Google Cloud Storage to store user uploaded images.
I use this Google App Engine api PHP function to create a url that serve the image through a Google service.
CloudStorageTools::getImageServingUrl($image_file, $options);
This service returns a public url to the image. This url takes some arguments that can be appended to the end of the url, which allows me to return different versions of the image. This could be a resized or cropped version of the image.
My problem is that this service does not always return the correct image.
Let me give you an example:
This is the url that is returned from the service. The problem is that this is not the correct image. It is not even one of my images.
http://lh3.googleusercontent.com/2oBRlAM8FZBuDm1Byp8fGAZhEwVSg2PlxRWWU2k3vNCn-DJwYWL41w2VnAZGfnLeHrJ-U4KgSSoEKpoduZ8QpRO9xNXRcAOoMrnZabSX6WMF
If I append some arguments at the end of the url, for example =w800 which should give me a resized version of the image with the width of 800px, then the service returns the correct image.
https://lh3.googleusercontent.com/2oBRlAM8FZBuDm1Byp8fGAZhEwVSg2PlxRWWU2k3vNCn-DJwYWL41w2VnAZGfnLeHrJ-U4KgSSoEKpoduZ8QpRO9xNXRcAOoMrnZabSX6WMF=w800
Until now I have discovered this problem on three images. I find it really alarming that the service sometimes is returning someone else's images.
Have any of you experienced a simular problem?
I am a bit worried about this problem, since I was hoping to use this service in a ios and android app that we are developing at my company.
I know that the problem is not related to caching on my computer since this is happening on multiple computers and mobile devices.
we would like to store uploaded images on cdn server in our website. We are thinking to add fallback in our system if cdn service goes down. so images are start to upload on server hard disk in this case. we can manage this fallback using general setting option.
what is efficient way to store upload images path in our database ? we have to store full image url or image name in database. so we can recover easily in fallback situation.