So, here's the problem. iPhones are awesome, but bandwidth and latency are serious issues with apps that have serverside requirements. My initial plan to solve this was to make multiple requests for bits of data (pun unintended) and have that be how the issue of lots of incoming//outgoing data was handled. This is a bad idea for a lot of reasons, most obvious to me is that my poor database (MySQL) can't handle this very well. From what I understand it's better to request large chunks all at once, especially if I'm going to ask for all of it anyways.
The problem is now I'm waiting again for a large amount of data to get through. I was wondering if there's a way to basically send the server a bunch of IDs to get from the database, and then that SINGLE request then sends a lot of little responses, each one containing all the information about a single db entry. Order is irrelevant, and ideally I'd be able to send another request to the server telling it to stop sending me things because I have what I need.
I realize this is probably NOT a simple thing to do so if you (awesome) guys could point me in the right direction that would also be incredible.
Current system is iPhone (Cocoa//Objective-C) -> PHP -> MySQL
Thanks a ton in advance.
AFAIK, a single request cannot get multiple responses. From what you are asking, it seems that you need to do this in two parts.
Part 1: Send a single call with the IDs.
Your server responds with a single message that contains the URLs or the information needed to call the unique "smaller" answers.
Part 2: Working from that list of responses, fire off multiple requests that run on their own threads.
I am thinking of this similar to how a web page works. You call the HTML URL in a web browser. The HTML tells the browser all the places/URLS it needs to get additional pieces (images, css, js, etc) to build the full page.
Hope this helps.
Related
Theres a ton of videos and websites trying to explain backend vs frontend, but unfortunately none of them explains it in a way that you know how to develop a backend - driven website (at least I haven't found anything good).
So, I wanted to ensure that I understood it and kindly ask you to confirm or correct me on this topic.
Example:
I wanted to build Mini - Google. I have a Database containing 1000 stored websites.
Assumption #1:
Everytime I type something into the search bar, the autofill suggestions change. This means, everytime i type, another website / API gets called returning the current autofill suggestions. On a developer site, this means the website e.g. is a Python script which gets called with the current word typed in as a Parameter and is returning all suggestions as e.g. JSON:
// Client Side Script
function ontype(input):
suggestions = get("https://api.googlemini.com/suggestions?q=" + str(input))
show(suggestions)
Assumption #2:
This also means I could manually call the website containing the Python script, providing a random word and it would always return a JSON containing the autofill suggestions for that word.
Question #1:
If A#1 turns out true but A#2 turns out false, how could I prevent a user from randomly accessing the "API" while still returning results when called by a script?
Assumption #3:
After pressing enter, my website googlemini.com/search?... would be called. As google.com/search reloads everytime searching for a new query (or going to page 2 etc.), I assume, instead of calling an API, when the server gets the client request, it first searches through its database, sorts the results and then returns a whole html as a static webpage:
// Server Side Script
#app.route("/search")
function oncall():
query = getparam("q")
results = searchdatabase(query)
html = buildhtml(results)
return html
Question #2:
Often, I hear (or at least understand it this way) that database and webserver are 2 seperate servers. How would that work? Wouldn't that mean the database server needs to be accessible to the web too (of course it would have security layers etc., but technically it would)? How could I access the database server from the webserver?
Question #3:
Are there, on a technical basis, any other ways to build backend services?
That's it. I would also appreciate any recommendations like videos, websites or others to learn how to technically setup and / or secure backend servers.
Thanks in advance.
For your first question you can yes there is a way to prevent miss use.
What you can do is add identifier to api like Auth token to identify a user and every time a user access the api you can save the count on the server n whenever the count has exceeded a limit within a time span you can reject the call. And the limit can be set in such a way that it doesn't trouble the honest user and punishes the wrong one. There are even more complex and effective methods but this is the basic idea.
For question number to let me explain you a simple concept a database is a very efficient, resourcefull and expensive data storage solution we never want it to be used in a general sense as varible store or something. We always want to access the database in call get the data process the data update the data. So we do it data way and its not necessary you make sepreate server for data base. The thing is we mostly make databse to be accessible to various platforms android, ios, windows. So its better to add some abstraction and keep data base as a separte entity.
For the last, I am not well aware about what you meant by other but I am listing some backend teechnologies, some of these might be used in isolation some of these not some other tools as well.
Django
FLask
Djnago rest
GraphQL
SQL
PHP
Node
Deno
I need some guidance on how to properly build out a system that will be able to scale. I will give you some information about what I am trying to do and then ask my specific question.
I have a site where I want visitors to send some data to be processed. They input the data into a textarea or upload it in a file. Simple. The data is somewhat preprocessed on the client side before a POST request is made to a REST endpoint.
What I am stuck on is what is a good way to take this posted data store it and then associate an id with it that references the user since I cannot process the data fast enough for it to be returned to the user in a reasonable amount of time?
This question is a bit vague and open to opinion, I admit it. I just need a push in the right direction to keep moving. What I have been considering is throwing the data into a message queue and then having some workers process the data elsewhere and when the data is processed alert the user as to where to find it with some sort of link to an S3 bucket or just a URL to a file. The other idea was to just run the request for each item to be processed against another end-point that already processes individual records in some sort of loop client side. The problem is as follows with this idea:
To process the data it may take somewhere from 30 minutes to 2 hours depending upon the amount that they want processed. It's not ideal for them to just sit there and wait for that to finish depending on the amount of records they need processed, so I have ruled this out mostly.
Any guidance would be very much appreciated as I don't have any coworkers to bounce things off of, nor do I know many people with the domain knowledge that I could freely ask. If this isn't the right place to ask this, could you point me in the right direction as to where it should be asked?
Chris
If I've got you right, your pipeline is:
Accept item from user
Possibly preprocess/validate it (?)
Put into some queue
Process data
Return result.
You man use one or several queues on stage (3). Entity from user gets added to one of the queues. If it's big enough, it could be stored in S3 or storage alike, and only info about it put into the queue: link, add date, user id (or email of alike). Processors can pull items from queue and give feedback to users.
If you have no strict requirements on order, things get much simpler: you don't need any sync between them. Treat all the components: upload acceptors, queues, storages and processors as independent pools of processes. Monitor each pool separately. If there's some bottlenecks - add machines to that pool.
I want to naI want to know if there is a real practical difference between different types of content in an HTTP response. Let me explain my self better.
Say I submit a POST request to a server with typical resource payload. Let's use a client with client_name, client_email, client_phone.
Would there be an actual difference if the server returns just an id:
{id:100}
Or if it returns the fully created resource without sensible data, like so:
{client_name: 'Some Client', client_email: 'email#sample.com', client_phone: '417-235-4622'}
Suppose that the application as a considerable amount of active users, creating resources at any given moment. Is there a significant cost in server resources associated with returning data from the server (just an ID or a full object)
Given the following scenarios when creating a resource:
Submit POST request, receive resource ID, complete all data visualization feedback with data in memory (info in form element).
Submit POST request, receive full object with id, email and phone. Continue with UI things.
If there is a difference in cost, and its significant, then the response ID is the way to go. But, I'm thinking that if I have lot's fields to submit, and most of them are required, and I'm only expecting an ID in return, then that'a a guarantee that te resource got created but it doesn't mean it was created completely. Suppose I submit the data, and one of those fields fails silently to submit to database (email for example), the server returns ID, the UI shows the user that the resource was created, the user reloads the page and the email is gone.
If the server returns the full object I get the feeling that the transaction is more atomic.
So, to wrap up. Is there a significante difference in terms of cost to the server ?
but it doesn't mean it was created completely. Suppose I submit the data, and one of those fields fails silently to submit to database (email for example)
Even if the email were to be saved in a different table than the rest of the data, it will still have to be done in a transactional manner (an indivisible operation that must succeed or fail as a complete unit; it can never be only partially complete). This could even mean rolling back changes if a failure is detected at any point during the save operation.
Now back to the main question, REST just says that you should conform to the uniform interface. In other words, it says you should do what POST is supposed to do as per the HTTP spec. Here is the quote from that spec that is relevant,
If a resource has been created on the
origin server, the response SHOULD
be 201 (Created) and contain an entity
which describes the status of the
request and refers to the new
resource, and a Location header
(see section 14.30).
I think it all depends on the use case scenarios. If the client immediately needs to display info regarding the newly created object, I really do not see any advantage to returning only the ID and doing a GET request after, to get the data you could have got with your initial POST.
Anyway as long as your API is consistent I think that you should choose the pattern that fits your needs the best. There is not any correct way of how to build a REST API, imo.
Is there a significante difference in terms of cost to the server ?
That's totally unanswerable by us. How powerful is the server? What software are you running on it? What's the breakdown of your expected traffic? What performance targets are you expected to hit? etc..
Performance problems should be solved through a combination of more better hardware and a sensible software architecture that still does everything you need it to. You don't even know if you have a problem yet and you're trying to fix it.
You're asking the wrong question. The question you should be asking is: when my clients create a user, are they likely to need server-created information beyond the URI immediately? Of course, we can't really answer that either. If the server isn't (and won't ever!) be creating anything, there's an obvious answer. If it is, or may, you may want to return a full representation even if the client doesn't need it now, so it's not a breaking change later if they decide they do. The pain there depends a lot on whether this is an internal- or external-facing API, and who owns the clients.
In addition to the other answers given, which are quite comprehensive, I would just like to add that it is contrary to the design of the web to provide object IDs and expect the client to know what to do with them. You should instead be providing URLs to the object in question. Clients can then do a GET request on the provided URL to fetch the full set of data for the object, should they want to. And I f the responses to these GET requests have already been cached, your server will not have to do any work at all to satisfy them!
On an iPhone app (or mobile in general) that constantly needs to send requests to a Web Service, is it better to work with one single requests that will fetch a large amount of data or multiple (possibly simultaneous) requests for each element with smaller amount of data fetched.
Example
I want to load a list of elements in a node. I have the node's ID. The 2 ways I can fetch the elements are the following :
send a single request with the node ID and get all the information about the n first elements in the node in a single response ;
send a first request with the node ID to get the IDs of the n first elements in the node, then for each one send another request to have one response per element.
I'm balanced between about that.
the heavyweight single response may cause more lag and timeouts because of the very unstable and slow mobile internet connection ;
the phone may have trouble handling too many responses at the same time.
What's your opinion ?
Since there is overhead for every request, one large request is generally faster than several small ones of the same size. This applies to high speed networks too, but in mobile networks the ratio between transfer speed and latency is even bigger.
I don't think the phone will have any problem handling the responses, so the multiple requests approach seems better for large requests/answers. However, depending on the size of your requests/responses, it may actually be faster to do it in a single request, in order to reduce the delay associated with multiple requests. The single request approach will also need to transfer slightly less data than the multiple request one.
Every call will have its overhead (i.e. network load), the number of connections might also be limited.
You might or might not be able to update your user interface during download, depending on how often your callbacks are called - you may be able to process partial data as it arrives.
If your data is easy to compress (typically text data), then using a single call might even reduce your total network usage footprint even more.
If the chunks of data are large, I'd go with several individual ones. This will also make things easier in case of network errors. Bottom line for me is to just get the right balance - make the packets reasonable sized and don't flood the server.
This is depend upon the situation. If you don't want to bother your user to waiting everytime throughout the app then you can use single request to load all the data at a time.
If you don't mind to let user wait then you can use multiple request on demand. For example if you just want to show title in tableview and detail when user tap on any title. So you can first get the title only and then when user tap you can get details for that title by ID. so that would be pretty good way to request on demand only.
Sometimes the situation merits use for only single requests for say a certain category. Say you have a twitter app and the tweets are seperated out into categories. Someone who has the app but only cares about sports may only look at the sports section which could be a single ajax call. Another user may only be intersted in two categories out of 15 categories. This means the user doesn't have to load unneccessary data. The important thing you need to determine is this.
Does all of the data need to be loaded all at once for the app to work correctly and are your users generally going to want all that data in the first place.
Working on an app where all the contents/data for it will be coming via JSON and occasionally i will display a HTML page.
The client is suggesting that maybe we should have some local database(MYSQL Lite) to cache the JSON data returned so we use less of the users data(if there search for the same item again) allowance and because it maybe slightly faster.
Are these good enough reasons for adding the extra complexity and potential problems of having a local DB on the phone?
I didn't think from my experience that the phone was particularly slow or that JSON or HTML were data heavy in there data usage. I'd prefer having a thin client.
Facebook/Twitter/etc work with very little problems using JSON and Html.
Would I be wrong to try steer away from the local DB idea?
Thanks,
-Code
Caching url request results can improve your application's latency over a slow connexion. You could use CoreData to manually manage a cache (key:url, value:request's answer)
Another more elegant solution would be (if you have write access to the webservices) to implement server-side the "if-modified-since" header so that your request data received would be kept at a minimum level.