Assume there is a system that accepts millions of simultaneous WebSocket connections from client applications. I was wondering if there is a way to route WebSocket connections to a specific instance behind a load balancer (or IP/Domain/etc) if clients provide some form of metadata, such as hash key, instance name, etc.
For instance, let's say each WebSocket client of the above system will always belong to a group (e.g. max group size of 100), and it will attempt to communicate with 99 other clients using the above system as a message gateway.
So the system's responsibility is to relay messages sent from clients in a group to other 99 clients in the same group. Clients won't ever need to communicate with other clients who belong to different groups.
Of course, one way to tackle this problem is to use Pubsub system, such that regardless of which instance clients are connected to, the server can simply publish the message to the Pubsub system with a group identifier and other clients can subscribe to the messages with a group identifier.
However, the Pubsub system can potentially encounter scaling challenges, excessive resource usage (single message getting published to thousands of instances), management overhead, latency increase, cost, and etc.
If it is possible to guarantee that WebSocket clients in a group will all be connected to the instance behind LB, we can skip using the Pubsub system and make things simpler, lower latency, and etc.
Would this be something that is possible to do, and if it isn't, what would be the best option?
(I am using Kubernetes in one of the cloud service providers if that matters.)
Routing in HTTP is generally based on the hostname and/or URL path. Sometimes to a lesser degree on other headers like cookies. But in this case it would mean that each group should have it's own unique URL.
But that part is easy, what I think you're really asking is "given arbitrary URLs, how can I get consistent routing?" which is much, much more complicated. The base concept is "consistent hashing", you hash the URL and use that to pick which endpoint to talk to. But then how to do deal with adding or removing replicas without scrambling the mapping entirely. That usually means using a hash ring and assigning portions of the hash space to specific replicas. Unfortunately this is the point where off-the-shelf tools aren't enough. These kinds of systems require deep knowledge of your protocol and system specifics so you'll probably need to rig this up yourself.
I m studying Microservices architecture and I m actually wondering something.
I m quite okay with the fact of using (back) service discovery to make request able on REST based microservices. I need to know where's the service (or at least the front of the server cluster) to make requests. So it make sense to be able to discover an ip:port in that case.
But I was wondering what could be the aim of using service registry / discovery when dealing with AMQP (based only, without HTTP possible calls) ?
I mean, using AMQP is just like "I need that, and I expect somebody to answer me", I dont have to know who's the server that sent me back the response.
So what is the aim of using service registry / discovery with AMQP based microservice ?
Thanks for your help
AMQP (any MOM, actually) provides a way for processes to communicate without having to mind about actual IP addresses, communication security, routing, among other concerns. That does not necessarily means that any process can trust or even has any information about the processes it communicates with.
Message queues do solve half of the process: how to reach the remote service. But they do not solve the other half: which service is the right one for me. In other words, which service:
has the resources I need
can be trusted (is hosted on a reliable server, has a satisfactory service implementation, is located in a country where the local laws are compatible with your requirements, etc)
charges what you want to pay (although people rarely discuss cost when it comes to microservices)
will be there during the whole time window needed to process your service -- keep in mind that servers are becoming more and more volatile. Some servers are actually containers that can last for a couple minutes.
Those two problems are almost linearly independent. To solve the second kind of problems, you have resource brokers in Grid computing. There is also resource allocation in order to make sure that the last item above is correctly managed.
There are some alternative strategies such as multicasting the intention to use a service and waiting for replies with offers. You may have reverse auction in such a case, for instance.
In short, the rule of thumb is that if you do not have an a priori knowledge about which service you are going to use (hardcoded or in some configuration file), your agent will have to negotiate, which includes dynamic service discovery.
I am doing a system architecture and my knowledge from college doesn't help me when it comes to understand the subtle differences between centralized, distributed and service oriented architecture/application.
If I take a typical client/server architecture, the client sends requests to a server, the server then sends responses to the client. That is a centralized architecture.
An application that handles both server and client sides will be a distributed application (because working on different platforms), but that is still a centralized architecture.
Therefore, a distributed architecture must involve a distributed application.
Questions: am I right? What does all that become when it comes to service oriented architectures / applications?
Distributed: The whole process involved in a computation task is divided into pieces and assigned to multiple computational nodes. Each node when doing its part of the processing does not have access to the whole information of the system that is necessary to achieve a globally optimized result. The aggregate of the results from multiple nodes will converge towards a global optimal result through usually multiple iterations of computations distributed across multiple nodes.
A good example is a router system in which each router has only the information it exchanges with its neighbours. At the start the neighbours known only part of the whole network system. Once a router gets more information from its neighbours it incorporates the new information into its view of the whole system, then spreads its view to its neighbours. Through multiple iteration of these steps, each separately computed by individual routers, all routers would settle on a consistent global view of the whole network system.
Another example could be a web ordering system where the browser initially gets a list of commonly order goods. The browser may have logic to track user viewing behaviour and make a decision to fetch from the server a different category of goods list, but does not send all the user behaviour parameters to the server. In this imaged example, the browser knows something the serer does not know, and the server does too. Thus the whole application would be a distributed system. In addition to this part of distributed nature, the user authentication could be done on one of the servers, the inventory is done on another server, and reservation on yet another one. Each of the servers involved would not have the whole information of the specific user browsing and ordering instance. But the aggregate work from all these nodes will fulfill the business need to sell more goods and satisfy more customers.
Opposite to the distributed, is the centralized, in that the computation logic would be always able to get the information of the whole picture.
Given this view, a client-server application can be a viewed as a distributed system if you think the client side involves non-trivial decision making. Or can be a centralized system if you think the client is dumb.
The service-oriented term is more about how the functional processing power is integrated into the system. In a service-oriented system, new capability may be introduced into the system at runtime by new API functionality discovery, or new logic capability discovery behind an unchanged API. Think about it, you could build an application that initially has no much built-in capability, then it expands its capability by discovering and incorporating new capability on the service providers. In contrast a traditional system needs to be built at build time, typically as a consequence of human-involving discussion-design-documentation activity. A service-oriented design is a good fit to a distributed system.
I've developed some web-based applications till now using PHP, Python and Java. But some fundamental but very important questions are still beyond my knowledge, so I made this post to get help and clarification from you guys.
Say I use some programming language as my backend language(PHP/Python/.Net/Java, etc), and I deploy my application with a web server(apache/lighttpd/nginx/IIS, etc). And suppose at time T, one of my page got 100 simultaneous requests from different users. So my questions are:
How does my web server handle such 100 simultaneous requests? Will web server generate one process/thread for each request? (if yes, process or thread?)
How does the interpreter of the backend language do? How will it handle the request and generate the proper html? Will the interpreter generate a process/thread for each request?(if yes, process or thread?)
If the interpreter will generate a process/thread for each request, how about these processes(threads)? Will they share some code space? Will they communicate with each other? How to handle the global variables in the backend codes? Or they are independent processes(threads)? How long is the duration of the process/thread? Will they be destroyed when the request is handled and the response is returned?
Suppose the web server can only support 100 simultaneous requests, but now it got 1000 simultaneous requests. How does it handle such situation? Will it handle them like a queue and handle the request when the server is available? Or other approaches?
I read some articles about Comet these days. And I found long connection may be a good way to handle the real-time multi-users usecase. So how about long connection? Is it a feature of some specific web servers or it is available for every web server? Long connection will require a long-existing interpreter process?
EDIT:
Recently I read some articles about CGI and fastcgi, which makes me know the approach of fastcgi should be a typical approach to hanlde request.
the protocol multiplexes a single transport connection between several independent FastCGI requests. This supports applications that are able to process concurrent requests using event-driven or multi-threaded programming techniques.
Quoted from fastcgi spec, which mentioned connection which can handle several requests, and can be implemented in mutli-threaded tech. I'm wondering this connection can be treated as process and it can generate several threads for each request. If this is true, I become more confused about how to handle the shared resource in each thread?
P.S thank Thomas for the advice of splitting the post to several posts, but I think the questions are related and it's better to group them together.
Thank S.Lott for your great answer, but some answers to each question are too brief or not covered at all.
Thank everyone's answer, which makes me closer to the truth.
Update, Spring 2018:
I wrote this response in 2010 and since then, a whole lot of things have changed in the world of a web backend developer. Namely, the advent of the "cloud" turning services such as one-click load balancers and autoscaling into commodities have made the actual mechanics of scaling your application much easier to get started.
That said, what I wrote in this article in 2010 still mostly holds true today, and understanding the mechanics behind how your web server and language hosting environment actually works and how to tune it can save you considerable amounts of money in hosting costs. For that reason, I have left the article as originally written below for anyone who is starting to get elbows deep in tuning their stack.
1. Depends on the webserver (and sometimes configuration of such). A description of various models:
Apache with mpm_prefork (default on unix): Process per request. To minimize startup time, Apache keeps a pool of idle processes waiting to handle new requests (which you configure the size of). When a new request comes in, the master process delegates it to an available worker, otherwise spawns up a new one. If 100 requests came in, unless you had 100 idle workers, some forking would need to be done to handle the load. If the number of idle processes exceeds the MaxSpare value, some will be reaped after finishing requests until there are only so many idle processes.
Apache with mpm_event, mpm_worker, mpm_winnt: Thread per request. Similarly, apache keeps a pool of idle threads in most situations, also configurable. (A small detail, but functionally the same: mpm_worker runs several processes, each of which is multi-threaded).
Nginx/Lighttpd: These are lightweight event-based servers which use select()/epoll()/poll() to multiplex a number of sockets without needing multiple threads or processes. Through very careful coding and use of non-blocking APIs, they can scale to thousands of simultaneous requests on commodity hardware, provided available bandwidth and correctly configured file-descriptor limits. The caveat is that implementing traditional embedded scripting languages is almost impossible within the server context, this would negate most of the benefits. Both support FastCGI however for external scripting languages.
2. Depends on the language, or in some languages, on which deployment model you use. Some server configurations only allow certain deployment models.
Apache mod_php, mod_perl, mod_python: These modules run a separate interpreter for each apache worker. Most of these cannot work with mpm_worker very well (due to various issues with threadsafety in client code), thus they are mostly limited to forking models. That means that for each apache process, you have a php/perl/python interpreter running inside. This severely increases memory footprint: if a given apache worker would normally take about 4MB of memory on your system, one with PHP may take 15mb and one with Python may take 20-40MB for an average application. Some of this will be shared memory between processes, but in general, these models are very difficult to scale very large.
Apache (supported configurations), Lighttpd, CGI: This is mostly a dying-off method of hosting. The issue with CGI is that not only do you fork a new process for handling requests, you do so for -every- request, not just when you need to increase load. With the dynamic languages of today having a rather large startup time, this creates not only a lot of work for your webserver, but significantly increases page load time. A small perl script might be fine to run as CGI, but a large python, ruby, or java application is rather unwieldy. In the case of Java, you might be waiting a second or more just for app startup, only to have to do it all again on the next request.
All web servers, FastCGI/SCGI/AJP: This is the 'external' hosting model of running dynamic languages. There are a whole list of interesting variations, but the gist is that your application listens on some sort of socket, and the web server handles an HTTP request, then sends it via another protocol to the socket, only for dynamic pages (static pages are usually handled directly by the webserver).
This confers many advantages, because you will need less dynamic workers than you need the ability to handle connections. If for every 100 requests, half are for static files such as images, CSS, etc, and furthermore if most dynamic requests are short, you might get by with 20 dynamic workers handling 100 simultaneous clients. That is, since the normal use of a given webserver keep-alive connection is 80% idle, your dynamic interpreters can be handling requests from other clients. This is much better than the mod_php/python/perl approach, where when your user is loading a CSS file or not loading anything at all, your interpreter sits there using memory and not doing any work.
Apache mod_wsgi: This specifically applies to hosting python, but it takes some of the advantages of webserver-hosted apps (easy configuration) and external hosting (process multiplexing). When you run it in daemon mode, mod_wsgi only delegates requests to your daemon workers when needed, and thus 4 daemons might be able to handle 100 simultaneous users (depends on your site and its workload)
Phusion Passenger: Passenger is an apache hosting system that is mostly for hosting ruby apps, and like mod_wsgi provides advantages of both external and webserver-managed hosting.
3. Again, I will split the question based on hosting models for where this is applicable.
mod_php, mod_python, mod_perl: Only the C libraries of your application will generally be shared at all between apache workers. This is because apache forks first, then loads up your dynamic code (which due to subtleties, is mostly not able to use shared pages). Interpreters do not communicate with each other within this model. No global variables are generally shared. In the case of mod_python, you can have globals stay between requests within a process, but not across processes. This can lead to some very weird behaviours (browsers rarely keep the same connection forever, and most open several to a given website) so be very careful with how you use globals. Use something like memcached or a database or files for things like session storage and other cache bits that need to be shared.
FastCGI/SCGI/AJP/Proxied HTTP: Because your application is essentially a server in and of itself, this depends on the language the server is written in (usually the same language as your code, but not always) and various factors. For example, most Java deployment use a thread-per-request. Python and its "flup" FastCGI library can run in either prefork or threaded mode, but since Python and its GIL are limiting, you will likely get the best performance from prefork.
mod_wsgi/passenger: mod_wsgi in server mode can be configured how it handles things, but I would recommend you give it a fixed number of processes. You want to keep your python code in memory, spun up and ready to go. This is the best approach to keeping latency predictable and low.
In almost all models mentioned above, the lifetime of a process/thread is longer than a single request. Most setups follow some variation on the apache model: Keep some spare workers around, spawn up more when needed, reap when there are too many, based on a few configurable limits. Most of these setups -do not- destroy a process after a request, though some may clear out the application code (such as in the case of PHP fastcgi).
4. If you say "the web server can only handle 100 requests" it depends on whether you mean the actual webserver itself or the dynamic portion of the webserver. There is also a difference between actual and functional limits.
In the case of Apache for example, you will configure a maximum number of workers (connections). If this number of connections was 100 and was reached, no more connections will be accepted by apache until someone disconnects. With keep-alive enabled, those 100 connections may stay open for a long time, much longer than a single request, and those other 900 people waiting on requests will probably time out.
If you do have limits high enough, you can accept all those users. Even with the most lightweight apache however, the cost is about 2-3mb per worker, so with apache alone you might be talking 3gb+ of memory just to handle the connections, not to mention other possibly limited OS resources like process ids, file descriptors, and buffers, and this is before considering your application code.
For lighttpd/Nginx, they can handle a large number of connections (thousands) in a tiny memory footprint, often just a few megs per thousand connections (depends on factors like buffers and how async IO apis are set up). If we go on the assumption most your connections are keep-alive and 80% (or more) idle, this is very good, as you are not wasting dynamic process time or a whole lot of memory.
In any external hosted model (mod_wsgi/fastcgi/ajp/proxied http), say you only have 10 workers and 1000 users make a request, your webserver will queue up the requests to your dynamic workers. This is ideal: if your requests return quickly you can keep handling a much larger user load without needing more workers. Usually the premium is memory or DB connections, and by queueing you can serve a lot more users with the same resources, rather than denying some users.
Be careful: say you have one page which builds a report or does a search and takes several seconds, and a whole lot of users tie up workers with this: someone wanting to load your front page may be queued for a few seconds while all those long-running requests complete. Alternatives are using a separate pool of workers to handle URLs to your reporting app section, or doing reporting separately (like in a background job) and then polling its completion later. Lots of options there, but require you to put some thought into your application.
5. Most people using apache who need to handle a lot of simultaneous users, for reasons of high memory footprint, turn keep-alive off. Or Apache with keep-alive turned on, with a short keep-alive time limit, say 10 seconds (so you can get your front page and images/CSS in a single page load). If you truly need to scale to 1000 connections or more and want keep-alive, you will want to look at Nginx/lighttpd and other lightweight event-based servers.
It might be noted that if you do want apache (for configuration ease of use, or need to host certain setups) you can put Nginx in front of apache, using HTTP proxying. This will allow Nginx to handle keep-alive connections (and, preferably, static files) and apache to handle only the grunt work. Nginx also happens to be better than apache at writing logfiles, interestingly. For a production deployment, we have been very happy with nginx in front of apache(with mod_wsgi in this instance). The apache does not do any access logging, nor does it handle static files, allowing us to disable a large number of the modules inside apache to keep it small footprint.
I've mostly answered this already, but no, if you have a long connection it doesn't have to have any bearing on how long the interpreter runs (as long as you are using external hosted application, which by now should be clear is vastly superior). So if you want to use comet, and a long keep-alive (which is usually a good thing, if you can handle it) consider the nginx.
Bonus FastCGI Question You mention that fastcgi can multiplex within a single connection. This is supported by the protocol indeed (I believe the concept is known as "channels"), so that in theory a single socket can handle lots of connections. However, it is not a required feature of fastcgi implementors, and in actuality I do not believe there is a single server which uses this. Most fastcgi responders don't use this feature either, because implementing this is very difficult. Most webservers will make only one request across a given fastcgi socket at a time, then make the next across that socket. So you often just have one fastcgi socket per process/thread.
Whether your fastcgi application uses processing or threading (and whether you implement it via a "master" process accepting connections and delegating or just lots of processes each doing their own thing) is up to you; and varies based on capabilities of your programming language and OS too. In most cases, whatever is the default the library uses should be fine, but be prepared to do some benchmarking and tuning of parameters.
As to shared state, I recommend you pretend that any traditional uses of in-process shared state do not exist: even if they may work now, you may have to split your dynamic workers across multiple machines later. For state like shopping carts, etc; the db may be the best option, session-login info can be kept in securecookies, and for temporary state something akin to memcached is pretty neat. The less you have reliant on features that share data (the "shared-nothing" approach) the bigger you can scale in the future.
Postscript: I have written and deployed a whole lot of dynamic applications in the whole scope of setups above: all of the webservers listed above, and everything in the range of PHP/Python/Ruby/Java. I have extensively tested (using both benchmarking and real-world observation) the methods, and the results are sometimes surprising: less is often more. Once you've moved away from hosting your code in the webserver process, You often can get away with a very small number of FastCGI/Mongrel/mod_wsgi/etc workers. It depends on how much time your application stays in the DB, but it's very often the case that more processes than 2*number of CPU's will not actually gain you anything.
How does my web server handle such 100 simultaneous requests? Does web server generate one process/thread for each request? (if yes, process or thread?)
It varies. Apache has both threads and processes for handling requests. Apache starts several concurrent processes, each one of which can run any number of concurrent threads. You must configure Apache to control how this actually plays out for each request.
How does the interpreter of the backend language do? How will it handle the request and generate the proper html? Will the interpreter generate a process/thread for each request?(if yes, process or thread?)
This varies with your Apache configuration and your language. For Python one typical approach is to have daemon processes running in the background. Each Apache process owns a daemon process. This is done with the mod_wsgi module. It can be configured to work several different ways.
If the interpreter will generate a process/thread for each request, how about these processes(threads)? Will they share some code space? Will they communicate with each other? How to handle the global variables in the backend codes? Or they are independent processes(threads)? How long is the duration of the process/thread? Will they be destroyed when the request is handled and the response is returned?
Threads share the same code. By definition.
Processes will share the same code because that's the way Apache works.
They do not -- intentionally -- communicate with each other. Your code doesn't have a way to easily determine what else is going on. This is by design. You can't tell which process you're running in, and can't tell what other threads are running in this process space.
The processes are long-running. They do not (and should not) be created dynamically. You configure Apache to fork several concurrent copies of itself when it starts to avoid the overhead of process creation.
Thread creation has much less overhead. How Apaches handles threads internally doesn't much matter. You can, however, think of Apache as starting a thread per request.
Suppose the web server can only support 100 simultaneous requests, but now it got 1000 simultaneous requests. How does it handle such situation? Will it handle them like a queue and handle the request when the server is available? Or other approaches?
This is the "scalability" question. In short -- how will performance degrade as the load increases. The general answer is that the server gets slower. For some load level (let's say 100 concurrent requests) there are enough processes available that they all run respectably fast. At some load level (say 101 concurrent requests) it starts to get slower. At some other load level (who knows how many requests) it gets so slow you're unhappy with the speed.
There is an internal queue (as part of the way TCP/IP works, generally) but there's no governor that limits the workload to 100 concurrent requests. If you get more requests, more threads are created (not more processes) and things run more slowly.
To begin with, requiring detailed answers to all your points is a bit much, IMHO.
Anyway, a few short answers about your questions:
#1
It depends on the architecture of the server. Apache is a multi-process, and optionally also, multi-threaded server. There is a master process which listens on the network port, and manages a pool of worker processes (where in the case of the "worker" mpm each worker process has multiple threads). When a request comes in, it is forwarded to one of the idle workers. The master manages the size of the worker pool by launching and terminating workers depending on the load and the configuration settings.
Now, lighthttpd and nginx are different; they are so-called event-based architectures, where multiple network connections are multiplexed onto one or more worker processes/threads by using the OS support for event multiplexing such as the classic select()/poll() in POSIX, or more scalable but unfortunately OS-specific mechanisms such as epoll in Linux. The advantage of this is that each additional network connection needs only maybe a few hundred bytes of memory, allowing these servers to keep open tens of thousands of connections, which would generally be prohibitive for a request-per-process/thread architecture such as apache. However, these event-based servers can still use multiple processes or threads in order to utilize multiple CPU cores, and also in order to execute blocking system calls in parallel such as normal POSIX file I/O.
For more info, see the somewhat dated C10k page by Dan Kegel.
#2
Again, it depends. For classic CGI, a new process is launched for every request. For mod_php or mod_python with apache, the interpreter is embedded into the apache processes, and hence there is no need to launch a new process or thread. However, this also means that each apache process requires quite a lot of memory, and in combination with the issues I explained above for #1, limits scalability.
In order to avoid this, it's possible to have a separate pool of heavyweight processes running the interpreters, and the frontend web servers proxy to the backends when dynamic content needs to be generated. This is essentially the approach taken by FastCGI and mod_wsgi (although they use custom protocols and not HTTP so perhaps technically it's not proxying). This is also typically the approach chosen when using the event-based servers, as the code for generating the dynamic content seldom is re-entrant which it would need to be in order to work properly in an event-based environment. Same goes for multi-threaded approaches as well if the dynamic content code is not thread-safe; one can have, say, frontend apache server with the threaded worker mpm proxying to backend apache servers running PHP code with the single-threaded prefork mpm.
#3
Depending on at which level you're asking, they will share some memory via the OS caching mechanism, yes. But generally, from a programmer perspective, they are independent. Note that this independence is not per se a bad thing, as it enables straightforward horizontal scaling to multiple machines. But alas, some amount of communication is often necessary. One simple approach is to communicate via the database, assuming that one is needed for other reasons, as it usually is. Another approach is to use some dedicated distributed memory caching system such as memcached.
#4
Depends. They might be queued, or the server might reply with some suitable error code, such as HTTP 503, or the server might just refuse the connection in the first place. Typically, all of the above can occur depending on how loaded the server is.
#5
The viability of this approach depends on the server architecture (see my answer to #1). For an event-based server, keeping connections open is no big issue, but for apache it certainly is due to the large amount of memory required for every connection. And yes, this certainly requires a long-running interpreter process, but as described above, except for classic CGI this is pretty much granted.
Web servers are multi-threaded environment; besides using application scoped variables, a user request doesn't interact with other threads.
So:
Yes, a new thread will be created for every user
Yes, HTML will be processed for every request
You'll need to use application scoped variables
If you get more requests than you can deal, they will be put on queue. If they got served before configured timeout period, user will get his response, or a "server busy" like error.
Comet isn't specific for any server/language. You can achieve same result by quering your server every n seconds, without dealing with other nasty threads issues.
I've been considering the advantages of REST services, the whole statelessness and session affinity "stuff". What strikes me is that if you have multiple deployed versions of your service on a number of machines in your infrastructure, and they all act on a given resource, where is the state of that resource stored?
Would it make sense to have a single host in the infrastructre that utilises a distributed cache, and any state that is change inside a service, it simply fetches/puts to the cache? This would allow any number of deployed services for loading balancing reasons to all see the same state views of resources.
If you're designing a system for high load (which usually implies high reliability), having a single point of failure is never a good idea. If the service providing the consistent view goes down, at best your performance decreases drastically as the database is queried for everything and at worst, your whole application stops working.
In your question, you seem to be worried about consistency. If there's something to be learned about eBay's architecture, it's that there is a trade-off to be made between availability/redundancy/performance vs consistency. You may find 100% consistency is not required and you can get away with a little "chaos".
A distributed cache (like memcache) can be used as a backing for a distributed hashtable which have been used extensively to create scalable infrastructures. If implemented correctly, caches can be redundant and caches can join and leave the ring dynamically.
REST is also inherently cacheable as the HTTP layer can be cached with the appropriate use of headers (ETags) and software (e.g. Squid proxy as a Reverse proxy). The one drawback of specifying caching through headers is that it relies on the client interpreting and respecting them.
However, to paraphrase Phil Karlton, caching is hard. You really have to be selective about the data that you cache, when you cache it and how you invalidate that cache. Invalidating can be done in the following ways:
Through a timer based means (cache for 2 mins, then reload)
When an update comes in, invalidating all caches containing the relevant data.
I'm partial to the timer based approach as its simpler to implement and you can say with relative certainty how long stale data will live in the system (e.g. Company details will be updated in 2 hours, Stock prices will be updated in 10 seconds).
Finally, high load also depends on your use case and depending on the amount of transactions none of this may apply. A methodology (if you will) may be the following:
Make sure the system is functional without caching (Does it work)
Does it meet performance criteria (e.g. requests/sec, uptime goals)
Optimize the bottlenecks
Implement caching where required
After all, you may not have a performance problem in the first place and you may able to get away with a single database and a good back up strategy.
I think the more traditional view of load balancing web applications is that you would have your REST service on multiple application servers and they would retrieve resource data from single database server.
However, with the use of hypermedia, REST services can easily vertically partition the application so that some resources come from one service and some from another service on a different server. This would allow you to scale to some extent, depending on your domain, without have a single data store. Obviously with REST you would not be able to do transactional updates across these services, but there are definitely scenarios where this partitioning is valuable.
If you are looking at architectures that need to really scale then I would suggest looking at Greg Young's stuff on CQS Architecture (video) before attempting to tackle the problems of a distributed cache.