I have this Play app that connects to a distant server in order to consume a given API. In order to load balance my requests to the distant server, I connect multiple accounts to that same server. Each account can query the API a given number of times. Each account is handled by an Akka actor and these actors are behind an Akka Round Robin router. Thus when wanting to consume the distant API, I "ask" the RR router for the wanted info.
This implementation runs fine until, one account gets disconnected. Basically, when one account is disconnected, the actor returns a given object that says "something was wrong with the connection", and then I send a second request to the RR router again to be handled by another account.
My question is, instead of having to have the "retry" logic outside the router-routee group, is there a way to do it inside? I am thinking that for example at router level, define a logic that handles these "something was wrong with the connection" messages by automatically forwarding the request to the next routee to be handled by it, and only return a final response once all routees have been tried and none worked?
Does Akka provide a simple way of achieving this or should I just carry on with my implementation?
I'm not sure if I fully understand your design but I think you should try using first complete model supported by the ScatterGatherFirstCompleted routing logic.
router.type-mapping {
...
scatter-gather-pool = "akka.routing.ScatterGatherFirstCompletedPool"
scatter-gather-group = "akka.routing.ScatterGatherFirstCompletedGroup"
..
}
in the simple form
---> Route
--> SGFC-1
RR ->
or possibly combined with round robin router.
---> Route
--> SGFC-1
RR ->
--> SGFC-2
---> Route
The same as in your proposal connections are represented by the routes. SGFC-1 and SGFC-2 should have access to the same pool of routees (conenctions.) or part of the pool.
Related
I have a complex problem and I can't figure out which one is the best solution to solve it.
this is the scenario:
I have N servers under a single load balancer and a Database.
All the servers connect to the database
All the servers run the same identical application
I want to implement a Cache in order to decrease the response time and reduce to the minimum the HTTP calls Server -> Database
I implemented it and works like a charm on a single server...but I need to find a mechanism to update all the other caches in the other servers when the data is not valid anymore.
example:
I have server A and server B, both have their own cache.
At the first request from the outside, for example, get user information, replies server A.
his cache is empty so he needs to get the information from the database.
the second request goes to B, also here server B cache is empty, so he needs to get information from the database.
the third request, again on server A, now the data is in the cache, it replies immediately without database request.
the fourth request, on server B, is a write request (for example change user name), server B can make the changes on the database and update his own cache, invalidating the old user.
but server A still has the old invalid user.
So I need a mechanism for server B to communicate to server A (or N other servers) to invalidate/update the data in the cache.
whats is the best way to do this, in scala play framework?
Also, consider that in the future servers can be in geo-redundancy, so in different geographical locations, in a different network, served by a different ISP.
would be great also to update all the other caches when one user is loaded (one server request from database update all the servers caches), this way all the servers are ready for future request.
Hope I have been clear.
Thanks
Since you're using Play, which under the hood, already uses Akka, I suggest using Akka Cluster Sharding. With this, the instances of your Play service would form a cluster (including failure detection, etc.) at startup, and organize between themselves which instance owns a particular user's information.
So proceeding through your requests, the first request to GET /userinfo/:uid hits server A. The request handler hashes uid (e.g. with murmur3: consistent hashing is important) and resolves it to, e.g., shard 27. Since the instances started, this is the first time we've had a request involving a user in shard 27, so shard 27 is created and let's say it gets owned by server A. We send a message (e.g. GetUserInfoFor(uid)) to a new UserInfoActor which loads the required data from the DB, stores it in its state, and replies. The Play API handler receives the reply and generates a response to the HTTP request.
For the second request, it's for the same uid, but hits server B. The handler resolves it to shard 27 and its cluster sharding knows that A owns that shard, so it sends a message to the UserInfoActor on A for that uid which has the data in memory. It replies with the info and the Play API handler generates a response to the HTTP request from the reply.
In this way, all subsequent requests (e.g. the third, the same GET hitting server A) for the user info will not touch the DB, no matter which server they hit.
For the fourth request, which let's say is POST /userinfo/:uid and hits server B, the request handler again hashes the uid to shard 27 but this time, we send, e.g., an UpdateUserInfoFor(uid, newInfo) message to that UserInfoActor on server A. The actor receives the message, updates the DB, updates its in-memory user info and replies (either something simple like Done or the new info). The request handler generates a response from that reply.
This works really well: I've personally seen systems using cluster sharding keep terabytes in memory and operate with consistent single-digit millisecond latency for streaming analytics with interactive queries. Servers crash, and the actors running on the servers get rebalanced to surviving instances.
It's important to note that anything matching your requirements is a distributed system and you're requiring strong consistency, i.e. you're requiring that it be unavailable under a network partition (if B is unable to communicate an update to A, it has no choice but to fail the request). Once you start talking about geo-redundancy and multiple ISPs, you're going to see partitions pretty regularly. The only way to get availability under a network partition is to relax the consistency demand and accept that sometimes the GET will not incorporate the latest PUT/POST/DELETE.
This is probably not something that you want to build yourself. But there are plenty of distributed caches out there that you can use, such as Ehcache or InfiniSpan. I suggest you look into one of those two.
one connection send many request to server
How to process request concurrently.
Please use a simple example like timeserver or echoserver in netty.io
to illustrate the operation.
One way I could find out is to create a separate threaded handler that will be called as in a producer/consumer way.
The producer will be your "network" handler, giving message to the consumers, therefore not waiting for any wanswear and being able then to proceed with the next request.
The consumer will be your "business" handler, one per connection but possibly multi-threaded, consuming with multiple instances the messages and being able to answer using the Netty's context from the connection from which it is attached.
Another option for the consumer would be to have only one handler, still multi-threaded, but then message will come in with the original Netty's Context such that it can answear to the client, whatever the connection attached.
But the difficulties will come soon:
How to deal with an answear among several requests on client side: let say the client sends 3 requests A, B and C and the answears will come back, due to speed of the Business handler, as C, A, B... You have to deal with it, and knowing for which request the answer is.
You have to ensure all the ways the context given in parameter is still valid (channel active), if you don't want to have too many errors.
Perhaps the best way would be to however handle your request in order (as Netty does), and keep the answear's action as quick as possible.
I am using pub/Sub Socket and currently the server subscribe byte[0] (all topics)
while client subscribe byte[16] - a specific header as topic
However, I cannot stop client to subscribe byte[0] which can receive all other messages.
My application is a like a app game which has one single server using ZMQ as connection
and many clients have a ZMQ sockets to talk with server.
What pattern or socket I should use in this case?
Thanks
" ... cannot stop client to subscribe byte[0] which can receive all other messages."
Stopping a "subscribe to all" mode of the SUB client
For the ZMQ PUB/SUB Formal Communication Pattern archetype, the SUB client has to submit it's subscription request ( via zmq_setsockopt() ).
PUB-side ( a Game Server ) has got no option to do that from it's side.
There is no-subscription state right on a creation of a new SUB socket, thus an absolutely restrictive filter, thas no message pass through. ( For furhter details on methods for SUBSCRIBE / UNSUBSCRIBE ref. below )
ZeroMQ specification details setting for this:
int zmq_setsockopt ( void *socket,
int option_name,
const void *option_value,
size_t option_len
);
Caution: only ZMQ_SUBSCRIBE
ZMQ_UNSUBSCRIBE
ZMQ_LINGER
take effect immediately,
other options are active only for subsequent socket bind/connects.
ZMQ_SUBSCRIBE: Establish message filter
The ZMQ_SUBSCRIBE option shall establish a new message filter on a ZMQ_SUB socket. Newly created ZMQ_SUB sockets shall filter out all incoming messages, therefore you should call this option to establish an initial message filter.
An empty option_value of length zero shall subscribe to all incoming messages.
A non-empty option_value shall subscribe to all messages beginning with the specified prefix.
Multiple filters may be attached to a single ZMQ_SUB socket, in which case a message shall be accepted if it matches at least one filter.
ZMQ_UNSUBSCRIBE: Remove message filter
The ZMQ_UNSUBSCRIBE option shall remove an existing message filter on a ZMQ_SUB socket. The filter specified must match an existing filter previously established with the ZMQ_SUBSCRIBE option. If the socket has several instances of the same filter attached the ZMQ_UNSUBSCRIBE option shall remove only one instance, leaving the rest in place and functional.
How to enforce an ad-hoc, server-dictated, ZMQ_SUBSCRIBE restrictions?
This is possible via extending the messaging layer and adding a control-mode socket, that will carry server-initiated settings for the client ZMQ_SUB messages filtering.
Upon receiving a new, the server-dictated, ZMQ_SUBSCRIBE/ZMQ_UNSUBSCRIBE setting, the ZMQ_SUB client side code will simply handle that request and add zmq_setsockopt() accordingly.
FSA-driven grammars for this approach are rich of further possibilites, so will allow any Game Server / Game Community to smoothly go this way.
What pattern or socket I should use?
ZeroMQ is rather a library of LEGO-style elements to get assembled into a bigger picture.
Expecting such a smart library to have a one-size-fits-all ninja-element is on a closer look an oxymoron.
So, to avoid a "Never-ending-story" of adding "although this ... and also that ..."
Review all requirements and & list features for the end-to-end scaleable solution,
Design a messaging concept & validate it to meet all the listed requirements & cover all features in [1]
Implement [2]
Test [3] & correct it for meeting 1:1 the end-to-end specification [1]
Enjoy it. You have done it end-to-end right.
Suppose I have an application that uses actors for processing User. So there is one UserActor per user. Also every user Actor is mapped to user via id, e.g. to process actions with concrete user you should get Actor like that:
ActorSelection actor = actorSystem.actorSelection("/user/1");
where 1 is user id.
So the problem is - how generate unique id inside cluster effectively? First it needs to check that new id will not duplicate an existent one. I can create one actor for generating id's which will live in one node, and before creating any new UserActor Generator is asked for id, but this leads to additional request inside cluster whenever user is created. Is there a way to do this more effective? Are there build-in akka techniques to do that?
P.S. May this architecture for using Actor is not effective any suggestion/best practice is welcome.
I won't say whether or not your approach is a good idea. That's going to be up to you to decide. If I do understand your problem correctly though, then I can suggest a high level approach to making it work for you. If I understand correctly, you have a cluster, and for any given userId, there should be an actor in the system that handles requests for it, and it should only be on one node and consistently reachable based on the user id of the user. If that's correct, then consider the following approach.
Let's start first with a simple actor, let's call it UserRequestForwarder. This actors job is to find an actor instance for a request for a particular user id and forward on to it. If that actor instance does not yet exist, then this actor will create it before forwarding onto it. A very rough sketch could look like this:
class UserRequestForwarder extends Actor{
def receive = {
case req # DoSomethingForUser(userId) =>
val childName = s"user-request-handler-$userId"
val child = context.child(childName).getOrElse(context.actorOf(Props[UserRequestHandler]))
child forward req
}
}
Now this actor would be deployed onto every node in the cluster via a ConsistentHashingPool router configured in such a way that there would be one instance per node. You just need to make sure that there is something in every request that needs to travel through this router that allows it to be consistently hashed to the node that handles requests for that user (hopefully using the user id)
So if you pass all requests through this router, they will always land on the node that is responsible for that user, ending up in the UserRequestForwarder which will then find the correct user actor on that node and pass the request on to it.
I have not tried this approach myself, but it might work for what you are trying to do provided I understood your problem correctly.
Not an akka expert, so I can't offer code, but shouldn't the following approach work:
Have a single actor being responsible for creating the actors. And have it keep a Hashset of actor names, for actors that it created, and that didn't die already.
If you have to spread the load between multiple actors you can dispatch the task based on the first n digits of the hashcode of the actor name that has to be created.
It seems like you have your answer on how to generate the unique ID. In terms of your larger question, this is what Akka cluster sharding is designed to solve. It will handle distributing shards among your cluster, finding or starting your actors within the cluster and even rebalancing.
http://doc.akka.io/docs/akka/2.3.5/contrib/cluster-sharding.html
There's also an activator with a really nice example.
http://typesafe.com/activator/template/akka-cluster-sharding-scala
I know, that if I use a consistent hashing group router, it will always rout to the same registered routees.
So I wrote my application, with a few routees on there own routee-nodes and a public-node with a router, which is reachable by the client.
If the client sends a message it is routed as it should be and it works fine.
Now what I want to do is, add more public-nodes with thier own router that provides the same sending/routing behavior as every other public-node.
What I mean is, that it should not matter if a client sends message XYZ to public-node A, B or C. It should always go to the same routee-node.
At first I thought that akka may provides this behavior by default, because:
I used a group and not a pool router, so everyone knows the same routees
I leared that cluster nodes could be ordered
So I assumed that the routees list is ordered and the different routers map the same keys to the same routees. But testing showed me that I was wrong.
So, is there way in akka to achieve this behavior? Thanks.
As I expected, this behavior shoud be the standart for consistent hashing group routers and it is a bug in the akka-cluster package (current version 2.3.0-RC1)
See this tickt and this google-group post for more details.