I want to use the gRPC streaming mechanism for the clients to get notified when the system has changed.
E.g. The db stores users. Clients can add and delete users via gRPC unary calls. There are also streaming methods such that the clients can get notified when other client has added or deleted a user.
In case that I have several intances of my gRPC service (e.g. in k8s) how can Client1 that has a durable connection to instance1 gets notified when a client2 that makes a unary delete call to an instance 3?
You need a way to publish events between all instances. One way is to use your database. Or use a messaging solution that supports publish/subscribe. A lightweight solution could be redis.
Related
I'm trying to develop a project with microservices.
I have some questions on this topic (something is not clear):
1) How to implement microservices communication?
A) HTTP : Every microservice expose HTTP API , an API GATEWAY broadcast requests.
B) MQTT : every microservice pub/sub to a broker
C) BOTH : but how to understand when one is better than the other ?
Have I to use pub/sub protocol as a standard even for classic operations usually performed over HTTP ? For example I have two microservices:
web-management and product-service. web-management is a panel that lets the administrator to add, modify, ... products in its ecommerce digital shop. Let's say we want to implement createProduct operation. It's a command (according to event /command distinction), a one-to-one communication.
I can open an API in product-service, let's say (POST, "/product") that add the new product. I also can implement this transforming the command in a productCreationRequest event. In this case: web-managemnet publish this event. product-service listen to productCreationRequest events (and also productUpdateRequest, productGetEvents, ...) once it is notified it performs the operation and emits productCreated event.
I find this case borderline. For example a last-occasion-service may listen to productCreated and immediately send a message (email or push notification) to customers. What do you think about this use case?
2) Which may be a valid broker (I will use docker-compose or kubernetes to orchestrate containerized microservices: language adopted probably java, javascript, python)?
Both is definitely a possibility! Choose a broker that allows you to easily mix-and-match between HTTP (synchronous) communication, and more async event-driven pub/sub. It should allow you to migrate your microservices between the two options as required.
HTTP APIs are great at the edge of your distributed application, where a customer wants to submit an order or something, and block waiting for a response (200 OK).
But internally within your application between microservices, a lot of them don't need a response... async, eventually consistent. And using pub/sub (like MQTT) allows for multiple downstream consumers easily. Another great use for MQTT is streaming updates to downstream consumers... like a data-feed from a bus or airline company or something, rather than having to poll a REST API for updates.
For your use-case and similar ones, I would almost always recommend using pub/sub communication, even if today it's a simple request-reply interaction with a single backend process. REST over HTTP is point-to-point, and perhaps in the future you want another process to be able to see/consume/monitor that event or interaction. If you're already using publish-subscribe, adding that 2nd (or more) consumer of that data flow is trivial. Harder with REST/HTTP.
In terms of performance, I would highly doubt a blocking protocol like HTTP is going to outperform something that is asynchronous and bidirectional, like MQTT which uses WebSockets for web communication.
As for a broker to glue all this together, check out the standard edition Solace PubSub+ event broker... can do both (and translate between) MQTT and HTTP. I even wrote a CodeLab for this (almost) exact use case haha!
(BTW, I work for Solace! FYI.)
Consider using SMF framework for Javascript/Node.js, it helps prototype pub/sub communications via a message broker (RabbitMQ) between microservices out of the box:
https://medium.com/#krawa76/bootstrap-node-js-microservice-stack-4a348db38e51
As for the message broker routes, use an event-driven naming convention, e.g. post a "web.new-product", where "web" is the sub-system name, "new-product" - event name.
I have a service exposing a REST endpoint that, after a couple of transformations, calls a third-party service also via its REST endpoint.
I would like to implement some sort of throttling on my service to avoid being throttled by this third-party service. Note that my service's endpoint accepts only one request and not a list of them. I'm using Play and we also have Akka Streams as dependency.
My first though was to have my service saving the requests into a database table and then have an Akka Streams Source, leveraging the throttle function, picking tasks, applying the transformations and then calling the external service.
Is this a reasanoble approach or does it have any severe drawbacks?
Thanks!
Why save the requests to the database? Does the queue need to survive restarts and/or do you run a load-balanced setup that needs to somehow synchronize the requests?
If you don't need the above I'd think using only Source.queue to store the task data would work just as well?
And maybe you already thought of this: If you want to make your endpoint more resilient you should allow your API to send a 'sorry, busy' response and drop the request instead of queuing it if your queue grows beyond a certain size.
I am dealing with communication between microservices.
For example (fictive example, just for the illustration):
Microservice A - Store Users (getUser, etc.)
Microservice B - Store Orders (createOrder, etc.)
Now if I want to add new Order from the Client app, I need to know user address. So the request would be like this:
Client -> Microservice B (createOrder for userId 5) -> Microservice A (getUser with id 5)
The microservice B will create order with details (address) from the User Microservice.
PROBLEM TO SOLVE: How effectively deal with communication between microservice A and microservice B, as we have to wait until the response come back?
OPTIONS:
Use RestAPI,
Use AMQP, like RabbitMQ and deal with this issue via RPC. (https://www.rabbitmq.com/tutorials/tutorial-six-dotnet.html)
I don't know what will be better for the performance. Is call faster via RabbitMQ, or RestAPI? What is the best solution for microservice architecture?
In your case using direct REST calls should be fine.
Option 1 Use Rest API :
When you need synchronous communication. For example, your case. This option is suitable.
Option 2 Use AMQP :
When you need asynchronous communication. For example when your order service creates order you may want to notify product service to reduce the product quantity. Or you may want to nofity user service that order for user is successfully placed.
I highly recommend having a look at http://microservices.io/patterns/index.html
It all depends on your service's communication behaviour to choose between REST APIs and Event-Based design Or Both.
What you do is based on your requirement you can choose REST APIs where you see synchronous behaviour between services
and go with Event based design where you find services needs asynchronous behaviour, there is no harm combining both also.
Ideally for inter-process communication protocol it is better to go with messaging and for client-service REST APIs are best fitted.
Check the Communication style in microservices.io
REST based Architecture
Advantage
Request/Response is easy and best fitted when you need synchronous environments.
Simpler system since there in no intermediate broker
Promotes orchestration i.e Service can take action based on response of other service.
Drawback
Services needs to discover locations of service instances.
One to one Mapping between services.
Rest used HTTP which is general purpose protocol built on top of TCP/IP which adds enormous amount of overhead when using it to pass messages.
Event Driven Architecture
Advantage
Event-driven architectures are appealing to API developers because they function very well in asynchronous environments.
Loose coupling since it decouples services as on a event of once service multiple services can take action based on application requirement. it is easy to plug-in any new consumer to producer.
Improved availability since the message broker buffers messages until the consumer is able to process them.
Drawback
Additional complexity of message broker, which must be highly available
Debugging an event request is not that easy.
Personally I am not a fan of using a message broker for RPC. It adds unnecessary complexity and overhead.
How do you host your long-lived RabbitMQ consumer in your Users web service? If you make it some static singleton, in your web service how do you deal with scaling and concurrency? Or do you make it a stand-alone daemon process? Now you have two User applications instead of one. What happens if your Users consumer slows down, by the time it consumes the request message the Orders service context might have timed-out and sent another message or given up.
For RPC I would suggest simple HTTP.
There is a pattern involving a message broker that can avoid the need for a synchronous network call. The pattern is for services to consume events from other services and store that data locally in their own database. Then when the time comes when the Orders service needs a user record it can access it from its own database.
In your case, your Users app doesn't need to know anything about orders, but your Orders app needs to know some details about your users. So every time a user is added, modified, removed etc, the Users service emits an event (UserCreated, UserModified, UserRemoved). The Orders service can subscribe to those events and store only the data it needs, such as the user address.
The benefit is that is that at request time, your Orders service has one less synchronous dependency on another service. Testing the service is easier as you have fewer request time dependencies. There are also drawbacks however such as some latency between user record changes occuring and being received by the Orders app. Something to consider.
UPDATE
If you do go with RabbitMQ for RPC then remember to make use of the message TTL feature. If the client will timeout, then set the message expiration to that period. This will help avoid wasted work on the part of the consumer and avoid a queue getting backed up under load. One issue with RPC over a message broker is that once a queue fills up it can add long latencies that take a while to recover from. Setting your message expiration to your client timeout helps avoid that.
Regarding RabbitMQ for RPC. Normally we use a message broker for decoupling and durability. Seeing as RPC is a synchronous communication, that is, we are waiting for a response, then durability is not a consideration. That leaves us decoupling. The question is does that decoupling buy you anything over the decoupling you can do with HTTP via a gateway or Docker service names?
I'm willing to use ejabberd / mongooseIm in a microservice network. XMPP should be our chat protocol aside from a REST API network. I want to send messages incoming at the xmpp server downstream to worker services. Has anybody done this or could lead me into the right direction?
My first thoughts are using RabbitMQ for sending the new incoming messages to the workers.
There are basically two choices to giving your workers access to the messages routed by ejabberd / MongooseIM. I'll focus on MongooseIM, since I know it better (DISCLAIMER: I'm in the dev team).
The first is to scan the message archive in an async / polling fashion. The Message Archive Management describes XMPP level protocol for accessing it, but for your use case the important part is message persistence - so just making sure the relevant module (mod_mam) is enabled in server config and the messages will hit the database. The databases supported for MAM are PostgreSQL and Riak, though there was also some work on a Cassandra backend (YMMV). This doesn't require tinkering with the server / in Erlang for as long as there's a DB driver for your language of choice available. Since PR#657 it's possible to store the messages in raw XML or even some custom format if you're willing to write the serialization module.
The second option is to use the server mechanism of hooks and handlers (also available in ejabberd), which can trigger a server action on events like "user sent a message", "user logged in", "user logged out", ... This, however, requires a server side extension written in Erlang. In the simplest case the extension could forward any interesting event (with message content and metadata) via AMQP or just call some external HTTP/REST API - that way the real work is carried out by the workers giving you the freedom with regard to implementation language. This options also doesn't require to enable mod_mam or set up a database for message persistency (which you could still have with a persistent message queue...).
In general, the idea is perfectly feasible.
Generally, the most common XMPP extension use to build messaging systems for machines-to-machines, internet of things, microservices, etc is PubSub, as defined in XEP-0060.
This is a module you can enable in ejabberd. It is API based, so you can even customize the behaviour of that module to your application specific.
Pubsub basically allows to decouple senders and receivers and is especially designed for that use case.
I am working on an application that uses Amazon Kinesis, and one of the things I was wondering about is how you can roll over an application during an upgrade without data loss on streams. I have heard about things like blue/green deployments and such, but I was wondering what is the best practice for upgrading a data streaming service so you don't loose data from your streams.
For example, my application has an HTTP endpoint that ingests data as a series of POST operations. If I want to replace the service with a newer version, how do I manage existing application streaming to my endpoint?
One common method is having a software load balancer (LB) with a virtual IP; behind this LB there would be at least two HTTP ingestion endpoints during normal operation. During upgrade, each endpoint is announced out and upgraded in turn. The LB ensures that no traffic is forwarded to an announced out endpoint.
(The endpoints themselves can be on separate VMs, Docker containers or physical nodes).
Of course, the stream needs to be finite; the TCP socket/HTTP stream is owned by one of the endpoints. However, as long as the stream can be stopped gracefully, the following flow works, assuming endpoint A owns the current ingestion:
Tell endpoint A not to accept new streams. All new streams will be redirected only to endpoint B by the LB.
Gracefully stop existing streams on endpoint A.
Upgrade A.
Announce A back in.
Rinse and repeat with endpoint B.
As a side point, you would need two endpoints with a load balanced (or master/slave) set-up if you require any reasonable uptime and reliability guarantees.
There are more bespoke methods which allow hot code swap on the same endpoint, but they are more bespoke and rely on specific internal design (e.g. separate process between networking and processing stack connected by IPC).