I'm trying to develop a project with microservices.
I have some questions on this topic (something is not clear):
1) How to implement microservices communication?
A) HTTP : Every microservice expose HTTP API , an API GATEWAY broadcast requests.
B) MQTT : every microservice pub/sub to a broker
C) BOTH : but how to understand when one is better than the other ?
Have I to use pub/sub protocol as a standard even for classic operations usually performed over HTTP ? For example I have two microservices:
web-management and product-service. web-management is a panel that lets the administrator to add, modify, ... products in its ecommerce digital shop. Let's say we want to implement createProduct operation. It's a command (according to event /command distinction), a one-to-one communication.
I can open an API in product-service, let's say (POST, "/product") that add the new product. I also can implement this transforming the command in a productCreationRequest event. In this case: web-managemnet publish this event. product-service listen to productCreationRequest events (and also productUpdateRequest, productGetEvents, ...) once it is notified it performs the operation and emits productCreated event.
I find this case borderline. For example a last-occasion-service may listen to productCreated and immediately send a message (email or push notification) to customers. What do you think about this use case?
2) Which may be a valid broker (I will use docker-compose or kubernetes to orchestrate containerized microservices: language adopted probably java, javascript, python)?
Both is definitely a possibility! Choose a broker that allows you to easily mix-and-match between HTTP (synchronous) communication, and more async event-driven pub/sub. It should allow you to migrate your microservices between the two options as required.
HTTP APIs are great at the edge of your distributed application, where a customer wants to submit an order or something, and block waiting for a response (200 OK).
But internally within your application between microservices, a lot of them don't need a response... async, eventually consistent. And using pub/sub (like MQTT) allows for multiple downstream consumers easily. Another great use for MQTT is streaming updates to downstream consumers... like a data-feed from a bus or airline company or something, rather than having to poll a REST API for updates.
For your use-case and similar ones, I would almost always recommend using pub/sub communication, even if today it's a simple request-reply interaction with a single backend process. REST over HTTP is point-to-point, and perhaps in the future you want another process to be able to see/consume/monitor that event or interaction. If you're already using publish-subscribe, adding that 2nd (or more) consumer of that data flow is trivial. Harder with REST/HTTP.
In terms of performance, I would highly doubt a blocking protocol like HTTP is going to outperform something that is asynchronous and bidirectional, like MQTT which uses WebSockets for web communication.
As for a broker to glue all this together, check out the standard edition Solace PubSub+ event broker... can do both (and translate between) MQTT and HTTP. I even wrote a CodeLab for this (almost) exact use case haha!
(BTW, I work for Solace! FYI.)
Consider using SMF framework for Javascript/Node.js, it helps prototype pub/sub communications via a message broker (RabbitMQ) between microservices out of the box:
https://medium.com/#krawa76/bootstrap-node-js-microservice-stack-4a348db38e51
As for the message broker routes, use an event-driven naming convention, e.g. post a "web.new-product", where "web" is the sub-system name, "new-product" - event name.
Related
I need to integrate 3-5 existing and ready services that are developed by different teams. That is something like integrating several independent monolithic applications.
The very wanted feature is having a central communication component where all requests can be logged (or partially logged in case they have a big payload) so that it can be quickly seen what service sent a request with what payload and when if something goes wrong.
The second task is security. It is needed to protect inter-service communication.
I have been researching this topic for several days. That is what I've come up with so far:
Use Enterprise Service Bus (ESB)
Use MessageBroker (ActiveMQ, RabbitMQ, Kafka, etc)
Simply use REST API communication
I've read about ESB and I am not sure whether this solution is OK to use.
The picture from Wikipedia shows the following:
The problem with ESB that it not only implements communication between separate independent services. It also changes the communication itself from synchronous request-response model to an asynchronous messaging style. Currently, we don't need asynchronous messaging (maybe we will in the future, but not right now).
ESB allows to make request-response communication, but in a very inconvenient and complex way with generating correlation IDs, creating a temporary response-topic and consumer. With this, I have doubts whether it has more advantages to use ESB with its super complex Request-Response messaging style or simply use plain old REST API calls (RPC). However, with REST API it is not possible to have a centralized component that can log all communication between services. A similar problem exists with Message Broker too (because it also involves asynchronous messaging).
Is there any ready solution/pattern to integrate several services (not microservices) with centralized logging, security configuration and having simple to implement synchronous request-response model (with a possibility to add messaging, if needed, later)?
Make your services talk to each other via a proxy that will take care of these concerns (Back in 2007 I called this "edge-component" but the today it known as sidecar pattern)
A common way to do that would be to containerize your services, deploy to Kubernetes and use a service mesh (like Console's or istio, etc)
I know that messaging system is non blocking and scalable and should be used in microservices environment.
The use case that i am questioning is:
Imagine that there's an admin dashboard client responsible for sending API request to create an Item object. There is a microservice that provides API endpoint which uses a MySQL database where the Item should be stored. There is another microservice which uses elastic search for text searching purposes.
Should this admin dashboard client :
A. Send 2 API Calls; 1 Call to MySQL service and another elasticsearch service
or
B. Send message to topic to be consumed by both MySQL service and elasticsearch service?
What are the pros and cons when considering A or B?
I'm thinking that it's a little overkill when only 2 microservices are consuming this topic. Also, the frequency of which the admin is creating Item object is very small.
Like many things in software architecture, it depends. Your requirements, SLAs and business needs should make it clearer.
As you noted, messaging system is not blocking and much more scalable, but, API communication got it pluses as well.
In general, REST APIs are best suited to request/response interactions where the client application sends a request to the API backend over HTTP.
Message streaming is best suited for notifications when new data or events occur that you may want to take action upon.
In you specific case, I would go with a messaging system with is much more scalable and non-blocking.
Your A approach is coupling the "routing" logic into your application. Pretend you need to perform an API call to audit your requests, then you will need to change the code and add another call to your application logic. As you said, the approach is synchronous and unless you're not providing threading logic, your calls will be lined up and won't scale, ie, call mysql --> wait response, then call elastic search --> wait response, ...
In any case you can prefer this approach if you need immediate consistency, ie, the result call of one action feeding the second action.
The B approach is decoupling that routing logic, so, any other service interested in the event can subscribe to the topic and perform the action expected. Totally asynchronous and scalable. Here you will have eventual consistency and you have to recover any possible failure.
I am dealing with communication between microservices.
For example (fictive example, just for the illustration):
Microservice A - Store Users (getUser, etc.)
Microservice B - Store Orders (createOrder, etc.)
Now if I want to add new Order from the Client app, I need to know user address. So the request would be like this:
Client -> Microservice B (createOrder for userId 5) -> Microservice A (getUser with id 5)
The microservice B will create order with details (address) from the User Microservice.
PROBLEM TO SOLVE: How effectively deal with communication between microservice A and microservice B, as we have to wait until the response come back?
OPTIONS:
Use RestAPI,
Use AMQP, like RabbitMQ and deal with this issue via RPC. (https://www.rabbitmq.com/tutorials/tutorial-six-dotnet.html)
I don't know what will be better for the performance. Is call faster via RabbitMQ, or RestAPI? What is the best solution for microservice architecture?
In your case using direct REST calls should be fine.
Option 1 Use Rest API :
When you need synchronous communication. For example, your case. This option is suitable.
Option 2 Use AMQP :
When you need asynchronous communication. For example when your order service creates order you may want to notify product service to reduce the product quantity. Or you may want to nofity user service that order for user is successfully placed.
I highly recommend having a look at http://microservices.io/patterns/index.html
It all depends on your service's communication behaviour to choose between REST APIs and Event-Based design Or Both.
What you do is based on your requirement you can choose REST APIs where you see synchronous behaviour between services
and go with Event based design where you find services needs asynchronous behaviour, there is no harm combining both also.
Ideally for inter-process communication protocol it is better to go with messaging and for client-service REST APIs are best fitted.
Check the Communication style in microservices.io
REST based Architecture
Advantage
Request/Response is easy and best fitted when you need synchronous environments.
Simpler system since there in no intermediate broker
Promotes orchestration i.e Service can take action based on response of other service.
Drawback
Services needs to discover locations of service instances.
One to one Mapping between services.
Rest used HTTP which is general purpose protocol built on top of TCP/IP which adds enormous amount of overhead when using it to pass messages.
Event Driven Architecture
Advantage
Event-driven architectures are appealing to API developers because they function very well in asynchronous environments.
Loose coupling since it decouples services as on a event of once service multiple services can take action based on application requirement. it is easy to plug-in any new consumer to producer.
Improved availability since the message broker buffers messages until the consumer is able to process them.
Drawback
Additional complexity of message broker, which must be highly available
Debugging an event request is not that easy.
Personally I am not a fan of using a message broker for RPC. It adds unnecessary complexity and overhead.
How do you host your long-lived RabbitMQ consumer in your Users web service? If you make it some static singleton, in your web service how do you deal with scaling and concurrency? Or do you make it a stand-alone daemon process? Now you have two User applications instead of one. What happens if your Users consumer slows down, by the time it consumes the request message the Orders service context might have timed-out and sent another message or given up.
For RPC I would suggest simple HTTP.
There is a pattern involving a message broker that can avoid the need for a synchronous network call. The pattern is for services to consume events from other services and store that data locally in their own database. Then when the time comes when the Orders service needs a user record it can access it from its own database.
In your case, your Users app doesn't need to know anything about orders, but your Orders app needs to know some details about your users. So every time a user is added, modified, removed etc, the Users service emits an event (UserCreated, UserModified, UserRemoved). The Orders service can subscribe to those events and store only the data it needs, such as the user address.
The benefit is that is that at request time, your Orders service has one less synchronous dependency on another service. Testing the service is easier as you have fewer request time dependencies. There are also drawbacks however such as some latency between user record changes occuring and being received by the Orders app. Something to consider.
UPDATE
If you do go with RabbitMQ for RPC then remember to make use of the message TTL feature. If the client will timeout, then set the message expiration to that period. This will help avoid wasted work on the part of the consumer and avoid a queue getting backed up under load. One issue with RPC over a message broker is that once a queue fills up it can add long latencies that take a while to recover from. Setting your message expiration to your client timeout helps avoid that.
Regarding RabbitMQ for RPC. Normally we use a message broker for decoupling and durability. Seeing as RPC is a synchronous communication, that is, we are waiting for a response, then durability is not a consideration. That leaves us decoupling. The question is does that decoupling buy you anything over the decoupling you can do with HTTP via a gateway or Docker service names?
I heard Amazon uses HTTP for its microservice based architecture. An alternative is to use a messaging system like RabbitMQ or Solace systems. I personally have experience with Solace based microservice architecture, but never with REST.
Any idea what do various big league implementations like Amazon, Netflix, UK Gov etc use?
Other aspect is, in microservices, following things are required (besides others):
* Pattern matching
* Async messaging.. receiving system may be down
* Publish subscribe
* Cache load event.. i.e. on start up, a service may need to load all data from a couple of other services, and should be notified when data is completely loaded, so that it can 'know' that it is now ready to service requests
These aspects are naturally done with messaging rather than REST. Why should anyone use REST (except for public API). Thanks.
A standard that I've followed in the past is to use web services when the key requirement is speed (and data loss isn't critical) and messaging when the key requirement is reliability. Like you've said, if the receiving system is down, a message will sit on a queue until the system comes back up to process it. If it's a REST endpoint and it's down, requests will simply fail.
REST API presumes use of HTTP only. it is quite stone age technology and does not accept async. messaging. To plugin messaging there, I would consider WebSockets Gateways
-sorry for eventually dummy statements
A little background.
Very big monolithic Django application. All components use the same database. We need to separate services so we can independently upgrade some parts of the system without affecting the rest.
We use RabbitMQ as a broker to Celery.
Right now we have two options:
HTTP Services using a REST interface.
JSONRPC over AMQP to a event loop service
My team is leaning towards HTTP because that's what they are familiar with but I think the advantages of using RPC over AMQP far outweigh it.
AMQP provides us with the capabilities to easily add in load balancing, and high availability, with guaranteed message deliveries.
Whereas with HTTP we have to create client HTTP wrappers to work with the REST interfaces, we have to put in a load balancer and set up that infrastructure in order to have HA etc.
With AMQP I can just spawn another instance of the service, it will connect to the same queue as the other instances and bam, HA and load balancing.
Am I missing something with my thoughts on AMQP?
At first,
REST, RPC - architecture patterns, AMQP - wire-level and HTTP - application protocol which run on top of TCP/IP
AMQP is a specific protocol when HTTP - general-purpose protocol, thus, HTTP has damn high overhead comparing to AMQP
AMQP nature is asynchronous where HTTP nature is synchronous
both REST and RPC use data serialization, which format is up to you and it depends of infrastructure. If you are using python everywhere I think you can use python native serialization - pickle which should be faster than JSON or any other formats.
both HTTP+REST and AMQP+RPC can run in heterogeneous and/or distributed environment
So if you are choosing what to use: HTTP+REST or AMQP+RPC, the answer is really subject of infrastructure complexity and resource usage. Without any specific requirements both solution will work fine, but i would rather make some abstraction to be able switch between them transparently.
You told that your team familiar with HTTP but not with AMQP. If development time is an important time you got an answer.
If you want to build HA infrastructure with minimal complexity I guess AMQP protocol is what you want.
I had an experience with both of them and advantages of RESTful services are:
they well-mapped on web interface
people are familiar with them
easy to debug (due to general purpose of HTTP)
easy provide API to third-party services.
Advantages of AMQP-based solution:
damn fast
flexible
cost-effective (in resources usage meaning)
Note, that you can provide RESTful API to third-party services on top of your AMQP-based API while REST is not a protocol but rather paradigm, but you should think about it building your AQMP RPC api. I have done it in this way to provide API to external third-party services and provide access to API on those part of infrastructure which run on old codebase or where it is not possible to add AMQP support.
If I am right your question is about how to better organize communication between different parts of your software, not how to provide an API to end-users.
If you have a high-load project RabbitMQ is damn good piece of software and you can easily add any number of workers which run on different machines. Also it has mirroring and clustering out of the box. And one more thing, RabbitMQ is build on top of Erlang OTP, which is high-reliable,stable platform ... (bla-bla-bla), it is good not only for marketing but for engineers too. I had an issue with RabbitMQ only once when nginx logs took all disc space on the same partition where RabbitMQ run.
UPD (May 2018):
Saurabh Bhoomkar posted a link to the MQ vs. HTTP article written by Arnold Shoon on June 7th, 2012, here's a copy of it:
I was going through my old files and came across my notes on MQ and thought I’d share some reasons to use MQ vs. HTTP:
If your consumer processes at a fixed rate (i.e. can’t handle floods to the HTTP server [bursts]) then using MQ provides the flexibility for the service to buffer the other requests vs. bogging it down.
Time independent processing and messaging exchange patterns — if the thread is performing a fire-and-forget, then MQ is better suited for that pattern vs. HTTP.
Long-lived processes are better suited for MQ as you can send a request and have a seperate thread listening for responses (note WS-Addressing allows HTTP to process in this manner but requires both endpoints to support that capability).
Loose coupling where one process can continue to do work even if the other process is not available vs. HTTP having to retry.
Request prioritization where more important messages can jump to the front of the queue.
XA transactions – MQ is fully XA compliant – HTTP is not.
Fault tolerance – MQ messages survive server or network failures – HTTP does not.
MQ provides for ‘assured’ delivery of messages once and only once, http does not.
MQ provides the ability to do message segmentation and message grouping for large messages – HTTP does not have that ability as it treats each transaction seperately.
MQ provides a pub/sub interface where-as HTTP is point-to-point.
UPD (Dec 2018):
As noticed by #Kevin in comments below, it's questionable that RabbitMQ scales better then RESTful servies. My original answer was based on simply adding more workers, which is just a part of scaling and as long as single AMQP broker capacity not exceeded, it is true, though after that it requires more advanced techniques like Highly Available (Mirrored) Queues which makes both HTTP and AMQP-based services have some non-trivial complexity to scale at infrastructure level.
After careful thinking I also removed that maintaining AMQP broker (RabbitMQ) is simpler than any HTTP server: original answer was written in Jun 2013 and a lot of changed since that time, but the main change was that I get more insight in both of approaches, so the best I can say now that "your mileage may vary".
Also note, that comparing both HTTP and AMQP is apple to oranges to some extent, so please, do not interpret this answer as the ultimate guidance to base your decision on but rather take it as one of sources or as a reference for your further researches to find out what exact solution will match your particular case.
The irony of the solution OP had to accept is, AMQP or other MQ solutions are often used to insulate callers from the inherent unreliability of HTTP-only services -- to provide some level of timeout & retry logic and message persistence so the caller doesn't have to implement its own HTTP insulation code. A very thin HTTP gateway or adapter layer over a reliable AMQP core, with option to go straight to AMQP using a more reliable client protocol like JSONRPC would often be the best solution for this scenario.
Your thoughts on AMQP are spot on!
Furthermore, since you are transitioning from a monolithic to a more distributed architecture, then adopting AMQP for communication between the services is more ideal for your use case. Here is why…
Communication via a REST interface and by extension HTTP is synchronous in nature — this synchronous nature of HTTP makes it a not-so-great option as the pattern of communication in a distributed architecture like the one you talk about. Why?
Imagine you have two services, service A and service B in that your Django application that communicate via REST API calls. This API calls usually play out this way: service A makes an http request to service B, waits idly for the response, and only proceeds to the next task after getting a response from service B. In essence, service A is blocked until it receives a response from service B.
This is problematic because one of the goals with microservices is to build small autonomous services that would always be available even if one or more services are down– No single point of failure. The fact that service A connects directly to service B and in fact, waits for some response, introduces a level of coupling that detracts from the intended autonomy of each service.
AMQP on the other hand is asynchronous in nature — this asynchronous nature of AMQP makes it great for use in your scenario and other like it.
If you go down the AMQP route, instead of service A making requests to service B directly, you can introduce an AMQP based MQ between these two services. Service A will add requests to the Message Queue. Service B then picks up the request and processes it at its own pace.
This approach decouples the two services and, by extension, makes them autonomous. This is true because:
If service B fails unexpectedly, service A will keep accepting requests and adding them to the queue as though nothing happened. The requests would always be in the queue for service B to process them when it’s back online.
If service A experiences a spike in traffic, service B won’t even notice because it only picks up requests from the Message Queues at its own pace
This approach also has the added benefit of being easy to scale— you can add more queues or create copies of service B to process more requests.
Lastly, service A does not have to wait for a response from service B, the end users don’t also have to wait for long— this leads to improved performance and, by extension, a better user experience.
Just in case you are considering moving from HTTP to AMQP in your distributed architecture and you are just not sure how to go about it, you can checkout this 7 parts beginner guide on message queues and microservices. It shows you how to use a message queue in a distributed architecture by walking you through a demo project.