My team is torn with how we should architect our microservices with using a message bus.
We currently have an API Gateway with many microservices behind it all communicating over http.
After looking into implementing Message Buses (Kafka) the team is torn on whether the consumer and API should live in the same service or if they should be two separate services.
Some think they should be separate as they have different scaling concerns, while others think they should be in the same service since they are communicating with the same database and have the same domain concerns. IE) Not duplicating code between two services.
What are your thoughts?
We prefer them to be in the same service as they logically are doing work on same objects.
This also highly depends upon how you write your business logic. Like we prefer here to write our business logic in Aggregates(Domain Driven Design) and that's why writing consumer in the same service makes sense.
In some case where they are just updating data for searching kind of things , you may write them in separate service.
You can also look at Lagom (microservices framework for java)
Related
I am unsure how to make use of event-driven architecture in real-world scenarios. Let's say there is a route planning platform consisting of the following back-end services:
user-service (manages user data and roles)
map-data-service (roads & addresses, only modified by admins)
planning-tasks-service
(accepts new route planning tasks, keeps track of background tasks, stores results)
The public website will usually request data from all 3 of those services. map-data-service needs information about user-roles on a data change request. planning-tasks-service needs information about users, as well as about map-data to validate new tasks.
Right now those services would just make a sync request to each other to get the needed data. What would be the best way to translate this basic structure into an event-driven architecture? Can dependencies be reduced by making use of events? How will the public website get the needed data?
Cosmin is 100% correct in that you need something to do some orchestration.
One approach to take, if you have a client that needs data from multiple services, is the Experience API approach.
Clients call the experience API, which performs the orchestration - pulling data from different sources and providing it back to the client. The design of the experience API is heavily, and deliberately, biased towards what the client needs.
Based on the details you've said so far, I can't see anything that cries out for event-based architecture. The communication between the client and ExpAPI can be a mix of sync and async, as can the ExpAPI to [Services] communication.
And for what it's worth, putting all of that on API gateway is not a bad idea, in that they are designed to host API's and therefore provide the desirable controls and observability for managing them.
Update based on OP Comment
I was really interested in how an event-driven architecture could
reduce dependencies between my microservices, as it is often stated
Having components (or systems) talk via events is sort-of the asynchronous equivalent of Inversion of Control, in that the event consumers are not tightly-coupled to the thing that emits the events. That's how the dependencies are reduced.
One thing you could do would be to do a little side-project just as a learning exercise - take a snapshot of your code and do a rough-n-ready conversion to event-based and just see how that went - not so much as an attempt to event-a-cise your solution but to see what putting events into a real-world solution looks like. If you have the time, of course.
The missing piece in your architecture is the API Gateway, which should be the only entry-point in your system, used by the public website directly.
The API Gateway would play the role of an orchestrator, which decides to which services to route the request, and also it assembles the final response needed by the frontend.
For scalability purposes, the communication between the API Gateway and individual microservices should be done asynchronously through an event-bus (or message queue).
However, the most important step in creating a scalable event-driven architecture which leverages microservices, is to properly define the bounded contexts of your system and understand the boundaries of each functionality.
More details about this architecture can be found here
Event storming is the first thing you need to do to identify domain events(a change in state in your system). For example, 'userCreated', 'userModified', 'locatinCreated', 'routeCreated', 'routeCompleted' etc. Then you can define topics that manage these events. Interested parties can consume these events by subscribing to published events(via topics/channel) and then act accordingly. Implementation of an event-driven architecture is often composed of loosely coupled microservices that communicate asynchronously through a message broker like Apache Kafka. Free EDA book is an excellent resource to know most of the things in EDA.
Tutorial: Even-driven-architecture pattern
In an event-driven microservices system, is it usual or a best practice that the microservices are also restful APIs ?
In an event driven microservices, it is often described as some events being raised and some other microservices to respond to those events and do some actions. In this case, it seems like there is no concept of "resource" as in restful API. If a system is built using restful APIs, can this system be called as a microservice system ?
In the context of event-driven microservices, is the concept of restful still apply? I found myself a bit mixed up on these two as I start to learn more about event-driven micro and not sure if I have grasped the concept correct.
In an event-driven architecture, microservices don't communicate through REST APIs, but via a message passing framework (RabbitMQ, Kafka, etc.). As you correctly pointed out, microservices react to a message received on a particular topic/channel they listen to.
If the microservice is down, the message accumulates in the message bus and it gets processed either by the same microservice (when it gets back up) or by another one.
If they were to communicate through REST APIs and a microservice is down, you need to retry the request with an exponential backoff policy and take care of many other things. This article explains the difference really well.
A microservice system can be entirely built with REST microservices. It just doesn't follow the event-driven approach, but more like a synchronous (request/response) model. The design of your microservice system should be directly correlated to the requirements of your application.
Usually, a hybrid approach is used: you communicate with some microservices through REST APIs (for example with autentication/autorization microservices), because you need the response from them ASAP. With other ones, you can communicate through events, usually when you have fire-and-forget events, like logging, metrics, maybe storage.
Event Driven and Restful API are 2 different concepts. Restful API is mostly used synchronous communication and event driven is asynchronous mode of communication. A microservice can be event driven and also can support Restful APIs but both serve different prospective.
I am learning about microservices and I don't understand what the real difference
between creating a REST API and creating microservices?
I’m working in Go, but my question applies over all languages.
The Microservices approach is about breaking your system ("pile of code") into many small services, each typically has its own:
Clear business-related responsibility
Running process
Database
Code version control (e.g. git) repository
API (the protocol how other services / clients will contact the Microservice)
UI
The services themselves are kept small so as your system grow, there are more services - rather than larger services.
Microservices can use REST, RPC, or any other method to communicate with one another, so REST or an API is really orthogonal to the topic of microservices...
Reference: What is an API? In English, please.
API = Application Programming Interface
Microservices = an architecture
In short
Microservice should expose a well-defined API.
Microservice is the way you may want to architect your solution
API is what your consumers see.
You can expose API without microservices in the backend (in fact, most non-training scenarios don't require microservices).
You may want to read http://samnewman.io/books/building_microservices/ before you decide on using microservices (unless this is for training purposes).
Microservice is well defined when you are following SOC - seperation of Concern on the entity/domain level ,where each entity / domain are independent of any other service.
for example user service will only be responsible for storing, updating and deleting user related informations.
Microservice backend and frontend microservice can further be splitted in 2 parts
frontend microservice which exposes rest endpoint just like Web API
backend microservice which actually perform all the operations.
Rest API is more of endpoints exposed to outer world and can be used with microservices as well, as explained above.
The majority of the answers is based on the old-school understanding of API as a programmatic interface. Nowadays, this meaning is melted and start confusing people becuase some developers started (for simplicit or by mistake) interpred the API of an application as the application per se. In such case, it is impossible to distinguish between the modern API and Microservices. Nonetheless, we can say that an API-application can comprise many Microservices, the most of which interact within the application via Microservice's APIs while others may expose their APIs as Applications's APIs. Also, a Microservice (as a service) may not include other Microservices (services), but may orchestrate a composition of Microservices via API-bases invocations. Applications may contain Microservices but, in the best practices, may not contain other Applications.
Microservices
A microservice architecture is about slicing an application logic into small pieces or "components" that can act between them and/or be exposed through an API.
API
A (web) application need to design the business logic with all set of object entities (model) and possible operations on them.
An (Application Programming Interface][https://en.wikipedia.org/wiki/Application_programming_interface) is a way of making the requests to an application by exposing specific entry-points that are in charge of invoking the appropriate application operations.
ReST(ful) APIs
("ReST" as in Representational State Transfer) are APIs complying with at least these 5 constraints:
User-interface is distinct from data storage and manipulation (Client-Server architecture)
No client context is stored on the server ("stateless")
Server responses must, implicitly or explicitly, define themselves as cacheable or not
Client does not have to be aware of the layers between him and the server
Response/request messages must be: be self-descriptive; allow to identify a resource; use representations allowing to manipulate the resources; announce available actions and resources ("Uniform interface").
"The real difference"
So, while these notions are obviously related, they are clearly distinct concepts:
Being ReSTful or not, an API exposes operations provided by a server that might (but not necessarily) be shelled into smaller components (microservices).
Also, while a typical web (ReST)API uses the HTTP protocol between the client and the server, components within a microservice architecture might communicate using other protocol(s) (e.g. WAMP, AMQP, JSON-RPC, XML-RPC, SOAP, ...)
In layman's term, if you have a web API server and you split them into several independent mini servers, use a proxy-server and load-balancer to clusterize them, and (optionally, give each a separate database entity), that is a microservice architecture.
I have a microService architecture with 10 microServices and each of those provides a client. Inside of that client which is managed/controlled by microService team we just receive the parameters and pass them to a generic http invoker which receives the endpoint and N params and then does the call.
All microService use http and web api (I guess technology doesn't matter).
For me doesn't make sense to be the microService team to provide a client, should be the responsibility of the consumer, if they want to create some abstractions or invoke it directly is their problem, not a microService problem. And the way I see a web API is as a contract. So I think we should delete all clients (pass responsibility to consumers) on the microService side and create a service layer on the consumer's side that uses the generic invoker to reach the endpoints.
The image below represents all components where the red line defines the boundaries, who is responsible for what:
The gateway has Adapter Layer
Adapter Layer references the microService client package
MicroService client package references Generic HTTP invoker package
The other side of that is because we might have N number of consumers and they are all repeating the code of the client. And if the microService provides a client, we have a unique/central place to control that.
Which approach is correct? Is the client a responsability of the microService or the consumer?
This is an internal product.
I have a similar setup at work, with several microservices (~40) and a dozen teams. I was asked the same question several times, and my answer is the consumer is responsible for consuming. If the API works as designed and expected, there is no point in making the providing team responsible for anything.
The team that provides the service (team a), may provide a client, if they want (in doubt, without warranty). The consuming team (team B) may use the client if they want (taking all the risks included).
The only contract should be the API, everything else should be a goodie a team may provide on top. If team a has to provide a client, why do they provide an api at all then?
Given that both teams are loosely coupled and may use different technologies (or e.g. different spring framework versions), providing a client library to the other team proves to bring more problems than solve any. In a Java+spring-boot world e.g. you get into dependency problems very fast, especially if you include several clients from different service providing teams that evolve differently in time.
And even worse: what if the client-library of team A makes the system of team B unstable and introduces bugs? Who is responsible to fix that now?
If you want to reduce the work needed for your consuming teams because re-coding the client is so much work, your API may be to complex and/or your microservice may be no more microservice at all.
Imagine using HATEOAS on a restful API - writing a client for that is just a few lines of code, even with included API-Browser, Documentation and whatnot. See e.g. spring-rest-docs, hal-browser, swagger and various other technologies that make reading/browsing/documenting an API and implementing a client a breeze.
Above cases are described with two teams, imagine that with 10. We had a "client library" provided by one team, used by 4 other teams. You can guess how fast it became a complete mess until it was just deleted :)
I'm new to CQRS but I understand the intent behind it. From the few documents that I went through, I did not understand how can I deploy a CQRS application/service in a shared asp.net hosting environment (hosting provided by GoDaddy or DiscountAsp.net). Does the hosting server need to have MSMQ or similar message processing application to have CQRS working? Or it can work via asynchronous communication which is available in MVC4, WCF Service or Asp.net application.
Feedback appreciated as well as any links that talk about the deployment aspect of CQRS in a shared environment.
Does the hosting server need to have MSMQ or similar message
processing application to have CQRS working?
Generally yes, but it of course depends on your definition of CQRS. Some would for instance consider a single data store architecture only using distinct read and write models as CQRS. In this case there is no need for explicit synchronization of read and write stores. In the the usual sense however, synchronization needs between and read and write is required and is usually implemented with a messaging system of which MSMQ is an instance.
Does the hosting server need to have MSMQ or similar message
processing application to have CQRS working?
Async in ASP.NET MVC is used for processing a single request asynchronously, not for performing background tasks or passing messages between processes and/or nodes such as, say, write and read stores.
Some suggestions:
Consider hosting the infrastructure required for CQRS on a provider such as Azure, AWS, AppHarbor, iron.io, etc.
Consider using something like RavenHQ for the data store since RavenDB can effectively support CQRS out of the box. In both this and the previous case you can still use GoDaddy or the like to host the front facing ASP.NET app, if absolutely neccessary.