I've recently been playing around with Spring Webflux and it looks extremely useful and efficient. Also, reading about Reactive Systems, it seems like one of the defining traits of such systems is that they are message-driven.
Came across this post on the web: https://www.captechconsulting.com/blogs/annotation-driven-reactive-web-apis-with-spring-webflux
This post also mentions,
Spring WebFlux contains support for Reactive HTTP Rest API(s),
WebSocket applications, and Server-Sent Events. Spring WebFlux is
responsive, resilient, scalable, and message-driven.
My question is that if a write a simple REST API, much like the post describes, performing CRUD operations backed by a MongoDB and using spring-boot-starter-data-mongodb-reactive, could I call my API service message-driven? I could also potentially add a Webclient to talk to some downstream services.
Does message driven in the context of a REST API even make sense?
No, your application is not message-driven instead your application are Reactive. Reactive applications is event-driven, non-blocking, scalable, resilient and elastic. It supports Publisher and Subscriber mechanism, means asynchronous communication is being done between Publisher and Subscriber. It supports two types of Publishers
Mono: Used when we produce only one item.
Flux: Used when we produce multiple items.
To make your application message-driven, you need to use any message broker like Kafka, RabbitMQ etc.
Related
I am unsure how to make use of event-driven architecture in real-world scenarios. Let's say there is a route planning platform consisting of the following back-end services:
user-service (manages user data and roles)
map-data-service (roads & addresses, only modified by admins)
planning-tasks-service
(accepts new route planning tasks, keeps track of background tasks, stores results)
The public website will usually request data from all 3 of those services. map-data-service needs information about user-roles on a data change request. planning-tasks-service needs information about users, as well as about map-data to validate new tasks.
Right now those services would just make a sync request to each other to get the needed data. What would be the best way to translate this basic structure into an event-driven architecture? Can dependencies be reduced by making use of events? How will the public website get the needed data?
Cosmin is 100% correct in that you need something to do some orchestration.
One approach to take, if you have a client that needs data from multiple services, is the Experience API approach.
Clients call the experience API, which performs the orchestration - pulling data from different sources and providing it back to the client. The design of the experience API is heavily, and deliberately, biased towards what the client needs.
Based on the details you've said so far, I can't see anything that cries out for event-based architecture. The communication between the client and ExpAPI can be a mix of sync and async, as can the ExpAPI to [Services] communication.
And for what it's worth, putting all of that on API gateway is not a bad idea, in that they are designed to host API's and therefore provide the desirable controls and observability for managing them.
Update based on OP Comment
I was really interested in how an event-driven architecture could
reduce dependencies between my microservices, as it is often stated
Having components (or systems) talk via events is sort-of the asynchronous equivalent of Inversion of Control, in that the event consumers are not tightly-coupled to the thing that emits the events. That's how the dependencies are reduced.
One thing you could do would be to do a little side-project just as a learning exercise - take a snapshot of your code and do a rough-n-ready conversion to event-based and just see how that went - not so much as an attempt to event-a-cise your solution but to see what putting events into a real-world solution looks like. If you have the time, of course.
The missing piece in your architecture is the API Gateway, which should be the only entry-point in your system, used by the public website directly.
The API Gateway would play the role of an orchestrator, which decides to which services to route the request, and also it assembles the final response needed by the frontend.
For scalability purposes, the communication between the API Gateway and individual microservices should be done asynchronously through an event-bus (or message queue).
However, the most important step in creating a scalable event-driven architecture which leverages microservices, is to properly define the bounded contexts of your system and understand the boundaries of each functionality.
More details about this architecture can be found here
Event storming is the first thing you need to do to identify domain events(a change in state in your system). For example, 'userCreated', 'userModified', 'locatinCreated', 'routeCreated', 'routeCompleted' etc. Then you can define topics that manage these events. Interested parties can consume these events by subscribing to published events(via topics/channel) and then act accordingly. Implementation of an event-driven architecture is often composed of loosely coupled microservices that communicate asynchronously through a message broker like Apache Kafka. Free EDA book is an excellent resource to know most of the things in EDA.
Tutorial: Even-driven-architecture pattern
In an event-driven microservices system, is it usual or a best practice that the microservices are also restful APIs ?
In an event driven microservices, it is often described as some events being raised and some other microservices to respond to those events and do some actions. In this case, it seems like there is no concept of "resource" as in restful API. If a system is built using restful APIs, can this system be called as a microservice system ?
In the context of event-driven microservices, is the concept of restful still apply? I found myself a bit mixed up on these two as I start to learn more about event-driven micro and not sure if I have grasped the concept correct.
In an event-driven architecture, microservices don't communicate through REST APIs, but via a message passing framework (RabbitMQ, Kafka, etc.). As you correctly pointed out, microservices react to a message received on a particular topic/channel they listen to.
If the microservice is down, the message accumulates in the message bus and it gets processed either by the same microservice (when it gets back up) or by another one.
If they were to communicate through REST APIs and a microservice is down, you need to retry the request with an exponential backoff policy and take care of many other things. This article explains the difference really well.
A microservice system can be entirely built with REST microservices. It just doesn't follow the event-driven approach, but more like a synchronous (request/response) model. The design of your microservice system should be directly correlated to the requirements of your application.
Usually, a hybrid approach is used: you communicate with some microservices through REST APIs (for example with autentication/autorization microservices), because you need the response from them ASAP. With other ones, you can communicate through events, usually when you have fire-and-forget events, like logging, metrics, maybe storage.
Event Driven and Restful API are 2 different concepts. Restful API is mostly used synchronous communication and event driven is asynchronous mode of communication. A microservice can be event driven and also can support Restful APIs but both serve different prospective.
Context
I am looking for postgres jdbc drivers which supports reactive programming. I came across https://r2dbc.io/ which is a spec for reactive api's for jdbc.There are two sections in the site
one is "Clients" and another is "Drivers"
The client section starts with
R2DBC encourages libraries to provide a “humane” API in the form of a client library. R2DBC avoids implementing user-space features in each driver, and leaves these for specific clients to implement.
Postrgres implementation of r2dbc - https://github.com/pgjdbc/r2dbc-postgresql starts with
This implementation is not intended to be used directly, but rather to be used as the backing implementation for a humane client library to delegate to
My Questions
What is the difference between client and driver in general, or at-least in above context ?
What is the "humane api" being referred here ?
An example of client and human api in Spring is the DatabaseClient in Spring 5.3.
The original R2dbc spec defines the APIs using reactive streams spec. But DatabaseClient is based on the project reactor, which provides richer APIs for developers.
Compare my example connection factories(I have to use Reactor APIs to wrap the R2dbc APIs to make it more easy for use) and database clients.
OpenAPI is good for RESTful services and at the moment, I'm hacking it to do it for asynchronous messaging system (specifically Kafka) by using POST to a /topic so that I can use redoc do create a website for the API.
I am trying to see if there's already established system of documenting for this. Especially since the GET /events which is used for event sourcing is getting larger and larger by the day.
It seems asyncAPI is basically what you are looking for: openapi but for topics instead of REST endpoints.
https://www.asyncapi.com/docs/getting-started/coming-from-openapi/
CloudEvents is a CNCF backed project for documenting event sourcing, one specification is for Kafka
https://github.com/cloudevents/sdk-java/blob/master/kafka/README.md
If you want a REST API, look at the Kafka REST Proxy
Consider using Protocol Buffers within Kafka.
https://developers.google.com/protocol-buffers/
Protocol Buffers require an API contract (".proto" file) if you want to call or implement a service. The contracts are both human and machine readable.
Protocol Buffers can also be used with other messaging systems and other protocols like HTTP (check out "gRPC" for that). So your documentation / contract is more portable.
Of course this only works for projects having the flexibility to change their payload format.
My team is torn with how we should architect our microservices with using a message bus.
We currently have an API Gateway with many microservices behind it all communicating over http.
After looking into implementing Message Buses (Kafka) the team is torn on whether the consumer and API should live in the same service or if they should be two separate services.
Some think they should be separate as they have different scaling concerns, while others think they should be in the same service since they are communicating with the same database and have the same domain concerns. IE) Not duplicating code between two services.
What are your thoughts?
We prefer them to be in the same service as they logically are doing work on same objects.
This also highly depends upon how you write your business logic. Like we prefer here to write our business logic in Aggregates(Domain Driven Design) and that's why writing consumer in the same service makes sense.
In some case where they are just updating data for searching kind of things , you may write them in separate service.
You can also look at Lagom (microservices framework for java)