We are building a Spring Boot based web application which consists of a central API server and multiple remote workers. We tried to embrace the idea of microservices so each component is built as a separate Spring Boot App. But they do share a lot of same data schema for the objects being handled across components, thus currently the JPA model definitions in code are duplicated in each component's project. Every time something changes we need to remember to change it everywhere and this results in poor compatibility between different versions across components.
So I'm wondering if this is the best we can do here, or is there better ways to manage the components code in such scenario?
To be more specific, we use both MySQL and Redis for data storage and both are accessed by all components. Redis is actually serving as one tool of data communication between components.
Related
Knowing that drools work with in memory data. Is there a way to distribute horizontally on different drools instances to enhance performance when performing CRUD operations on rules, fact types, etc? I guess the instances would need to be on sync with each other in some way, so they all have the same data in memory or share in some way a knowledge base. I'm kinda new on drools and trying to research on a way to move a monolith on a cloud environment (gcp) so it can take advantage on load balancing, scaling, etc. Want to know if there is any feature on drools itself that supports this or if there is any way to implement this myself, thanks in advance for any information/documentation/use case on this matter.
Currently I haven't tried a way to do this, but my goal is to improve performance and availability by using automatic scaling or support multiple instances of my app.
I'm not sure what kind of "CRUD" you're doing on Drools (or how). But if you just want to deploy new rules (for example), then this is identical to pushing any data or application changes to your deployment in a distributed system -- either your nodes are gradually updated, so during the upgrade process you have some mix of old and new logic/code; or you deploy new instances with the new logic/code and then transition traffic to your new instances and away from the old ones -- either all at once or in a controlled blue/green (or similar) fashion.
If you want to split a monolith, I think the best approach for you would be to consider Kogito [1] and microservice architecture. With microservices, you could even consider using the Function as a service approach - having small immutable service instances, that are just executed and disposed. Kogito mainly targets Quarkus platform, but there are also some Spring Boot examples. There is also OpenShift operator available.
As far as sharing the working memory, there was a project in the KIE community called HACEP [2]. Unfortunately that is now deprecated and we are researching other solutions to make the working memory persisted.
[1] https://kogito.kie.org/
[2] https://github.com/kiegroup/openshift-drools-hacep
The term "entry point" is related to the fact that we have multiple partitions in a Working Memory and you can choose which one you are inserting into. If you can organize your business logic to work with different entry points you can process 'logical partitions' on different machines in parallel safely. At a glance drools entry points gives you something like table partitioning in Oracle which implies the same options.
Use load balancer with sticky sessions if you can (from business point of view) partition 'by client'
you question looks more like an architecture question.
As a start, I would have a look into the Kie Execution Server component provided with Drools that helps you to create microservice decisions based on Drools rulesets.
Kie Execution Server (used in stateless mode by clients) could be embedded in different pods/instances/servers to ensure horizontal scalability.
As mentioned by #RoddyoftheFrozenPeas , one of the problem you'll face will be the simultaneous hot deploy of new rulesets on the "swarm" of kieserver that hosts your services.
That would have to be handled using a proper devops strategy.
Best
Emmanuel
I am a beginner in Redis. I am interested to introduce Redis into my system to make it a small-scale microservice. I have read the high level concept of Read-Through cache strategy and then imagine the following picture in my mind.
Let me explain briefly, I will have 2 (or more) domain driven microservices (Payment, Customer) responsible for UPDATING (i.e. the "command" part in CQRS) data in their isolated PostgreSql DB schema. For the QUERY part, I would like that all of "GET" API requests from my mobile app fetch data from Redis using the Read-Through strategy by having some kind of "PG to Redis Converter" behind the scene (as in the label (6) in the picture).
This is what I think the Read-Through cache is about as far as I understand. But I am trying to search for an example of this kind of converter to integrate with my NodeJS or Java REST API, but I cannot find one. Most example I can find only talk about the concept. And the ones that show the implementation turn out to be more like Cache-Aside strategy.
Please help suggest which tools to use for such converter. Or if it can be configured directly into Redis itself (e.g. using Lua script?). It would be great if it can be done serverlessly using AWS service, but not necessary, thank you.
I am taking care of extraction of microservice out of a monolith written in Java with help of Spring Boot. We are planning to divide the whole monolith into a few smaller microservices.
I have to enable communication between the monolith and the new microservice as it needs entities from the new microservice (it has its own database) to perform certain actions. I thought of exposing REST endpoints but then I would have to duplicate entities.
Is it acceptable?
If so, then REST controllers based in monolith which retrieve entities from microservice should be placed in the same layer as repositories?
This solution would increase coupling which should be avoided, are there any other approaches?
I'll be grateful for any responses as well as articles which in your opinion may help here. Thank you in advance.
It totally depends on your Use-Case. There is no general solution and it is totally okay to maintain a repository where shared resources are placed.
If you take Microservice very strict, it would maintain the entities on its own. When transmitting the data you will use an intermediate format like JSON or XML which doesn't care about the structure of the data. The Microservice lives in its own world and only adresses his use-case. If another service undergoes a change and the entities change the other service should not be affected by this change. Each Service should only have the data it needs everything else is not his concern.
Therefore I would not use a central repository but as stated before there are Use-Cases where this is the way to go. Maybe some specific endpoints could help you by solving this issue.
We have a web2py application that we want to connect to an EmberJS client. The idea is to use the responsive capabilities of EmberJS to keep the client updated writing minimal code.
We have (REST) primitives which are in charge of creating / updating the underlying datastore (CouchDB). These primitives are sometimes complex and covering corner cases, involving the creation of several documents, connecting them, validating configuration parameters, ... This is implemented in the backend. We would like to avoid duplicating the full modelling of the data in our EmberJS application, and avoid duplicating the logic implemented by those primitives.
I have some questions:
does it make sense in EmberJS to just model a subset of the data in the documents? We would just create models for the small amount of properties that the user is able to interact with. The client would not see the full CouchDB documents, just the data necessary for display / interaction.
is it possible to connect EmberJS to a REST interface, without having to fully model the underlying data in the database?
does it make sense in EmberJS to just model a subset of the data in the documents?
Yes. There is no need to create ember models for objects/properties that user will not need to interact with.
is it possible to connect EmberJS to a REST interface, without having to fully model the underlying data in the database?
Definitely that is possible, it's a fairly common use case. The best way to get started is by building a small MVP that works with just couple of models. Once you've got that wired up it will be easy to add more domain objects.
The tricky part (especially at first) will be mapping your rest endpoints to the ember-data REST adapter. The adapter will work out-of-box with some REST endpoints - see the REST Adapter - but connecting a CouchDB datastore will probably require some customization. The tools for this are still evolving, have a look at ember-data integration tests to see what is available.
Basically I started to design my project to be like that:
Play! Framework for web gui (consuming RESTful service)
Spray Framework for RESTful service, connects to database, process incoming data, serves data for web gui
Database. Only service have rights to access it
Now I'm wondering, if it's really best possible design.
In fact with Play! I could easily have both web gui and service hosted at once.
That would be much easier to test and deploy in simple cases probably.
In complicated cases when high performance is needed I still can run one instance purely for gui and few next just to work as services (even if each of them could still serve full functionality).
On the other side I'm not sure if it won't hit performance too hard (services will be processing a lot of data, not only from the web gui). Also, isn't it mixing things which I should keep separated?
If I'll decide to keep them separated, should I allow to connect to database only through RESTful service? How to resolve problem with service and web gui trying to use different version of database? Should I use versioned REST protocol in that case?
----------------- EDIT------------------
My current system structure looks like that:
But I'm wondering if it wouldn't make sense to simplify it by putting RESTful service inside Play! gui web server directly.
----------------- EDIT 2------------------
Here is the diagram which illustrates my main question.
To say it clearly in other words: would it be bad to connect my service and web gui and share the model? And why?
Because there is also few advantages:
less configuration between service and gui needed
no data transfer needed
no need to create separate access layer (that could be disadvantage maybe, but in what case?)
no inconsistencies between gui/service model (for example because of different protocol versions)
easier to test and deploy
no code duplication (normally we need to duplicate big part of the model)
That said, here is the diagram:
Why do you need the RESTful service to connect to the database? Most Play! applications access the database directly from the controllers. The Play! philosophy considers accessing your models through a service layer to be an anti-pattern. The service layer could be handy if you intend to share that data with other (non-Play!) applications or external systems outside your control, but otherwise it's better to keep things simple. But you could also simply expose the RESTful interface from the Play! application itself for the other systems.
Play! is about keeping things simple and avoiding the over-engineered nonsense that has plagued Java development in the past.
Well, after few more hours of thinking about this, I think I found solution which will satisfy my needs. The goals which I want to be fulfilled are:
Web GUI cannot make direct calls to the database; it need to use proper model which will in turn use some objects repository
It must be possible to test and deploy whole thing as one packet with minimum configuration (at least for development phase, and then it should be possible to easy switch to more flexible solution)
There should be no code duplication (i.e. the same code in the service and web gui model)
If one approach will appear to be wrong I need to be able to switch to other one
What I forget to say before is that my service will have some embedded cache used to aggregate and process the data, and then make commits to database with bigger chunks of them. It's also present on the diagram.
My class structure will look like this:
|
|- models
|- IElementsRepository.scala
|- ElementsRepositoryJSON.scala
|- ElementsRepositoryDB.scala
|- Element.scala
|- Service
|- Element.scala
|- Web
|- Element.scala
|- controlers
|- Element.scala
|- views
|- Element
|- index.scala.html
So it's like normal MVC web app except the fact that there are separate model classes for service and web gui inheriting from main one.
In the Element.scala I will have IElementsRepository object injected using DI (probably using Guice).
IElementsRepository have two concrete implementations:
ElementsRepositoryJSON allows to retrieve data from service through JSON
ElementsRepositoryDB allows to retrieve data from local cache and DB.
This means that depending on active DI configuration both service and web gui can get the data from other service or local/external storage.
So for early development I can keep everything in one Play! instance and use direct cache and DB access (through ElementsRepositoryDB) and later reconfigure web gui to use JSON (through ElementsRepositoryJSON). This also allows me to run gui and service as separated instances if I want. I can even configure service to use other services as data providers (however for now I don't have such a needs).
More or less it will look like that:
Well, I think there's no objectively right or wrong answer here, but I'll offer my opinion: I think the diagram you've provided is exactly right. Your RESTful service is the single point of access for all clients including your web front-end, and I'd say that's the way it should be.
Without saying anything about Play!, Spray or any other web frameworks (or, for that matter, any database servers, HTML templating libraries, JSON parsers or whatever), the obvious rule of thumb is to maintain a strict separation of concerns by keeping implementation details from leaking into your interfaces. Now, you raised two concerns:
Performance: The process of marshalling and unmarshalling objects into JSON representations and serving them over HTTP is plenty fast (compared to JAXB, for example) and well supported by Scala libraries and web frameworks. When you inevitably find performance bottlenecks in a particular component, you can deal with those bottlenecks in isolation.
Testing and Deployment: The fact that the Play! framework shuns servlets does complicate things a bit. Normally I'd suggest for testing/staging, that you just take both the WAR for your front-end stuff and the WAR for your web service, and put them side-by-side in the same servlet container. I've done this in the past using the Maven Cargo plugin, for example. This isn't so straightforward with Play!, but one module I found (and have never used) is the play-cargo module... Point being, do whatever you need to do to keep the layers decoupled and then glue the bits together for testing however you want.
Hope this is useful...