Related
In the past I worked with what I believe were "microservices". We had a service discovery and the applications communicated with each other through REST sync calls, which I believe is called request / response.
Now we are working on an application that is simply broken down into multiple small applications which work together through publish / subscribe using Kafka, there's no direct communication between them nor a service discovery, per say.
Is it safe to say that these are "Microservices" too or what should I call them?
As one commenter pointed out- An answer will always be opinionated (and stackoverflow is generally not the best place for opinion based questions), but I am not afraid to give mine in an attempt to help.
My view goes along the lines of what Martin Fowler and others have taught us in last decade, as well as the insight from the pioneering work of Werner Vogels at Amazon. Their definitions are that microservices are not a well defined single standard, but an architectural style that can encompass a range of different approaches in distributed computing. What they have in common are the goals of providing scalability in many different dimensions, but mostly system complexity, organizational productivity, throughput and resilience - All while being affordable. To achieve that the designs should be centered around:
Service boundaries cut according to business goals (a type of Domain Driven Design).
Sizing / Scope of services to be limited (the famous two-pizza team size idea).
Clear separation in terms of organizational responsibilities and maintenance. This includes independent teams and release-/ maintenance cycles for each service.
Embrace of automation on all layers (how cloud virtualization, devops, CICD, infrastructure as code, etc. became popular tools).
Design focus on failure-resilient operation, e.g. avoid cascading failures, defined failure states, fallbacks, etc.
A consequence of those goals is also the embrace of distributed system architectures and design for horizontal scalability (statelessness, separation of compute and storage, etc.).
So if you feel your designs are along those lines then in my opinion you would have applied the microservices architectural style successfully.
If you want to learn more about this school of thought I recommend this read.
I am fairly new to building applications using the RESTful architecture. As a matter of fact, all I have done so far is categorized as Level 2 REST by Leonard Richardson and that I know Fielding would happily categorize as Non-RESTful.
I have spent hours trying to understand HATEOAS and how to reach level 4. And I see it more clearly now. I conceptualize the application as a series of state transitions, and the resources will dynamically provide links with information on how to move from one state to another.
But everything related to HATEOAS seem to be inherent of a human-computer interaction. I mean, even when the resources provide the links that enable the application user to move to the next state, it is ultimately the user the one that drives the application from one state to the other by causing the use of of the provided links.
But how are things supposed to work when we are dealing with computer-to-computer interaction? After all when it comes to service-orientation the idea of service composition is key, and we cannot naively assume that the client is always going to be a human being? Many services are designed to be consumed by non-human users, and some interactions/orchestrations might be fairly complex, the type of things that are typically modeled with things like BPM, or BPEL.
Is REST and particularly HATEOAS only usable in applications that imply human intervention and if not how is this supposed to work otherwise?
I am getting this vibe that REST is only good for certain type of solutions and inadequate for others, but literature out there has failed to explain those inadequacies and sell REST as the cure of all evil, but I just don't quite get how to use for proper service composition when humans are not the drivers.
I'd really appreciate any references or insights on this, because believe me I have two days straight reading all I have been able to find on this topic and I have not yet being able to reach any reasonable and well documented conclusions.
Well, your client app can parse the response to get possible actions. In this case actual urls are obtained not from knowledge of the API, but upon calling the initial method (usually GET). All human-less.
It sounds almost as if you're comparing SOA to REST/Hypermedia and fail to see that SOA is a strategy, for designing a complex system made out of other systems, while REST/Hypermedia is a software architecture style applying a bunch of constraints on client-server communication. The client, however, can be both a server or a human, it doesn't matter.
To use or not to use REST/Hypermedia is not something to bother with when outlining/designing service composition. It's a question that comes into play when trying to achieve syntactic interoperability. Many times it comes down to comparing REST to Soap and other technical details.
"A process virtual machine (also, language virtual machine) is
designed to run a single program, which means that it supports a
single process. Such virtual machines are usually closely suited to
one or more programming languages and built with the purpose of
providing program portability and flexibility (amongst other things).
An essential characteristic of a virtual machine is that the software
running inside is limited to the resources and abstractions provided
by the virtual machine—it cannot break out of its virtual
environment. quote from the Wikipedia Article"
I've been studying usage of virtual machines, especially with their importance in Cloud Computing, and I want to know if it would be possible to develop a VM based operating system that could be scaled dynamically to use the processing power of connected servers? Use it's own local hardware for fast processing, but also boost it's performance by sending processes that don't need to return immediate responses to a cloud service.
Is this possible, or is that concept flawed?
Basically, the OS scaling with the connected cloud servers. The processes that are ok to send to the cloud servers for a latent response would be up to the developers of each program.
At first, I could see this being effective only for corporations in need of cost-effective massive computation. But as internet speeds increase, even front-end interface animation calculations might be possible, having less local hardware, relying more heavily on cloud services.
If it's possible, it would allow many scientific simulations that would otherwise need super computer time to be possible from any location in the world, at no more cost than exactly what processing is done at a specific speed. And would lead eventually to consumer devices being extremely small, 'scaleably' powerful, and very cheap, allowing people to pay for processing the same way they pay for internet service today.
Is this possible, or is that concept flawed?
Both. ;)
What you are talking about seems like what used to be called "Grid Computing". (Sun even sold it in the early 90's.) The concept was that you put a magic library on all your boxes, and your app will be able to scale out with no further work by the programmer.
This is useful -- but only if your problem is "embarrassingly parallel" (i.e. lots of independent calculations that don't affect each other.).
MPI is one such popular way to do this: http://www.linux-mag.com/id/5759/
Unfortunately, most of the time people have problems that are more lumpy (grab a bunch of data from database, do some calculations, generate PDF.) In these cases, it's simpler to figure out a good strategy and manually code it up, than try to use a magic library which can be hard to debug, and even harder to work around performance problems. I know a lot of people using AWS, and none of them use a 'magic grid library' like you are talking about. They communicate between servers using simple protocols like Queues or HTTP interfaces.
That's not because your idea won't work. It's just that their needs can be satisfied by something much simpler to debug/run/tune.
Another neat idea in the same vein: http://www.gocircuit.org/
I'm auditing a project that uses what is called a Rules Engine. In short, it's a way to externalize business logic from application code.
This concept is entirely new to me and I'm pretty skeptical about it. After hearing people talk about Anemic Domain Models for the past few years, I'm questioning the Rules Engine Approach. To me they seem like a great way to WEAKEN a domain model. For example say I'm doing a java webapp interacting with a Rules Engine. Then I decide I want to have an Android app based on the same domain. Unless I want the Android app to interact with the Rules Engine as well, I'm going to have to miss out on whatever business logic was already written.
As I don't have any experience with them yet, just curiosity, I was interested to hear about the pros and cons are in using a Rules Engine? The only pro that I can think of is that you don't need to rebuild your entire Application just to change some business rule (but really, how many apps really have that many changes?). But using a Rules Engine to solve that problem kind of sounds to me like putting a band-aid over a shotgun wound.
UPDATE - since writing this, the god himself, Martin Fowler, has blogged about using a Rules engine.
Most rule engines that I have seen are viewed as a black box by system code. If I were to build a domain model, I would probably want certain business rules to be intrinsic to the domain model, e.g. business rules that tell me when an object has invalid values. This allows multiple systems to share the domain model without duplicating business logic. I could have each system use the same rule service to validate my domain model, but this appears to weaken my domain model (as was pointed out in the question). Why? Because instead of consistently enforcing my business rules across all systems at all times, I am relying on system programmers to determine when the business rules should be enforced (by calling the rule service). This may not be a problem if the domain model comes to you completely populated, but can be problematic if you're dealing with a user interface or system that changes values in the domain model over its lifetime.
There is another class of business rules: decision making. For example, an insurance company may need to classify the risk of underwriting an applicant and arrive at a premium. You could place these types of business rules in your domain model, but a centralized decision for scenarios like this are usually desirable and, actually, fit quite well into a service-oriented architecture. This does beg the question of why a rule engine and not system code. The place where a rule engine may be a better choice is where business rules responsible for the decision change over time (as some other answers have pointed out).
Rule engines usually allow you to change rules without restarting your system or deploying new executable code (regardless of what promises you receive from a vendor, do make sure you test your changes in a non-production environment because, even if the rule engine is flawless, humans are still changing the rules). If you're thinking, "I can do that by using a database to store values that change", you're right. A rule engine is not a magical box that does something new . It is intended to be a tool that provides a higher level of abstraction so you can focus less on reinventing the wheel. Many vendors take this a step further by letting you create templates so that business users can fill in the blanks instead of learning a rule language.
One parting caution about templates: templates can never take less time than writing a rule without a template because the template must, at the bare minimum, describe the rule. Plan for a higher initial cost (the same as if you were to build a system that used a database to store values that change vs. writing the rules in directly in system code) - the ROI is because you save on future maintenance of system code.
I think your concerns about anemic domain models are valid.
I've seen two applications of a well-known commercial Rete rules engine running in production where I work. I'd consider one a success and the other a failure.
The successful application is a decision tree app, consisting of ~10 trees of ~30 branch points each. The rules engine has a UI that does allow business folks to maintain the rules.
The less successful application has ~3000 rules slammed into a rules database. No one has any idea if there are conflicting rules when a new one is added. There is little understanding of the Rete algorithm, and the expertise with the product has left the firm, so it's become a black box that's untouchable and unrefactorable. The deployment cycle is still affected by rules changes - a complete regression test must be done when rules are changed. Memory was an issue, too.
I'd tread lightly. When a rule set is modest in size it's easy to understand changes, like the simplistic e-mail sample given above. Once the number of rules climbs into the hundreds I think you might have a problem.
I'd also worry about a rules engine becoming a singleton bottleneck in your application.
I see nothing wrong with using objects as a way to partition that rules engine space. Embedding behavior in objects that defer to a private rules engine seems okay to me. Problems will hit you when the rules engine requires state that isn't part of its object to fire properly. But that's just another example of design being difficult.
Rule Engines can offer a lot of value, in certain instances.
First, many rule engines work in a more declarative way. A very crude example would be AWK, where you can assign regexes to blocks of code. When the regex is seen by the file scanner, the block of code is executed.
You can see that in this case, if you had, say, a large AWK file and you wanted to add Yet Another "rule", you can readily go to the bottom of the file, add you regex and logic, and be done with it. Specifically, for many applications, you're not particularly concerned with what the other rules are doing, and the rules don't really interoperate with each other.
Thus the AWK file becomes more like a "rule soup". This "rule soup" nature lets folks focus very tightly on their domain with little concern for all of the other rules that may be in the system.
For example, Frank is interested in orders with a total of more than $1000, so he puts in to the rule system that he's interested. "IF order.total > 1000 THEN email Frank".
Meanwhile, Sally wants all orders from the west coast: "IF order.source == 'WEST_COAST' THEN email Sally".
So, you can see in this trivial, contrived case, that an order can satisfy both rules, yet both rules are independent of each other. A $1200 order from the West Coast notifies both Frank and Sally. When Frank is no longer concerned, he'll simply yank his rule out from the soup.
For many situations, this flexibility can be very powerful. It can also, like this case, be exposed to end users for simple rules. Using high level expressions and perhaps lightweight scripting.
Now, clearly, in a complicated system there are all sorts of interrelationships that can happen, an this is why the entire system is not "Done with rules". Someone, somewhere is going to be in charge of the rules not getting out of hand. But that doesn't necessarily lessen the value such a system can provide.
Mind this doesn't even go in to things like expert systems, where rules fire on data that rules can create, but a simpler rules system.
Anyway, I hope this example shows how a rules system can help augment a larger application.
The biggest pro that I've seen for rules engines is that it lets the Business Rule owners implement the business rules, instead of putting the onus on programmers. Even if you have an agile process where you are constantly getting feedback from stakeholders and going through rapid iterations, it's still not going to achieve the level of efficiency that can be achieved by having the people making the business rules implement them as well.
Also, you can't under-emphasize the value in removing the recompile-retest-redeploy cycle that can result from a simple rule change, if the rules are embedded in code. There are often several teams that are involved in putting the blessing on a build, and using a Rules Engine can make much of that unnecessary.
I've written a rules engine for a client. The biggest win was including all the stakeholders. The engine could run (or replay) a query and explain what was happening in text. The business people could look at the text description and quickly point out nuances in rules, exceptions, and other special cases. Once the business side was involved, the validation got much better because it was easy to get their input. Additionally, the rules engine can live separately from other parts of an application code base so you can use it across applications.
The con is that some programmers don't like to learn too much. Rule engines and the rules you put into them, along with the stuff that implements them, can be a bit hairy. Although a good system can easily handle sick and twisted webs of logic (or illogic often ;), it's not as simple as coding a bunch of if statements (no matter what some of the simple-minded rule engines do). The rules engine gives you the tools to handle rule relationships, but you still have to be able to imagine all of that in your mind. Sometimes it's like living in the movie Brazil. :)
It (as everything else) depends on your application. For some applications (usually the ones that never change or the rules are best on real life constants, i.e. won't change noticeably in eons, for instance physical properties and formulae) it doesn't make sense to use a rule engine, it just introduces additional complexity and requires the developer to have a larger skill set.
For other applications it's really a good idea. Take for instance order processing (orders being anything from invoicing to processing currency transactions), every now and then there's a minute change to some relevant law or code (in the judicial sense) that requires you to fulfil a new requirement (for instance sales tax, a classic).
Rather than trying to force your old application into this new situation where all the sudden you have to think about sales tax, where as before you didn't, it is easier to adapt your rule set rather than having to meddle about in potentially a large set of your code.
Then the next amendment from your local government requires reporting of all sales within a certain criteria, rather than you have to go in and add that, too. In the end you'll end up with very complex code that will prove pretty difficult to manage when you turn around and want to revert effect of one of the rules, without influencing all the others...
Everybody thus far has been very positive about rules engines, but I advise the reader to be wary. When a problem becomes a little bit more complicated, you may suddenly find that an entire rules engine has been rendered unsuitable, or much more complicated than in a more powerful language. Also, for many problems, rules engines will not be able to easily detect properties that greatly reduce the runtime and memory footprint of evaluating the condition. There are relatively few situations in which I would prefer a rule engine to a dependency injection framework or a more dynamic programming language.
"but really, how many apps really have that many changes?"
Honestly, every app I have worked on has gone through serious workflow and/or logic changes from concept until well after deployment. It's the number one reason for "maintenance" programming...
The reality is that you can't think of everything up front, hence the reason for Agile processes. Further, the BA's always seem to miss something vital until it's found in testing.
Rule Engines force you to truly separate business logic from presentation and storage. Further, if using the right engine, your BA's can add and remove logic as necessary. As Chris Marasti-Georg said, it puts the onus on the BA. But more than that, it allows the BA to get exactly what they are asking for.
A rules engine is a win on a configurable application where you don't want to have to do custom builds if it can be avoided. They are also good at centralising large bases of rules and algorithms like Rete are efficient for quickly matching against large rule sets.
Lots of good answers already but wanted to add a couple of things:
in automating a decision of any complexity the critical thing rapidly becomes your ability to manage rather than execute the logic involved. A rules engine will not help with this - you need to think about the rule management capabilities that a business rules management system has. Most commercial and open source rules engines have evolved into rule management systems with repositories, reporting on rule usage, versioning etc. A repository of rules, structured into coherent rule sets that can be orchestrated to make business decisions is a lot easier to manage than either thousands of lines of code or a rule soup.
There are many ways to use a declarative, rules-based approach. Using rules to manage a UI or as part of defining a process can be very effective. The most valuable use of a rules approach, however, is to automate business decisions and to deliver this as loosely coupled decision services that take input, execute rules and return an answer - a decision. Thing of these as services that answer questions for other services like "is this customer a good credit risk" or "what discount should I give this customer for this order or "what's the best cross sell for this customer at this time. These decision services can be very effectively built using a rules management system and allow for easy integration of analytics over time, something many decisions benefit from.
I see rule, process and data engines (a.k.a. databases) as essentially being similar. But, for some reason, we never say that blackboxing the persistence subsystem is bad.
Secondly, from my POV, an anemic model is not one that is light in implementation of behaviors, it is one that is light in behaviors itself. The actual method that describes an available behavior in a domain model object does not have to be done by the object itself.
The biggest complexity from my experience in Rule Engines is that:
from OOP POV it's a real pain to refactor and test rules written in a declarative language while you are refactoring code that affects them.
Often we should always think about the execution order of rules which turns into a mess when there are lots of them.
Some minor changes may trigger incorrect behaviour of rules leading to production bugs. In practice it's not always possible to cover all cases with tests up front.
Rules mutating objects used in other ones also increase complexity causing developers to break them into stages.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 7 years ago.
Improve this question
Our system (exotic commodity derivative trade capture and risk management) is being redeveloped shortly. One proposal that I have heard is that a rule engine will be incorporated to make it easier for the end-users (commodities traders, so fairly sophisticated) to make certain changes to the business logic.
I am a little skeptical of rules engines. The agilist in me wonders if they are just a technical solution to a process problem... ie. it takes too long for our developers to respond to the business's need for change. The solution to that problem should be a more collaborative approach to development, better test coverage, more agile practices all around.
Hearing about situations where a rule engine was truly a boon (especially in a trading environment) would be certainly helpful.
I've seen two applications that used the Blaze Rete engine from Fair Issac.
One application slammed thousands of rules into a single knowledge base, had terrible memory problems, has become a black box that few understand. I would not call that a success, but it is running in production.
Another application used decision trees to represent on the order of hundreds of questions on a medical form to disposition clients. It was done so elegantly that business people can update the rules as needed, without having to involve a developer. (Still has to be deployed by one, though.) I'd call that a great success.
So it depends on how well focused the problem is, the size of the rule set, the knowledge of the developers. My prejudice is that simply making a rules engine a single point of failure and dumping rules into it probably isn't a good approach. I'd start with a data-driven or table-driven approach and grow that until a rules engine was needed. I'd also strive to encapsulate the rules engine as part of the behavior of an object. I'd hide the rules engine from users and try to partition the rules space into the domain model.
I don't know if I'd say they're ever truly a boon, but I think they can certainly be valuable. I worked on a system for a few years in the insurance industry, where a rules engine was employed quite successfully to allow the business users to create rules that determined what policies were legal, depending on the state.
For instance, if you had to have a copay in certain states, or certain combinations of deductible and copay were not allowed, either because of product considerations, or because it was simply illegal due to state law.
The number of states that the company operated in, along with the constant change in rules (quarterly) would make this a dizzying coding practice. More importantly, it's not in the expertise of a programmer. It adds extra pointless communication where the end user is describing the rule to be put in effect to a programmer who is not an insurance industry expert like they are.
Designed correctly, a rules engine can still enable a workflow system that allows for good testing. In this case, the rules were stored in a database, and there were QA and PROD databases. So the BA's could test their rules in QA, and then promote them to PROD.
As with anything, its usually about the implementation, and not the actual technique.
Yes, Microsoft has a Business Rule Engine (BRE) in BizTalk that has been used successfully for years. I've heard that they've had clients buy BizTalk (very expensive) just for the BRE.
In my experience, the practicality of having a business user update the rules is slim to none. It usually takes a technical person to work the business rules editor.
A rule engine is little more than something that executes declarative statements. They come with two primary advantages (that I see):
Your business logic is maintained from a single place instead of being sprinkled throughout application code. Technically, a well-designed application should already do this with the architecture, regardless of a rule engine being present or not.
You need to worry [less] about dependencies between declarative statements. The rule engine should be smart enough to decide the order to run rules based on dependencies. You may find that some rule engines support a sequential ordering of rules within a ruleset or calling rulesets (groups of rules) in a particular order, but this isn't really in the spirit of declarative programming. Many rule engines use Rete (an algorithm) to decide when to schedule the execution of declarative statements.
I suspect most, if not all, rule engines add more overhead than if you were to write the best possible program that doesn't use a rule engine. This is similar to how writing code in assembly is generally faster than a compiler (but you usually don't write assembly because it's more convenient and productive to use higher-level abstractions).
If you were to stop here, then you would probably use programmers to maintain rules and use a rule engine as a convenient way to build a business logic tier in your application. Some rule engines offer something called templates that let you define templates for rules. The advantage here is that non-technical users are supposed to be able to write their own rules and modify existing rules.
A rule engine is one more tool in your tool chest that, when used properly, can be valuable.
The problem with many of these rule engines are the lack of speed and the fact that replacing or augmenting rules can break existing working rules in subtle ways. So you still have to re-test the system thoroughly after each rule change. So you're basically just exchanging one computer language for another one - one with a much smaller base of users. As another poster mentioned, I've yet to see a business analyst successfully use a rule engine. You need a programmer anyway.
I certainly have, but can't publicly talk about them, but its likely you have interacted with one several times this year ;)
I see it in 2 camps: the logic programmers and the business users. Different tools target different sets, some both. The successful cases of business users have only worked when it was a subset of the logic, and they also had a way to define test cases and run them themselves (and they are prepared to think logically).
Logic programmers are rarer, but can often be found coming from non imperative programming backgrounds (they are also the sort of people that find functional programming intuitive).
Keep in mind at the end of the day even with visual tools, if you are telling a computer to do something it is still programming.
I work with lots of vendors in the space and one of the great things about this is that I get to talk to lots of their customers. So, yes, hundreds of companies have got exactly the benefits they were promised - increased agility, better business/IT collaboration, easier regulatory compliance, better consistency of decision making, lower maintenance costs, faster times to market etc.
Over and over again, across all the major vendors and the open source players, I see that used correctly - to automate and improve high-volume operational decisions with many rules, rules that change a lot, rules that interact in complex ways or rules with a high business domain content - business rules management systems work.
Really.
My experience is limited to (i)not much and (ii)prolog; but I can safely say that a rule engine can help you express propositional concepts much cleaner than procedural code.
Rules engines are routinely used in the insurance business. I've worked on systems with hundreds (600ish) rules that were implemented in a rules engine. It worked very well.
Do you have a credit rating? A FICO score, perhaps? That's Fair Isaac COrporation, the developers of the Blaze rules engine.
For a while I worked for the PEATE distributed computing project which was developing a system for calculation of large scale, high volume atmospheric data. The system had three parts to it: the data manager, the scheduler, and the algorithm execution component. There could be any number of any of these components, all done through web services, but what it allowed for was for different researchers to execute arbitrary jobs against arbitrary data, and also allowed for different scheduling mechanisms to be plugged in as requirements changed.
I left the project before it got too far off the ground, but this seems like it could potentially fit the scenario, and serve as another example for some kind of rule engine. That being said, however, if the original developers are still going to be the one's making the algorithms to run, I can't see too much benefit in having a rule engine unless it handled a substantial overhead that each rule or algorithm would incur on it's own.
This sounds like a bit more involved than a simple rule engine, but such an architecture could feasibly apply to a rule engine as well.