I am designing a solution and would like to double check if this is according to the microservices architecture.
We have clients, accounts and transactions like a normal bank account.
Clients have basic data like name and address.
Accounts might be for savings or current
Transactions are money transfers between 2 accounts
So I am designing the following way:
1 microservice to manage client data (will manage just client basic data and their addresses)
1 microservice to manage account data (will manage account basic, the client id is part of the account data)
1 microservice to manage money data (will have the account's balance and all transfers)
Please let me know if this is according to the microservice architecture and if you have another understanding.
As per my understanding, the main goal of a microservices architecture is to support faster and continuous releases of different parts of a big system without waiting on each other. There are two approaches to design a new system, microservices first approach and microservices later approach. In the first approach, the system is designed from ground up to follow microservices architecture, the system is initially itself broken down into services and the services talk to each other typically over HTTP & REST. In the other approach, system is initially is not built as microservices and all the modules are under single application. Both of these approaches has it pros and cons which is a separate discussion.
In your case, you are taking the first approach, to design the new system with separate services for each functionality. I am not an expert in banking domain but from what I understand, the client (customer) system can be definitely a separate service and be responsible for maintaining customer master data. The account service can be responsible for maintaining accounts data and serve out account related information. However, account balance is an integral property of an account and it should be always associated with an account. Finally the transfer can be a separate service that can record the transfers between accounts. Whenever there is a transfer, it can query the accounts for their current balance and if the transfer is valid one, then it can record the transfer.
However as this involves financial transaction, you would have to make sure that each transaction follows the ACID rules. Maintaining ACID properties among distributed systems is tricky and there are several ways to mitigate this such as only having ACID transactions on the most critical areas and having eventual consistency on the others. For example, banks do not immediately reflect all the transactions to the customer as "completed" and instead show the message saying "pending to be processed" (eventual consistency) so that customer is aware of the exact status.
Related
Imagine you have a user registration form, where you fill in the form:
First Name, Last Name, Age, Address, Prefered way of communication: Sms, Email (radiobuttons). You have 2 microservices:
UserManagement service
Communication service
When user is registered we should create 2 aggregates in 2 services: User in UserManagementContext and UserCommunicationSettings in Communication. There are three ways I could think of achieving this:
Perform 2 different requests from UI. What if one of them fails?
Put all that data in User and then raise integration event with all that data, catch it in CommunicationContext. Fat events, integration events shouldn't contain domain data, just the ids of aggreagates.
Put the message in the queue, so both contexts would have the adapters to take the needed info.
What is the best option to split this data and save the information?
Perform 2 different requests from UI. What if one of them fails?
I don't think this would do. You are going to end up in inconsistent state.
I'm for approach 3#:
User is persisted (created) in your user store.
UserRegistered event is sent around, containing the ID of the user.
All interested parties handle UserRegistered event.
I'd opt for slim events, because your services may need different user data and its better to let them to get this data on their own rather than putting all the things into the event.
As you mentioned to store communication settings, assuming that communication data are supposedly not part of the bounded context of UserManagement Service.
I see a couple of problems with approach #3 proposed in other answer. Though I think approach #3 in original answer is quite similar to my answer.
What if more communication modes are added? Naturally, it should only cause Communication Service to change, not UserManagement Service. SRP. Communication settings MS should store all communication settings related data in its own datastore.
What if user updates his communication settings preference only? Why user management service should take burden of handling that? Communication settings change should just trigger changes in its corresponding micro-service that is Communication Service in our case.
I just find it better to use some natural keys to identify & correlate entities across micro-services rather than internal IDs generated by DB. Consider that tomorrow you decide to use completely different strategy to create "ids" of user for UserManagement service e.g. non-numeric IDs, different id generation algorithm etc. I would want to keep other micro-services unaffected of any such decisions.
Proposed approach:
Include API Gateway in architecture. Frontend always talks to API Gateway.
API Gateway sends commands to message queue like RegisterUser to be consumed by interested micro-services.
If you wish to keep architecture simple, you may go with publishing a single message with all data that can be consumed by any interested micro-services. If you strictly want individual micro-services to see only its relevant data, create a message queue per unique data structure expected by consuming services.
Background
I am looking to design the back-end for an on-demand service app, akin to Uber. In this app, and most others like it, we have many buyers who send requests for a service that must be forwarded to many service-providers providers, etc. - all in real time.
I am planning to create a single REST API for the back-end, implemented in Javascript/Node over a single Postgres database. This back-end would mediate between the buyers and service-providers by receiving buyer requests, matching buyers with available service-provider based on custom criteria, forwarding the requests to service-providers, etc.
With a real-time on-demand service like this, there are many race-conditions that can occur i.e. when multiple buyers are matched with the same service-provider at the same time. I was thinking to handle all of these cases with a web-API flavor of optimistic concurrency control (inspired by this blog post).
Questions
Is a single REST API the right backend solution for this case?
Are there any schemes, algorithms, frameworks, solutions out there
that could handle concurrent service requests while also being
reasonable performant and scalable*?
*Scalable to a reasonable degree, i.e. up to a magnitude of 1000s rather that Uber's millions
I am doing a system architecture and my knowledge from college doesn't help me when it comes to understand the subtle differences between centralized, distributed and service oriented architecture/application.
If I take a typical client/server architecture, the client sends requests to a server, the server then sends responses to the client. That is a centralized architecture.
An application that handles both server and client sides will be a distributed application (because working on different platforms), but that is still a centralized architecture.
Therefore, a distributed architecture must involve a distributed application.
Questions: am I right? What does all that become when it comes to service oriented architectures / applications?
Distributed: The whole process involved in a computation task is divided into pieces and assigned to multiple computational nodes. Each node when doing its part of the processing does not have access to the whole information of the system that is necessary to achieve a globally optimized result. The aggregate of the results from multiple nodes will converge towards a global optimal result through usually multiple iterations of computations distributed across multiple nodes.
A good example is a router system in which each router has only the information it exchanges with its neighbours. At the start the neighbours known only part of the whole network system. Once a router gets more information from its neighbours it incorporates the new information into its view of the whole system, then spreads its view to its neighbours. Through multiple iteration of these steps, each separately computed by individual routers, all routers would settle on a consistent global view of the whole network system.
Another example could be a web ordering system where the browser initially gets a list of commonly order goods. The browser may have logic to track user viewing behaviour and make a decision to fetch from the server a different category of goods list, but does not send all the user behaviour parameters to the server. In this imaged example, the browser knows something the serer does not know, and the server does too. Thus the whole application would be a distributed system. In addition to this part of distributed nature, the user authentication could be done on one of the servers, the inventory is done on another server, and reservation on yet another one. Each of the servers involved would not have the whole information of the specific user browsing and ordering instance. But the aggregate work from all these nodes will fulfill the business need to sell more goods and satisfy more customers.
Opposite to the distributed, is the centralized, in that the computation logic would be always able to get the information of the whole picture.
Given this view, a client-server application can be a viewed as a distributed system if you think the client side involves non-trivial decision making. Or can be a centralized system if you think the client is dumb.
The service-oriented term is more about how the functional processing power is integrated into the system. In a service-oriented system, new capability may be introduced into the system at runtime by new API functionality discovery, or new logic capability discovery behind an unchanged API. Think about it, you could build an application that initially has no much built-in capability, then it expands its capability by discovering and incorporating new capability on the service providers. In contrast a traditional system needs to be built at build time, typically as a consequence of human-involving discussion-design-documentation activity. A service-oriented design is a good fit to a distributed system.
I'm trying to study about high frequency trading systems. Whats the mechanism that HFT use to connect with the exchange and whats the procedure (does it has to go through a broker or is it direct access, if it's direct access what sort of connection information that i require)
Thanks in advance for your answers.
Understand that there are two different "connections" in an HFT engine. The first is the connection to a market data source. The second is to a clearing resource. As mentioned in kpavlov's answer, a very expensive COLO (co-location) is needed to get as close to the data source/target as possible. Depending on their nominal latency these COLO resources cost thousands of dollars per month.
With both connections, your trading engine must be certified by the provider (ICE, CME, etc) to comply with their requirements. With CME the certification process is automated, with ICE it employs human review. In any case, the certification requires that your software demonstrate conformance to standards and freedom from undesirable network side effects.
You must also subscribe to your data source(s) and clearing service, neither is inexpensive and pricing varies over a pretty wide range. During the subscription process you'll gain access to the service providers technical data specification(s)-- a critical part of designing your trading engine. Using old data that you find on the Internet for design purposes is a recipe for problems later. Subscription also gets you access to the provider(s) test sites. It is on these test sites that you test and debug your engine.
After you think you engine is ready for deployment you begin connecting to the data/clearing production servers. This connection will get you into a place of shadows-- port roulette. Not every port at the provider's network edge has the same latency. Here you'll learn that you can have the shortest latency yet seldom have orders filled first. Traditional load balancing does little to help this and CME has begun deployment of FPGA-based systems to ensure correct temporal sequencing of inbound orders, but it's still early in its deployment process.
Once you're running you then get to learn that mistakes can be very expensive. If you place an order prior to a market pre-open event the order is automatically rejected. Do it too often and the clearing provider will charge you a very stiff penalty. Other things can also get you penalized or even kicked-off the service if your systems are determined to be implementing strategies to block others from access, etc.
All the major exchanges web sites have links to public data and educational resources to help decide if HFT is "for you" and how to go about it.
It usually requires an approval from exchange to grant access from outside. They protect their servers by firewalls so your server/network need to be authorized to access.
Special certification procedure with technician (by phone) is usually required before they authorize you.
Most liquidity providers use FIX protocol or custom APIs. You may consider starting implementing your connector with QuickFix, but it may become a bottleneck later, when your traffic will grow.
Information you need to access by FIX is:
Server IP
Server port
FIX protocol credentials:
SenderCompID
TargetCompID
Username
Password
Other fields
I'm in the planning stages of a .NET service which continually processes incoming messages, which involves various transformations, database inserts and updates, etc. As a whole, the service is huge and complicated, but the individual tasks it performs are small, simple, and well-defined.
For this reason, and in order to allow for easy expansion in future, I want to split the service into several smaller services which basically perform part of the processing before passing it onto the next service in the chain.
In order to achieve this, I need some kind of intermediary messaging system that will pass messages from one service to another. I want this to happen in such a way that if a link in the chain crashing or is taken offline briefly, the messages will begin to queue up and get processed once the destination comes back online.
I've always used message queuing for this type of thing, but have recently been made aware of SQL Service Broker which appears to do something similar. Is SQLSB a viable alternative for this scenario and, if so, would I see any performance benefits by using that instead of standard Message Queuing?
Thanks
It sounds to me like you may be after a service bus architecture. This would provide you with the coordination and fault tolerance you are looking for. I'm most familiar and partial to NServiceBus, but there are others including Mass Transit and Rhino Service Bus.
If most of these steps initiate from a database state and end up in a database update, then merging your message storage with your data storage makes a lot of sense:
a single product to backup/restore
consistent state backups
a single high-availability/disaster recoverability solution (DB mirroring, clustering, log shipping etc)
database scale storage (IO capabilities, size and capacity limitations etc as per the database product characteristics, not the limits of message store products).
a single product to tune, troubleshoot, administer
In addition there are also serious performance considerations, as having your message store be the same as the data store means you are not required to do two-phase commit on every message interaction. Using a separate message store requires you to enroll the message store and the data store in a distributed transaction (even if is on the same machine) which requires two-phase commit and is much slower than the single-phase commit of database alone transactions.
In addition using a message store in the database as opposed to an external one has advantages like queryability (run SELECT over the message queues).
Now if we translate the abstract terms 'message store in the database as being Service Broker and 'non-database message store' as being MSMQ, you can see my point why SSB will run circles any time around MSMQ.
My recent experiences with both approaches (starting with Sql Server Service Broker) led me to the situation in which I cry for getting my messages out of SQL server. The problem is quasi-political but you might want to consider it: SQL server in my organisation is managed by a specialized DBA while application servers (i.e. messaging like NServiceBus) by developers and network team. Any change to database servers requires painful performance analysis from DBA and is immersed in fear that we might get standard SQL responsibilities down by our queuing engine living in the same space.
SSSB is pretty difficult to manage (not unlike messaging middleware) but the difference is that I am more allowed to screw something up in the messaging world (the worst that may happen is some pile of messages building up somewhere and logs filling up) and I can't afford for any mistakes in SQL world, where customer transactional data live and is vital for business (including data from legacy systems). I really don't want to get those 'unexpected database growth' or 'wait time alert' or 'why is my temp db growing without end' emails anymore.
I've learned that application servers are cheap. Just add message handlers, add machines... easy. Virtually no license costs. With SQL server it is exactly opposite. It now appears to me that using Service Broker for messaging is like using an expensive car to plow potato field. It is much better for other things.