I am trying to implement queues into our microservice architecture, to be specific AWS SNS/SQS.
For example I have this scenarion.
After order is created Orders MS raises OrderCreated event and this event publishes message to AWS OrderCreated SNS. SQS queue InvoiceCreate is subscribed to OrderCreated SNS and will get this message.
Evertyhing makes scence so far. If Invoicing MS is listening to InvoiceCreate queueu and retrieves all new messages - Invoicing MS should create an invoice, but my question is with what data?
a) contact Order MS (to order data relevant for creating invoice). If unable to do so, message will be left in queue until Invoicing MS is able to collect the relevant data
b) message published should contain all the relevant data needed to create an invoice.
If choosing A Invoicing MS will not be decoupled and it will be depending on Order MS, but on the other hand it can collect additional data other then the data packed with original message.
If choosing B, since OrderCreated event and OrderCreated SNS doesnt really know who will use message data ie. OrderCreated could be also used to perform different actions, I am confused how to preciselly decide what data should be stuffed in this message
Our architecture is set up more like your option B. To use your example, the Order service would publish it's OrderCreated event and attach - as a payload - most (or even all) of the Order information in the Payload section of the message. We format message and payload as JSON for compatibility, but you can do whatever.
In some cases, we don't publish all info, just specific fields for the Added/Edited entity - it depends on the service and the sensitivity of the information. So long as you only ever add fields to a message (don't remove any), you are honoring the contract and aren't really tightly coupled to it.
Again, to your example, the InvoiceService could get its information from one or more of several options:
Pull it directly from the OrderCreated message if you include everything needed
Pull what it can from the OrderCreated message, publish an InvoiceStarted event that triggers the OrderService (and/or others) to send it an OrderInvoiceComplete message with the rest of the details it needs
Keep a local copy of whatever key data it needs - populated by subscribing to other events - so that it can combine OrderCreated data with some local data to flesh out an invoice
It's best to avoid the InvoiceService responding to a message by making a call directly back to the OrderService - this is a pretty tight coupling that can be avoided by simply messaging back if you have to.
So, there are lots of options. I personally prefer the technique of putting all data that might be useful into the messages when things are created/updated and letting consuming services decide what to use/ignore. For our scenario, that works well but we have only a few well-contained clients accessing our services so there may be more secure ways to do it that aren't relevant for us.
Related
we are building chat application similar to messenger. There is required behavior:
User log in
User should see last N messages, and he should be able to load older messages
New messages should be appended as well
My solution:
I would like to use websockets for this purpose with combination of REST. My idea was that client application decide by message id which messages need. So REST will be used for initial fetching of messages and fetching older messages.
New messages will received by websockets
Possible issue which I should handle:
Application starts subscribing websocket channel for new messages and send request for old messages without initial message id
There is chance that after calling GET request new message come, and will be stored in DB
Client application started subscribing websocket channel so message will received by websockets.
GET request didn't know about this message and fetch last N messages where this new messages will occured and client application will have duplicate record and have to filtered this messages
Can you give me advice if there is some elegant way how to handle this case? Thank you.
I would resolve your task having in mind the following:
The client application should know only about the topic to which to listen. And not the ID of the message starting from which to listen.
It is up to the server to decide what to return (even time should always be tracked server-side).
The WebSocket is used as a transport for STOMP (simply to not reinvent the wheel). The WebSocket connection could be opened once the client application is loaded and not when it is entering the "listen for messages" state. But topic subscription should be performed when necessary.
You can always send GET request and initiate a STOMP subscription simultaneously (almost simultaneously, well with a delay of 1-2 nano-second). And those always should be processed in different promises. But I would align those in the following way: first, the STOMP subscription is initiated, And a specific message on subscription with the initial timestamp of the start of subscription is delivered; second, REST request to get previous 10-100 messages for the TOPIC prior to a specific timestamp (received from STOMP) is performed.
Getting the last 10 messages (which are prior to subscription moment) could be delivered as by REST as by STOMP approach: you can always react to a subscription event on your server-side, and deliver client-specific messages.
Regarding the problem of multiple identical messages from different "data channels", it is easily resolvable: your client (hope that is not jquery, but rather Angular or React or Vue or anything else) will be storing all the data in a single collection in a controller, and filtering and checking by message-id that only unique entries are stored is easy.
BUT if your system will produce hundreds of thousands of messages per second: I guess HTTP-based protocols are not your choice in this case.
Let's suppose a simplified scenario like this:
There are two Kafka topics, users and orders and three microservices user-service, order-service and shipping-service.
When an order is placed through the order service, an OrderCreated event is added to the orders topic and listened by the shipping service. This service needs to get the user information to send the order. According to my requirements I can't make a REST call to user-service but use a stateful approach. That is to say, the shipping service is a Kafka Streams application that listens to the users topic, having a KTable backed by a local store with the full user table information. Thus, when processing the order it already has the user information available locally.
However, one concern of this approach is the consistency of the local user information in the shipping service, e.g:
A user updates its shipping address in the user-service, it updates its local SQL database and publishes an event in the user topic with this change.
The user places an order, so order-service publishes it in the order topic.
For whatever reason shipping service could process the OrderCreated event from order topic before reading the UserUpdated information from the user topic so it would use an address which is not valid anymore.
How could I guarantee that the shipping service always has an updated user information in this event-carried state transfer scenario?
If you need ordering guarantees, you would need to write both the user information update as well as the order into the same topic (and in particular into the same partition) because Kafka only guarantees order within a single partition.
You could call this topic "user_action" with a unique user-id as key (both an user information update as well as an user order is an user action). In your case, all three services would consume the "user_action" topic. While the user service only considers user updates and the order service only considers orders, the shipping service considers both.
This blog post might help, too: https://www.confluent.io/blog/put-several-event-types-kafka-topic/
I have a scenario for my app which is similar to sending friend request in Facebook.
When user A sends friend request to user B, internally a new friend request document is created. At a later time when user B also wants to send friend request to A, system would find out that a friend request document existed and so they should be friend of each other, no new friend request document would be created.
I'm trying to figure out the case when user A and user B both simultaneously sends friend request to each other which will then create 2 friend request documents and leading to undetermined behaviour...
Thanks for your suggestions.. Really appreciated!
Edit:
A few had suggested to use a request queue to solve this; however,
I'm confused about using queue because i thought it would make my rest api endpoint process requests sequentially. Wouldn't I lose all the benefit of multi-threading by using queue? I can't help but imagine how bad it would be if my service has millions of requests queued and waiting to be executed one by one just due to this issue. Has anyone seen something along similar problems seen in production?
I had similar situation with my client which has concurrent writes in the database, What I have implemented is a Queue service.
Create a request in the queue rather than writing in the database, a separate reader will
read one message from the queue at a time and check if it is valid to write it to
database, write only if there is no previous request.
You can implement your own queue or you can use service like AWS-SQS, rabbitmq, MSMQ etc.
// Specific to your case
In mongodb write operations on a single document are atomic.
mongodb has a feature of unique index.
Hence if you insert the document with an _id(or any other unique index) with person names A and B by creating unique index for both (such as "A_B" by lexicographically sorting the names) before doing insertion. You will inherently be able to insert only one instance of that document.
// General
What essentially we would like to have are transactions but since mongodb doesn't support such, as of now. There are a few tricks to achieve this:
2 phase commits :
https://docs.mongodb.org/v3.0/tutorial/perform-two-phase-commits/
Using an external source to maintain a flag, for example using memcache which supports insertion in transactional manner/Compare and Swap.
Here if you use system calls method in frontend then you should fire one request to frontend from Database when some user like, I send you request then within a sec database send you one system call and your frontend code immediate correct the button text like
"Add a friend" to "incoming request"
or else.
if you are only setting up database then just make a system call which send it to UI when friend request arrives or as you say Document created, the further process will be handled by UI Developer.
Thank you.
if you don't like the answer then I m apologize for that but don't downvote me because I M new in Stack Overflow Community.
I am currently looking to buildout a messaging service where users can send and receive messages privately between each other. I may have a need for multi-user chat, but for the most part, I only want single recipients to be able to read messages sent to them.
With looking at RabbitMQ, does it make sense to use one exchange, and create a queue for each user when they login and destroy each queue on logout? Are there major performance issues with creating a queue for each user or are there better alternatives?
I am building a REST API and plan on having users send messages to others through an endpoint (/send) and subscribe to their own message streams via websockets or something similar. I will probably store messages in MongoDB as well, so users can access all of their previous messages. Any suggestions on structure are appreciated.
I think your approach is correct. You event don't need an exchange if you will use the default exchange (AMQP Default). And during login create a new queue and keep queue name same as user name. (Just need to make sure user names are unique) And if you publish message to the default exchange with username (ie: queue name) as routing key, RbbitMQ will route that message to that queue only. And on logout if you delete the queue then user is going to miss the messages when he is not online. If it is OK then create queue after login and use the configuration exclusive which says queue gets deleted when there is no consumer. But if you want to keep offline messages then you need to create queue permanently during user signup.
Reading up on CQRS there is a lot of talk of email notification - i'm wondering where to get the data from. Imagine a senario where one user invites other users to an event. To inform a user that he has been invited to an event, he is sent an email.
The concrete steps might go like this:
A CreateEvent command with an associated collection of users to invite, is received by the server.
A new Meeting aggregate is created and a method InviteUser is called for each user that is to be invited.
Each time a user is invited to an event, a domain event UserWasInvitedToEvent is raised.
An email notification sender picks up the domain event and sends out the notification email.
Now my question is this: Where do I go for information to include in the email?
Say I want to include a description of the event as well as the user's name. Since this is CQRS I can't get it thru my domain model; All the properties of the domain objects are private! Should I then query the read side? Or maybe move email notification to a different service entirely?
In CQRS, you're separating the command from the query side. You will always want to go to the query side in order to get data for a given event handler. The write database is going to be a separate database that contains the data necessary for building up your domain objects and will not be optimized for reads, but for writes.
The domain should register and send an EventCreated event to the event handlers / processors. This could be raised from the constructor of the Meeting aggregate.
The event processing component would pick up the EventCreated event, and update the query database with the data contained in the event (ie, the Id of the event and its name).
The domain could register and send a UserWasInvitedToEvent event to the event processors.
The event processors would pick up the UserWasInvitedToEvent and update the query store with any reporting data necessary.
Another event processing component would also pick up the UserWasInvitedToEvent event. This process could have access to the query database and pull back all of the data necessary for sending the email.
The query database is nothing more than a reporting database, so you could even have a specific table that stores all of the data required for the email in one place.
In order to orchestrate several different events into a single handler (assuming the events might be processed in a different order at different times), you could utilize the concept of a Saga in your messaging bus. NServiceBus is an example of a messaging bus that supports Saga's. See this StackOverflow question as well: NServiceBus Delayed Message Processing.