we are building chat application similar to messenger. There is required behavior:
User log in
User should see last N messages, and he should be able to load older messages
New messages should be appended as well
My solution:
I would like to use websockets for this purpose with combination of REST. My idea was that client application decide by message id which messages need. So REST will be used for initial fetching of messages and fetching older messages.
New messages will received by websockets
Possible issue which I should handle:
Application starts subscribing websocket channel for new messages and send request for old messages without initial message id
There is chance that after calling GET request new message come, and will be stored in DB
Client application started subscribing websocket channel so message will received by websockets.
GET request didn't know about this message and fetch last N messages where this new messages will occured and client application will have duplicate record and have to filtered this messages
Can you give me advice if there is some elegant way how to handle this case? Thank you.
I would resolve your task having in mind the following:
The client application should know only about the topic to which to listen. And not the ID of the message starting from which to listen.
It is up to the server to decide what to return (even time should always be tracked server-side).
The WebSocket is used as a transport for STOMP (simply to not reinvent the wheel). The WebSocket connection could be opened once the client application is loaded and not when it is entering the "listen for messages" state. But topic subscription should be performed when necessary.
You can always send GET request and initiate a STOMP subscription simultaneously (almost simultaneously, well with a delay of 1-2 nano-second). And those always should be processed in different promises. But I would align those in the following way: first, the STOMP subscription is initiated, And a specific message on subscription with the initial timestamp of the start of subscription is delivered; second, REST request to get previous 10-100 messages for the TOPIC prior to a specific timestamp (received from STOMP) is performed.
Getting the last 10 messages (which are prior to subscription moment) could be delivered as by REST as by STOMP approach: you can always react to a subscription event on your server-side, and deliver client-specific messages.
Regarding the problem of multiple identical messages from different "data channels", it is easily resolvable: your client (hope that is not jquery, but rather Angular or React or Vue or anything else) will be storing all the data in a single collection in a controller, and filtering and checking by message-id that only unique entries are stored is easy.
BUT if your system will produce hundreds of thousands of messages per second: I guess HTTP-based protocols are not your choice in this case.
Related
I am trying to implement queues into our microservice architecture, to be specific AWS SNS/SQS.
For example I have this scenarion.
After order is created Orders MS raises OrderCreated event and this event publishes message to AWS OrderCreated SNS. SQS queue InvoiceCreate is subscribed to OrderCreated SNS and will get this message.
Evertyhing makes scence so far. If Invoicing MS is listening to InvoiceCreate queueu and retrieves all new messages - Invoicing MS should create an invoice, but my question is with what data?
a) contact Order MS (to order data relevant for creating invoice). If unable to do so, message will be left in queue until Invoicing MS is able to collect the relevant data
b) message published should contain all the relevant data needed to create an invoice.
If choosing A Invoicing MS will not be decoupled and it will be depending on Order MS, but on the other hand it can collect additional data other then the data packed with original message.
If choosing B, since OrderCreated event and OrderCreated SNS doesnt really know who will use message data ie. OrderCreated could be also used to perform different actions, I am confused how to preciselly decide what data should be stuffed in this message
Our architecture is set up more like your option B. To use your example, the Order service would publish it's OrderCreated event and attach - as a payload - most (or even all) of the Order information in the Payload section of the message. We format message and payload as JSON for compatibility, but you can do whatever.
In some cases, we don't publish all info, just specific fields for the Added/Edited entity - it depends on the service and the sensitivity of the information. So long as you only ever add fields to a message (don't remove any), you are honoring the contract and aren't really tightly coupled to it.
Again, to your example, the InvoiceService could get its information from one or more of several options:
Pull it directly from the OrderCreated message if you include everything needed
Pull what it can from the OrderCreated message, publish an InvoiceStarted event that triggers the OrderService (and/or others) to send it an OrderInvoiceComplete message with the rest of the details it needs
Keep a local copy of whatever key data it needs - populated by subscribing to other events - so that it can combine OrderCreated data with some local data to flesh out an invoice
It's best to avoid the InvoiceService responding to a message by making a call directly back to the OrderService - this is a pretty tight coupling that can be avoided by simply messaging back if you have to.
So, there are lots of options. I personally prefer the technique of putting all data that might be useful into the messages when things are created/updated and letting consuming services decide what to use/ignore. For our scenario, that works well but we have only a few well-contained clients accessing our services so there may be more secure ways to do it that aren't relevant for us.
I am developing an application where I have several Consumers (an project which will consume a message and process it and after processing it send back to the topic where it comes from)
An Getway project which has two REST method Post method, this Post method will produce a messaage to one of the consumer and that particular consumer will process and send back to the Getway
now, I have a Get method that will consume the message which is sent back by the consumer.
but I want to remove this Get call, and I want all this done in single Post call, where I can produce a message to consumer and once the consumer processed I want to read it
how to achieve this? should I use Poll?
If so then where should I write Poll and is there a way to handle this on event basis like once message send by Getway it reads the response on some event.
In my project we used spring kafka template.
I have java server application. And I need to monitor a lot of gmail accounts to be able to send push notifications to mobile devices about new Inbox messages.
I need to know sender email and message subject to send push notification.
And I tried Gmail push notifications system (webhooks option)
If I understood everything correctly in order to get needed info for each new message for each user there is a following scenario:
Google sends me email and history id via https request.
I call history API and get new message ids for user
I request message info by message id
That means that I need 2 additional requests for each new message of each user. And it looks quite hard if server needs to handle several new messages each second. And I still don't see other way.
Is there any way to make it shorter? (e.g. to make google send me not only history id but needed new message details or at least make one additional request, but not two)
Thanks!
We tried using Push notifications but the volume of requests generated has meant that we poll on a timed basis instead. It is worth noting that you will get Push notifications for any change to the message such as a label change, a read status change etc. As you say there are many requests.
If you need the notifications real-time I don't see how you can avoid the process that you outline.
If you don't need it real-time then you can poll or at the least check for Push notifications per account on a timed basis and if some have been received retrieve any new messages in a batch rather than a single request.
I was facing the same problem. History API was not giving recent messageId. I solved this problem by hitting THREAD API where i set Q = last 5 min timestamp.
Webhook push notification is helping to notify a new mail has come. After i get all messages last 10 min with THREAD API.
I am currently looking to buildout a messaging service where users can send and receive messages privately between each other. I may have a need for multi-user chat, but for the most part, I only want single recipients to be able to read messages sent to them.
With looking at RabbitMQ, does it make sense to use one exchange, and create a queue for each user when they login and destroy each queue on logout? Are there major performance issues with creating a queue for each user or are there better alternatives?
I am building a REST API and plan on having users send messages to others through an endpoint (/send) and subscribe to their own message streams via websockets or something similar. I will probably store messages in MongoDB as well, so users can access all of their previous messages. Any suggestions on structure are appreciated.
I think your approach is correct. You event don't need an exchange if you will use the default exchange (AMQP Default). And during login create a new queue and keep queue name same as user name. (Just need to make sure user names are unique) And if you publish message to the default exchange with username (ie: queue name) as routing key, RbbitMQ will route that message to that queue only. And on logout if you delete the queue then user is going to miss the messages when he is not online. If it is OK then create queue after login and use the configuration exclusive which says queue gets deleted when there is no consumer. But if you want to keep offline messages then you need to create queue permanently during user signup.
I am developing a xmpp client and currently working on "pubsub".
I created a node in pubsub and subscribed two users to it.
But when a subscriber login(show presence) i get the last published item twice.
I am using ejabberd server.
Can anyone help??
Thank you.
Same here,
There seems to be two different queues. With notification_type=normal, messages are stored offline. When receiver becomes online, it receives the message twice:
from offline spool, without headline attribute
from pubsub send loop (as node is still configured with send_last_published_item = on_sub_and_presence)
As stated here when configuring pubsub node with notification_type=normal, it's best to disable send_last_published_item or set it to on_sub only in order to avoid receiving the message twice.