We are implementing read receipts in a chat system. We want to store the timestamp when each message was read by the participants. (i.e messageId, participantId and timestamp), and display this when the user wants to see message info on the frontend. Other than this we also want to display ticks next to each message, depending on whether all participants have seen the message.
We are using sockets for sending the event when the message(s) is seen.
As in a chat system, the number of messages will keep growing, and we are concerned about the scalability of the system.
We have thought of the following schemas/approaches as of now:
Creating entries for the participant and the message in the 'ReadMessages' collection when a message is seen. When fetching the messages, it has to be populated with ‘ReadMessages'.
Embedding this data in the messages collection itself. This approach would require frequent updates to the 'Messages' collection, and probably slowing down reads on this collection?
Creating entries in the 'ReadMessages' collection when a message is seen by a participant, and storing a flag named 'Read' in the 'Messages' collection, that will be re-calculated and updated (if required), whenever a message is seen. This approach doesn't require populating while fetching messages, and also minimizes updates.
Similar as 3. Just the status 'Read' in the messages collection will be updated in a cron job. This will require lesser processing when a message seen event occurs, but this cron job will have to scan through all messages to see which are not marked as 'Read' yet (as the index is only on _id). Also, read receipts on the client will be updated with a delay depending upon the interval in which the cron job is run.
Which of these would be better in the long run?
Or is there any other better way schema for this can be designed?
Also, could multiple concurrent inserts to the collection cause performance problems?
I am trying to understand how durable subscription works in ActiveMQ Artemis. Currently my biggest question is about the storage.
I want to know if the messages are duplicated, which means for each consumer, the message is stored to disk or if the messages are stored in one place and consumers only knows the message at which they were disconnected and need to resume.
From my tests, i can see that : with a fixed message size and number of messages published, the space taken on disk is the same, no matter if I have 1,2 or 3 durable subscription. I took care of disconnecting them so that the messages are stored, don't worry. This would lead me to think that the queues only know about the index of the message that the consumer will need to start to consume when he comes back.
But on the opposite, i checked write Bytes per secondes with iostat cmd, and the more i have durable subscription queues, the more this stat grows. Which would mean messages are duplicated.
Do you guys have more informations, even the source code that is responsible for this.
By default ActiveMQ Artemis will store durable messages in a set of local files known as a "journal." In situations where more than one queue has the same message (e.g. in the case of a multiple durable subscriptions on the same JMS topic) the actual message data is only stored once and each queue gets a "reference" to that message. Storing the exact same message data multiple times would be hugely inefficient.
However, it's worth noting that the ActiveMQ Artemis journal files are initialized with zeroes when they are created which means that even an "empty" journal takes up space on the disk. Therefore, when messages are stored in the journal the amount of disk space they take will not change. The existing zeroes will simply be overwritten with the message data.
You can find the source code for ActiveMQ Artemis on GitHub.
If you want to see proof of this behavior you can use the artemis data print command. This command prints the raw records from the journal in a human readable format. If you were to have 2 durable subscriptions using client IDs of durable-client1 and durable-client2 and subscription names of subscriber-1 and subscriber-2 respectively on a JMS topic named exampleTopic and you send one message you would end up with an address, 2 queues, 1 actual message, and 2 references in the journal. You'd see something like this in the data print command output:
********************************************
B I N D I N G S J O U R N A L
********************************************
...
### Surviving Records Summary ###
...
recordID=2;userRecordType=44;isUpdate=false;compactCount=0;PersistentAddressBindingEncoding [id=2, name=exampleTopic, routingTypes={MULTICAST}, autoCreated=false]
recordID=18;userRecordType=21;isUpdate=false;compactCount=0;PersistentQueueBindingEncoding [id=18, name=durable-client1.subscriber-1, address=exampleTopic, filterString=null, user=null, autoCreated=false, maxConsumers=-1, purgeOnNoConsumers=false, exclusive=false, lastValue=false, lastValueKey=null, nonDestructive=false, consumersBeforeDispatch=0, delayBeforeDispatch=-1, routingType=0, configurationManaged=false, groupRebalance=false, groupBuckets=-1, groupFirstKey=null, autoDelete=false, autoDeleteDelay=0, autoDeleteMessageCount=0]
recordID=23;userRecordType=21;isUpdate=false;compactCount=0;PersistentQueueBindingEncoding [id=23, name=durable-client1.subscriber-2, address=exampleTopic, filterString=null, user=null, autoCreated=false, maxConsumers=-1, purgeOnNoConsumers=false, exclusive=false, lastValue=false, lastValueKey=null, nonDestructive=false, consumersBeforeDispatch=0, delayBeforeDispatch=-1, routingType=0, configurationManaged=false, groupRebalance=false, groupBuckets=-1, groupFirstKey=null, autoDelete=false, autoDeleteDelay=0, autoDeleteMessageCount=0]
...
********************************************
M E S S A G E S J O U R N A L
********************************************
...
### Surviving Records Summary ###
recordID=27;userRecordType=45;isUpdate=false;compactCount=0;Message(messageID=27;userMessageID=41705395-b2d1-11e9-91f9-a0afbd82eaba;msg=CoreMessage[messageID=27,durable=true,userID=41705395-b2d1-11e9-91f9-a0afbd82eaba,priority=4, timestamp=Tue Jul 30 08:52:22 CDT 2019,expiration=0, durable=true, address=exampleTopic,size=232,properties=TypedProperties[__AMQ_CID=durable-client1,_AMQ_ROUTING_TYPE=0]]#454305524
recordID=27;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding [queueID=18]
recordID=27;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding [queueID=23]
...
We are aggregating in session windows by using the following code:
.windowedBy(SessionWindows.with(...))
.aggregate(..., ..., ...)
The state store that is created for us automatically is backed by a changelog topic with cleanup.policy=compact.
When redeploying our topology, we found that restoring the state store took much longer than expected (10+ minutes). The explanation seems to be that even though a session has been closed, it is still present in the changelog topic.
We noticed that session windows have a default maintain duration of one day but even after the inactivity + maintain durations have been exceeded, it does not look like messages are removed from the changelog topic.
a) Do we need to manually delete "old" (by our definition) messages to keep the size of the changelog topic under control? (This may be the case as hinted to by [1].)
b) Would it be possible to somehow have the changelog topic created with cleanup.policy=compact,delete and would that even make sense?
[1] A session store seems to be created internally by Kafka Stream's UnwindowedChangelogTopicConfig (and not WindowedChangelogTopicConfig) which may make this comment from Kafka Streams - reducing the memory footprint for large state stores relevant: "For non-windowed store, there is no retention policy. The underlying topic is compacted only. Thus, if you know, that you don't need a record anymore, you would need to delete it via a tombstone. But it's a little tricky to achieve... – Matthias J. Sax Jun 27 '17 at 22:07"
You are hitting a bug. I just created a ticket for this: https://issues.apache.org/jira/browse/KAFKA-7101.
I would recommend that you modify the topic config manually for fix the issue in your deployment.
I am receiving a large number of correlated HL7 messages in Mirth. They contain an ID which is always the same for all correlated messages and they always come within a minute. Multiple batches can be received at the same time. It's hard to say when the batch ends, but when there are no more messages for a minute, it's safe to assume that the batch has finished.
How could I implement an aggregator pattern in Mirth that would keep reading and completing correlated messages and send the completed message after it didn't receive any new messages with the same ID within a defined time interval?
You may drop all incoming message to a folder and store the message ID in a Global Map. Once new messages start to arrive with the message ID different than the one stored in the map (meaning that the next sequence is started), trigger another channel either by sending the message ID it needs to look for or in some other way. After that replace the message ID in the Global Map with the message ID of a new sequence.
If that sounds too complicated, you may do the same, but the second channel will constantly scan the folder (File Reader) and grab only files having the same message ID and older than a minute from a current time (which is in my mind is too vague qualifier).
I've implemented this by saving all messages in a folder using an ID inside the message (that identifies the sequence) as the file name. The file gets updated with each new message. Several sequences live together in the same folder.
The next channel is using a simple file reader that only fetches the files that are a minute or more old.
Our Transactional Dead-letter queue is filling up in MSMQ. I can't find documentation on particular.net that points to why this could be happening.
It looks like every single message that is being processed (successfully) is ending up at the that queue.
What is the reason items are sent to the Transactional Dead Letter queue?
If you open dead letter queue in Computer Management, each message has a reason why it ended up there. It's under "class" column. That should point you where to look. For example one of reasons could be "The time-to-be-received has elapsed.", if message wasn't received within time specified in its "TimeToBeReceived" property.