What are the strategies for Pusher channel structures in social status update applications? - publish-subscribe

When building a social application it's common to follow other users or topics as an indication of interest in updates by the user or topic. For example, following other users on Twitter, Friending other people on Facebook or liking a product or brand on Facebook.
Pusher has the concept of channels that you subscribe to. Channels are a human readable string that provide a logical identifier to information (e.g. "some-channel-name") and therefore seems to naturally suggest that in a social application any updates on a user or topic should be sent on a channel specific to that item (e.g. "userX-status-updates" or "myBrand-status-updates").
However, this raises concerns about how efficient it is to subscribe to multiple channels if a user is following a high number of other users or topic.
Therefore, what are the appropriate strategies for structuring channels in an social status update style application that uses Pusher?

The first thing to clarify is that you need a mapping of who you are following so for the purposes of this answer I'm going to assume that it's stored in a DB on the server. It also assumed that status updates are triggered as follows:
Client (userX posts status update) -> Your Server (sanitize & validate)
Your Server -> Pusher
Pusher -> Clients (users interested in updates from UserX)
There are two possible solutions to the channel information architecture problem:
Channel Per User Status: A user subscribes to a userX-status-updates channel for all the users that they follow and users trigger update events on their own status update channel.
Users I'm Following Channel: When a user posts a status update you look up who is following that user and publish the update on a users-you-follow-updates channel.
Strategy 1. is the most optimal solution as it keeps interactions with your own infrastructure an Pusher to a minimum.
Here's the detail on these two strategies:
1. Channel Per User Status
The assumption here is that subscribing to channels is costly but that not entirely correct. Channels are simply a way of routing events. However, if you are using authenticated channels (private & presence) you need to authenticate the subscription via your own server. If you use the Pusher WebSocket libraries "out of the box" each subscription will result in a request to your server. So, a user is following 1,000 users that's 1,000 requests to your server.
But, for the pusher-js library there is a multi-auth plugin that can batch the authentication requests into a single call.
There is also a BatchAuthorizer for the Pusher WebSocket Java library, but it's only a sample solution to this scenario.
2. Users I'm Following Channel
Note: although this is an option it's probably only appropriate for smaller numbers of users
In this scenario a user sends their status update to the server, the server performs a lookup of which users are interested in the update and triggers and update even on a channel for each interested user.
For example, give users UserA, UserB and UserC each of those users will subscribe to their own update channel; UserA-followers-updates, UserB-followers-updates, and UserC-followers-updates respectively. If each of these users follows UserZ then when UserZ makes as status update that update is published on each of those channels.
This may also sound inefficient, however it is possible to trigger the same event on 10 channels at a time. So in the above example it would only require one call to the Pusher HTTP API to send the status update to all interested users. More information on multi-channel event publishing here.

Related

What's the best way to join and leave pubnub channels dynamically

I'm working on this chat type scenario and the need is to track presence -
Note: every client in this scenario will subscribe with option withPresence set to true.
Pubnub cofig-
Max announce is set to 1, so i rely on interval event type.
phoneA will ask server for a channel name, Server will generate a unique channel name (lets say ch-chat) and return to PhoneA.
Now both phoneA and server will add the channel to their own channel groups (cg-phoneA and cg-server)and subscribe to their own channel groups.
PhoneA shares the channel name with phoneB and phoneC and they both add the channel to their own channel groups, cg-phoneB and cg-phoneC. Bkth B and C are also subscribed to their own channel group.
Now,
When phoneA removes channel (ch-chat) from its own channel group (cg-phoneA), server does not gets phobeA in leave uuids list of interval event. But after ~5mins server gets timeout event for phoneA.
What i want -
phoneA should present in the leave list of the interval event since phoneA removed the channel from its own channel group, theres no way it should be subscribed anymore to it.
Presence Modes: Announce vs Interval
One issue is that your Announce Max setting of 1 will mute all presence events except for state-change. In other words, you are in Interval mode all the time for every channel. Read more about Presence Event Modes here.
When channels are added to a channel group, you should receive the join event (if in Announce mode and not Interval), if you are subscribed to the channel group withPresence: true. Be sure you want to receive presence events for all channels in that channel group though. You can create a second channel group for channels you do not need to receive presence events.
Managing Channel Groups
As for managing Channels in your Channel Groups, be sure that only your Server is the one adding/removing channels for all channel groups. This has to do with security when you enable Access Manager (and everyone should be enabled/implementing Access Manager to protect your keys and your users).
Presence Webhooks
As for Presence Webhooks, this is the best way for your Server to monitor presence events. It is not recommended that you subscribe from your server. It will be difficult/complex to scale to multiple server instances. This is why the Presence Webhooks exist. And your clients and POST directly to the Server for any requests it needs, like your "generate new channel" request. And your server and reply to those posts and then publish all further updates to the channel the client is subscribed to.
Channel Naming Conventions
It is great that you are using "generated channel names" (like a UUID pattern) but prefix your channels to identity them to provide a means to group them with pattern match through regex. I would recommend using a "dot" so that you can also potentially take advantage of PubNub wildcarding features:
PubNub Functions channel binding
Presence ACLs (custom presence channel behavior configuration)
Access Manager wildcard granting
Wildcard Subscribe
For example:
chat.4849-ut83-83jd...
notify.tuy7-87er-27fn...
NOTE: Channel Groups do not allow "dots" in the name and do not have any wildcarding features.

How to choose which events are sent to the user on an occasionally connected system using CQRS with event sourcing?

I am building a web app that users can edit and share notes. Users should be connected to notes with roles (owner, read, read-write). This is an occasionally connected system so I chose to do the syncing using CQRS and event sourcing. Following Greg Young's presentation [36:20 - 38:40], the flow would be as follows:
Client does changes while offline.
Client connects to the Internet.
The "store and forward" sends the events that occurred while the client was offline.
Client compares the local events with the received events and does a merge, deciding what commands to keep. Then updates local view model.
Client sends the stored commands (created offline) to the server.
Server executes the commands and generates events that are stored in event store.
"store and forward" holds the events each user is interested in, until the users come back online.
The question is: How does the "store and foreword" decide what events should be sent to each user?
Obviously sending all events would compromise the security of other users.
Since your client knows which aggregates it displays, then it can just tell backend "hey, are there events for aggregateIds [...] since [timestamp]?".
This is how reSolve framework keeps UI reactive - client subscribes to events for particular aggregateId and receives them in real time via websockets.
So one answer to your question could be "let user ask for events (aggregateIds) he is interested in"

AppSync/Graphql Multiple subscriptions or one subscriptions for multiple ids?

Problem :
We are trying to make a chat application using AWS product AppSync and we want to achive the best performance but we're facing problem with real time subscriptions in AppSync and Graphql where a single user will need to handle hundereds of subscription in some cases which we think is not the best solution, what do you suggest ?
Problem Example:
Mutation{
addMessage(conversation_id=Int!, content:String!) : Message
}
Subscription{
subscribeForNewMessages(convesration_id: Int!):Message
#aws_subscribe(mutations: ["addMessage"])
}
the problem with this design is that the user need to invoke this subscription and keep listening for every single conversation, which we expect to be overwheelming the client in case if the conversations quantity is huge.
Questions :
Q1 :
What we are striving to achieve is one subscription for multiple (conversation_id)s, how this will be possible?
These folks (https://github.com/apollographql/apollo-client/issues/2633) are talking about something similar, we tested it and it doesn't work, is it a valid solution?
Q2:
Regarding Amplify; Will amplify perform well when listening for hundereds of subscription simulanuosly? does it make some sort of merging subscription and websockets or it will deal them separately?
Q3:
what are your comments about these designs? where there will be a service that will braodcast(invoke mutations with clients ids) the messages for chat participants , and the client will subscribe only for a single channel . like the following:
src2 : AWS AppSync for chatting application
src2 : Subscribe to a List of Group / Private Chats in AWS AppSync
Q1/Q2
You'll have to make multiple subscriptions and the aws ios/android/amplify sdks can handle subscription handshake protocols for real-time updates to data.
Take a look here
Q3
I recommend allowing clients to subscribe to specific channels (even if that means multiple subscriptions) so that the filtering logic can be done in the service rather than client side, reducing client side code which also means you don't have to worry about maintenance or scalability.

How to set up an e-commerce messaging solution using ably.io

I have an e-commerce website and I want to implement a messaging solution between my clients, using ably.io. I am new to ably.io and also to pub/sub message pattern and need some advice to put me on the right track.
Let’s say I have 75,000 users on my website. Users can create an advertisement... if anyone is interested in the advertisement they should be able to message the advertiser.
This is the solution that comes to my mind after reading ably's documentation:
Since I have 75,000 users, I create 75,000 channels, 1 channel per user.
An user named Brian creates an advertisement on the website. Brian subscribes to BrianChannel to receive all messages on his own channel. Now Amy and Tom have seen Brian's advertisement and want to contact him, they publish a message on BrianChannel and Brian would receive the messages.
I am not sure if the above solution is the right approach?
If the above solution is the right way, I have 2 further questions:
Let's Brian uses JavaScript to subscribe to BrianChannel. Now Brian is obviously not online 24/7 - how should I handle his messages while he is offline?
I have 75,000 users registered on my website. At any point of time 5,000 users are online. Out of these 5000 users online, 20 users are actually messaging each other. Using my solution above, I should have 5,000 active channels for all the online users... Is there a way to reduce the number of active channels? Perhaps by periodically polling the channels?
This is Srushtika, Dev Advocate at Ably.
Your approach is correct and is in-fact the most commonly implemented one. i.e, each of your 75,000 users will have their own channels to which only they will be subscribed to. Any other client who wishes to send them a message, will attach to 'their' channel and publish messages to it. Since all users are always subscribed to their own channels, they will be able to receive any messages intended for them.
If any of your subscriber clients are not online at the time a message is to be sent to them, Ably's Realtime platform will, by default, retain it for 2min after which, if the subscriber client is still offline, the messages will be discarded. However, using the Persisted History feature, you can increase the time for which messages are retained to 24-72hrs.
Unfortunately, as of now, Ably doesn't offer a mechanism to detach from channels with no activity while the client is still online. But, this feature is coming soon. Keep an eye on our Engineering blog where we post all such updates.
Hope that answers your question.

OpenFire + FastPath WebChat - Pre Fill the User Chat Window with ongoing Conversation

Currently I am using Olark for live chat on my website.
I am planning to replace it with an in house OpenFire installation.
However, there is one problem.
With Olark live chat snippet (which I embed on my website) - if a user opens the website in multiple tabs - it's prefilled with the ongoing conversation. For e.g. - you can try it on (moonclerk.com).
How do I achieve the same pre population of chat window with the ongoing conversation with OpenFire + FP WebChat?
The way we achieve it at Olark is not exactly trivial. We don't actually use an XMPP client on the end-user's side, which makes it a bit easier, but basically our transport layer is able to grab conversation events for a conversation in-progress based on a session UUID that is stored in the user's cookies. XMPP isn't involved in that process at all, for us (it is only concerned with final delivery to and from Operators).
I don't know what FastPath's architecture is like, but if you were looking for this functionality, or to add it yourself, XMPP supports retrieving some n number of messages from the message history for a client. Check out XMPPFramework - Retrieve Archived Messages From Openfire Server for more on that.