Detect load balancer/server changes in WebSocket connection from client - sockets

Question

Current configuration
I have a client application connected to a Cloud server through a WebSocket (node.js) connection. I need WebSocket to get real time notifications of incoming messages.
Let use abc.example.com the domain name for the Cloud server for this example.
Cloud server configuration
The Cloud is powered by Amazon Elastic Load Balancer.
This Cloud server have this underlying architecture:
Cloud architecture
On Cloud update, the load balancer switches to another one so all new data posted to the Cloud is handled by a new load balancer and server.
So abc.example.com is always accessible even if the load balancer/server changes. (e.g. Doing an HTTP call)
WebSocket configuration
The WebSocket configuration is connecting to abc.example.com, which connects to a certain server and it stays connected to this one server until something closes it.
Problems
When connected, the WebSocket connection stays open to a server on the Cloud and doesn't detect when the load balancer switches to another one (e.g. on Cloud updates)
So if I send new data to the server for my client (e.g. new message), the connected client doesn't receive it through the WebSocket connection.
Although an HTTP GET query does the work because it resolves the right server.
From my understanding, this is a normal behavior since the server to which the client is connected with WebSocket is still on and didn't close the connection ; also nothing wrong happened.
Actually, I tried to switch the load balancer and the server (the initial server to which the client is connected) still send a pong response to the client when pinged periodically.
So is there any way to detect when the load balancer has switched from the client side ? I'm not allowed to modify the Cloud configuration but I can suggest it if there is a fairly easy solution.
Bottom line is: I don't want to miss any notifications when the Cloud updates.
Other observations:
At t0:
Client app is connected to server1 through WebSocket thanks to ELB1
Reception succeeds through WebSocket when sending new messages to the Cloud
At t1:
Cloud update: Switch from ELB1 to ELB2
Failure to receives new messages through WebSocket
At t2:
Cloud update: Switch from ELB2 to ELB1
Reception succeeds through WebSocket when sending new messages to the Cloud
Any suggestions/help is appreciated,
*This answer helped me understand the network structure but I'm still running out of ideas.
*Apologies if the terminology is not entirely appropriate.

Answer 1

Did you consider using a Pub/Sub server/database such as Redis?
This will augment the architecture in a way that allows Websocket connections to be totally independent from HTTP connections, so events on one server can be pushed to a websocket connection on a different server.
This is a very common network design for horizontal scaling and should be easy enough to implement using Redis or MongoDB.
Another approach (which I find as less effective but could offer scaling advantages for specific databases and designs) would be for each server to "poll" the database (of "subscribe" to database changes), allowing the server to emulate a pub/sub subscription and push data to connected clients.
A third approach, which is by far the most complicated, is to implement a "gossip" protocol and an internal pub/sub service.
As you can see, all three approaches have one thing in common - they never assume that HTTP connections and Websocket connections are routed to the same server.
EDIT (a quick example using redis):
Using Ruby and the iodine HTTP/Websocket server, here's a quick example for an application that uses the first approach to push events to clients (a common Redis Database for Pub/Sub).
Notice that it doesn't matter which server originates an event, the event is pushed to the waiting client.
The application is quite simple and uses a single event "family" (pub/sub channel called "chat"), but it's easy to filter events using multiple channels, such as a channel per user or a channel per resource (i.e. blog post etc').
It's also possible for clients to listen to multiple event types (subscribe to multiple channels) or use glob matching to subscribe to all the (existing and future) matching channels.
save the following to config.ru:
require 'uri'
require 'iodine'
# initialize the Redis engine for each Iodine process.
if ENV["REDIS_URL"]
uri = URI(ENV["REDIS_URL"])
Iodine.default_pubsub = Iodine::PubSub::RedisEngine.new(uri.host, uri.port, 0, uri.password)
else
puts "* No Redis, it's okay, pub/sub will support the process cluster."
end
# A simple router - Checks for Websocket Upgrade and answers HTTP.
module MyHTTPRouter
# This is the HTTP response object according to the Rack specification.
HTTP_RESPONSE = [200, { 'Content-Type' => 'text/html',
'Content-Length' => '32' },
['Please connect using websockets.']]
WS_RESPONSE = [0, {}, []]
# this is function will be called by the Rack server (iodine) for every request.
def self.call env
# check if this is an upgrade request.
if(env['upgrade.websocket?'.freeze])
env['upgrade.websocket'.freeze] = WS_RedisPubSub.new(env['PATH_INFO'] && env['PATH_INFO'].length > 1 ? env['PATH_INFO'][1..-1] : "guest")
return WS_RESPONSE
end
# simply return the RESPONSE object, no matter what request was received.
HTTP_RESPONSE
end
end
# A simple Websocket Callback Object.
class WS_RedisPubSub
def initialize name
#name = name
end
# seng a message to new clients.
def on_open
subscribe channel: "chat"
# let everyone know we arrived
# publish channel: "chat", message: "#{#name} entered the chat."
end
# send a message, letting the client know the server is suggunt down.
def on_shutdown
write "Server shutting down. Goodbye."
end
# perform the echo
def on_message data
publish channel: "chat", message: "#{#name}: #{data}"
end
def on_close
# let everyone know we left
publish channel: "chat", message: "#{#name} left the chat."
# we don't need to unsubscribe, subscriptions are cleared automatically once the connection is closed.
end
end
# this function call links our HelloWorld application with Rack
run MyHTTPRoute
Make sure you have the iodine gem installed (gem install ruby).
Make sure you have a Redis database server running (mine is running on localhost in this example).
From the terminal, run two instances of the iodine server on two different ports (use two terminal windows or add the & to demonize the process):
$ REDIS_URL=redis://localhost:6379/ iodine -t 1 -p 3000 redis.ru
$ REDIS_URL=redis://localhost:6379/ iodine -t 1 -p 3030 redis.ru
In this example, I'm running two separate server processes, using ports 3000 and 3030.
Connect to the two ports from two browser windows. For example (a quick javascript client):
// run 1st client app on port 3000.
ws = new WebSocket("ws://localhost:3000/Mitchel");
ws.onmessage = function(e) { console.log(e.data); };
ws.onclose = function(e) { console.log("closed"); };
ws.onopen = function(e) { e.target.send("Yo!"); };
// run 2nd client app on port 3030 and a different browser tab.
ws = new WebSocket("ws://localhost:3000/Jane");
ws.onmessage = function(e) { console.log(e.data); };
ws.onclose = function(e) { console.log("closed"); };
ws.onopen = function(e) { e.target.send("Yo!"); };
Notice that events are pushed to both websockets, without any concern as to the event's origin.
If we don't define the REDIS_URL environment variable, the application won't use the Redis database (it will use iodine's internal engine instead) and the scope for any events will be limited to a single server (a single port).
You can also shut down the Redis database and notice how events are suspended / delayed until the Redis server restarts (some events might be lost in these instances while the different servers reconnect, but I guess network failure handling is something we have to decide on one way or another)...
Please note, I'm iodine's author, but this architectural approach isn't Ruby or iodine specific - it's quite a common approach to solve the issue of horizontal scaling.