Question:
Does Knative expose low-level network components that allow me to configure the stack in such a way, that each instance has a unique IP address available for outbound networking?
Info
I have a workload that has to happen on queue event. The incoming event will start the fetching on an API. Due to rate limiting and amount of request (around 100), the process is long-running and with wait / request / wait / request / wait / .. . What the code (JS) basically does is, hitting an API endpoint with parameters from the queues message and sending the result of the 100 API requests back with another queue.
Serverless on Lamdba is therefore expensive, also on AWS multiple instances are likely to be spawned on the same VM (tested), resulting in the same IP for outbound traffic. Therefore Lambda is not an option for me.
I read a lot about Knative lately and I imagine that the Kubernetes stack offers better configurability. I need to have concurrent instances of my service, but I need to have a unique outbound IP per instance.
Currently, the solution is deployed on AWS Beanstalk where I scale them out based on queue-length. Therefore 1 - 10 instances exist at the same time and perform the API requests. I use micro instances since CPU/../.. load is really low. There have been multiple issues with Beanstalk, that's why we'd like to move.
I do not expect a monthly cost advantage (IPs are expensive, that's ok), I am just unhappy with the deployment on Beanstalk.
IMHO, going with KNative/Kubernetes is probably not the way to proceed here. You will have to manage a ton of complexity just to get some IP addresses. Beanstalk will seem like a walk in the park.
Depending on how many IPs you need, you can just setup a few EC2 instances loaded up with IP addresses. One cheap t3.small instance can host 12 IPv4 addresses (ref) and your JS code can simply send requests from each of the different IP addresses. (Depending on your JS http client, usually there's a localAddress option you can set.)
Related
I read this article about the API Gateway pattern. I realize that API Gateways typically serve as reverse proxies, but this forces a bottleneck situation. If all requests to an application's public services go through a single gateway, or even a single load balancer across multiple replicas of a gateway (perhaps a hardware load balancer which can handle large amounts of bandwidth more easily than an API gateway), then that single access point is the bottleneck.
I also understand that it is a wide bottleneck, as it simply has to deliver messages in proxy, as the gateways and load balancers themselves are not responsible for any processing or querying. However, imagining a very large application with many users, one would require extremely powerful hardware to not notice the massive bandwidth traveling over the gateway or load balancer, given that every request to every microservice exposed by the gateway travels through that single access point.
If the API gateway instead simply redirected the client to publicly exposed microservices (sort of like a custom DNS lookup), the hardware requirements would be much lower. This is because the messages traveling to and from the API Gateway would be very small, the requests consisting only of a microservice name, and the responses consisting only of the associated public IP address.
I recognize that this pattern would involve greater latency due to increased external requests. It would also be more difficult to secure, as every microservice is publicly exposed, rather than providing authentication at a single entrypoint. However, it would allow for bandwidth to be distributed much more evenly, and provide a much wider bottleneck, thus making the application much more scalable. Is this a valid strategy?
A DNS/Public IP based approach not good from a lot of perspectives:
Higher attack surface area as you have too many exposed points and each need to be protected
No. of public IPs needed is higher
You may need more DNS settings with subdomains or domains to be there for these APIs
A lot of times your APIs will run on a root path but you may want to expose them on a folder path example.com/service1, which requires
you to use some gateway for the same
Handling SSL certificates for these public exposures
Security on a focussed set of nodes vs securing every publically exposed service becomes a very difficult task
While theoretically it is possible to redirect clients directly to the nodes, there are a few pitfalls.
Security, Certificate and DNS management has been covered by #Tarun
Issues with High Availability
DNS's cache the domains vs IP's they serve fairly aggressively because these seldom change. If we use DNS to expose multiple instances of services publicly, and one of the servers goes down, or if we're doing a deployment, DNS's will continue routing the requests to the nodes which are down for a fair amount of time. We have no control over external DNS's and their policies.
Using reverse proxies, we avoid hitting those nodes based on health checks.
I m studying Microservices architecture and I m actually wondering something.
I m quite okay with the fact of using (back) service discovery to make request able on REST based microservices. I need to know where's the service (or at least the front of the server cluster) to make requests. So it make sense to be able to discover an ip:port in that case.
But I was wondering what could be the aim of using service registry / discovery when dealing with AMQP (based only, without HTTP possible calls) ?
I mean, using AMQP is just like "I need that, and I expect somebody to answer me", I dont have to know who's the server that sent me back the response.
So what is the aim of using service registry / discovery with AMQP based microservice ?
Thanks for your help
AMQP (any MOM, actually) provides a way for processes to communicate without having to mind about actual IP addresses, communication security, routing, among other concerns. That does not necessarily means that any process can trust or even has any information about the processes it communicates with.
Message queues do solve half of the process: how to reach the remote service. But they do not solve the other half: which service is the right one for me. In other words, which service:
has the resources I need
can be trusted (is hosted on a reliable server, has a satisfactory service implementation, is located in a country where the local laws are compatible with your requirements, etc)
charges what you want to pay (although people rarely discuss cost when it comes to microservices)
will be there during the whole time window needed to process your service -- keep in mind that servers are becoming more and more volatile. Some servers are actually containers that can last for a couple minutes.
Those two problems are almost linearly independent. To solve the second kind of problems, you have resource brokers in Grid computing. There is also resource allocation in order to make sure that the last item above is correctly managed.
There are some alternative strategies such as multicasting the intention to use a service and waiting for replies with offers. You may have reverse auction in such a case, for instance.
In short, the rule of thumb is that if you do not have an a priori knowledge about which service you are going to use (hardcoded or in some configuration file), your agent will have to negotiate, which includes dynamic service discovery.
I'm using third party service that has own notion of session and expects all requests in session to come from same IP. They claim that it's a required security measure and suggest to use proxy, LOL.
Is there any way to use same IP for several requests with Socket API?
Interval between requests in session is ~10 seconds, so keeping connection alive and reusing it should work. I've tried to setup proxy module that runs single instance and uses HttpClient with connection pool. Logging shows that after first request connection is properly released and stored in pool. While doing second request I see that pool has 1 connection, but HttpClient says that there in no free connection for route and opens new. Probably route changes somehow?
It goes against the way App Engine is meant to work, scale-ability. Spawning instances closets to the consumers and multiple instance will mean different IP's. If you want a static IP you will need to switch to Compute Engine. Its a server VM that can have a static IP address. Or if your 3rd party service can take range you get get them from the link.
App Engine does not currently provide a way to map static IP addresses to an application. In order to optimize the network path between an end user and an App Engine application, end users on different ISPs or geographic locations might use different IP addresses to access the same App Engine application. DNS might return different IP addresses to access App Engine over time or from different network locations.
Actually, I solved this issue. Solution that I described in question was missing one step. As my connection was SSL authenticated, I had to use same context for all requests that I make.
I have built a WebSockets server that acts as a chat message router (i.e. receiving messages from clients and pushing them to other clients according to a client ID).
It is a requirement that the service be able to scale to handle many millions of concurrent open socket connections, and I wish to be able to horizontally scale the server.
The architecture I have had in mind is to put the websocket server nodes behind a load balancer, which will create a problem because clients connected to different nodes won't know about each other. While both clients A and B enter via the LoadBalancer, client A might have an open connection with node 1 while client B is connected to node 2 - each node holds it's own dictionary of open socket connections.
To solve this problem, I was thinking of using some MQ system like ZeroMQ or RabbitMQ. All of the websocket server nodes will be subscribers of the MQ server, and when a node gets a request to route a message to a client which is not in the local connections dictionary, it will pub-lish a message to the MQ server, which will tell all the sub-scriber nodes to look for this client and issue the message if it's connected to that node.
Q1: Does this architecture make sense?
Q2: Is the pub-sub pattern described here really what I am looking for?
ZeroMQ would be my option - both architecture-wise & performance-wise
-- fast & low latency ( can measure your implementation performance & overheads, down to sub [usec] scale )
-- broker-less ( does not introduce another point-of-failure, while itself can have { N+1 | N+M } self-healing architecture )
-- smart Formal Communication Pattern primitives ready to be used ( PUB / SUB is the least cardinal one )
-- fair-queue & load balancing architectures built-in ( invisible for external observer )
-- many transport Classes for server-side internal multi-process / multi-threading distributed / parallel processing
-- ready to almost linear scaleability
Adaptive node re-discovery
This is a bit more complex subject. Your intention to create a feasible architecture will have to drill down into more details to solve.
Node authentication vs. peer-to-peer messaging
Node (re)-discovery vs. legal & privacy issues
Node based autonomous self-organising Agents vs. needs for central policy enforcement
To update this for 2021, we just solved this problem where we needed to design a system that could handle millions of simultaneous WS connections from IoT devices. The WS server just relays messages to our Serverless API backend that handles the actual logic. We chose to use docker and the node ws package using an auto-scaling AWS ECS Fargate cluster with an ALB in front of it.
This solved the main problem of routing messages, but then we had the same issue of how do we route response messages from the server. We initially thought of just keeping a central DB of connections, but routing messages to a specific Fargate instance behind an ALB didn't seem feasible.
Instead, we set up a simple sub/pub pattern using AWS SNS (https://aws.amazon.com/pub-sub-messaging/). Every WS server receives the response and then searches its own WS connections. Since each Fargate instance handles just routing (no logic), they can handle a lot of connections when we vertically scale them.
Update: To make this even more performant, you can use a persistent connection like Redis Pub/Sub to allow the response message to only go to one single server instead of every server.
Recently I've added some load-balancing capabilities to a piece of software that I wrote. It is a networked application that does some data crunching based on input coming from a SQL database. Since the crunching can be pretty intensive I've added the capability to have multiple instances of this application running on different servers to split the load but as it is now the load balancing is a manual act. A user must specify which instances take which portion of the input domain.
I would like to take that to the next level and program the instances to automatically negotiate the diving up of the input data and to recognize if one of them "disappears" (has crashed or has been powered down) so that the remaining instances can take on the failed instance's workload.
In order to implement this I'm considering using a simple heartbeat protocol between the instances to determine who's online and who isn't and while this is not terribly complicated I'd like to know if there are any established heartbeat network protocols (based on UDP, TCP or both).
Obviously this happens a lot in the networking world with clustering, fail-over and high-availability technologies so I guess in the end I'd like to know if maybe there are any established protocols or algorithms that I should be aware of or implement.
EDIT
It seems, based on the answers, that either there are no well established heart-beat protocols or that nobody knows about them (which would imply that they aren't so well established after all) in which case I'm just going to roll my own.
While none of the answers offered what I was looking for specifically I'm going to vote for Matt Davis's answer since it was the closest and he pointed out a good idea to use multicast.
Thank you all for your time~
Distribued Interactive Simulation (DIS), which is defined under IEEE Standard 1278, uses a default heartbeat of 5 seconds via UDP broadcast. A DIS heartbeat is essentially an Entity State PDU, which fully defines the state, including the position, of the given entity. Due to its application within the simulation community, DIS also uses a concept referred to as dead-reckoning to provide higher frequency heartbeats when the actual position, for example, is outside a given threshold of its predicted position.
In your case, a DIS Entity State PDU would be overkill. I only mention it to make note of the fact that heartbeats can vary in frequency depending on the circumstances. I don't know that you'd need something like this for the application you described, but you never know.
For heartbeats, use UDP, not TCP. A heartbeat is, by nature, a connectionless contrivance, so it goes that UDP (connectionless) is more relevant here than TCP (connection-oriented).
The thing to keep in mind about UDP broadcasts is that a broadcast message is confined to the broadcast domain. In short, if you have computers that are separated by a layer 3 device, e.g., a router, then broadcasts are not going to work because the router will not transmit broadcast messages from one broadcast domain to another. In this case, I would recommend using multicast since it will span the broadcast domains, providing the time-to-live (TTL) value is set high enough. It's also a more automated approach than directed unicast, which would require the sender to know the IP address of the receiver in order to send the message.
Broadcast a heartbeat every t using UDP; if you haven't heard from a machine in more than k*t, then it's assumed down. Be careful that the aggregate bandwidth used isn't a drain on resources. You can use IP broadcast addresses, or keep a list of specific IPs you're doing work for.
Make sure the heartbeat includes a "reboot count" as well as "machine ID" so that you know previous server state isn't around.
I'd recommend using MapReduce if it fits. It would save a lot of work.
I'm not sure this will answer the question but you might be interested by the way Weblogic Server clustering work under the hood. From the book Mastering BEA WebLogic Server:
[...] WebLogic Server clustering provides a loose coupling of the servers in the cluster. Each server in the cluster is independent and does not rely on any other server for any fundamental operations. Even if contact with every other server is lost, each server will continue to run and be able to process the requests it receives. Each server in the cluster maintains its own list of other servers in the cluster through periodic heartbeat messages. Every 10 seconds, each server sends a heartbeat message to the other servers in the cluster to let them know it is still alive. Heartbeat messages are sent using IP multicast technology built into the JVM, making this mechanism efficient and scalable as the number of servers in the cluster gets large. Each server receives these heartbeat messages from other servers and uses them to maintain its current cluster membership list. If a server misses receiving three heartbeat messages in a row from any other server, it takes that server out of its membership list until it receives another heartbeat message from that server. This heartbeat technology allows servers to be dynamically added and dropped from the cluster with no impact on the existing servers’ configurations.
Cisco content switches are a hardware solution for this problem. They implement a virtual IP address as a front end to multiple real servers, whose real IP addresses are known to the switch. The switch periodically sends HTTP HEAD requests to the web servers, to verify they are still running (which the switch software calls a "keepalive", although this doesn't keep the server itself alive). The Cisco switch accepts traffic on the virtual IP and forwards it to the actual web servers, using configurable load balancing such as round-robin, or user-defined load balancing.
These switches retail in the $3-10K range, although my business partner picked one up on eBay for about $300 a year ago. If you can afford one, they do represent a proven hardware solution to the question of how to have a service spread transparently across multiple servers. Redhat includes a built-in port configuration so that you could implement your own Cisco switch using a cheap RedHat box. Google for "virtual ip address" and "cisco content router" for more information.
In addition to trying hardware load-balancers, you can also try a free-open-source load-balancing software application such as HAProxy, available for Linux and the BSDs.